How to decode a string

[Lad]

To be able to decode a string successfully, I need to know what coding
it is in.
The string can be coded in utf8 or in windows-1250 or in another
coding.
Is there a method how to find out the string coding.
Thank you for help
L.

[Fredrik Lundh]

Lad wrote:

To be able to decode a string successfully, I need to know what coding
it is in.

ask whoever provided the string.

The string can be coded in utf8 or in windows-1250 or in another
coding. Is there a method how to find out the string coding.

in general, no. if you have enough text, you may guess, but the right
approach for that depends on the application.

</F>

[Lad]

Fredrik Lundh wrote:

Lad wrote:

To be able to decode a string successfully, I need to know what coding
it is in.

ask whoever provided the string.

The string can be coded in utf8 or in windows-1250 or in another
coding. Is there a method how to find out the string coding.

in general, no. if you have enough text, you may guess, but the right
approach for that depends on the application.

</F>

Fredrik,
Thank you for your reply
The text is from Mysql table field that uses utf8_czech_ci collation,
but when I try
`RealName`.decode('utf8'),where RealName is that field of MySQL

I will get:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3:
ordinal
not in range(128)

Can you please suggest the solution?
Thank you
L.

[Marc 'BlackJack' Rintsch]

In <11**********************@m73g2000cwd.googlegroups .com>, Lad wrote:

The text is from Mysql table field that uses utf8_czech_ci collation,
but when I try
`RealName`.decode('utf8'),where RealName is that field of MySQL

I will get:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3:
ordinal
not in range(128)

Can you please suggest the solution?

Do you get this from converting the value from the database or from trying
to print the unicode string? Can you give us the output of

print repr(RealName)

Ciao,
Marc 'BlackJack' Rintsch

[Lad]

Marc 'BlackJack' Rintsch wrote:

In <11**********************@m73g2000cwd.googlegroups .com>, Lad wrote:

The text is from Mysql table field that uses utf8_czech_ci collation,
but when I try
`RealName`.decode('utf8'),where RealName is that field of MySQL

I will get:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3:
ordinal
not in range(128)

Can you please suggest the solution?

Do you get this from converting the value from the database or from trying
to print the unicode string? Can you give us the output of

print repr(RealName)

Ciao,

Marc 'BlackJack' Rintsch

for
print repr(RealName) command
I will get

P?ibylov\xe1 Ludmila
where instead of ? should be also a character
Thank you for help
L.

[Fredrik Lundh]

Lad wrote:

for
print repr(RealName) command
I will get

P?ibylov\xe1 Ludmila
where instead of ? should be also a character

that's not very likely; repr() always includes quotes, always escapes
non-ASCII characters, and optionally includes a Unicode prefix.

please try this

print "*", repr(RealName), type(RealName), "*"

and post the entire output; that is, *everything* between the asterisks.

</F>

[Lad]

Fredrik Lundh wrote:

Lad wrote:

for
print repr(RealName) command
I will get

P?ibylov\xe1 Ludmila
where instead of ? should be also a character

that's not very likely; repr() always includes quotes, always escapes
non-ASCII characters, and optionally includes a Unicode prefix.

please try this

print "*", repr(RealName), type(RealName), "*"

and post the entire output; that is, *everything* between the asterisks.

The result of print "*", repr(RealName), type(RealName), "*" is

* 'Fritschov\xe1 Laura' <type 'str'*
Best regards,
L

[Fredrik Lundh]

"Lad" wrote:

The result of print "*", repr(RealName), type(RealName), "*" is

* 'Fritschov\xe1 Laura' <type 'str'*

looks like the MySQL interface is returning 8-bit strings using ISO-8859-1
encoding (or some variation of that; \xE1 is "LATIN SMALL LETTER A
WITH ACUTE" in 8859-1).

have you tried passing "use_unicode=True" to the connect() call ?

</F>

[Lad]

Fredrik Lundh wrote:

"Lad" wrote:

The result of print "*", repr(RealName), type(RealName), "*" is

* 'Fritschov\xe1 Laura' <type 'str'*

looks like the MySQL interface is returning 8-bit strings using ISO-8859-1
encoding (or some variation of that; \xE1 is "LATIN SMALL LETTER A
WITH ACUTE" in 8859-1).

have you tried passing "use_unicode=True" to the connect() call ?

</F>

Frederik,
Thank you for your reply.
I found out that if I do not decode the string at all, it looks
correct. But I do not know why it is ok without decoding.
I use Django and I do not use use_unicode=True" to the connect() call.