469,338 Members | 8,424 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,338 developers. It's quick & easy.

Python Unicode to String conversion

Hi everyone,

I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>", the value is probably an e with one of the french accents)

I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...

If someone could find me a solution, that'd really be a lifesaver.
I've been losing hours and hours over this one :s

thijs

Aug 31 '07 #1
9 15394
On 8/31/07, th*********@gmail.com <th*********@gmail.comwrote:
Hi everyone,

I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>", the value is probably an e with one of the french accents)

I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...

encode().

You didn't post the code that was failing, I can encode that value
into UTF-8 (and unless I'm very much mistaken, you should be able to
encode any unicode string to UTF-8).
Sep 1 '07 #2
On Sep 1, 8:55 am, thijs.br...@gmail.com wrote:
Hi everyone,

I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>",
I'm a pig-ignorant Anglo; it's news to me that Python error messages
varied by locale; I thought they always came out in ASCII as G(od|
uido) intended :-) Does that message emanate from Python or psycopg?
In either case, it is saying that it is expecting a UTF8-encoded
string, but the string given to it is not a valid UTF8-encoded string.
the value is probably an e with one of the french accents)
PROBABLY?? (1) Please try to understand that computers are quite
deterministic. (2) If you want help, stop guessing and use something
like
print repr(the_value)
and tell us what it *actually* is. Also show us the *relevant* parts
of your code, so that we can see how your are trying to convert your
data, and how you are trying to pass it to psycopg. Also show us the
full traceback that you get.
>
I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...

If someone could find me a solution, that'd really be a lifesaver.
I've been losing hours and hours over this one :s
1. Find out what your input data actually is (e.g. unicode)
2. Find out what form psycopg requires (e.g. utf8-encoded str).
3. unicode to utf8 is quite simple:
>>useq = u"S\xe9quence"
# that's "Sequence" with an acute accent on the first "e"
>>useq8 = useq.encode('utf8')
print repr(useq)
u'S\xe9quence'
>>print repr(useq8)
'S\xc3\xa9quence'
>>useq8.decode('utf8')
u'S\xe9quence'
# round trip works as expected

Here is what ASCII-Python says about malformed UTF8:
>>"\xe0\x20\x63".decode('utf8')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\python25\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-2:
invalid data

Cheers,
John

Sep 1 '07 #3
On Sep 1, 9:56 am, "Chris Mellon" <arka...@gmail.comwrote:
On 8/31/07, thijs.br...@gmail.com <thijs.br...@gmail.comwrote:
Hi everyone,
I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>", the value is probably an e with one of the french accents)
I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...

encode().

You didn't post the code that was failing, I can encode that value
into UTF-8
What is "that value"?
(1) unichr(0xe02063)? You must have a wide unicode build of Python ...
(2) u"\xe0\x20\x63"? Of course you can encode it; so what?
(and unless I'm very much mistaken, you should be able to
encode any unicode string to UTF-8).
That is true, by definition. However you are barking this truism up
the wrong tree. The unknown complainant's whinge is that it is
expecting a sequence of octets (an 8-bit string) that is valid UTF8,
but the actuality is something else. It is *NOT* trying to say that a
unicode input can't be converted to UTF8.

Sep 1 '07 #4
On Fri, 2007-08-31 at 15:55 -0700, th*********@gmail.com wrote:
Hi everyone,

I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
I'm guessing that you are passing a latin-1 encoded string and pretend
(or psycopg assumes) incorrectly that it's UTF-8 encoded. In latin-1
encoding, 0xe0 is a small letter a with a grave accent, 0x20 is a space,
and 0x63 is a small letter c. While this is a perfectly valid latin-1
encoded character string, it doesn't represent a valid UTF-8 encoded
character string.

It seems that you need to pass a UTF-8 encoded string to the database.
To give you specific advice on how to do that, we'd have to see your
code. For now, I'll give you the generic advice of taking a look at
http://www.amk.ca/python/howto/unicode .

HTH,

--
Carsten Haese
http://informixdb.sourceforge.net
Sep 1 '07 #5
In message <11**********************@d55g2000hsg.googlegroups .com>,
th*********@gmail.com wrote:
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
It would be useful to see some actual code snippet, traceback listing etc.
Sep 1 '07 #6
First make sure your DB encoding is UTF-8 not the latin1
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
then try this:

def smart_str(s, encoding='utf-8', errors='strict'):
"""
Returns a bytestring version of 's', encoded as specified in
'encoding'.
"""
if not isinstance(s, basestring):
try:
return str(s)
except UnicodeEncodeError:
return unicode(s).encode(encoding, errors)
elif isinstance(s, unicode):
return s.encode(encoding, errors)
elif s and encoding != 'utf-8':
return s.decode('utf-8', errors).encode(encoding, errors)
else:
return s

Sep 1 '07 #7
Sorry for answering so late. Thanks a million! This code snippet
helped me solve the problem.

I think I will be using SQLAlchemy for these sorts of things from now
on though, it seems to be taking care of these things itself, on top
of being one hell of a handy ORM of course :)

thijs

On 1 sep, 09:17, iapain <iap...@gmail.comwrote:
First make sure your DB encoding is UTF-8 not the latin1
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

then try this:

def smart_str(s, encoding='utf-8', errors='strict'):
"""
Returns a bytestring version of 's', encoded as specified in
'encoding'.
"""
if not isinstance(s, basestring):
try:
return str(s)
except UnicodeEncodeError:
return unicode(s).encode(encoding, errors)
elif isinstance(s, unicode):
return s.encode(encoding, errors)
elif s and encoding != 'utf-8':
return s.decode('utf-8', errors).encode(encoding, errors)
else:
return s

Sep 16 '07 #8
On 1 sep, 09:17, iapain <iap...@gmail.comwrote:
>
First make sure your DB encoding is UTF-8 not the latin1
It took me days to figure out what was going on when dealing with
unicode, ascii, latin1, utf8, decodeerrors, etc, so I'm just chiming
in to echo something similar iapain's comments:

When dealing with unicode, i've run into situations where I have
multiple encodings in the same string, usually latin1 and utf8
(latin1 != ascii, and latin1 != utf8, and they don't play nice
together). So, for future readers, if you have problems dealing with
unicode encode and decode, try using a mix of latin1 and utf8
encodings to figure out whats going on, and what characters are
fubar'ing the process.

Sep 17 '07 #9
En Mon, 17 Sep 2007 01:33:14 -0300, Richard Levasseur
<ri********@gmail.comescribi�:
When dealing with unicode, i've run into situations where I have
multiple encodings in the same string, usually latin1 and utf8
(latin1 != ascii, and latin1 != utf8, and they don't play nice
together). So, for future readers, if you have problems dealing with
unicode encode and decode, try using a mix of latin1 and utf8
encodings to figure out whats going on, and what characters are
fubar'ing the process.
Life is easier if you follow these guidelines:
- work internally always in Unicode (not strings)
- All input data (read from files, coming from an Internet connection,
typed by user...) should be decoded from byte strings into unicode as
early as possible. (You should know which encoding your data comes in, in
each case)
- All output data (written to files, printing to screen, etc) is encoded
from unicode into byte strings as late as possible.

This way, unless your input data is garbage, you never could mix strings
from different encodings.
For further information, read the Unicode Howto
<http://www.amk.ca/python/howto/unicodeand this excerpt form the "Python
Cookbook", by Alex Martelli
<http://www.onlamp.com/pub/a/python/excerpt/pythonckbk_chap1/index.html>

--
Gabriel Genellina

Sep 17 '07 #10

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

6 posts views Thread by nico | last post: by
3 posts views Thread by Shrii | last post: by
8 posts views Thread by Preben Randhol | last post: by
2 posts views Thread by willie | last post: by
5 posts views Thread by Martin Landa | last post: by
4 posts views Thread by vcnewbie | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by zhoujie | last post: by
1 post views Thread by haryvincent176 | last post: by
1 post views Thread by Marylou17 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.