473,576 Members | 3,366 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Python Unicode to String conversion

Hi everyone,

I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>", the value is probably an e with one of the french accents)

I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...

If someone could find me a solution, that'd really be a lifesaver.
I've been losing hours and hours over this one :s

thijs

Aug 31 '07 #1
9 15726
On 8/31/07, th*********@gma il.com <th*********@gm ail.comwrote:
Hi everyone,

I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>", the value is probably an e with one of the french accents)

I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...

encode().

You didn't post the code that was failing, I can encode that value
into UTF-8 (and unless I'm very much mistaken, you should be able to
encode any unicode string to UTF-8).
Sep 1 '07 #2
On Sep 1, 8:55 am, thijs.br...@gma il.com wrote:
Hi everyone,

I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>",
I'm a pig-ignorant Anglo; it's news to me that Python error messages
varied by locale; I thought they always came out in ASCII as G(od|
uido) intended :-) Does that message emanate from Python or psycopg?
In either case, it is saying that it is expecting a UTF8-encoded
string, but the string given to it is not a valid UTF8-encoded string.
the value is probably an e with one of the french accents)
PROBABLY?? (1) Please try to understand that computers are quite
deterministic. (2) If you want help, stop guessing and use something
like
print repr(the_value)
and tell us what it *actually* is. Also show us the *relevant* parts
of your code, so that we can see how your are trying to convert your
data, and how you are trying to pass it to psycopg. Also show us the
full traceback that you get.
>
I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...

If someone could find me a solution, that'd really be a lifesaver.
I've been losing hours and hours over this one :s
1. Find out what your input data actually is (e.g. unicode)
2. Find out what form psycopg requires (e.g. utf8-encoded str).
3. unicode to utf8 is quite simple:
>>useq = u"S\xe9quenc e"
# that's "Sequence" with an acute accent on the first "e"
>>useq8 = useq.encode('ut f8')
print repr(useq)
u'S\xe9quence'
>>print repr(useq8)
'S\xc3\xa9quenc e'
>>useq8.decode( 'utf8')
u'S\xe9quence'
# round trip works as expected

Here is what ASCII-Python says about malformed UTF8:
>>"\xe0\x20\x63 ".decode('utf8' )
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\python25\li b\encodings\utf _8.py", line 16, in decode
return codecs.utf_8_de code(input, errors, True)
UnicodeDecodeEr ror: 'utf8' codec can't decode bytes in position 0-2:
invalid data

Cheers,
John

Sep 1 '07 #3
On Sep 1, 9:56 am, "Chris Mellon" <arka...@gmail. comwrote:
On 8/31/07, thijs.br...@gma il.com <thijs.br...@gm ail.comwrote:
Hi everyone,
I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>", the value is probably an e with one of the french accents)
I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...

encode().

You didn't post the code that was failing, I can encode that value
into UTF-8
What is "that value"?
(1) unichr(0xe02063 )? You must have a wide unicode build of Python ...
(2) u"\xe0\x20\x63" ? Of course you can encode it; so what?
(and unless I'm very much mistaken, you should be able to
encode any unicode string to UTF-8).
That is true, by definition. However you are barking this truism up
the wrong tree. The unknown complainant's whinge is that it is
expecting a sequence of octets (an 8-bit string) that is valid UTF8,
but the actuality is something else. It is *NOT* trying to say that a
unicode input can't be converted to UTF8.

Sep 1 '07 #4
On Fri, 2007-08-31 at 15:55 -0700, th*********@gma il.com wrote:
Hi everyone,

I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).

The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
I'm guessing that you are passing a latin-1 encoded string and pretend
(or psycopg assumes) incorrectly that it's UTF-8 encoded. In latin-1
encoding, 0xe0 is a small letter a with a grave accent, 0x20 is a space,
and 0x63 is a small letter c. While this is a perfectly valid latin-1
encoded character string, it doesn't represent a valid UTF-8 encoded
character string.

It seems that you need to pass a UTF-8 encoded string to the database.
To give you specific advice on how to do that, we'd have to see your
code. For now, I'll give you the generic advice of taking a look at
http://www.amk.ca/python/howto/unicode .

HTH,

--
Carsten Haese
http://informixdb.sourceforge.net
Sep 1 '07 #5
In message <11************ **********@d55g 2000hsg.googleg roups.com>,
th*********@gma il.com wrote:
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
It would be useful to see some actual code snippet, traceback listing etc.
Sep 1 '07 #6
First make sure your DB encoding is UTF-8 not the latin1
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
then try this:

def smart_str(s, encoding='utf-8', errors='strict' ):
"""
Returns a bytestring version of 's', encoded as specified in
'encoding'.
"""
if not isinstance(s, basestring):
try:
return str(s)
except UnicodeEncodeEr ror:
return unicode(s).enco de(encoding, errors)
elif isinstance(s, unicode):
return s.encode(encodi ng, errors)
elif s and encoding != 'utf-8':
return s.decode('utf-8', errors).encode( encoding, errors)
else:
return s

Sep 1 '07 #7
Sorry for answering so late. Thanks a million! This code snippet
helped me solve the problem.

I think I will be using SQLAlchemy for these sorts of things from now
on though, it seems to be taking care of these things itself, on top
of being one hell of a handy ORM of course :)

thijs

On 1 sep, 09:17, iapain <iap...@gmail.c omwrote:
First make sure your DB encoding is UTF-8 not the latin1
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063

then try this:

def smart_str(s, encoding='utf-8', errors='strict' ):
"""
Returns a bytestring version of 's', encoded as specified in
'encoding'.
"""
if not isinstance(s, basestring):
try:
return str(s)
except UnicodeEncodeEr ror:
return unicode(s).enco de(encoding, errors)
elif isinstance(s, unicode):
return s.encode(encodi ng, errors)
elif s and encoding != 'utf-8':
return s.decode('utf-8', errors).encode( encoding, errors)
else:
return s

Sep 16 '07 #8
On 1 sep, 09:17, iapain <iap...@gmail.c omwrote:
>
First make sure your DB encoding is UTF-8 not the latin1
It took me days to figure out what was going on when dealing with
unicode, ascii, latin1, utf8, decodeerrors, etc, so I'm just chiming
in to echo something similar iapain's comments:

When dealing with unicode, i've run into situations where I have
multiple encodings in the same string, usually latin1 and utf8
(latin1 != ascii, and latin1 != utf8, and they don't play nice
together). So, for future readers, if you have problems dealing with
unicode encode and decode, try using a mix of latin1 and utf8
encodings to figure out whats going on, and what characters are
fubar'ing the process.

Sep 17 '07 #9
En Mon, 17 Sep 2007 01:33:14 -0300, Richard Levasseur
<ri********@gma il.comescribiï¿ ½:
When dealing with unicode, i've run into situations where I have
multiple encodings in the same string, usually latin1 and utf8
(latin1 != ascii, and latin1 != utf8, and they don't play nice
together). So, for future readers, if you have problems dealing with
unicode encode and decode, try using a mix of latin1 and utf8
encodings to figure out whats going on, and what characters are
fubar'ing the process.
Life is easier if you follow these guidelines:
- work internally always in Unicode (not strings)
- All input data (read from files, coming from an Internet connection,
typed by user...) should be decoded from byte strings into unicode as
early as possible. (You should know which encoding your data comes in, in
each case)
- All output data (written to files, printing to screen, etc) is encoded
from unicode into byte strings as late as possible.

This way, unless your input data is garbage, you never could mix strings
from different encodings.
For further information, read the Unicode Howto
<http://www.amk.ca/python/howto/unicodeand this excerpt form the "Python
Cookbook", by Alex Martelli
<http://www.onlamp.com/pub/a/python/excerpt/pythonckbk_chap 1/index.html>

--
Gabriel Genellina

Sep 17 '07 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
5481
by: nico | last post by:
In my python scripts, I use a lot of accented characters as I work in french. In order to do this, I put the line # -*- coding: UTF-8 -*- at the beginning of the script file. Then, when I need to store accented characters in a string, I used to prefix the literal string with 'u', like this: mystring = u"prénom" But if I understand well,...
3
3888
by: Shrii | last post by:
1.I read a unicode file by using codec 2.I want to pass that string to exec() statement 3.But one of my character (U+0950) in that string is not showing properly in the output got by that exec() statement could anyone help me to get proper output. ? somesh
18
34100
by: Ger | last post by:
I have not been able to find a simple, straight forward Unicode to ASCII string conversion function in VB.Net. Is that because such a function does not exists or do I overlook it? I found Encoding.Convert, but that needs byte arrays. Thanks, /Ger
2
10288
by: aurora | last post by:
I have some unicode string with some characters encode using python notation like '\n' for LF. I need to convert that to the actual LF character. There is a 'unicode_escape' codec that seems to suit my purpose. >>> encoded = u'A\\nA' >>> decoded = encoded.decode('unicode_escape') >>> print len(decoded) 3 Note that both encoded and...
8
20738
by: Preben Randhol | last post by:
Hi If I use len() on a string containing unicode letters I get the number of bytes the string uses. This means that len() can report size 6 when the unicode string only contains 3 characters (that one would write by hand or see on the screen). Is there a way to calculate in characters and not in bytes to represent the characters. The...
2
401
by: willie | last post by:
Martin v. Löwis: Thanks for the thorough explanation. One last question about terminology then I'll go away :) What is the proper way to describe "ustr" below? <type 'unicode'>
5
2619
by: Martin Landa | last post by:
Hi all, sorry for a newbie question. I have unicode string (or better say latin2 encoding) containing non-ascii characters, e.g. s = "Ukázka_možnosti_využití_programu_OpenJUMP_v_SOA" I would like to convert this string to plain ascii (using some lookup table for latin2)
4
2228
by: vcnewbie | last post by:
Hi I'm maintaining a VisualC++ project to increase its security regarding stored passwords. I thought about using SHA256Managed to create a hash for the password when creating a user and when this new user tries to login, a new hash will be created for the given password and compared to the stored hash. I guess this is quite common.
1
6533
by: Chrysie | last post by:
Hi, I am new to python. I am using python 2.6.6 with pyodbc-2.1.8 and pywin32-216 on Windows Vista. I was able to connect to MS Access with pyodbc and execute my SELECT statement to retrieve data from MS Access. However, what I have retrieved appeared to be in Unicode strings (e.g., u'xyz', etc.) which I could not use as keys to build a...
0
7776
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
8121
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8286
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7869
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
8143
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
1
5664
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
3779
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3797
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2287
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.