Hi everyone,
I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>", the value is probably an e with one of the french accents)
I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...
If someone could find me a solution, that'd really be a lifesaver.
I've been losing hours and hours over this one :s
thijs 9 15727
On 8/31/07, th*********@gma il.com <th*********@gm ail.comwrote:
Hi everyone,
I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>", the value is probably an e with one of the french accents)
I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...
encode().
You didn't post the code that was failing, I can encode that value
into UTF-8 (and unless I'm very much mistaken, you should be able to
encode any unicode string to UTF-8).
On Sep 1, 8:55 am, thijs.br...@gma il.com wrote:
Hi everyone,
I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>",
I'm a pig-ignorant Anglo; it's news to me that Python error messages
varied by locale; I thought they always came out in ASCII as G(od|
uido) intended :-) Does that message emanate from Python or psycopg?
In either case, it is saying that it is expecting a UTF8-encoded
string, but the string given to it is not a valid UTF8-encoded string.
the value is probably an e with one of the french accents)
PROBABLY?? (1) Please try to understand that computers are quite
deterministic. (2) If you want help, stop guessing and use something
like
print repr(the_value)
and tell us what it *actually* is. Also show us the *relevant* parts
of your code, so that we can see how your are trying to convert your
data, and how you are trying to pass it to psycopg. Also show us the
full traceback that you get.
>
I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...
If someone could find me a solution, that'd really be a lifesaver.
I've been losing hours and hours over this one :s
1. Find out what your input data actually is (e.g. unicode)
2. Find out what form psycopg requires (e.g. utf8-encoded str).
3. unicode to utf8 is quite simple:
>>useq = u"S\xe9quenc e"
# that's "Sequence" with an acute accent on the first "e"
>>useq8 = useq.encode('ut f8') print repr(useq)
u'S\xe9quence'
>>print repr(useq8)
'S\xc3\xa9quenc e'
>>useq8.decode( 'utf8')
u'S\xe9quence'
# round trip works as expected
Here is what ASCII-Python says about malformed UTF8:
>>"\xe0\x20\x63 ".decode('utf8' )
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\python25\li b\encodings\utf _8.py", line 16, in decode
return codecs.utf_8_de code(input, errors, True)
UnicodeDecodeEr ror: 'utf8' codec can't decode bytes in position 0-2:
invalid data
Cheers,
John
On Sep 1, 9:56 am, "Chris Mellon" <arka...@gmail. comwrote:
On 8/31/07, thijs.br...@gma il.com <thijs.br...@gm ail.comwrote:
Hi everyone,
I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
(sorry, locale is french, it means "byte sequence invalid for encoding
<<utf8>>", the value is probably an e with one of the french accents)
I've found lots of stuff about this googling the error, but I don't
seem to be able to find a "works always"-function just to convert a
unicode variable back to string...
encode().
You didn't post the code that was failing, I can encode that value
into UTF-8
What is "that value"?
(1) unichr(0xe02063 )? You must have a wide unicode build of Python ...
(2) u"\xe0\x20\x63" ? Of course you can encode it; so what?
(and unless I'm very much mistaken, you should be able to
encode any unicode string to UTF-8).
That is true, by definition. However you are barking this truism up
the wrong tree. The unknown complainant's whinge is that it is
expecting a sequence of octets (an 8-bit string) that is valid UTF8,
but the actuality is something else. It is *NOT* trying to say that a
unicode input can't be converted to UTF8.
On Fri, 2007-08-31 at 15:55 -0700, th*********@gma il.com wrote:
Hi everyone,
I'm having quite some troubles trying to convert Unicode to String
(for use in psycopg, which apparently doesn't know how to cope with
unicode strings).
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
I'm guessing that you are passing a latin-1 encoded string and pretend
(or psycopg assumes) incorrectly that it's UTF-8 encoded. In latin-1
encoding, 0xe0 is a small letter a with a grave accent, 0x20 is a space,
and 0x63 is a small letter c. While this is a perfectly valid latin-1
encoded character string, it doesn't represent a valid UTF-8 encoded
character string.
It seems that you need to pass a UTF-8 encoded string to the database.
To give you specific advice on how to do that, we'd have to see your
code. For now, I'll give you the generic advice of taking a look at http://www.amk.ca/python/howto/unicode .
HTH,
--
Carsten Haese http://informixdb.sourceforge.net
In message <11************ **********@d55g 2000hsg.googleg roups.com>, th*********@gma il.com wrote:
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
It would be useful to see some actual code snippet, traceback listing etc.
First make sure your DB encoding is UTF-8 not the latin1
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
then try this:
def smart_str(s, encoding='utf-8', errors='strict' ):
"""
Returns a bytestring version of 's', encoded as specified in
'encoding'.
"""
if not isinstance(s, basestring):
try:
return str(s)
except UnicodeEncodeEr ror:
return unicode(s).enco de(encoding, errors)
elif isinstance(s, unicode):
return s.encode(encodi ng, errors)
elif s and encoding != 'utf-8':
return s.decode('utf-8', errors).encode( encoding, errors)
else:
return s
Sorry for answering so late. Thanks a million! This code snippet
helped me solve the problem.
I think I will be using SQLAlchemy for these sorts of things from now
on though, it seems to be taking care of these things itself, on top
of being one hell of a handy ORM of course :)
thijs
On 1 sep, 09:17, iapain <iap...@gmail.c omwrote:
First make sure your DB encoding is UTF-8 not the latin1
The error I keep having is something like this:
ERREUR: Séquence d'octets invalide pour le codage «UTF8» : 0xe02063
then try this:
def smart_str(s, encoding='utf-8', errors='strict' ):
"""
Returns a bytestring version of 's', encoded as specified in
'encoding'.
"""
if not isinstance(s, basestring):
try:
return str(s)
except UnicodeEncodeEr ror:
return unicode(s).enco de(encoding, errors)
elif isinstance(s, unicode):
return s.encode(encodi ng, errors)
elif s and encoding != 'utf-8':
return s.decode('utf-8', errors).encode( encoding, errors)
else:
return s
On 1 sep, 09:17, iapain <iap...@gmail.c omwrote:
>
First make sure your DB encoding is UTF-8 not the latin1
It took me days to figure out what was going on when dealing with
unicode, ascii, latin1, utf8, decodeerrors, etc, so I'm just chiming
in to echo something similar iapain's comments:
When dealing with unicode, i've run into situations where I have
multiple encodings in the same string, usually latin1 and utf8
(latin1 != ascii, and latin1 != utf8, and they don't play nice
together). So, for future readers, if you have problems dealing with
unicode encode and decode, try using a mix of latin1 and utf8
encodings to figure out whats going on, and what characters are
fubar'ing the process.
En Mon, 17 Sep 2007 01:33:14 -0300, Richard Levasseur
<ri********@gma il.comescribiï¿ ½:
When dealing with unicode, i've run into situations where I have
multiple encodings in the same string, usually latin1 and utf8
(latin1 != ascii, and latin1 != utf8, and they don't play nice
together). So, for future readers, if you have problems dealing with
unicode encode and decode, try using a mix of latin1 and utf8
encodings to figure out whats going on, and what characters are
fubar'ing the process.
Life is easier if you follow these guidelines:
- work internally always in Unicode (not strings)
- All input data (read from files, coming from an Internet connection,
typed by user...) should be decoded from byte strings into unicode as
early as possible. (You should know which encoding your data comes in, in
each case)
- All output data (written to files, printing to screen, etc) is encoded
from unicode into byte strings as late as possible.
This way, unless your input data is garbage, you never could mix strings
from different encodings.
For further information, read the Unicode Howto
<http://www.amk.ca/python/howto/unicodeand this excerpt form the "Python
Cookbook", by Alex Martelli
<http://www.onlamp.com/pub/a/python/excerpt/pythonckbk_chap 1/index.html>
--
Gabriel Genellina This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: nico |
last post by:
In my python scripts, I use a lot of accented characters as I work in
french.
In order to do this, I put the line
# -*- coding: UTF-8 -*-
at the beginning of the script file.
Then, when I need to store accented characters in a string, I used to
prefix the literal string with 'u', like this:
mystring = u"prénom"
But if I understand well, prefixing a unicode string literal with 'u'
|
by: Shrii |
last post by:
1.I read a unicode file by using codec
2.I want to pass that string to exec() statement
3.But one of my character (U+0950) in that string is not showing
properly in the output got by that exec() statement
could anyone help me to get proper output. ?
somesh
|
by: Ger |
last post by:
I have not been able to find a simple, straight forward Unicode to ASCII
string conversion function in VB.Net.
Is that because such a function does not exists or do I overlook it?
I found Encoding.Convert, but that needs byte arrays.
Thanks,
/Ger
|
by: aurora |
last post by:
I have some unicode string with some characters encode using python
notation like '\n' for LF. I need to convert that to the actual LF
character. There is a 'unicode_escape' codec that seems to suit my purpose.
>>> encoded = u'A\\nA'
>>> decoded = encoded.decode('unicode_escape')
>>> print len(decoded)
3
Note that both encoded and decoded are unicode string. I'm trying to use
|
by: Preben Randhol |
last post by:
Hi
If I use len() on a string containing unicode letters I get the number
of bytes the string uses. This means that len() can report size 6 when
the unicode string only contains 3 characters (that one would write by
hand or see on the screen). Is there a way to calculate in characters
and not in bytes to represent the characters.
The reason for asking is that PyGTK needs number of characters to set
the width of Entry widgets to a...
| |
by: willie |
last post by:
Martin v. Löwis:
Thanks for the thorough explanation. One last question
about terminology then I'll go away :)
What is the proper way to describe "ustr" below?
<type 'unicode'>
|
by: Martin Landa |
last post by:
Hi all,
sorry for a newbie question. I have unicode string (or better say
latin2 encoding) containing non-ascii characters, e.g.
s = "Ukázka_možnosti_využití_programu_OpenJUMP_v_SOA"
I would like to convert this string to plain ascii (using some lookup
table for latin2)
|
by: vcnewbie |
last post by:
Hi
I'm maintaining a VisualC++ project to increase its security regarding
stored passwords.
I thought about using SHA256Managed to create a hash for the password
when creating a user and when this new user tries to login, a new hash
will be created for the given password and compared to the stored
hash. I guess this is quite common.
|
by: Chrysie |
last post by:
Hi,
I am new to python. I am using python 2.6.6 with pyodbc-2.1.8 and pywin32-216 on Windows Vista.
I was able to connect to MS Access with pyodbc and execute my SELECT statement to retrieve data from MS Access. However, what I have retrieved appeared to be in Unicode strings (e.g., u'xyz', etc.) which I could not use as keys to build a dictionary, and I could not use as strings to write regular expressions to match with certain patterns....
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |