Strange problems with encoding

[Sebastian Meyer]

Hi newsgroup,

i am trying to replace german special characters in strings like
str = re.sub('ö', 'oe', str)

When i work with this, i always get the message
UniCode Error: ASCII decoding error : ordinal not in range(128)

Yes i have googled, i searched the faq, manual and python library and
searched all known soruces of information. I played with the python
builtin function encode to enforce the rigth encoding, but the error
stays the same. I ve read a lot about UniCode and internal conversion
about Strings done by python, but somehow i ve missed the clue.
Nope, python says Huuups... ordinal not in range(128), ;-(

Anyone of you having any idea?? Seems like i am too stupid to read
documentation carefully., perhaps i misunderstand something...

thanks for your help in advance

Sebastian

[Rudy Schockaert]

Sebastian Meyer wrote:

Hi newsgroup,

i am trying to replace german special characters in strings like
str = re.sub('ö', 'oe', str)

When i work with this, i always get the message
UniCode Error: ASCII decoding error : ordinal not in range(128)

Yes i have googled, i searched the faq, manual and python library and
searched all known soruces of information. I played with the python
builtin function encode to enforce the rigth encoding, but the error
stays the same. I ve read a lot about UniCode and internal conversion
about Strings done by python, but somehow i ve missed the clue.
Nope, python says Huuups... ordinal not in range(128), ;-(

Anyone of you having any idea?? Seems like i am too stupid to read
documentation carefully., perhaps i misunderstand something...

thanks for your help in advance

Sebastian

I'm experiencing something similar for the moment. I try to
base64-encode Unicode strings and I get the exact same errormessage.

s = u'ö'
s u'\xf6' s.encode('base6 4') Traceback (most recent call last):
File "<interacti ve input>", line 1, in ?
File "C:\Python23\li b\encodings\bas e64_codec.py", line 24, in
base64_encode
output = base64.encodest ring(input)
File "C:\Python23\li b\base64.py", line 39, in encodestring
pieces.append(b inascii.b2a_bas e64(chunk))
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xe9' in
position 0: ordinal not in range(128)

When I don't specify it's unicode it works: s = 'ö'
s '\xf6' s.encode('base6 4')

'9g==\n'

The reason I want to base64-encode these unicode strings is because I
get those as input and want to store them in a MySQL database using
SQLObject.

[Michael Hudson]

"Sebastian Meyer" <s.*****@techno logy-network.de> writes:

Hi newsgroup,

i am trying to replace german special characters in strings like
str = re.sub('ö', 'oe', str)

1) str is the name of a builtin -- often a bad idea to use that as a
variable name.

2) I presume `str' is a unicode string? Try writing the literal as
u'ö' instead (and adding the appropriate coding cookie to your
source file if using Python 2.3). Or I guess you could write it

u'\N{LATIN SMALL LETTER O WITH DIAERESIS}'

Cheers,
mwh

--
Usenet is like a herd of performing elephants with diarrhea --
massive, difficult to redirect, awe-inspiring, entertaining, and
a source of mind-boggling amounts of excrement when you least
expect it. -- spaf (1992)

[Sebastian Meyer]

On Thu, 06 Nov 2003 13:39:25 +0000, Michael Hudson wrote:

"Sebastian Meyer" <s.*****@techno logy-network.de> writes:

Hi newsgroup,

i am trying to replace german special characters in strings like
str = re.sub('ö', 'oe', str)
1) str is the name of a builtin -- often a bad idea to use that as a
variable name.

it was only the example name for the variable, be sure that dont
use any builtins as variable names
maybe not a good example ... thanks for the hint

2) I presume `str' is a unicode string? Try writing the literal as
u'ö' instead (and adding the appropriate coding cookie to your
source file if using Python 2.3). Or I guess you could write it

u'\N{LATIN SMALL LETTER O WITH DIAERESIS}'
i ll try and report back...

Cheers,
mwh

[Joe Fromm]

"Sebastian Meyer" <s.*****@techno logy-network.de> wrote in message
news:pa******** *************** ****@technology-network.de...

Hi newsgroup,

i am trying to replace german special characters in strings like
str = re.sub('ö', 'oe', str)

When i work with this, i always get the message
UniCode Error: ASCII decoding error : ordinal not in range(128)

Try adding

sys.setdefaulte ncoding( 'latin-1' )

to your site.py module, or rewrite your fragment as

from = 'ö'
to = 'oe'
s = re.sub( from.encode('la tin-1'), to.encode('lati n-1', s )

If you are running on Windows you might want to change 'latin-1' to 'mbcs',
as that seems to be the most forgiving codec, but it is Windows only.

Joe

[Michael Hudson]

Rudy Schockaert <ru************ *@pandoraSTOPSP AM.be> writes:

Sebastian Meyer wrote:

Hi newsgroup,
i am trying to replace german special characters in strings like
str = re.sub('ö', 'oe', str)
When i work with this, i always get the message
UniCode Error: ASCII decoding error : ordinal not in range(128)
Yes i have googled, i searched the faq, manual and python library
and
searched all known soruces of information. I played with the python
builtin function encode to enforce the rigth encoding, but the error
stays the same. I ve read a lot about UniCode and internal conversion
about Strings done by python, but somehow i ve missed the clue.
Nope, python says Huuups... ordinal not in range(128), ;-(
Anyone of you having any idea?? Seems like i am too stupid to read
documentation carefully., perhaps i misunderstand something...
thanks for your help in advance
Sebastian
I'm experiencing something similar for the moment. I try to
base64-encode Unicode strings and I get the exact same errormessage.

"base64-encoding Unicode strings" is not a particularly well defined
operation. "base64-encoding" is a way of turning *binary data* into a
particularly "safe" sequence of ascii characters.

Unicode (in some sense) is a family of ways of representing strings of
characters as binary data.

So to base-64 encode a Unicode string, you need to choose *which*
member of this family you're going to use, which is to say the
encoding. UTF-8 would seem a good bet.

But...

>>> s = u'ö'
>>> s u'\xf6' >>> s.encode('base6 4') Traceback (most recent call last):
File "<interacti ve input>", line 1, in ?
File "C:\Python23\li b\encodings\bas e64_codec.py", line 24, in
base64_encode
output = base64.encodest ring(input)
File "C:\Python23\li b\base64.py", line 39, in encodestring
pieces.append(b inascii.b2a_bas e64(chunk))
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xe9' in
position 0: ordinal not in range(128)

u'ö'.encode('ut f-8').encode('bas e64') 'w7Y=\n'
When I don't specify it's unicode it works:

>>> s = 'ö'
>>> s '\xf6' >>> s.encode('base6 4')

'9g==\n'

Well, this works because your terminal seems to be latin-1:
u'ö'.encode('la tin-1').encode('bas e64')

'9g==\n'

What would you like to do with a character that isn't in latin-1?
The reason I want to base64-encode these unicode strings is because I
get those as input and want to store them in a MySQL database using
SQLObject.

! Why can't you just encode them as utf-8 strings? (Or, thinking
about it, why doesn't SQLObject support unicode?)

Cheers,
mwh

--
I think if we have the choice, I'd rather we didn't explicitly put
flaws in the reST syntax for the sole purpose of not insulting the
almighty. -- /will on the doc-sig

[Peter Otten]

Sebastian Meyer wrote:

Hi newsgroup,

i am trying to replace german special characters in strings like
str = re.sub('ö', 'oe', str)

When i work with this, i always get the message
UniCode Error: ASCII decoding error : ordinal not in range(128)

Yes i have googled, i searched the faq, manual and python library and
searched all known soruces of information. I played with the python
builtin function encode to enforce the rigth encoding, but the error
stays the same. I ve read a lot about UniCode and internal conversion
about Strings done by python, but somehow i ve missed the clue.
Nope, python says Huuups... ordinal not in range(128), ;-(

Anyone of you having any idea?? Seems like i am too stupid to read
documentation carefully., perhaps i misunderstand something...

thanks for your help in advance

Sebastian

Works here, even with my older snake:

Python 2.2.1 (#1, Sep 10 2002, 17:49:17)
[GCC 3.2] on linux2
Type "help", "copyright" , "credits" or "license" for more information.

import re
re.sub("ö", "oe", "Döspaddel" ) 'Doespaddel' re.sub("ö", "oe", u"Döspaddel" ) u'Doespaddel' re.sub("ö", u"oe", u"Döspaddel" ) u'Doespaddel' re.sub(u"ö", u"oe", u"Döspaddel" ) u'Doespaddel'

To provoke a UnicodeError, I have to convert a unicode string with umlauts
to str without providing the encoding:
str(u"Döspaddel ") Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeError: ASCII encoding error: ordinal not in range(128)

I suspect that you have something similar hidden in your code (i. e.
characters >= 128 that are not converted). The remedy is to explicitly
decode with the appropriate encoding:
u"Döspaddel".en code("latin-1") 'D\xf6spaddel'

Try to build a minimal script that shows the reported behaviour and fix it
or post it for more detailed advice. By the way, don't use str as a
variable name, it's the type of "ordinary" strings.

Peter

[Rudy Schockaert]

Joe Fromm wrote:

Try adding

sys.setdefaulte ncoding( 'latin-1' )

to your site.py module, or rewrite your fragment as

At the end of site.py you can enable a piece of code that sets your
default encoding to the current locale of your computer:

if 1:
# Enable to support locale aware default string encodings.
import locale
loc = locale.getdefau ltlocale()
if loc[1]:
encoding = loc[1]

This works great for me.

Thanks for pointing me to site.py

P.S. I really need some weeks off so I can read all the available
documentation ;-)

[Rudy Schockaert]

>

u'ö'.encode ('utf-8').encode('bas e64')

'w7Y=\n'

This works indeed. And thanks to Joe Fromm's hint (site.py) I don't have
to worry about it anymore.
What would you like to do with a character that isn't in latin-1?
Actually, I don't care as long as the encode and decode on the same
machine give me back the original value.

The reason I want to base64-encode these unicode strings is because I
get those as input and want to store them in a MySQL database using
SQLObject.

! Why can't you just encode them as utf-8 strings? (Or, thinking
about it, why doesn't SQLObject support unicode?)

The actual input strings don't really contain unicode text values, but
rather binary values i get as result from calling win32.NetUserEn um.

The manual of SQLObject (great product btw) explains how you can easily
store binary data in a SQL table by encoding it when setting and
decoding it when getting the value. Tha is just what I was trying to do.

[Michael Hudson]

Rudy Schockaert <ru************ *@pandoraSTOPSP AM.be> writes:

>u'ö'.encode ('utf-8').encode('bas e64')

'w7Y=\n'

This works indeed. And thanks to Joe Fromm's hint (site.py) I don't
have to worry about it anymore.

Well, I'm from the setdefaultencod ing-is-evil camp, but it sounds like
you're in a pretty icky situation.

What would you like to do with a character that isn't in latin-1?

Actually, I don't care as long as the encode and decode on the same
machine give me back the original value.

Huh?

The reason I want to base64-encode these unicode strings is because I
get those as input and want to store them in a MySQL database using
SQLObject.

! Why can't you just encode them as utf-8 strings? (Or, thinking
about it, why doesn't SQLObject support unicode?)

The actual input strings don't really contain unicode text values, but
rather binary values i get as result from calling win32.NetUserEn um.

Oh, so they're not really unicode strings at all? Blech. That's
really really nasty. Binary data should really be represented as
(narrow) strings in Python. Perhaps the utf-16-le codec would be the
most appropriate...

Cheers,
mwh

--
Q: What are 1000 lawyers at the bottom of the ocean?
A: A good start.
(A lawyer told me this joke.)
-- Michael Ströder, comp.lang.pytho n