unicode encoding usablilty problem

[aurora]

I have long find the Python default encoding of strict ASCII frustrating.
For one thing I prefer to get garbage character than an exception. But the
biggest issue is Unicode exception often pop up in unexpected places and
only when a non-ASCII or unicode character first found its way into the
system.

Below is an example. The program may runs fine at the beginning. But as
soon as an unicode character u'b' is introduced, the program boom out
unexpectedly.

sys.getdefaulte ncoding() 'ascii' a='\xe5'
# can print, you think you're ok .... print a
å b=u'b'
a==b Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xe5 in position 0:
ordinal not in range(128)

One may suggest the correct way to do it is to use decode, such as

a.decode('latin-1') == b
This brings up another issue. Most references and books focus exclusive on
entering unicode literal and using the encode/decode methods. The fallacy
is that string is such a basic data type use throughout the program, you
really don't want to make a individual decision everytime when you use
string (and take a penalty for any negligence). The Java has a much more
usable model with unicode used internally and encoding/decoding decision
only need twice when dealing with input and output.

I am sure these errors are a nuisance to those who are half conscious to
unicode. Even for those who choose to use unicode, it is almost impossible
to ensure their program work correctly.

[Fredrik Lundh]

anonymous coward <au******@gmail .com> wrote:

This brings up another issue. Most references and books focus exclusive on entering unicode
literal and using the encode/decode methods. The fallacy is that string is such a basic data type
use throughout the program, you really don't want to make a individual decision everytime when
you use string (and take a penalty for any negligence). The Java has a much more usable model
with unicode used internally and encoding/decoding decision only need twice when dealing with
input and output.
that's how you should do things in Python too, of course. a unicode string
uses unicode internally. decode on the way in, encode on the way out, and
things just work.

the fact that you can mess things up by mixing unicode strings with binary
strings doesn't mean that you have to mix unicode strings with binary strings
in your program.
Even for those who choose to use unicode, it is almost impossible to ensure their program work
correctly.

well, if you use unicode the way it was intended to, it just works.

</F>

[aurora]

On Fri, 18 Feb 2005 19:24:10 +0100, Fredrik Lundh <fr*****@python ware.com>
wrote:

that's how you should do things in Python too, of course. a unicode
string
uses unicode internally. decode on the way in, encode on the way out, and
things just work.

the fact that you can mess things up by mixing unicode strings with
binary
strings doesn't mean that you have to mix unicode strings with binary
strings
in your program.

I don't want to mix them. But how could I find them? How do I know this
statement can be potential problem

if a==b:

where a and b can be instantiated individually far away from this line of
code that put them together?

In Java they are distinct data type and the compiler would catch all
incorrect usage. In Python, the interpreter seems to 'help' us to promote
binary string to unicode. Things works fine, unit tests pass, all until
the first non-ASCII characters come in and then the program breaks.

Is there a scheme for Python developer to use so that they are safe from
incorrect mixing?

[Walter Dörwald]

aurora wrote:

[...]
In Java they are distinct data type and the compiler would catch all
incorrect usage. In Python, the interpreter seems to 'help' us to
promote binary string to unicode. Things works fine, unit tests pass,
all until the first non-ASCII characters come in and then the program
breaks.

Is there a scheme for Python developer to use so that they are safe
from incorrect mixing?

Put the following:

import sys
sys.setdefaulte ncoding("undefi ned")

in a file named sitecustomize.p y somewhere in your Python path and
Python will complain whenever there's an implicit conversion between
str and unicode.

HTH,
Walter Dörwald

[Jarek Zgoda]

Fredrik Lundh napisa³(a):

This brings up another issue. Most references and books focus exclusive on entering unicode
literal and using the encode/decode methods. The fallacy is that string is such a basic data type
use throughout the program, you really don't want to make a individual decision everytime when
you use string (and take a penalty for any negligence). The Java has a much more usable model
with unicode used internally and encoding/decoding decision only need twice when dealing with
input and output.

that's how you should do things in Python too, of course. a unicode string
uses unicode internally. decode on the way in, encode on the way out, and
things just work.

There are implementations of Python where it isn't so easy, Python for
iSeries (http://www.iseriespython.com/) is one of them. The code written
for "normal" platform doesn't work on AS/400, even if all strings used
internally are unicode objects, also unicode literals don't work as
expected.

Of course, this is implementation fault but this makes a headache if you
need to write portable code.

--
Jarek Zgoda
http://jpa.berlios.de/ | http://www.zgodowie.org/

[Thomas Heller]

=?ISO-8859-15?Q?Walter_D=F 6rwald?= <wa****@livingl ogic.de> writes:

aurora wrote:

> [...]
In Java they are distinct data type and the compiler would catch all
incorrect usage. In Python, the interpreter seems to 'help' us to
promote binary string to unicode. Things works fine, unit tests
pass, all until the first non-ASCII characters come in and then the
program breaks.
Is there a scheme for Python developer to use so that they are safe
from incorrect mixing?

Put the following:

import sys
sys.setdefaulte ncoding("undefi ned")

in a file named sitecustomize.p y somewhere in your Python path and
Python will complain whenever there's an implicit conversion between
str and unicode.

Sounds cool, so I did it.
And started a program I was currently working on.
The first function in it is this:

if sys.platform == "win32":

def _locate_gccxml( ):
import _winreg
for subkey in (r"Software\gcc xml", r"Software\Kitw are\GCC_XML"):
for root in (_winreg.HKEY_C URRENT_USER, _winreg.HKEY_LO CAL_MACHINE):
try:
hkey = _winreg.OpenKey (root, subkey, 0, _winreg.KEY_REA D)
except WindowsError, detail:
if detail.errno != 2:
raise
else:
return _winreg.QueryVa lueEx(hkey, "loc")[0] + r"\bin"

loc = _locate_gccxml( )
if loc:
os.environ["PATH"] = loc

All strings in that snippet are text strings, so the first approach was
to convert them to unicode literals. Doesn't work. Here is the final,
working version (changes are marked):

if sys.platform == "win32":

def _locate_gccxml( ):
import _winreg
for subkey in (r"Software\gcc xml", r"Software\Kitw are\GCC_XML"):
for root in (_winreg.HKEY_C URRENT_USER, _winreg.HKEY_LO CAL_MACHINE):
try:
hkey = _winreg.OpenKey (root, subkey, 0, _winreg.KEY_REA D)
except WindowsError, detail:
if detail.errno != 2:
raise
else:
return _winreg.QueryVa lueEx(hkey, "loc")[0] + ur"\bin"
#-----------------------------------------------------------------^
loc = _locate_gccxml( )
if loc:
os.environ["PATH"] = loc.encode("mbc s")
#--------------------------------^

So, it appears that:

- the _winreg.QueryVa lue function is strange: it takes ascii strings,
but returns a unicode string.
- _winreg.OpenKey takes ascii strings
- the os.environ["PATH"] accepts an ascii string.

And I won't guess what happens when there are registry entries with
unlauts (ok, they could be handled by 'mbcs' encoding), and with chinese
or japanese characters (no way to represent them in ascii strings with a
western locale and mbcs encoding, afaik).
I suggest that 'sys.setdefault encoding("undef ined")' be the standard
setting for the core developers ;-)

Thomas

[Martin v. Löwis]

aurora wrote:

The Java
has a much more usable model with unicode used internally and
encoding/decoding decision only need twice when dealing with input and
output.

In addition to Fredrik's comment (that you should use the same model
in Python) and Walter's comment (that you can enforce it by setting
the default encoding to "undefined" ), I'd like to point out the
historical reason: Python predates Unicode, so the byte string type
has many convenience operations that you would only expect of
a character string.

We have come up with a transition strategy, allowing existing
libraries to widen their support from byte strings to character
strings. This isn't a simple task, so many libraries still expect
and return byte strings, when they should process character strings.
Instead of breaking the libraries right away, we have defined
a transitional mechanism, which allows to add Unicode support
to libraries as the need arises. This transition is still in
progress.

Eventually, the primary string type should be the Unicode
string. If you are curious how far we are still off that goal,
just try running your program with the -U option.

Regards,
Martin

[Jarek Zgoda]

Walter Dörwald napisa³(a):

Is there a scheme for Python developer to use so that they are safe
from incorrect mixing?

Put the following:

import sys
sys.setdefaulte ncoding("undefi ned")

in a file named sitecustomize.p y somewhere in your Python path and
Python will complain whenever there's an implicit conversion between
str and unicode.

This will help in your code, but there is big pile of modules in stdlib
that are not unicode-friendly. From my daily practice come shlex
(tokenizer works only with encoded strings) and logging (you cann't
specify encoding for FileHandler).

--
Jarek Zgoda
http://jpa.berlios.de/ | http://www.zgodowie.org/

[Thomas Heller]

=?ISO-8859-15?Q?=22Martin_ v=2E_L=F6wis=22 ?= <ma****@v.loewi s.de> writes:

We have come up with a transition strategy, allowing existing
libraries to widen their support from byte strings to character
strings. This isn't a simple task, so many libraries still expect
and return byte strings, when they should process character strings.
Instead of breaking the libraries right away, we have defined
a transitional mechanism, which allows to add Unicode support
to libraries as the need arises. This transition is still in
progress.

Eventually, the primary string type should be the Unicode
string. If you are curious how far we are still off that goal,
just try running your program with the -U option.

Is it possible to specify a byte string literal when running with the -U option?

Thomas

[Thomas Heller]

=?ISO-8859-15?Q?=22Martin_ v=2E_L=F6wis=22 ?= <ma****@v.loewi s.de> writes:

Eventually, the primary string type should be the Unicode
string. If you are curious how far we are still off that goal,
just try running your program with the -U option.

Not very far - can't even call functions ;-)

c:\>py -U
Python 2.5a0 (#60, Dec 29 2004, 11:27:13) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright" , "credits" or "license" for more information.

def f(**kw): .... pass
.... f(**{"a": 0}) Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: f() keywords must be strings

Thomas