Unicode and Zipfile problems

[Gerson Kurz]

AAAAAAAARG I hate the way python handles unicode. Here is a nice
problem for y'all to enjoy: say you have a variable thats unicode

directory = u"c:\temp"

Its unicode not because you want it to, but because its for example
read from _winreg which returns unicode.

You do an os.listdir(directory). Note that all filenames returned are
now unicode. (Change introduced I believe in 2.3).

You add the filenames to a zipfile.ZipFile object. Sometimes, you will
get this exception:

Traceback (most recent call last):
File "collect_trace_info.py", line 65, in CollectTraceInfo
z.write(pathname)
File "C:\Python23\lib\zipfile.py", line 416, in write
self.fp.write(zinfo.FileHeader())
File "C:\Python23\lib\zipfile.py", line 170, in FileHeader
return header + self.filename + self.extra
UnicodeDecodeError: 'ascii' codec can't decode byte 0x88 in position
12:
ordinal not in range(128)

After you have regained your composure, you find the reason: "header"
is a struct.pack() generated byte string. self.filename is however a
unicode string because it is returned by os.listdir as unicode. If
"header" generates anything above 0x7F - which can but need not
happen, depending on the type of file you have an exception waiting
for yourself - sometimes. Great. (The same will probably occur if
filename contains chars > 0x7F). The problem does not occur if you
have "str" type filenames, because then no backandforth conversion is
being made.

There is a simple fix, before calling z.write() byte-encode it. Here
is a sample code:

import os, zipfile, win32api

def test(directory):
z =
zipfile.ZipFile(os.path.join(directory,"temp.zip") ,"w",zipfile.ZIP_DEFLATED)
for filename in os.listdir(directory):
z.write(os.path.join(directory, filename))
z.close()

if __name__ == "__main__":
test(unicode(win32api.GetSystemDirectory()))

Note: It might work on your system, depending on the types of files.
To fix it, use

z.write(os.path.join(directory, filename).encode("latin-1"))

But to my thinking, this is a bug in zipfile.py, really.

Now, could anybody please just write a
"i-don't-care-if-my-app-can-display-klingon-characters" raw byte
encoding which doesn't throw any assertions and doesn't care whether
or not the characters are in the 0x7F range? Its ok if I cannot port
my batchscripts to swaheli, really.

[Gerson Kurz]

Here is my "dontcare"-codec, ugly on purpose:

import codecs

def E(i,e=''):
l=lambda c:chr(min(ord(c),255))
r="".join(map(l,i))
return (r,len(r))

def D(i,e=''):
l=lambda c: unichr(ord(c))
r=u"".join(map(l,i))
return (r,len(r))

class c(codecs.Codec):
def encode(self, i,e=''):
return E(i,e)
def decode(self, i,e=''):
return D(i,e)

class w(c,codecs.StreamWriter): pass
class r(c,codecs.StreamReader): pass

getregentry = lambda: (E, D, r, w)

To install, save as python23\lib\encodings\dontcare.py and enable it
in site.py. Here is a testcode:

try:
print unicode(chr(0xFF))
except UnicodeDecodeError, e:
print e

try:
print unichr(12345)
except UnicodeEncodeError, e:
print e

[vincent wehren]

"Gerson Kurz" <ge*********@t-online.de> schrieb im Newsbeitrag
news:3f*************@news.t-online.de...
| AAAAAAAARG I hate the way python handles unicode. Here is a nice
| problem for y'all to enjoy: say you have a variable thats unicode
|
| directory = u"c:\temp"
|
| Its unicode not because you want it to, but because its for example
| read from _winreg which returns unicode.
|
| You do an os.listdir(directory). Note that all filenames returned are
| now unicode. (Change introduced I believe in 2.3).

Wrong.

That's only true if type(directory) gives you <type 'unicode'>
If you call str(directory) before doing os.listdir(directory)
you (in most cases) want even notice and can continue doing what you want to
do
just fine - plus, and that's the good part - you can forget about
those hacks you suggest later and which some would consider *evil*.
It'll save yourself some time too.

Hey, and leave my Swahili friends alone will ya! ;)

HTH,
Vincent Wehren

|
| You add the filenames to a zipfile.ZipFile object. Sometimes, you will
| get this exception:
|
| Traceback (most recent call last):
| File "collect_trace_info.py", line 65, in CollectTraceInfo
| z.write(pathname)
| File "C:\Python23\lib\zipfile.py", line 416, in write
| self.fp.write(zinfo.FileHeader())
| File "C:\Python23\lib\zipfile.py", line 170, in FileHeader
| return header + self.filename + self.extra
| UnicodeDecodeError: 'ascii' codec can't decode byte 0x88 in position
| 12:
| ordinal not in range(128)
|
| After you have regained your composure, you find the reason: "header"
| is a struct.pack() generated byte string. self.filename is however a
| unicode string because it is returned by os.listdir as unicode. If
| "header" generates anything above 0x7F - which can but need not
| happen, depending on the type of file you have an exception waiting
| for yourself - sometimes. Great. (The same will probably occur if
| filename contains chars > 0x7F). The problem does not occur if you
| have "str" type filenames, because then no backandforth conversion is
| being made.
|
| There is a simple fix, before calling z.write() byte-encode it. Here
| is a sample code:
|
| import os, zipfile, win32api
|
| def test(directory):
| z =
|
zipfile.ZipFile(os.path.join(directory,"temp.zip") ,"w",zipfile.ZIP_DEFLATED)
| for filename in os.listdir(directory):
| z.write(os.path.join(directory, filename))
| z.close()
|
| if __name__ == "__main__":
| test(unicode(win32api.GetSystemDirectory()))
|
| Note: It might work on your system, depending on the types of files.
| To fix it, use
|
| z.write(os.path.join(directory, filename).encode("latin-1"))
|
| But to my thinking, this is a bug in zipfile.py, really.
|
| Now, could anybody please just write a
| "i-don't-care-if-my-app-can-display-klingon-characters" raw byte
| encoding which doesn't throw any assertions and doesn't care whether
| or not the characters are in the 0x7F range? Its ok if I cannot port
| my batchscripts to swaheli, really.
|
|

[vincent wehren]

"vincent wehren" <vi*****@visualtrans.de> schrieb im Newsbeitrag
news:bo**********@news4.tilbu1.nb.home.nl...
|
| "Gerson Kurz" <ge*********@t-online.de> schrieb im Newsbeitrag
| news:3f*************@news.t-online.de...
| | AAAAAAAARG I hate the way python handles unicode. Here is a nice
| | problem for y'all to enjoy: say you have a variable thats unicode
| |
| | directory = u"c:\temp"
| |
| | Its unicode not because you want it to, but because its for example
| | read from _winreg which returns unicode.
| |
| | You do an os.listdir(directory). Note that all filenames returned are
| | now unicode. (Change introduced I believe in 2.3).
|
| Wrong.
|
| That's only true if type(directory) gives you <type 'unicode'>
| If you call str(directory) before doing os.listdir(directory)
| you (in most cases) want even notice and can continue doing what you want
to

And when I say "in most cases", I mean all those cases where "directory"
doesn't have characters that map to a single-byte value outside of the ASCII
range. In other cases you'll just go :

directory =
directory.encode(your_favorite_and_hoepfully_the_r ight_single_byte_legacy_en
coding_here)

before calling os.listdir(directory)
Regards,

Vincent
| do
| just fine - plus, and that's the good part - you can forget about
| those hacks you suggest later and which some would consider *evil*.
| It'll save yourself some time too.
|
| Hey, and leave my Swahili friends alone will ya! ;)
|
| HTH,
| Vincent Wehren
|
|
|
| |
| | You add the filenames to a zipfile.ZipFile object. Sometimes, you will
| | get this exception:
| |
| | Traceback (most recent call last):
| | File "collect_trace_info.py", line 65, in CollectTraceInfo
| | z.write(pathname)
| | File "C:\Python23\lib\zipfile.py", line 416, in write
| | self.fp.write(zinfo.FileHeader())
| | File "C:\Python23\lib\zipfile.py", line 170, in FileHeader
| | return header + self.filename + self.extra
| | UnicodeDecodeError: 'ascii' codec can't decode byte 0x88 in position
| | 12:
| | ordinal not in range(128)
| |
| | After you have regained your composure, you find the reason: "header"
| | is a struct.pack() generated byte string. self.filename is however a
| | unicode string because it is returned by os.listdir as unicode. If
| | "header" generates anything above 0x7F - which can but need not
| | happen, depending on the type of file you have an exception waiting
| | for yourself - sometimes. Great. (The same will probably occur if
| | filename contains chars > 0x7F). The problem does not occur if you
| | have "str" type filenames, because then no backandforth conversion is
| | being made.
| |
| | There is a simple fix, before calling z.write() byte-encode it. Here
| | is a sample code:
| |
| | import os, zipfile, win32api
| |
| | def test(directory):
| | z =
| |
|
zipfile.ZipFile(os.path.join(directory,"temp.zip") ,"w",zipfile.ZIP_DEFLATED)
| | for filename in os.listdir(directory):
| | z.write(os.path.join(directory, filename))
| | z.close()
| |
| | if __name__ == "__main__":
| | test(unicode(win32api.GetSystemDirectory()))
| |
| | Note: It might work on your system, depending on the types of files.
| | To fix it, use
| |
| | z.write(os.path.join(directory, filename).encode("latin-1"))
| |
| | But to my thinking, this is a bug in zipfile.py, really.
| |
| | Now, could anybody please just write a
| | "i-don't-care-if-my-app-can-display-klingon-characters" raw byte
| | encoding which doesn't throw any assertions and doesn't care whether
| | or not the characters are in the 0x7F range? Its ok if I cannot port
| | my batchscripts to swaheli, really.
| |
| |
|
|

[Martin v. Löwis]

ge*********@t-online.de (Gerson Kurz) writes:

Here is my "dontcare"-codec, ugly on purpose:

Of course, this codec does not work for your original problem: Just
see try it on your original data, and then see how Winzip
misinterprets the file names.

zipfiles encode file names in code page 437.

Regards,
Martin

[Thomas Heller]

ma****@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) writes:

ge*********@t-online.de (Gerson Kurz) writes:

Here is my "dontcare"-codec, ugly on purpose:

Of course, this codec does not work for your original problem: Just
see try it on your original data, and then see how Winzip
misinterprets the file names.

zipfiles encode file names in code page 437.

So is this a bug in the zipfile module?

Thomas

[Martin v. Löwis]

Thomas Heller <th*****@python.net> writes:

zipfiles encode file names in code page 437.

So is this a bug in the zipfile module?

Yes. It should auto-encode Unicode strings as cp437.

For byte strings, there is not much it could do - the zipfile will
just contain mojibake if encoding of the byte string is not cp437.

Regards,
Martin

[Gerson Kurz]

>Of course, this codec does not work for your original problem: Just

see try it on your original data, and then see how Winzip
misinterprets the file names.

You are of course right (although my original problem was not with the
filenames - I am using an english version of Windows - but with the
header information).

But, if you look at the number of people that have run into problems
with pythons strictness in unicode matters - isn't it time to offer
some more "relaxed" way of handling all this? I mean, come on, Python
is the language that will change the meaning of 7/3 because people had
problems with integer division. And, although some people use Python
for large multilanguage applications - I would bet that by far more
people use Python for lots of small utility scripts, and couldn't care
less about whether or not its "international". It just has to work, on
my machine(s). Its not rocket science.

[Peter Otten]

Gerson Kurz wrote:

Of course, this codec does not work for your original problem: Just
see try it on your original data, and then see how Winzip
misinterprets the file names.
You are of course right (although my original problem was not with the
filenames - I am using an english version of Windows - but with the
header information).

But, if you look at the number of people that have run into problems
with pythons strictness in unicode matters - isn't it time to offer
some more "relaxed" way of handling all this? I mean, come on, Python

So you've never run into trouble with different encodings and editors not
aware of the encoding? For me this annoyance dates back to Windows 3.1 vs
DOS - RIP. I do remember a book on Apple Pascal that had all y and z
characters accidentally switched.

I'm not aware if there has been a discussion before, but I think it would be
worth the overhead if every string were aware of its encoding, so that
together with the -*- comment in the script you'd never again have to
explicitly go through the decode/encode routine *and* could avoid those
funny filenames - or two-character umlauts that accidentally made it into
your ISO-8859-1 files.
is the language that will change the meaning of 7/3 because people had
problems with integer division. And, although some people use Python
for large multilanguage applications - I would bet that by far more
people use Python for lots of small utility scripts, and couldn't care
less about whether or not its "international". It just has to work, on
my machine(s). Its not rocket science.

One byte per character encodings might or might not vanish, but unicode
*will* stay, so you better be aware of the pitfalls.
Remember that Python is also the language of "explicit is better implicit".

Peter

[Martin v. Löwis]

Peter Otten <__*******@web.de> writes:

I'm not aware if there has been a discussion before, but I think it would be
worth the overhead if every string were aware of its encoding, so that
together with the -*- comment in the script you'd never again have to
explicitly go through the decode/encode routine *and* could avoid those
funny filenames - or two-character umlauts that accidentally made it into
your ISO-8859-1 files.

Bill Janssen first suggested that a before Unicode was introduced. I
believe it won't help much, as, in many cases, Python can't know what
encoding a byte string is, e.g. if you read from a file, or a socket.
In some cases, you have binary data proper.

Regards,
Martin

[Radovan Garabik]

vincent wehren <vi*****@visualtrans.de> wrote:

"Gerson Kurz" <ge*********@t-online.de> schrieb im Newsbeitrag
news:3f*************@news.t-online.de...
| AAAAAAAARG I hate the way python handles unicode. Here is a nice
| problem for y'all to enjoy: say you have a variable thats unicode
|
| directory = u"c:\temp"
|
| Its unicode not because you want it to, but because its for example
| read from _winreg which returns unicode.
|
| You do an os.listdir(directory). Note that all filenames returned are
| now unicode. (Change introduced I believe in 2.3).

Wrong.

That's only true if type(directory) gives you <type 'unicode'>
If you call str(directory) before doing os.listdir(directory)
you (in most cases) want even notice and can continue doing what you want to
do
just fine - plus, and that's the good part - you can forget about
those hacks you suggest later and which some would consider *evil*.
It'll save yourself some time too.

Hey, and leave my Swahili friends alone will ya! ;)
Uhm, AFAIK Swahili does not use any special characters in the alphabet
(if you do not count digraphs, that is). Plain old ASCII is sufficient
Not that my remark is useful for anything, though.
| Now, could anybody please just write a
| "i-don't-care-if-my-app-can-display-klingon-characters" raw byte
| encoding which doesn't throw any assertions and doesn't care whether
| or not the characters are in the 0x7F range? Its ok if I cannot port
| my batchscripts to swaheli, really.

but you HAVE TO care, since on MS Windows, if a filename is unicode,
it is UTF-16 and you just cannot convert it into stream of bytes
without messing up with encodings. UTF-16 is not even ASCII compatible,
after all.
--
-----------------------------------------------------------
| Radovan GarabÃ*k http://melkor.dnp.fmph.uniba.sk/~garabik/ |
| __..--^^^--..__ garabik @ kassiopeia.juls.savba.sk |
-----------------------------------------------------------
Antivirus alert: file .signature infected by signature virus.
Hi! I'm a signature virus! Copy me into your signature file to help me spread!

[Gerson Kurz]

>but you HAVE TO care, since on MS Windows, if a filename is unicode,

it is UTF-16 and you just cannot convert it into stream of bytes
without messing up with encodings. UTF-16 is not even ASCII compatible,
after all.

Well, you know, the strange thing is: I have written C and C++
programms on Win32 ever since WinNT 3.x (Unicode was not part of
16-bit Windows) and a few of these actually work. Sort of. Even though
they cannot work because they don't support UTF-16, right?

And the same goes for many commercial or wellknown applications.
Antivir Professional? Kerio Personal Firewall? Free Agent? Paint Shop
Pro? Winzip? Winamp? Of course I don't have the source for these, but
the Dependency Viewer (from the Microsoft SDK) will show you that all
of these link with the ASCII-Versions of the Windows API. Seems like
there is a lot of broken apps out there! And the most shocking of all
- this holds true even of python23.dll: ShellExecuteA,
RegQueryValueExA, LoadStringA, LoadLibraryExA - its all ASCII!
Somebody better call for a major unicode cleanup!

But OK, I agree, the subject is somewhat boring - even though every
week somebody else runs into problems with this (see the thread
"Strange problem with encoding" from today) there will probably be no
change introduced in Python at this point on this subject.

[Neil Hodgson]

Gerson Kurz:

Of course I don't have the source for these, but
the Dependency Viewer (from the Microsoft SDK) will show you that all
of these link with the ASCII-Versions of the Windows API.
No, these applications link to what Microsoft calls the "ANSI" versions
of the Windows API. That is important because your initial problem would not
have occurred if all your file names were ASCII. Instead they contained
character 0x88 which was probably a circumflex modifier although it could
have been a Euro symbol.
Seems like
there is a lot of broken apps out there! And the most shocking of all
- this holds true even of python23.dll: ShellExecuteA,
RegQueryValueExA, LoadStringA, LoadLibraryExA - its all ASCII!
Somebody better call for a major unicode cleanup!
We will gradually work on increasing the scope of Unicode support in
Python. For example, os.popen would be a good candidate for receiving
Unicode support.
But OK, I agree, the subject is somewhat boring - even though every
week somebody else runs into problems with this (see the thread
"Strange problem with encoding" from today) there will probably be no
change introduced in Python at this point on this subject.

If you are interested in enhancing Python then produce a concrete
proposal.

From my point of view, Unicode has been a great source of simplification
as it has reduced the need for code conversion and potential for loss of
information. In the future, more of the software infrastructure will be able
to handle Unicode. ZIP files produced by some tools can already store
Unicode file names although there is no published standard for this.

Example product:
http://www.componentsource.com/Catal...ary_505440.htm
"Stores and retrieves the latest zip file format extensions, allowing
Unicode filenames and NT file attributes, extra time stamps and security
permissions to be stored in the zip file"

ZIP format definition:
http://www.pkware.com/products/enter...s/appnote.html

Neil

[Gerson Kurz]

> No, these applications link to what Microsoft calls the "ANSI" versions

of the Windows API. That is important because your initial problem would not
have occurred if all your file names were ASCII. Instead they contained
character 0x88 which was probably a circumflex modifier although it could
have been a Euro symbol.
The filename was DENOMALG.INI. The problem was not with the filename,
it was that the binary struct.packed header in front of it contained
0x88. The problem occured because Python thinks that if one tiny part
of an expression is unicode - even if all unicode characters of that
tiny part are <0x7F - everything in the expression has to be promoted
to unicode, BUT woe is you if the byte strings have anything >0x7F.
You can get an exception if you simply write chr(0xFF)+u"" - an empty
string!!!
If you are interested in enhancing Python then produce a concrete
proposal.

I am going to think about that. Here are some ideas that immediately
come to my mind.

There should be two modes, easily configurable: "dontcare" and
"international".

"dontcare" can be implemented easily enough by changing site.py (I
only have to find out how to remove that s***** DepracationWarning
introduced in 2.3 for source with german comments, for christs sake!
comments!).

For "international" mode, its ok to be as strict as now. But I also
think there should be a more elaborate concept. You know, you don't
internationalize your application simply by using unicode strings.

- there is no monetary data type in Python (I see that this is a
discussion that has been on the list a few days ago)
- in my "dayjob" C++ application that is used across europe and
america, I use SCU (smallest currency unit = cent, pfennig) but I need
different ways of displaying that information. For example, take the
amount 100000. In the standard output, you want to see 1.000,00 Euro.
If you edit a field that contains this value, you want to edit 1000,00
(that is, you don't want either the Euro or the "."). If you have a
value that represents notes, you want to see 200 Euro and not 200,00
Euro (nobody talks that way about notes).
- the locale module relys on the C implementation, not the OS
implementation (the OS settings should take higher priority), has
documentation errors, and is generally, well, ugly. strftime, need I
say more?
- the gettext API sucks, especially if you need to change text along
the way. Python is an interpreted language, why not simply use one or
more module "strings.py" which contain the strings which are refered
to by identifiers?

But I digress.

You know, Python is a great language, it is a lot better than C or C++
and I love it and everything. And the core developers are doing a
really good job, and I am very thankful for that, and I mean it.

It is just that sometimes I get carried away because things turn out
not to be as intuitive as they should be - and Python spoils you, it
really does, because for *most* things, Python is as intuitive as it
can get, so things that are maybe too complex to be intuitively simple
suddenly start to annoy you.

[Martin v. Löwis]

ge*********@t-online.de (Gerson Kurz) writes:

There should be two modes, easily configurable: "dontcare" and
"international".
I'd be curious what precisely it is that has two modes.
"dontcare" can be implemented easily enough by changing site.py (I
only have to find out how to remove that s***** DepracationWarning
introduced in 2.3 for source with german comments, for christs sake!
comments!).
You can add a warning filter to site.py. However, I'm curious what
else "dontcare" would do, specifically.
- the locale module relys on the C implementation, not the OS
implementation (the OS settings should take higher priority), has
documentation errors, and is generally, well, ugly. strftime, need I
say more?
Yes, please do. What is the difference between the C implementation
and the OS implementation?
- the gettext API sucks, especially if you need to change text along
the way. Python is an interpreted language, why not simply use one or
more module "strings.py" which contain the strings which are refered
to by identifiers?

Because that is more difficult to process for translators.

Regards,
Martin

[Gerson Kurz]

ma****@v.loewis.de wrote:

I'd be curious what precisely it is that has two modes.
The Python interpreter.

"dontcare" can be implemented easily enough by changing site.py (I
only have to find out how to remove that s***** DepracationWarning
introduced in 2.3 for source with german comments, for christs sake!
comments!).

You can add a warning filter to site.py.

Thanks, but could you please elaborate a little on that? Are you
suggesting I write my own import hook to filter the warning there? Is
there any other mode? It seems the warning is generated inside the
python core (tokenizer.c, 478) and not in the lib python files.
else "dontcare" would do, specifically.
It would be like a Python build without Unicode support. A cursory
glance at the Python source reveals that probably

#ifdef Py_WIN_WIDE_FILENAMES

is used to encapsulate the W-API of windows; but it seems that unicode
support is rather deeply entrenched in the source (at least there are
no obvious #ifdefs in unicodeobject.c / unicodectype.c) - damnit I was
hoping this to go fast ;)

Basically, what is annoying about the way python handles unicode is
this:

a) you get warnings when you do stuff you've been doing for years
without ever getting any warning.

b) it forces you to be correct - even when you don't care.

Now of course there is nothing wrong in being correct: it is only that
sometimes it is not worth the effort and you don't care and you are
STILL forced to care about it.

Like, you want to write a small script that dumps some registry keys
to stdout. Bang, you get an exception because there is a German umlaut
in one of these. Now, a C ANSI code that dumps the registry will
perhaps not display the right character, screw the console CP, but at
least you can read the thing, because you're a human and not a stupid
computer and that is all that matters. And normally you don't even
know WHAT encoding to use.

Maybe its time for a "UNICODE for dummies" section in the python
manual. But maybe its also time for a more relaxed way of handling all
that?

So, back to the two ways in which the Python unicode handling is
annoying - it would be fine if you could easily change "strict"
encoding to "relaxed" (I'm not sure about that, but toying with my
dontcare.py I see that there is a parameter to the en/coding
functions, so maybe one could set default encoding = OS locale
encoding (see below) and disable exceptions when something goes wrong.

That way

a) you DONT get warnings when you do stuff you've been doing for years
without ever getting any warning.

b) it DOES NOT force you to be correct - even when you don't care.

so I at least would be happy with that.
Yes, please do. What is the difference between the C implementation
and the OS implementation?

a) Last time I checked, strftime gives you a date and time
representation for the current locale. As in: one date and time
representation ("%x %X"). However, you have like long and short dates.
Ask the simple question: do you put the time before the date or after?
You have times that require millisecond precision (e.g. timestamps in
a tracefile) and you have times that are just hours. If you want to
include milliseconds, you will have to resort to a manual guess as to
what the time format is, and so on. Not including the fact that, at
least on windows, there are at least three different time "classes"
used in python, from the time module, the datetime module, and
win32api.GetLocalTime() / win32api.GetSystemTime().

b) The C implementation is part of the C implementation (as the name
would indicate) and you can read for example in the CRT sources; if
you install DevStudio6, you will find it e.g. here

C:\Program Files\Microsoft Visual Studio\VC98\CRT\SRC\STRFTIME.C

(The default install doesn't copy these files, you have to set
"advanced" options during setup IIRC). The OS version is a set of API
functions called the "National Language Support Functions", which
contains the functions GetDateFormat and GetTimeFormat which have a
completely different syntax and are used by other applications (such
as, yuk, VB). If you look at the API documentation, you'll notice that
the two versions have different options.

c) I run an english version of Windows 2000, but I have german locale
settings. Windows distinguishes between "system locale" and "user
locale". Many applications, virtually all of them, use the user lcoale
settings (that is, german). Python uses the C default which is - well
I'm not really sure whether or not its english, but it certainly isn't
german by (OS) default.

d) The documentation for the locale format says you should set "de" or
"de_DE", but "GERMAN" is the actual locale for "german". But how do
you know? And how do you add functionality to your application to
always use the users locale (ie German on my english system - as any
other app including stupid MFC apps can do)?

[Martin v. Löwis]

ge*********@t-online.de (Gerson Kurz) writes:

Thanks, but could you please elaborate a little on that? Are you
suggesting I write my own import hook to filter the warning there?
import warnings
warnings.filterwarnings('ignore', 'Non-ASCII character .*/peps/pep-0263',
DeprecationWarning)

(see http://groups.google.com/groups?selm...40news1.tin.it)
Basically, what is annoying about the way python handles unicode is
this:
We are back to square zero now. What specifically do you suggest to
change?
a) you get warnings when you do stuff you've been doing for years
without ever getting any warning.
Yes, but that happens not only for Unicode strings; try importing
regex for another example.
b) it forces you to be correct - even when you don't care.
Yes, but it does so all over the place:

x = 3.14

is different, in Python, from

x = 3,14

You have to know whether you mean a decimal point, or a tuple comma,
and you have to know how to spell either.

Python does not automatically correct mistakes that you make.
Maybe its time for a "UNICODE for dummies" section in the python
manual. But maybe its also time for a more relaxed way of handling all
that?
Contributions are welcome.
So, back to the two ways in which the Python unicode handling is
annoying - it would be fine if you could easily change "strict"
encoding to "relaxed" (I'm not sure about that, but toying with my
dontcare.py I see that there is a parameter to the en/coding
functions, so maybe one could set default encoding = OS locale
encoding (see below) and disable exceptions when something goes wrong.
You can replace "strict" with "replace", in Unicode error handling, if
this is what you are suggesting.

Of course, that would not have helped in your original problem, as it
would have replaced the header bytes of the zip header with question
marks, when converting the header to a Unicode object.

But let's assume we support a way of setting the default error
handling to "replace".
a) you DONT get warnings when you do stuff you've been doing for years
without ever getting any warning.
Right.
b) it DOES NOT force you to be correct - even when you don't care.
Wrong. The code that you had been using for years still would stop
working, and you would not get an exception; instead, you would get a
corrupted zip file.
so I at least would be happy with that.
I don't believe you would.

Yes, please do. What is the difference between the C implementation
and the OS implementation?

a) Last time I checked, strftime gives you a date and time
representation for the current locale. As in: one date and time
representation ("%x %X"). However, you have like long and short dates.
Ask the simple question: do you put the time before the date or after?

Time first. What does that have to do with C implementation and OS
implementation?
The OS version is a set of API functions called the "National
Language Support Functions", which contains the functions
GetDateFormat and GetTimeFormat which have a completely different
syntax and are used by other applications (such as, yuk, VB).
You are apparently talking about Microsoft Windows here. Yes, that
particular OS has an API that is different from standard
C. Fortunately, Microsoft compilers use the underlying API to expose
OS functionality. So by calling the C library, you call OS routines.
c) I run an english version of Windows 2000, but I have german locale
settings. Windows distinguishes between "system locale" and "user
locale". Many applications, virtually all of them, use the user lcoale
settings (that is, german). Python uses the C default which is - well
I'm not really sure whether or not its english, but it certainly isn't
german by (OS) default.
Yes, on startup, Python is not locale-aware.
d) The documentation for the locale format says you should set "de" or
"de_DE", but "GERMAN" is the actual locale for "german". But how do
you know? And how do you add functionality to your application to
always use the users locale (ie German on my english system - as any
other app including stupid MFC apps can do)?

You don't give any locale name. If you want the user's settings to be
used, invoke

locale.setlocale(locale.LC_ALL, "")

That gives you what I think you mean by "OS functions".

Regards,
Martin

[David Eppstein]

In article <3f**************@news.t-online.de>,
ge*********@t-online.de (Gerson Kurz) wrote:

Basically, what is annoying about the way python handles unicode is
this:

it forces you to be correct - even when you don't care.

You should always care.

(He says, after having to deal with automatically guessing which
encoding was used, for a file format that explicitly calls for one
particular encoding, because the d*mn users didn't care and used a
different one instead some of the time.)

--
David Eppstein http://www.ics.uci.edu/~eppstein/
Univ. of California, Irvine, School of Information & Computer Science

[Gerson Kurz]

ma****@v.loewis.de wrote:

b) it forces you to be correct - even when you don't care.
Yes, but it does so all over the place:

x = 3.14

is different, in Python, from

x = 3,14

Why is it that the number of people that complain in clp about having
trouble spelling fractions is way, way below the number of people that
come here with yet another unicode problem?
You are apparently talking about Microsoft Windows here.
It is not like if you support something Windowish, you are wandering
off into obscureness.
particular OS has an API that is different from standard
C. Fortunately, Microsoft compilers use the underlying API to expose
OS functionality. So by calling the C library, you call OS routines.
You are right. I stepped into the locale code, and you are right.

Of course, the C api still does not expose the full functionality of
the underlying OS api (like short and long date formats for a locale).

locale.setlocale(locale.LC_ALL, "")

Thanks, I didn't know that.

[Martin v. Löwis]

ge*********@t-online.de (Gerson Kurz) writes:

x = 3.14

is different, in Python, from

x = 3,14

Why is it that the number of people that complain in clp about having
trouble spelling fractions is way, way below the number of people that
come here with yet another unicode problem?

Because people understand almost understand what floating point
numbers are, but they fail to understand what a character set is.
This, in turn, is because they have been exposed to floating point
numbers for a long time, but never had to deal with character sets.

Regards,
Martin