Unicode and Zipfile problems

[Gerson Kurz]

AAAAAAAARG I hate the way python handles unicode. Here is a nice
problem for y'all to enjoy: say you have a variable thats unicode

directory = u"c:\temp"

Its unicode not because you want it to, but because its for example
read from _winreg which returns unicode.

You do an os.listdir(dire ctory). Note that all filenames returned are
now unicode. (Change introduced I believe in 2.3).

You add the filenames to a zipfile.ZipFile object. Sometimes, you will
get this exception:

Traceback (most recent call last):
File "collect_trace_ info.py", line 65, in CollectTraceInf o
z.write(pathnam e)
File "C:\Python23\li b\zipfile.py", line 416, in write
self.fp.write(z info.FileHeader ())
File "C:\Python23\li b\zipfile.py", line 170, in FileHeader
return header + self.filename + self.extra
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0x88 in position
12:
ordinal not in range(128)

After you have regained your composure, you find the reason: "header"
is a struct.pack() generated byte string. self.filename is however a
unicode string because it is returned by os.listdir as unicode. If
"header" generates anything above 0x7F - which can but need not
happen, depending on the type of file you have an exception waiting
for yourself - sometimes. Great. (The same will probably occur if
filename contains chars > 0x7F). The problem does not occur if you
have "str" type filenames, because then no backandforth conversion is
being made.

There is a simple fix, before calling z.write() byte-encode it. Here
is a sample code:

import os, zipfile, win32api

def test(directory) :
z =
zipfile.ZipFile (os.path.join(d irectory,"temp. zip"),"w",zipfi le.ZIP_DEFLATED )
for filename in os.listdir(dire ctory):
z.write(os.path .join(directory , filename))
z.close()

if __name__ == "__main__":
test(unicode(wi n32api.GetSyste mDirectory()))

Note: It might work on your system, depending on the types of files.
To fix it, use

z.write(os.path .join(directory , filename).encod e("latin-1"))

But to my thinking, this is a bug in zipfile.py, really.

Now, could anybody please just write a
"i-don't-care-if-my-app-can-display-klingon-characters" raw byte
encoding which doesn't throw any assertions and doesn't care whether
or not the characters are in the 0x7F range? Its ok if I cannot port
my batchscripts to swaheli, really.

[Gerson Kurz]

Here is my "dontcare"-codec, ugly on purpose:

import codecs

def E(i,e=''):
l=lambda c:chr(min(ord(c ),255))
r="".join(map(l ,i))
return (r,len(r))

def D(i,e=''):
l=lambda c: unichr(ord(c))
r=u"".join(map( l,i))
return (r,len(r))

class c(codecs.Codec) :
def encode(self, i,e=''):
return E(i,e)
def decode(self, i,e=''):
return D(i,e)

class w(c,codecs.Stre amWriter): pass
class r(c,codecs.Stre amReader): pass

getregentry = lambda: (E, D, r, w)

To install, save as python23\lib\en codings\dontcar e.py and enable it
in site.py. Here is a testcode:

try:
print unicode(chr(0xF F))
except UnicodeDecodeEr ror, e:
print e

try:
print unichr(12345)
except UnicodeEncodeEr ror, e:
print e

[vincent wehren]

"Gerson Kurz" <ge*********@ t-online.de> schrieb im Newsbeitrag
news:3f******** *****@news.t-online.de...
| AAAAAAAARG I hate the way python handles unicode. Here is a nice
| problem for y'all to enjoy: say you have a variable thats unicode
|
| directory = u"c:\temp"
|
| Its unicode not because you want it to, but because its for example
| read from _winreg which returns unicode.
|
| You do an os.listdir(dire ctory). Note that all filenames returned are
| now unicode. (Change introduced I believe in 2.3).

Wrong.

That's only true if type(directory) gives you <type 'unicode'>
If you call str(directory) before doing os.listdir(dire ctory)
you (in most cases) want even notice and can continue doing what you want to
do
just fine - plus, and that's the good part - you can forget about
those hacks you suggest later and which some would consider *evil*.
It'll save yourself some time too.

Hey, and leave my Swahili friends alone will ya! ;)

HTH,
Vincent Wehren

|
| You add the filenames to a zipfile.ZipFile object. Sometimes, you will
| get this exception:
|
| Traceback (most recent call last):
| File "collect_trace_ info.py", line 65, in CollectTraceInf o
| z.write(pathnam e)
| File "C:\Python23\li b\zipfile.py", line 416, in write
| self.fp.write(z info.FileHeader ())
| File "C:\Python23\li b\zipfile.py", line 170, in FileHeader
| return header + self.filename + self.extra
| UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0x88 in position
| 12:
| ordinal not in range(128)
|
| After you have regained your composure, you find the reason: "header"
| is a struct.pack() generated byte string. self.filename is however a
| unicode string because it is returned by os.listdir as unicode. If
| "header" generates anything above 0x7F - which can but need not
| happen, depending on the type of file you have an exception waiting
| for yourself - sometimes. Great. (The same will probably occur if
| filename contains chars > 0x7F). The problem does not occur if you
| have "str" type filenames, because then no backandforth conversion is
| being made.
|
| There is a simple fix, before calling z.write() byte-encode it. Here
| is a sample code:
|
| import os, zipfile, win32api
|
| def test(directory) :
| z =
|
zipfile.ZipFile (os.path.join(d irectory,"temp. zip"),"w",zipfi le.ZIP_DEFLATED )
| for filename in os.listdir(dire ctory):
| z.write(os.path .join(directory , filename))
| z.close()
|
| if __name__ == "__main__":
| test(unicode(wi n32api.GetSyste mDirectory()))
|
| Note: It might work on your system, depending on the types of files.
| To fix it, use
|
| z.write(os.path .join(directory , filename).encod e("latin-1"))
|
| But to my thinking, this is a bug in zipfile.py, really.
|
| Now, could anybody please just write a
| "i-don't-care-if-my-app-can-display-klingon-characters" raw byte
| encoding which doesn't throw any assertions and doesn't care whether
| or not the characters are in the 0x7F range? Its ok if I cannot port
| my batchscripts to swaheli, really.
|
|

[vincent wehren]

"vincent wehren" <vi*****@visual trans.de> schrieb im Newsbeitrag
news:bo******** **@news4.tilbu1 .nb.home.nl...
|
| "Gerson Kurz" <ge*********@ t-online.de> schrieb im Newsbeitrag
| news:3f******** *****@news.t-online.de...
| | AAAAAAAARG I hate the way python handles unicode. Here is a nice
| | problem for y'all to enjoy: say you have a variable thats unicode
| |
| | directory = u"c:\temp"
| |
| | Its unicode not because you want it to, but because its for example
| | read from _winreg which returns unicode.
| |
| | You do an os.listdir(dire ctory). Note that all filenames returned are
| | now unicode. (Change introduced I believe in 2.3).
|
| Wrong.
|
| That's only true if type(directory) gives you <type 'unicode'>
| If you call str(directory) before doing os.listdir(dire ctory)
| you (in most cases) want even notice and can continue doing what you want
to

And when I say "in most cases", I mean all those cases where "directory"
doesn't have characters that map to a single-byte value outside of the ASCII
range. In other cases you'll just go :

directory =
directory.encod e(your_favorite _and_hoepfully_ the_right_singl e_byte_legacy_e n
coding_here)

before calling os.listdir(dire ctory)
Regards,

Vincent
| do
| just fine - plus, and that's the good part - you can forget about
| those hacks you suggest later and which some would consider *evil*.
| It'll save yourself some time too.
|
| Hey, and leave my Swahili friends alone will ya! ;)
|
| HTH,
| Vincent Wehren
|
|
|
| |
| | You add the filenames to a zipfile.ZipFile object. Sometimes, you will
| | get this exception:
| |
| | Traceback (most recent call last):
| | File "collect_trace_ info.py", line 65, in CollectTraceInf o
| | z.write(pathnam e)
| | File "C:\Python23\li b\zipfile.py", line 416, in write
| | self.fp.write(z info.FileHeader ())
| | File "C:\Python23\li b\zipfile.py", line 170, in FileHeader
| | return header + self.filename + self.extra
| | UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0x88 in position
| | 12:
| | ordinal not in range(128)
| |
| | After you have regained your composure, you find the reason: "header"
| | is a struct.pack() generated byte string. self.filename is however a
| | unicode string because it is returned by os.listdir as unicode. If
| | "header" generates anything above 0x7F - which can but need not
| | happen, depending on the type of file you have an exception waiting
| | for yourself - sometimes. Great. (The same will probably occur if
| | filename contains chars > 0x7F). The problem does not occur if you
| | have "str" type filenames, because then no backandforth conversion is
| | being made.
| |
| | There is a simple fix, before calling z.write() byte-encode it. Here
| | is a sample code:
| |
| | import os, zipfile, win32api
| |
| | def test(directory) :
| | z =
| |
|
zipfile.ZipFile (os.path.join(d irectory,"temp. zip"),"w",zipfi le.ZIP_DEFLATED )
| | for filename in os.listdir(dire ctory):
| | z.write(os.path .join(directory , filename))
| | z.close()
| |
| | if __name__ == "__main__":
| | test(unicode(wi n32api.GetSyste mDirectory()))
| |
| | Note: It might work on your system, depending on the types of files.
| | To fix it, use
| |
| | z.write(os.path .join(directory , filename).encod e("latin-1"))
| |
| | But to my thinking, this is a bug in zipfile.py, really.
| |
| | Now, could anybody please just write a
| | "i-don't-care-if-my-app-can-display-klingon-characters" raw byte
| | encoding which doesn't throw any assertions and doesn't care whether
| | or not the characters are in the 0x7F range? Its ok if I cannot port
| | my batchscripts to swaheli, really.
| |
| |
|
|

[Martin v. Löwis]

ge*********@t-online.de (Gerson Kurz) writes:

Here is my "dontcare"-codec, ugly on purpose:

Of course, this codec does not work for your original problem: Just
see try it on your original data, and then see how Winzip
misinterprets the file names.

zipfiles encode file names in code page 437.

Regards,
Martin

[Thomas Heller]

ma****@v.loewis .de (Martin v. =?iso-8859-15?q?L=F6wis?=) writes:

ge*********@t-online.de (Gerson Kurz) writes:

Here is my "dontcare"-codec, ugly on purpose:

Of course, this codec does not work for your original problem: Just
see try it on your original data, and then see how Winzip
misinterprets the file names.

zipfiles encode file names in code page 437.

So is this a bug in the zipfile module?

Thomas

[Martin v. Löwis]

Thomas Heller <th*****@python .net> writes:

zipfiles encode file names in code page 437.

So is this a bug in the zipfile module?

Yes. It should auto-encode Unicode strings as cp437.

For byte strings, there is not much it could do - the zipfile will
just contain mojibake if encoding of the byte string is not cp437.

Regards,
Martin

[Gerson Kurz]

>Of course, this codec does not work for your original problem: Just

see try it on your original data, and then see how Winzip
misinterpret s the file names.

You are of course right (although my original problem was not with the
filenames - I am using an english version of Windows - but with the
header information).

But, if you look at the number of people that have run into problems
with pythons strictness in unicode matters - isn't it time to offer
some more "relaxed" way of handling all this? I mean, come on, Python
is the language that will change the meaning of 7/3 because people had
problems with integer division. And, although some people use Python
for large multilanguage applications - I would bet that by far more
people use Python for lots of small utility scripts, and couldn't care
less about whether or not its "international" . It just has to work, on
my machine(s). Its not rocket science.

[Peter Otten]

Gerson Kurz wrote:

Of course, this codec does not work for your original problem: Just
see try it on your original data, and then see how Winzip
misinterpre ts the file names.
You are of course right (although my original problem was not with the
filenames - I am using an english version of Windows - but with the
header information).

But, if you look at the number of people that have run into problems
with pythons strictness in unicode matters - isn't it time to offer
some more "relaxed" way of handling all this? I mean, come on, Python

So you've never run into trouble with different encodings and editors not
aware of the encoding? For me this annoyance dates back to Windows 3.1 vs
DOS - RIP. I do remember a book on Apple Pascal that had all y and z
characters accidentally switched.

I'm not aware if there has been a discussion before, but I think it would be
worth the overhead if every string were aware of its encoding, so that
together with the -*- comment in the script you'd never again have to
explicitly go through the decode/encode routine *and* could avoid those
funny filenames - or two-character umlauts that accidentally made it into
your ISO-8859-1 files.
is the language that will change the meaning of 7/3 because people had
problems with integer division. And, although some people use Python
for large multilanguage applications - I would bet that by far more
people use Python for lots of small utility scripts, and couldn't care
less about whether or not its "international" . It just has to work, on
my machine(s). Its not rocket science.

One byte per character encodings might or might not vanish, but unicode
*will* stay, so you better be aware of the pitfalls.
Remember that Python is also the language of "explicit is better implicit".

Peter

[Martin v. Löwis]

Peter Otten <__*******@web. de> writes:

I'm not aware if there has been a discussion before, but I think it would be
worth the overhead if every string were aware of its encoding, so that
together with the -*- comment in the script you'd never again have to
explicitly go through the decode/encode routine *and* could avoid those
funny filenames - or two-character umlauts that accidentally made it into
your ISO-8859-1 files.

Bill Janssen first suggested that a before Unicode was introduced. I
believe it won't help much, as, in many cases, Python can't know what
encoding a byte string is, e.g. if you read from a file, or a socket.
In some cases, you have binary data proper.

Regards,
Martin