473,574 Members | 3,101 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Unicode and Zipfile problems

AAAAAAAARG I hate the way python handles unicode. Here is a nice
problem for y'all to enjoy: say you have a variable thats unicode

directory = u"c:\temp"

Its unicode not because you want it to, but because its for example
read from _winreg which returns unicode.

You do an os.listdir(dire ctory). Note that all filenames returned are
now unicode. (Change introduced I believe in 2.3).

You add the filenames to a zipfile.ZipFile object. Sometimes, you will
get this exception:

Traceback (most recent call last):
File "collect_trace_ info.py", line 65, in CollectTraceInf o
z.write(pathnam e)
File "C:\Python23\li b\zipfile.py", line 416, in write
self.fp.write(z info.FileHeader ())
File "C:\Python23\li b\zipfile.py", line 170, in FileHeader
return header + self.filename + self.extra
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0x88 in position
12:
ordinal not in range(128)

After you have regained your composure, you find the reason: "header"
is a struct.pack() generated byte string. self.filename is however a
unicode string because it is returned by os.listdir as unicode. If
"header" generates anything above 0x7F - which can but need not
happen, depending on the type of file you have an exception waiting
for yourself - sometimes. Great. (The same will probably occur if
filename contains chars > 0x7F). The problem does not occur if you
have "str" type filenames, because then no backandforth conversion is
being made.

There is a simple fix, before calling z.write() byte-encode it. Here
is a sample code:

import os, zipfile, win32api

def test(directory) :
z =
zipfile.ZipFile (os.path.join(d irectory,"temp. zip"),"w",zipfi le.ZIP_DEFLATED )
for filename in os.listdir(dire ctory):
z.write(os.path .join(directory , filename))
z.close()

if __name__ == "__main__":
test(unicode(wi n32api.GetSyste mDirectory()))

Note: It might work on your system, depending on the types of files.
To fix it, use

z.write(os.path .join(directory , filename).encod e("latin-1"))

But to my thinking, this is a bug in zipfile.py, really.

Now, could anybody please just write a
"i-don't-care-if-my-app-can-display-klingon-characters" raw byte
encoding which doesn't throw any assertions and doesn't care whether
or not the characters are in the 0x7F range? Its ok if I cannot port
my batchscripts to swaheli, really.
Jul 18 '05 #1
19 11866
Here is my "dontcare"-codec, ugly on purpose:

import codecs

def E(i,e=''):
l=lambda c:chr(min(ord(c ),255))
r="".join(map(l ,i))
return (r,len(r))

def D(i,e=''):
l=lambda c: unichr(ord(c))
r=u"".join(map( l,i))
return (r,len(r))

class c(codecs.Codec) :
def encode(self, i,e=''):
return E(i,e)
def decode(self, i,e=''):
return D(i,e)

class w(c,codecs.Stre amWriter): pass
class r(c,codecs.Stre amReader): pass

getregentry = lambda: (E, D, r, w)

To install, save as python23\lib\en codings\dontcar e.py and enable it
in site.py. Here is a testcode:

try:
print unicode(chr(0xF F))
except UnicodeDecodeEr ror, e:
print e

try:
print unichr(12345)
except UnicodeEncodeEr ror, e:
print e

Jul 18 '05 #2

"Gerson Kurz" <ge*********@ t-online.de> schrieb im Newsbeitrag
news:3f******** *****@news.t-online.de...
| AAAAAAAARG I hate the way python handles unicode. Here is a nice
| problem for y'all to enjoy: say you have a variable thats unicode
|
| directory = u"c:\temp"
|
| Its unicode not because you want it to, but because its for example
| read from _winreg which returns unicode.
|
| You do an os.listdir(dire ctory). Note that all filenames returned are
| now unicode. (Change introduced I believe in 2.3).

Wrong.

That's only true if type(directory) gives you <type 'unicode'>
If you call str(directory) before doing os.listdir(dire ctory)
you (in most cases) want even notice and can continue doing what you want to
do
just fine - plus, and that's the good part - you can forget about
those hacks you suggest later and which some would consider *evil*.
It'll save yourself some time too.

Hey, and leave my Swahili friends alone will ya! ;)

HTH,
Vincent Wehren

|
| You add the filenames to a zipfile.ZipFile object. Sometimes, you will
| get this exception:
|
| Traceback (most recent call last):
| File "collect_trace_ info.py", line 65, in CollectTraceInf o
| z.write(pathnam e)
| File "C:\Python23\li b\zipfile.py", line 416, in write
| self.fp.write(z info.FileHeader ())
| File "C:\Python23\li b\zipfile.py", line 170, in FileHeader
| return header + self.filename + self.extra
| UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0x88 in position
| 12:
| ordinal not in range(128)
|
| After you have regained your composure, you find the reason: "header"
| is a struct.pack() generated byte string. self.filename is however a
| unicode string because it is returned by os.listdir as unicode. If
| "header" generates anything above 0x7F - which can but need not
| happen, depending on the type of file you have an exception waiting
| for yourself - sometimes. Great. (The same will probably occur if
| filename contains chars > 0x7F). The problem does not occur if you
| have "str" type filenames, because then no backandforth conversion is
| being made.
|
| There is a simple fix, before calling z.write() byte-encode it. Here
| is a sample code:
|
| import os, zipfile, win32api
|
| def test(directory) :
| z =
|
zipfile.ZipFile (os.path.join(d irectory,"temp. zip"),"w",zipfi le.ZIP_DEFLATED )
| for filename in os.listdir(dire ctory):
| z.write(os.path .join(directory , filename))
| z.close()
|
| if __name__ == "__main__":
| test(unicode(wi n32api.GetSyste mDirectory()))
|
| Note: It might work on your system, depending on the types of files.
| To fix it, use
|
| z.write(os.path .join(directory , filename).encod e("latin-1"))
|
| But to my thinking, this is a bug in zipfile.py, really.
|
| Now, could anybody please just write a
| "i-don't-care-if-my-app-can-display-klingon-characters" raw byte
| encoding which doesn't throw any assertions and doesn't care whether
| or not the characters are in the 0x7F range? Its ok if I cannot port
| my batchscripts to swaheli, really.
|
|
Jul 18 '05 #3
"vincent wehren" <vi*****@visual trans.de> schrieb im Newsbeitrag
news:bo******** **@news4.tilbu1 .nb.home.nl...
|
| "Gerson Kurz" <ge*********@ t-online.de> schrieb im Newsbeitrag
| news:3f******** *****@news.t-online.de...
| | AAAAAAAARG I hate the way python handles unicode. Here is a nice
| | problem for y'all to enjoy: say you have a variable thats unicode
| |
| | directory = u"c:\temp"
| |
| | Its unicode not because you want it to, but because its for example
| | read from _winreg which returns unicode.
| |
| | You do an os.listdir(dire ctory). Note that all filenames returned are
| | now unicode. (Change introduced I believe in 2.3).
|
| Wrong.
|
| That's only true if type(directory) gives you <type 'unicode'>
| If you call str(directory) before doing os.listdir(dire ctory)
| you (in most cases) want even notice and can continue doing what you want
to

And when I say "in most cases", I mean all those cases where "directory"
doesn't have characters that map to a single-byte value outside of the ASCII
range. In other cases you'll just go :

directory =
directory.encod e(your_favorite _and_hoepfully_ the_right_singl e_byte_legacy_e n
coding_here)

before calling os.listdir(dire ctory)
Regards,

Vincent
| do
| just fine - plus, and that's the good part - you can forget about
| those hacks you suggest later and which some would consider *evil*.
| It'll save yourself some time too.
|
| Hey, and leave my Swahili friends alone will ya! ;)
|
| HTH,
| Vincent Wehren
|
|
|
| |
| | You add the filenames to a zipfile.ZipFile object. Sometimes, you will
| | get this exception:
| |
| | Traceback (most recent call last):
| | File "collect_trace_ info.py", line 65, in CollectTraceInf o
| | z.write(pathnam e)
| | File "C:\Python23\li b\zipfile.py", line 416, in write
| | self.fp.write(z info.FileHeader ())
| | File "C:\Python23\li b\zipfile.py", line 170, in FileHeader
| | return header + self.filename + self.extra
| | UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0x88 in position
| | 12:
| | ordinal not in range(128)
| |
| | After you have regained your composure, you find the reason: "header"
| | is a struct.pack() generated byte string. self.filename is however a
| | unicode string because it is returned by os.listdir as unicode. If
| | "header" generates anything above 0x7F - which can but need not
| | happen, depending on the type of file you have an exception waiting
| | for yourself - sometimes. Great. (The same will probably occur if
| | filename contains chars > 0x7F). The problem does not occur if you
| | have "str" type filenames, because then no backandforth conversion is
| | being made.
| |
| | There is a simple fix, before calling z.write() byte-encode it. Here
| | is a sample code:
| |
| | import os, zipfile, win32api
| |
| | def test(directory) :
| | z =
| |
|
zipfile.ZipFile (os.path.join(d irectory,"temp. zip"),"w",zipfi le.ZIP_DEFLATED )
| | for filename in os.listdir(dire ctory):
| | z.write(os.path .join(directory , filename))
| | z.close()
| |
| | if __name__ == "__main__":
| | test(unicode(wi n32api.GetSyste mDirectory()))
| |
| | Note: It might work on your system, depending on the types of files.
| | To fix it, use
| |
| | z.write(os.path .join(directory , filename).encod e("latin-1"))
| |
| | But to my thinking, this is a bug in zipfile.py, really.
| |
| | Now, could anybody please just write a
| | "i-don't-care-if-my-app-can-display-klingon-characters" raw byte
| | encoding which doesn't throw any assertions and doesn't care whether
| | or not the characters are in the 0x7F range? Its ok if I cannot port
| | my batchscripts to swaheli, really.
| |
| |
|
|
Jul 18 '05 #4
ge*********@t-online.de (Gerson Kurz) writes:
Here is my "dontcare"-codec, ugly on purpose:


Of course, this codec does not work for your original problem: Just
see try it on your original data, and then see how Winzip
misinterprets the file names.

zipfiles encode file names in code page 437.

Regards,
Martin
Jul 18 '05 #5
ma****@v.loewis .de (Martin v. =?iso-8859-15?q?L=F6wis?=) writes:
ge*********@t-online.de (Gerson Kurz) writes:
Here is my "dontcare"-codec, ugly on purpose:


Of course, this codec does not work for your original problem: Just
see try it on your original data, and then see how Winzip
misinterprets the file names.

zipfiles encode file names in code page 437.


So is this a bug in the zipfile module?

Thomas
Jul 18 '05 #6
Thomas Heller <th*****@python .net> writes:
zipfiles encode file names in code page 437.


So is this a bug in the zipfile module?


Yes. It should auto-encode Unicode strings as cp437.

For byte strings, there is not much it could do - the zipfile will
just contain mojibake if encoding of the byte string is not cp437.

Regards,
Martin
Jul 18 '05 #7
>Of course, this codec does not work for your original problem: Just
see try it on your original data, and then see how Winzip
misinterpret s the file names.


You are of course right (although my original problem was not with the
filenames - I am using an english version of Windows - but with the
header information).

But, if you look at the number of people that have run into problems
with pythons strictness in unicode matters - isn't it time to offer
some more "relaxed" way of handling all this? I mean, come on, Python
is the language that will change the meaning of 7/3 because people had
problems with integer division. And, although some people use Python
for large multilanguage applications - I would bet that by far more
people use Python for lots of small utility scripts, and couldn't care
less about whether or not its "international" . It just has to work, on
my machine(s). Its not rocket science.
Jul 18 '05 #8
Gerson Kurz wrote:
Of course, this codec does not work for your original problem: Just
see try it on your original data, and then see how Winzip
misinterpre ts the file names.
You are of course right (although my original problem was not with the
filenames - I am using an english version of Windows - but with the
header information).

But, if you look at the number of people that have run into problems
with pythons strictness in unicode matters - isn't it time to offer
some more "relaxed" way of handling all this? I mean, come on, Python


So you've never run into trouble with different encodings and editors not
aware of the encoding? For me this annoyance dates back to Windows 3.1 vs
DOS - RIP. I do remember a book on Apple Pascal that had all y and z
characters accidentally switched.

I'm not aware if there has been a discussion before, but I think it would be
worth the overhead if every string were aware of its encoding, so that
together with the -*- comment in the script you'd never again have to
explicitly go through the decode/encode routine *and* could avoid those
funny filenames - or two-character umlauts that accidentally made it into
your ISO-8859-1 files.
is the language that will change the meaning of 7/3 because people had
problems with integer division. And, although some people use Python
for large multilanguage applications - I would bet that by far more
people use Python for lots of small utility scripts, and couldn't care
less about whether or not its "international" . It just has to work, on
my machine(s). Its not rocket science.


One byte per character encodings might or might not vanish, but unicode
*will* stay, so you better be aware of the pitfalls.
Remember that Python is also the language of "explicit is better implicit".

Peter
Jul 18 '05 #9
Peter Otten <__*******@web. de> writes:
I'm not aware if there has been a discussion before, but I think it would be
worth the overhead if every string were aware of its encoding, so that
together with the -*- comment in the script you'd never again have to
explicitly go through the decode/encode routine *and* could avoid those
funny filenames - or two-character umlauts that accidentally made it into
your ISO-8859-1 files.


Bill Janssen first suggested that a before Unicode was introduced. I
believe it won't help much, as, in many cases, Python can't know what
encoding a byte string is, e.g. if you read from a file, or a socket.
In some cases, you have binary data proper.

Regards,
Martin
Jul 18 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
2915
by: LC | last post by:
Hi, I'm having a problem using the zipfile module in Windows 2000 sp4. When I use it to zip a small file it works fine, but large file doesnt. Here's the error msg i get... --------------------------------------------------------------------- dynamite4_db_archive.py", line 45, in ? file.write(name, os.path.basename(name),...
1
4208
by: Waitman Gobble | last post by:
Hello, I am new to Python. I am having trouble with zipfile.py. On a Linux machine with python 2.4.2 I have trouble opening a zipfile. Python is complaining about the bit where it does a seek(-22,2). Looks to me like zipfile.py is trying to come back 22 bytes from the end of file. # python
5
2446
by: Waguy | last post by:
Hi all, I am new to python and want to create a process to unzip large numbers of zip files I get from a SOAP application. The files all have a ZIP extention and can be unzipped using WinZip. However when I try opening the files using zlib or zipfile modules I get the following error: Traceback (most recent call last):
5
6620
by: Martin | last post by:
I get below error when trying to write unicode xml to a zipfile. zip.writestr('content.xml', content.toxml()) File "/usr/lib/python2.4/zipfile.py", line 460, in writestr zinfo.CRC = binascii.crc32(bytes) # CRC-32 checksum UnicodeEncodeError: 'ascii' codec can't encode character u'\u25cf' in position 2848: ordinal not in range(128) ...
12
4654
by: xamdam | last post by:
Hi fellas, I am experiencing problems reading a 2GB zipfile consisting of multiple zipped files. I found a thread http://mail.python.org/pipermail/python-dev/2005-April/053027.html that mentions a problem on the writing side, does such a problem exist on a reading side? I am using 2.4.1, perhaps there is a fix in a later version?
3
4207
by: towers | last post by:
Hi I'm probably doing something stupid but I've run into a problem whereby I'm trying to add a csv file to a zip archive - see example code below. The csv just has several rows with carriage return line feeds (CRLF). However after adding it to an archive and then decompressing the line endings have been converted to just line feeds...
5
5142
by: Neil Crighton | last post by:
I'm using the zipfile library to read a zip file in Windows, and it seems to be adding too many newlines to extracted files. I've found that for extracted text-encoded files, removing all instances of '\r' in the extracted file seems to fix the problem, but I can't find an easy solution for binary files. The code I'm using is something...
1
2184
by: John Machin | last post by:
On Jun 4, 8:06 pm, jwesonga <crazylun...@gmail.comwrote: Nothing is ever as it seems. Let's try to work backwards from the error message ... and we don't need your magnificent script, just the traceback will do for now, so: The error says that you are trying to seek 22 bytes backwards from the
3
2615
by: =?ISO-2022-JP?B?Ik1hcnRpbiB2LiBMbyJ3aXMi?= | last post by:
Step 4: Either wait for Python 2.7 or apply the patch to your own copy Actually, this is released in Python 2.6, see r62724. Regards, Martin
0
7805
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7726
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
8052
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8234
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
0
6452
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5620
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
3740
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
1
1339
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
1060
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.