unpickle from URL problem

[Alan Isaac]

I am on a Windows box.

I pickle a tuple of 2 simple objects with the pickle module.
It pickles fine. It unpickles fine.

I upload to a server.
I try to unpickle from the URL. No luck. Try it:
x1, x2 = pickle.load(urllib.urlopen('http://www.american.edu/econ/notes/hw/example1'))

I change the filetype to unix. I upload again.
I try to unpickle from the URL. Now it works. Try it:
x1, x2 = pickle.load(urllib.urlopen('http://www.american.edu/econ/notes/hw/example2'))

Why the difference?

Thank you,
Alan Isaac

[Marc 'BlackJack' Rintsch]

On Wed, 10 Oct 2007 05:58:51 +0000, Alan Isaac wrote:

I am on a Windows box.

I pickle a tuple of 2 simple objects with the pickle module.
It pickles fine. It unpickles fine.

I upload to a server.
I try to unpickle from the URL. No luck. Try it:
x1, x2 = pickle.load(urllib.urlopen('http://www.american.edu/econ/notes/hw/example1'))

I change the filetype to unix. I upload again.
I try to unpickle from the URL. Now it works. Try it:
x1, x2 = pickle.load(urllib.urlopen('http://www.american.edu/econ/notes/hw/example2'))

Why the difference?

Pickles are *binary* files, not text files, so make sure you always treat
them as binary, e.g. opening the files with mode 'rb' and 'wb' and don't
transmit them in text mode over FTP etc.

Ciao,
Marc 'BlackJack' Rintsch

[Alan Isaac]

Marc 'BlackJack' Rintsch wrote:

Pickles are *binary* files, not text files

Actually not:
http://docs.python.org/lib/node316.html

These were created with protocol 0.

But my question is about the different outcomes
I observed.

Thank you,
Alan Isaac

[Hrvoje Niksic]

Alan Isaac <ai****@american.eduwrites:

I upload to a server.
I try to unpickle from the URL. No luck. Try it:
x1, x2 = pickle.load(urllib.urlopen('http://www.american.edu/econ/notes/hw/example1'))

I change the filetype to unix. I upload again.
I try to unpickle from the URL. Now it works. Try it:
x1, x2 = pickle.load(urllib.urlopen('http://www.american.edu/econ/notes/hw/example2'))

Why the difference?

The first upload breaks the file. You uploaded it in (presumably
FTP's) text mode, which changes \n -\r\n. But you download it using
http, which specifies no such conversion in the opposite direction.

It is true that the lowest pickle protocol is textual in nature, but
it just means that you can open a file in text mode, write the pickle
into the file, and reliably read it back on the same platform. The
newlines will be converted to the platform newline representation
(such as \r\n on Windows or \r on Mac) upon write, and back to the
original single-byte (\n) representation upon read. But if you feed
the unpickler a string that already contains \r\n, there is nothing to
convert it back to \n.

[Alan Isaac]

Hrvoje Niksic wrote:

The first upload breaks the file. You uploaded it in (presumably
FTP's) text mode, which changes \n -\r\n. But you download it using
http, which specifies no such conversion in the opposite direction.

No: I used binary upload both time.
(Unless my ftp client is broken, and I think not.)

The first example simply keeps the Windows eols,
which are present in the pickled file on my Windows machine.
This is the one created by pickle.dump.
This file unpickles just fine.

I altered the 2nd file before upload,
changing to the Unix eol convention.
This also unpickles just fine on my machine,
but in addition the urllib download of this
file unpickles just fine.

Alan Isaac

[Marc 'BlackJack' Rintsch]

On Wed, 10 Oct 2007 15:21:06 +0000, Alan Isaac wrote:

Marc 'BlackJack' Rintsch wrote:

>Pickles are *binary* files, not text files

Actually not:
http://docs.python.org/lib/node316.html

These were created with protocol 0.

Actually yes, the docs are wrong. It's a binary file with bytes
constraint to ASCII values. Nevertheless it breaks if you don't open the
files in binary mode and try to use it on a platform that treats line
endings differently.

Ciao,
Marc 'BlackJack' Rintsch

[Hrvoje Niksic]

Alan Isaac <ai****@american.eduwrites:

Hrvoje Niksic wrote:

>The first upload breaks the file. You uploaded it in (presumably
FTP's) text mode, which changes \n -\r\n. But you download it using
http, which specifies no such conversion in the opposite direction.

No: I used binary upload both time.
(Unless my ftp client is broken, and I think not.)

The first example simply keeps the Windows eols,
which are present in the pickled file on my Windows machine.

That's why it doesn't work on Unix, the \r's are a problem when not
removed by the platform's IO layer. Because of the newline silliness,
the notion of a "text file" is, unfortunately, quite platform-specific
even when the file is pure ASCII, so pickle protocol 0 being textual
is a bit misleading regarding the resulting file's portability. :-(