471,344 Members | 1,544 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,344 software developers and data experts.

unpickle from URL problem

I am on a Windows box.

I pickle a tuple of 2 simple objects with the pickle module.
It pickles fine. It unpickles fine.

I upload to a server.
I try to unpickle from the URL. No luck. Try it:
x1, x2 = pickle.load(urllib.urlopen('http://www.american.edu/econ/notes/hw/example1'))

I change the filetype to unix. I upload again.
I try to unpickle from the URL. Now it works. Try it:
x1, x2 = pickle.load(urllib.urlopen('http://www.american.edu/econ/notes/hw/example2'))

Why the difference?

Thank you,
Alan Isaac
Oct 10 '07 #1
6 5164
On Wed, 10 Oct 2007 05:58:51 +0000, Alan Isaac wrote:
I am on a Windows box.

I pickle a tuple of 2 simple objects with the pickle module.
It pickles fine. It unpickles fine.

I upload to a server.
I try to unpickle from the URL. No luck. Try it:
x1, x2 = pickle.load(urllib.urlopen('http://www.american.edu/econ/notes/hw/example1'))

I change the filetype to unix. I upload again.
I try to unpickle from the URL. Now it works. Try it:
x1, x2 = pickle.load(urllib.urlopen('http://www.american.edu/econ/notes/hw/example2'))

Why the difference?
Pickles are *binary* files, not text files, so make sure you always treat
them as binary, e.g. opening the files with mode 'rb' and 'wb' and don't
transmit them in text mode over FTP etc.

Ciao,
Marc 'BlackJack' Rintsch
Oct 10 '07 #2
Marc 'BlackJack' Rintsch wrote:
Pickles are *binary* files, not text files
Actually not:
http://docs.python.org/lib/node316.html

These were created with protocol 0.

But my question is about the different outcomes
I observed.

Thank you,
Alan Isaac
Oct 10 '07 #3
Alan Isaac <ai****@american.eduwrites:
I upload to a server.
I try to unpickle from the URL. No luck. Try it:
x1, x2 = pickle.load(urllib.urlopen('http://www.american.edu/econ/notes/hw/example1'))

I change the filetype to unix. I upload again.
I try to unpickle from the URL. Now it works. Try it:
x1, x2 = pickle.load(urllib.urlopen('http://www.american.edu/econ/notes/hw/example2'))

Why the difference?
The first upload breaks the file. You uploaded it in (presumably
FTP's) text mode, which changes \n -\r\n. But you download it using
http, which specifies no such conversion in the opposite direction.

It is true that the lowest pickle protocol is textual in nature, but
it just means that you can open a file in text mode, write the pickle
into the file, and reliably read it back on the same platform. The
newlines will be converted to the platform newline representation
(such as \r\n on Windows or \r on Mac) upon write, and back to the
original single-byte (\n) representation upon read. But if you feed
the unpickler a string that already contains \r\n, there is nothing to
convert it back to \n.
Oct 10 '07 #4
Hrvoje Niksic wrote:
The first upload breaks the file. You uploaded it in (presumably
FTP's) text mode, which changes \n -\r\n. But you download it using
http, which specifies no such conversion in the opposite direction.
No: I used binary upload both time.
(Unless my ftp client is broken, and I think not.)

The first example simply keeps the Windows eols,
which are present in the pickled file on my Windows machine.
This is the one created by pickle.dump.
This file unpickles just fine.

I altered the 2nd file before upload,
changing to the Unix eol convention.
This also unpickles just fine on my machine,
but in addition the urllib download of this
file unpickles just fine.

Alan Isaac
Oct 10 '07 #5
On Wed, 10 Oct 2007 15:21:06 +0000, Alan Isaac wrote:
Marc 'BlackJack' Rintsch wrote:
>Pickles are *binary* files, not text files

Actually not:
http://docs.python.org/lib/node316.html

These were created with protocol 0.
Actually yes, the docs are wrong. It's a binary file with bytes
constraint to ASCII values. Nevertheless it breaks if you don't open the
files in binary mode and try to use it on a platform that treats line
endings differently.

Ciao,
Marc 'BlackJack' Rintsch
Oct 10 '07 #6
Alan Isaac <ai****@american.eduwrites:
Hrvoje Niksic wrote:
>The first upload breaks the file. You uploaded it in (presumably
FTP's) text mode, which changes \n -\r\n. But you download it using
http, which specifies no such conversion in the opposite direction.

No: I used binary upload both time.
(Unless my ftp client is broken, and I think not.)

The first example simply keeps the Windows eols,
which are present in the pickled file on my Windows machine.
That's why it doesn't work on Unix, the \r's are a problem when not
removed by the platform's IO layer. Because of the newline silliness,
the notion of a "text file" is, unfortunately, quite platform-specific
even when the file is pure ASCII, so pickle protocol 0 being textual
is a bit misleading regarding the resulting file's portability. :-(
Oct 10 '07 #7

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

1 post views Thread by Jesse Bloom | last post: by
7 posts views Thread by Yun Mao | last post: by
3 posts views Thread by draghuram | last post: by
4 posts views Thread by Ted Zeng | last post: by
10 posts views Thread by krustymonkey | last post: by
2 posts views Thread by Danny Shevitz | last post: by
reply views Thread by Ronak mishra | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.