binary alternative to pickle?

[Darren Dale]

I am analysing medium sized datasets with Python. At the moment, I am
processing the ASCII files into Python objects every time I run the
script, because it takes about the same time as unpickling the same data.

I think I must have overlooked some utility. In Matlab, you can save
some or all existing variables into a binary file, which loads very
quickly. Is it possible to do the same with Python? Ive looked at
cPickle, which is really slow, and marshal claims that my objects are
unmarshable. Are there other alternatives to save work in progress?

Thanks,
Darren

[Erik Max Francis]

Darren Dale wrote:

I think I must have overlooked some utility. In Matlab, you can save
some or all existing variables into a binary file, which loads very
quickly. Is it possible to do the same with Python? Ive looked at
cPickle, which is really slow, and marshal claims that my objects are
unmarshable. Are there other alternatives to save work in progress?

Use cPickle with one of the latest protocol versions.

--
__ Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
/ \ San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
\__/ I grow with each breath that I take / I cherish my freedom
-- Chante Moore

[Darren Dale]

>

Use cPickle with one of the latest protocol versions.

Oh, yeah. I overlooked that. Thanks.

[Pierre-Frédéric Caillaud]

I have a huge dataset which contains a lot of individual records
represented by class instances.

I pickle this to a file :

way #1 :
for object in objects :
cPickle.dump( object, myfile, -1 )

way #2 :
p = cPickle.Pickler( myfile, -1 )
When I try to unpickle this big file :

p = cPickle.Unpickler( open( ...
many times p.load()... display a progress counter...

Loading the file generated by #1 works fine, with linear speed.
Loading the file generated by #2 :
- the progress counter runs as fast as #1
- eats all memory, then swap
- when eating swap, the progress counter slows down a lot (of course)
- and the process must be killed to save the machine.

I'm talking lots of memory here. The pickled file is about 80 MB, when
loaded it fits into RAM no problem. However I killed the #2 when it had
already hogged about 700 Mb of RAM, and showed no sign of wanting to stop.

What's the problem ?