469,327 Members | 1,290 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,327 developers. It's quick & easy.

pickling multiple dictionaries

Hi,

I am running a script that produces about 450,000 dictionaries. I tried
putting them into a tuple and then pickling the tuple, but the tuple
gets too big. Can I pickle dictionaries one after another into the same
file and then read them out again?

Cheers,
Matthew

May 24 '06 #1
3 2698
Hello Matthew,

You can try either http://docs.python.org/lib/module-shelve.html or any
other database bindings with blobs.

HTH,
Miki
http://pythonwise.blogspot.com

May 24 '06 #2
manstey wrote:
Hi,

I am running a script that produces about 450,000 dictionaries. I tried
putting them into a tuple and then pickling the tuple, but the tuple
gets too big. Can I pickle dictionaries one after another into the same
file and then read them out again?

Cheers,
Matthew

If you don't know, just try it:

In [1]:import pickle
In [2]:d1={'a':1}
In [3]:d2={'b':2}
In [4]:pfile=file('test.p','wb')
In [5]:pickle.dump(d1,pfile)
In [6]:pickle.dump(d2,pfile)
In [7]:pfile.close()
In [8]:del d1
In [9]:del d2
In [10]:pfile=file('test.p','rb')
In [11]:d1=pickle.load(pfile)
In [12]:d1
Out[12]:{'a': 1}
In [13]:d2=pickle.load(pfile)
In [14]:d2
Out[14]:{'b': 2}

If your data is *really* large, have a look to PyTables
(http://www.pytables.org/moin).

Regards,
Hans Georg
May 24 '06 #3
Thanks very much. How large is *really* large for making pytables
worthwhile. Our python script generates an xml file of about 450Mb. Is
pytables worth using then?

May 24 '06 #4

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

1 post views Thread by Edward Loper | last post: by
8 posts views Thread by Hans Georg Krauthaeuser | last post: by
1 post views Thread by fedor | last post: by
1 post views Thread by Erik Max Francis | last post: by
7 posts views Thread by Philipp H. Mohr | last post: by
5 posts views Thread by Livin | last post: by
reply views Thread by Irmen de Jong | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by zhoujie | last post: by
reply views Thread by suresh191 | last post: by
reply views Thread by Purva khokhar | last post: by
reply views Thread by haryvincent176 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.