473,320 Members | 1,883 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Memory error while saving dictionary of size 65000X50 using pickle

I am trying to save a dictionary of size 65000X50 to a local file and
I get the memory error problem.

How do I go about resolving this? Is there way to partition the pickle
object and combine later if this is a problem due to limited resources
(memory) on the machine (it is 32 bit machine Win XP, with 4GB RAM).
Here is the detail description of the error:

Traceback (most recent call last):
File "<pyshell#12>", line 1, in <module>
s = pickle.dumps(itemsim)
File "C:\Python25\lib\pickle.py", line 1366, in dumps
Pickler(file, protocol).dump(obj)
File "C:\Python25\lib\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python25\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python25\lib\pickle.py", line 663, in _batch_setitems
save(v)
File "C:\Python25\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\lib\pickle.py", line 600, in save_list
self._batch_appends(iter(obj))
File "C:\Python25\lib\pickle.py", line 615, in _batch_appends
save(x)
File "C:\Python25\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\lib\pickle.py", line 562, in save_tuple
save(element)
File "C:\Python25\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\lib\pickle.py", line 477, in save_float
self.write(FLOAT + repr(obj) + '\n')
MemoryError: out of memory
Jul 7 '08 #1
2 4425
Nagu wrote:
I am trying to save a dictionary of size 65000X50 to a local file and
I get the memory error problem.

How do I go about resolving this? Is there way to partition the pickle
object and combine later if this is a problem due to limited resources
(memory) on the machine (it is 32 bit machine Win XP, with 4GB RAM).
Here is the detail description of the error:

Traceback (most recent call last):
File "<pyshell#12>", line 1, in <module>
s = pickle.dumps(itemsim)
File "C:\Python25\lib\pickle.py", line 1366, in dumps
Pickler(file, protocol).dump(obj)
File "C:\Python25\lib\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python25\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python25\lib\pickle.py", line 663, in _batch_setitems
save(v)
File "C:\Python25\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\lib\pickle.py", line 600, in save_list
self._batch_appends(iter(obj))
File "C:\Python25\lib\pickle.py", line 615, in _batch_appends
save(x)
File "C:\Python25\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\lib\pickle.py", line 562, in save_tuple
save(element)
File "C:\Python25\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\lib\pickle.py", line 477, in save_float
self.write(FLOAT + repr(obj) + '\n')
MemoryError: out of memory
I've generated some rather large pickled dicts before and have never seen this
before. You can, of course, split your dictionary into several smaller ones,
pickle them individually and combine them back together on unpickle/read using
the dictionary update method.

BTW - 32-bit Windows can only address 3.5Gb of memory maximum. If you have more
installed, it is ignored.

-Larry
Jul 7 '08 #2
Nagu wrote:
I am trying to save a dictionary of size 65000X50 to a local file and
I get the memory error problem.
What do you mean by this size specification? When I interpreter X as
multiplication, I can't see a problem: the code

import pickle

d = {}

for i in xrange(65000*50):
d[i]=i
print "Starting dump"
s = pickle.dumps(d)

works just fine for me. Can you please modify it so that it does cause
a problem?

Regards,
Martin
Jul 7 '08 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: Benjamin Scott | last post by:
Hello. I attempted to build a compound dictionary: len(Lst)=1000 len(nuerLst)=250 len(nuestLst)=500 Dict={}
6
by: Byron | last post by:
Hello, I am a newbie and would like to know if it is possible to convert a string back to a dictionary? For example, I can convert a dictionary to a string by doing this: >>> names =...
8
by: Hans Georg Krauthaeuser | last post by:
Dear all, I have a long running application (electromagnetic compatibility measurements in mode-stirred chambers over GPIB) that use pickle (cPickle) to autosave a class instance with all the...
10
by: Luis P. Mendes | last post by:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, my program builds a dictionary that I would like to save in a file. My question is what are the simple ways to do it? The first solution...
6
by: Ganesan selvaraj | last post by:
I using C# .net. i want to split the text files based of the some condition. my source text file size may be 4 kb to 4 gb. some time when i split the i got the "out of memory exception. when i...
3
by: RubenDV | last post by:
I am writing code for a dictionary type in c, but i have some problems allocating the memory for it. The dictionary type itself is a struct with the following contents: typedef struct { char **...
0
by: Nagu | last post by:
I am trying to save a dictionary of size 65000X50 to a local file and I get the memory error problem. How do I go about resolving this? Is there way to partition the pickle object and combine...
1
by: Nagu | last post by:
I didn't have the problem with dumping as a string. When I tried to save this object to a file, memory error pops up. I am sorry for the mention of size for a dictionary. What I meant by...
3
by: Catherine Moroney | last post by:
I'm writing a python program that reads in a very large "pickled" file (consisting of one large dictionary and one small one), and parses the results out to several binary and hdf files. The...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.