Memory error while saving dictionary of size 65000X50 using pickle

[Nagu]

I am trying to save a dictionary of size 65000X50 to a local file and
I get the memory error problem.

How do I go about resolving this? Is there way to partition the pickle
object and combine later if this is a problem due to limited resources
(memory) on the machine (it is 32 bit machine Win XP, with 4GB RAM).
Here is the detail description of the error:

Traceback (most recent call last):
File "<pyshell#1 2>", line 1, in <module>
s = pickle.dumps(it emsim)
File "C:\Python25\li b\pickle.py", line 1366, in dumps
Pickler(file, protocol).dump( obj)
File "C:\Python25\li b\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python25\li b\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\li b\pickle.py", line 649, in save_dict
self._batch_set items(obj.iteri tems())
File "C:\Python25\li b\pickle.py", line 663, in _batch_setitems
save(v)
File "C:\Python25\li b\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\li b\pickle.py", line 600, in save_list
self._batch_app ends(iter(obj))
File "C:\Python25\li b\pickle.py", line 615, in _batch_appends
save(x)
File "C:\Python25\li b\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\li b\pickle.py", line 562, in save_tuple
save(element)
File "C:\Python25\li b\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\li b\pickle.py", line 477, in save_float
self.write(FLOA T + repr(obj) + '\n')
MemoryError: out of memory

[Larry Bates]

Nagu wrote:

I am trying to save a dictionary of size 65000X50 to a local file and
I get the memory error problem.

How do I go about resolving this? Is there way to partition the pickle
object and combine later if this is a problem due to limited resources
(memory) on the machine (it is 32 bit machine Win XP, with 4GB RAM).
Here is the detail description of the error:

Traceback (most recent call last):
File "<pyshell#1 2>", line 1, in <module>
s = pickle.dumps(it emsim)
File "C:\Python25\li b\pickle.py", line 1366, in dumps
Pickler(file, protocol).dump( obj)
File "C:\Python25\li b\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python25\li b\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\li b\pickle.py", line 649, in save_dict
self._batch_set items(obj.iteri tems())
File "C:\Python25\li b\pickle.py", line 663, in _batch_setitems
save(v)
File "C:\Python25\li b\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\li b\pickle.py", line 600, in save_list
self._batch_app ends(iter(obj))
File "C:\Python25\li b\pickle.py", line 615, in _batch_appends
save(x)
File "C:\Python25\li b\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\li b\pickle.py", line 562, in save_tuple
save(element)
File "C:\Python25\li b\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python25\li b\pickle.py", line 477, in save_float
self.write(FLOA T + repr(obj) + '\n')
MemoryError: out of memory

I've generated some rather large pickled dicts before and have never seen this
before. You can, of course, split your dictionary into several smaller ones,
pickle them individually and combine them back together on unpickle/read using
the dictionary update method.

BTW - 32-bit Windows can only address 3.5Gb of memory maximum. If you have more
installed, it is ignored.

-Larry

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Nagu wrote:

I am trying to save a dictionary of size 65000X50 to a local file and
I get the memory error problem.

What do you mean by this size specification? When I interpreter X as
multiplication, I can't see a problem: the code

import pickle

d = {}

for i in xrange(65000*50 ):
d[i]=i
print "Starting dump"
s = pickle.dumps(d)

works just fine for me. Can you please modify it so that it does cause
a problem?

Regards,
Martin