pickling subclasses of dict/list

[Edward Loper]

I'm having trouble pickling subclasses of dict when they contain cycles.
In particular:

import pickle
class D(dict): pass
d = D()
d[1] = d # add a cycle.
print d {1: {...}} pickle.dump(d, open('d.pickle', 'w')) Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/tmp/python-rSB6hN", line 6, in ?
pickle.dump(d, open('d.pickle', 'w'))
File "/usr/lib/python2.3/pickle.py", line 1382, in dump
Pickler(file, protocol, bin).dump(obj)
[...]
File "/usr/lib/python2.3/pickle.py", line 414, in save_reduce
save(func)
RuntimeError: maximum recursion depth exceeded

Note that pickling dict's that contain cycles works just fine, as does
pickling classes that contain cycles in their instance variables. E.g.:
d = {}
d[1] = d
print d
pickle.dump(d, open('d.pickle', 'w'))
print pickle.load(open('d.pickle'))

{1: {...}}

I tried several things with __getstate__, __reduce__, etc., but couldn't
get this to work. Is there some magic that I'm missing? What's the
best way to get around this? (And should I file this as a bug in
pickle? Or am I just not seeing the right way to do it?)

-Edward

[Greg Chapman]

On Sat, 03 Jul 2004 14:37:28 -0400, Edward Loper
<ed*****@gradient.cis.upenn.edu> wrote:

I'm having trouble pickling subclasses of dict when they contain cycles.

Note that pickling dict's that contain cycles works just fine, as does
pickling classes that contain cycles in their instance variables. E.g.:

>>> d = {}
>>> d[1] = d
>>> print d
>>> pickle.dump(d, open('d.pickle', 'w'))
>>> print pickle.load(open('d.pickle')) {1: {...}}

I tried several things with __getstate__, __reduce__, etc., but couldn't
get this to work. Is there some magic that I'm missing? What's the
best way to get around this? (And should I file this as a bug in
pickle? Or am I just not seeing the right way to do it?)

I think the answer to this problem is to use the extended __reduce__ API
documented here:

http://www.python.org/peps/pep-0307.html

Your subclass's __reduce__ can return a dictitems (or listitems) iterator, the
items of which will get pickled after the subclass instance is memoized, thus
avoiding the infinite recursion. The following works for your simple example:

class D(dict): .... def __reduce__(self):
.... return (D, (), None, None, self.iteritems())
.... d = D()
d[1] = d
pickle.dumps(d) 'c__main__\nD\np0\n(tRp1\nI1\ng1\ns.' s = _
dd = pickle.loads(s)
dd.items()

[(1, {1: {...}})]

---
Greg Chapman