467,864 Members | 1,718 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 467,864 developers. It's quick & easy.

pickling subclasses of dict/list

I'm having trouble pickling subclasses of dict when they contain cycles.
In particular:
import pickle
class D(dict): pass
d = D()
d[1] = d # add a cycle.
print d {1: {...}} pickle.dump(d, open('d.pickle', 'w')) Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/tmp/python-rSB6hN", line 6, in ?
pickle.dump(d, open('d.pickle', 'w'))
File "/usr/lib/python2.3/pickle.py", line 1382, in dump
Pickler(file, protocol, bin).dump(obj)
[...]
File "/usr/lib/python2.3/pickle.py", line 414, in save_reduce
save(func)
RuntimeError: maximum recursion depth exceeded

Note that pickling dict's that contain cycles works just fine, as does
pickling classes that contain cycles in their instance variables. E.g.:
d = {}
d[1] = d
print d
pickle.dump(d, open('d.pickle', 'w'))
print pickle.load(open('d.pickle'))

{1: {...}}

I tried several things with __getstate__, __reduce__, etc., but couldn't
get this to work. Is there some magic that I'm missing? What's the
best way to get around this? (And should I file this as a bug in
pickle? Or am I just not seeing the right way to do it?)

-Edward

Jul 18 '05 #1
  • viewed: 2064
Share:
1 Reply
On Sat, 03 Jul 2004 14:37:28 -0400, Edward Loper
<ed*****@gradient.cis.upenn.edu> wrote:
I'm having trouble pickling subclasses of dict when they contain cycles.

Note that pickling dict's that contain cycles works just fine, as does
pickling classes that contain cycles in their instance variables. E.g.:
>>> d = {}
>>> d[1] = d
>>> print d
>>> pickle.dump(d, open('d.pickle', 'w'))
>>> print pickle.load(open('d.pickle')) {1: {...}}

I tried several things with __getstate__, __reduce__, etc., but couldn't
get this to work. Is there some magic that I'm missing? What's the
best way to get around this? (And should I file this as a bug in
pickle? Or am I just not seeing the right way to do it?)


I think the answer to this problem is to use the extended __reduce__ API
documented here:

http://www.python.org/peps/pep-0307.html

Your subclass's __reduce__ can return a dictitems (or listitems) iterator, the
items of which will get pickled after the subclass instance is memoized, thus
avoiding the infinite recursion. The following works for your simple example:
class D(dict): .... def __reduce__(self):
.... return (D, (), None, None, self.iteritems())
.... d = D()
d[1] = d
pickle.dumps(d) 'c__main__\nD\np0\n(tRp1\nI1\ng1\ns.' s = _
dd = pickle.loads(s)
dd.items()

[(1, {1: {...}})]

---
Greg Chapman

Jul 18 '05 #2

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

1 post views Thread by Thomas Guettler | last post: by
1 post views Thread by Thomas Guettler | last post: by
3 posts views Thread by Nicolas Fleury | last post: by
reply views Thread by benevilent | last post: by
5 posts views Thread by Charles Krug | last post: by
7 posts views Thread by Daniel Nogradi | last post: by
2 posts views Thread by Peter Bengtsson | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.