473,578 Members | 3,177 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

pickling subclasses of dict/list

I'm having trouble pickling subclasses of dict when they contain cycles.
In particular:
import pickle
class D(dict): pass
d = D()
d[1] = d # add a cycle.
print d {1: {...}} pickle.dump(d, open('d.pickle' , 'w')) Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/tmp/python-rSB6hN", line 6, in ?
pickle.dump(d, open('d.pickle' , 'w'))
File "/usr/lib/python2.3/pickle.py", line 1382, in dump
Pickler(file, protocol, bin).dump(obj)
[...]
File "/usr/lib/python2.3/pickle.py", line 414, in save_reduce
save(func)
RuntimeError: maximum recursion depth exceeded

Note that pickling dict's that contain cycles works just fine, as does
pickling classes that contain cycles in their instance variables. E.g.:
d = {}
d[1] = d
print d
pickle.dump(d, open('d.pickle' , 'w'))
print pickle.load(ope n('d.pickle'))

{1: {...}}

I tried several things with __getstate__, __reduce__, etc., but couldn't
get this to work. Is there some magic that I'm missing? What's the
best way to get around this? (And should I file this as a bug in
pickle? Or am I just not seeing the right way to do it?)

-Edward

Jul 18 '05 #1
1 2287
On Sat, 03 Jul 2004 14:37:28 -0400, Edward Loper
<ed*****@gradie nt.cis.upenn.ed u> wrote:
I'm having trouble pickling subclasses of dict when they contain cycles.

Note that pickling dict's that contain cycles works just fine, as does
pickling classes that contain cycles in their instance variables. E.g.:
>>> d = {}
>>> d[1] = d
>>> print d
>>> pickle.dump(d, open('d.pickle' , 'w'))
>>> print pickle.load(ope n('d.pickle')) {1: {...}}

I tried several things with __getstate__, __reduce__, etc., but couldn't
get this to work. Is there some magic that I'm missing? What's the
best way to get around this? (And should I file this as a bug in
pickle? Or am I just not seeing the right way to do it?)


I think the answer to this problem is to use the extended __reduce__ API
documented here:

http://www.python.org/peps/pep-0307.html

Your subclass's __reduce__ can return a dictitems (or listitems) iterator, the
items of which will get pickled after the subclass instance is memoized, thus
avoiding the infinite recursion. The following works for your simple example:
class D(dict): .... def __reduce__(self ):
.... return (D, (), None, None, self.iteritems( ))
.... d = D()
d[1] = d
pickle.dumps(d) 'c__main__\nD\n p0\n(tRp1\nI1\n g1\ns.' s = _
dd = pickle.loads(s)
dd.items()

[(1, {1: {...}})]

---
Greg Chapman

Jul 18 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
1996
by: Thomas Guettler | last post by:
Hi! If I pickle a class which inherits from dict, I don't get the same data again, if I unpickle it. def test_pickle(): home="..." server=WorkflowServer(home) server.save()
1
1517
by: Thomas Guettler | last post by:
Hi! After unpickling the objects are not the same any more. Is this a bug or feature? import pickle class MyDictContainer(dict): def __init__(self): dict.__init__(self)
3
1851
by: Nicolas Fleury | last post by:
Hi, Does anyone know if arrays would be picklable in python 2.4? Until then, I tried to derive from array.array and add __setstate__ and __getstate__ with the following code, but it seems I'm not doing it correctly. Any idea what I'm doing wrong? Thx and Regards, Nicolas import array as arr
0
1159
by: benevilent | last post by:
Hey, I am creating an extension type in C for Python, which I want it such that instances can be pickled. I basically want the functionality of the 'object' type such that subclasses of the type which I define, can define methods including __getstate__ which will get called on pickling/unpickling. To do this I need to define the...
5
1266
by: Charles Krug | last post by:
List: I have this: # classC.py class C(object): pass class D(C): pass
7
2069
by: Daniel Nogradi | last post by:
What is the simplest way to instantiate all classes that are subclasses of a given class in a module? More precisely I have a module m with some content: # m.py class A: pass class x( A ): pass
6
1234
by: Marco Lierfeld | last post by:
Hello there, I want to save an instance of a class containing a dictionary with the pickle-module. The class looks like this: class subproject: configuration = {} build_steps = # some functions
0
1033
by: dackz | last post by:
>>class ListyThing(list): pass .... Traceback (most recent call last): File "<stdin>", line 1, in <module> AssertionError <type 'list'> I don't find this intuitive. Is this intentional? I believe this could be avoided if list.__getitem__ used "self.__class__()" to make a new instance, instead of "list()", but I don't know how that works...
2
5474
by: Peter Bengtsson | last post by:
Hi, I'm trying to pickle an object instance of a class that is like a dict but with a __getattr__ and I'm getting pickling errors. This works but is not good enough. $ python2.4 .... pass .... {'age': 40, 'name': 'Zahid'} {'age': 40, 'name': 'Zahid'}
0
7847
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
8125
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8290
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
0
8148
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
1
5664
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
3781
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3798
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2292
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1389
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.