473,230 Members | 1,481 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,230 software developers and data experts.

pickling a subclass of tuple

Hi all, happy new year,

I was trying to pickle a instance of a subclass of a tuple when I ran
into a problem. Pickling doesn't work with HIGHEST_PROTOCOL. How should
I rewrite my class so I can pickle it?

Thanks ,

Fedor

#!/usr/bin/env python
import pickle
class A(tuple):
def __new__(klass, arg1,arg2):
return super(A,klass).__new__(klass, (arg1,arg2))
a=A(1,2)
print "no pickle",a
print "normal pickle",pickle.loads(pickle.dumps(a))
print "highest protocol",
pickle.loads(pickle.dumps(a,pickle.HIGHEST_PROTOCO L))

This is the output:
'''
no pickle (1, 2)
normal pickle (1, 2)
highest protocol
Traceback (most recent call last):
File "./test.py", line 9, in ?
print "highest
protocol",pickle.loads(pickle.dumps(a,pickle.HIGHE ST_PROTOCOL))
File "/usr/lib/python2.3/pickle.py", line 1394, in loads
return Unpickler(file).load()
File "/usr/lib/python2.3/pickle.py", line 872, in load
dispatch[key](self)
File "/usr/lib/python2.3/pickle.py", line 1097, in load_newobj
obj = cls.__new__(cls, *args)
TypeError: __new__() takes exactly 3 arguments (2 given)
'''


Jul 18 '05 #1
1 3187
fedor <no****@here.com> wrote:
Hi all, happy new year,

I was trying to pickle a instance of a subclass of a tuple when I ran
into a problem. Pickling doesn't work with HIGHEST_PROTOCOL. How should
I rewrite my class so I can pickle it?


You're falling afoul of an optimization in pickle's protocol 2, which is
documented in pickle.py as follows:

# A __reduce__ implementation can direct protocol 2 to
# use the more efficient NEWOBJ opcode, while still
# allowing protocol 0 and 1 to work normally. For this to
# work, the function returned by __reduce__ should be
# called __newobj__, and its first argument should be a
# new-style class. The implementation for __newobj__
# should be as follows, although pickle has no way to
# verify this:
#
# def __newobj__(cls, *args):
# return cls.__new__(cls, *args)
#
# Protocols 0 and 1 will pickle a reference to __newobj__,
# while protocol 2 (and above) will pickle a reference to
# cls, the remaining args tuple, and the NEWOBJ code,
# which calls cls.__new__(cls, *args) at unpickling time
# (see load_newobj below). If __reduce__ returns a
# three-tuple, the state from the third tuple item will be
# pickled regardless of the protocol, calling __setstate__
# at unpickling time (see load_build below).

Essentially, and simplifying just a little...: you're inheriting
__reduce_ex__ (because you're not overriding it), but you ARE overriding
__new__ *and changing its signature* -- so, the inherited __reduce__ex__
is used, and, with this protocol 2 optimization, it essentially assumes
that __new__ is similarly used -- or, at least, that a __new__ is used
which does not arbitrarily change the signature!

So, if you want to change __new__'s signature, and yet be picklable by
protocol 2, you have to override __reduce_ex__ to return the right
"args"... those your class's __new__ expects!
For example, you could consider something like...:

def __newobj__(cls, *args):
return cls.__new__(cls, *args)

class A(tuple):
def __new__(klass, arg1, arg2):
return super(A, klass).__new__(klass, (arg1, arg2))

def __reduce_ex__(self, proto=0):
if proto >= 2:
return __newobj__, (A, self[0], self[1])
else:
return super(A, self).__reduce_ex__(proto)

Note the key difference in A's __reduce_ex__ (for proto=2) wrt tuple's
(which is the same as object's) -- that's after an "import a" where a.py
has this code as well as an 'a = A(1, 2)'...:
a.a.__reduce_ex__(2) (<function __newobj__ at 0x3827f0>, (<class 'a.A'>, 1, 2)) tuple.__reduce_ex__(a.a, 2) (<function __newobj__ at 0x376770>, (<class 'a.A'>, (1, 2)), {}, None,
None)
Apart from the additional tuple items (not relevant here), tuple's
reduce returns args as (<class 'a.A'>, (1, 2)) -- two items: the class
and the tuplevalue; so with protocol 2 this ends up calling A.__new__(A,
(1,2))... BOOM, because, differently from tuple.__new__, YOUR override
doesn't accept this signature! So, I suggest tweaking A's reduce so it
returns args as (<class 'a.A'>, 1, 2)... apparently the only signature
you're willing to accept in your A.__new__ method.

Of course, if A.__new__ can have some flexibility, you COULD have it
accept the same signature as tuple.__new__ and then you wouldn't have to
override __reduce_ex__. Or, you could override __reduce_ex__ in other
ways, say:

def __reduce_ex__(self, proto=0):
if proto >= 2:
proto = 1
return super(A, self).__reduce_ex__(proto)

this would avoid the specific optimization that's tripping you up due to
your signature-change in __new__.

The best solution may be to forget __reduce_ex__ and take advantage of
the underdocumented special method __getnewargs__ ...:

class A(tuple):
def __new__(klass, arg1, arg2):
return super(A, klass).__new__(klass, (arg1, arg2))

def __getnewargs__(self):
return self[0], self[1]

This way, you're essentially choosing to explicitly tell the "normal"
__reduce_ex__ about the particular arguments you want to be used for the
__new__ call needed to reconstruct your object on unpickling! This
highlights even better the crucial difference, due strictly to the
change in __new__'s signature...:
a.a.__getnewargs__() (1, 2) tuple.__getnewargs__(a.a)

((1, 2),)

It IS, I guess, somewhat unfortunate that you have to understand
pickling in some depth to let you change __new__'s signature and yet
fully support pickling... on the other hand, when you're overriding
__new__ you ARE messing with some rather deep infrastructure,
particularly if you alter its signature so that it doesn't accept
"normal" calls any more, so it's not _absurd_ that compensatory depth of
understanding is required;-).
Alex
Jul 18 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Edward Loper | last post by:
I'm having trouble pickling subclasses of dict when they contain cycles. In particular: >>> import pickle >>> class D(dict): pass >>> d = D() >>> d = d # add a cycle. >>> print d {1: {...}}...
0
by: Skip Montanaro | last post by:
I have a class that inherits from PyGTK's gobject.GObject. I thought to pickle it I'd be able to just define __etstate__ methods. When I attempt to dump an instance of this class I get a...
4
by: harold fellermann | last post by:
Hi all, I have a problem pickling an extension class. As written in the Extending/Embedding Manual, I provided a function __reduce__ that returns the appropreate tuple. This seams to work fine,...
2
by: Kirk Strauser | last post by:
I have a module that defines a Search class and a SearchResult class. I use these classes by writing other modules that subclass both of them as needed to interface with particular search engines....
9
by: Alex | last post by:
I have a serious problem and I hope there is some solution. It is easier to illustrate with a simple code: >>> class Parent(object): __slots__= def __init__(self, a, b): self.A=a; self.B=b...
18
by: Sandra-24 | last post by:
Can you create an instance of a subclass using an existing instance of the base class? Such things would be impossible in some languages or very difficult in others. I wonder if this can be done...
3
by: manstey | last post by:
Hi, I am running a script that produces about 450,000 dictionaries. I tried putting them into a tuple and then pickling the tuple, but the tuple gets too big. Can I pickle dictionaries one after...
2
by: Alan Isaac | last post by:
I am probably confused about immutable types. But for now my questions boil down to these two: - what does ``tuple.__init__`` do? - what is the signature of ``tuple.__init__``? These...
1
by: Mike Rooney | last post by:
Hi everyone, this is my first post to this list. I am trying to create a subclass of datetime.date and pickle it, but I get errors on loading it back. I have created a VERY simple demo of this: ...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.