Why can't you pickle instancemethods?

[Chris]

Why can pickle serialize references to functions, but not methods?

Pickling a function serializes the function name, but pickling a
staticmethod, classmethod, or instancemethod generates an error. In
these cases, pickle knows the instance or class, and the method, so
what's the problem? Pickle doesn't serialize code objects, so why can't
it serialize the name as it does for functions? Is this one of those
features that's feasible, but not useful, so no one's ever gotten
around to implementing it?

Regards,
Chris

>>import pickle

def somefunc():

.... return 1
....

>>class Functions(object):

.... @staticmethod
.... def somefunc():
.... return 1
....

>>class Foo(object):

.... pass
....

>>f = Foo()
f.value = somefunc
print pickle.dumps(f)

ccopy_reg
_reconstructor
p0
(c__main__
Foo
p1
c__builtin__
object
p2
Ntp3
Rp4
(dp5
S'value'
p6
c__main__
somefunc
p7
sb.

>>>
f.value = Functions.somefunc
print pickle.dumps(f)

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "C:\Program Files\Python24\lib\pickle.py", line 1386, in dumps
Pickler(file, protocol, bin).dump(obj)
File "C:\Program Files\Python24\lib\pickle.py", line 231, in dump
self.save(obj)
File "C:\Program Files\Python24\lib\pickle.py", line 338, in save
self.save_reduce(obj=obj, *rv)
File "C:\Program Files\Python24\lib\pickle.py", line 433, in
save_reduce
save(state)
File "C:\Program Files\Python24\lib\pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Program Files\Python24\lib\pickle.py", line 663, in
save_dict
self._batch_setitems(obj.iteritems())
File "C:\Program Files\Python24\lib\pickle.py", line 677, in
_batch_setitems
save(v)
File "C:\Program Files\Python24\lib\pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Program Files\Python24\lib\pickle.py", line 765, in
save_global
raise PicklingError(
pickle.PicklingError: Can't pickle <function somefunc at 0x009EC5F0>:
it's not the same object as __
main__.somefunc

[mdsteele]

Chris wrote:

Why can pickle serialize references to functions, but not methods?

Pickling a function serializes the function name, but pickling a
staticmethod, classmethod, or instancemethod generates an error. In
these cases, pickle knows the instance or class, and the method, so
what's the problem? Pickle doesn't serialize code objects, so why can't
it serialize the name as it does for functions? Is this one of those
features that's feasible, but not useful, so no one's ever gotten
around to implementing it?

I have often wondered this myself. I'm convinced that it would in fact
be useful -- more than once I've written a program that has lots of
objects with function pointers, and where it was inconvenient that the
method pointers could not be pickled. One compromise that I have used
before is to write a class such as:

class InstanceMethodSet(object):
def __init__(self,methods):
self.methods = set(methods)
def __getstate__(self):
return [(method.im_self, method.im_func.func_name)
for method in self.method]
def __setstate__(self,state):
self.methods = set(getattr(obj,name) for obj,name in state)

Obviously, this particular example is crude and not terribly robust,
but it seems to do the job -- it effectively lets you pickle a set of
instance method pointers. I don't know of any reason why instance
methods (or class or static methods) couldn't be pickled directly,
unless perhaps there exists some kind of pathological corner case that
would create Badness?

-Matt

[Gabriel Genellina]

At Friday 20/10/2006 18:33, Chris wrote:

>Why can pickle serialize references to functions, but not methods?

Because all references must be globally accessible.

>Pickling a function serializes the function name, but pickling a
staticmethod, classmethod, or instancemethod generates an error. In
these cases, pickle knows the instance or class, and the method, so
what's the problem? Pickle doesn't serialize code objects, so why can't
it serialize the name as it does for functions? Is this one of those
features that's feasible, but not useful, so no one's ever gotten
around to implementing it?

Well, it's not so common to take a method from one class and glue it
into an *instance* of another class...
(Note that if you copy&paste a method from one *class* into another
*class*, pickle works OK, as long as in the unpickling environment
you rebuild your classes the same way)

For static methods there is no way you can retrieve a globally usable
name (like 'Functions.somefunc' in your example) - at least I don't
know how to do it, they appear to have lost any reference to the
defining class.
For class methods you can build such reference using im_self and
im_func.func_name
For instance methods use im_class and im_func.func_name
Then define your own __getstate__ and __setstate__.
--
Gabriel Genellina
Softlab SRL

__________________________________________________
Correo Yahoo!
Espacio para todos tus mensajes, antivirus y antispam ¡gratis!
¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar

[Chris]

md******@gmail.com wrote:

Chris wrote:

Why can pickle serialize references to functions, but not methods?

Pickling a function serializes the function name, but pickling a
staticmethod, classmethod, or instancemethod generates an error. In
these cases, pickle knows the instance or class, and the method, so
what's the problem? Pickle doesn't serialize code objects, so why can't
it serialize the name as it does for functions? Is this one of those
features that's feasible, but not useful, so no one's ever gotten
around to implementing it?

I have often wondered this myself. I'm convinced that it would in fact
be useful -- more than once I've written a program that has lots of
objects with function pointers, and where it was inconvenient that the
method pointers could not be pickled. One compromise that I have used
before is to write a class such as:

class InstanceMethodSet(object):
def __init__(self,methods):
self.methods = set(methods)
def __getstate__(self):
return [(method.im_self, method.im_func.func_name)
for method in self.method]
def __setstate__(self,state):
self.methods = set(getattr(obj,name) for obj,name in state)

Obviously, this particular example is crude and not terribly robust,
but it seems to do the job -- it effectively lets you pickle a set of
instance method pointers. I don't know of any reason why instance
methods (or class or static methods) couldn't be pickled directly,
unless perhaps there exists some kind of pathological corner case that
would create Badness?

-Matt

Thanks, that's quite clever. Although I think you'd still have to
explicitly specify im_self for staticmethods. I could imagine a similar
callable proxy that simply removes the direct method reference from the
equation:

class MethodProxy(object):
def __init__(self, obj, method):
self.obj = obj
if isinstance(method, basestring):
self.methodName = method
else:
assert callable(method)
self.methodName = method.func_name
def __call__(self, *args, **kwargs):
return getattr(self.obj, self.methodName)(*args, **kwargs)

picklableMethod = MethodProxy(someObj, someObj.method)

[Steven Bethard]

Chris wrote:

Why can pickle serialize references to functions, but not methods?

Here's the recipe I use::

def _pickle_method(method):
func_name = method.im_func.__name__
obj = method.im_self
cls = method.im_class
return _unpickle_method, (func_name, obj, cls)

def _unpickle_method(func_name, obj, cls):
for cls in cls.mro():
try:
func = cls.__dict__[func_name]
except KeyError:
pass
else:
break
return func.__get__(obj, cls)

import copy_reg
import types
copy_reg.pickle(types.MethodType, _pickle_method, _unpickle_method)

There may be some special cases where this fails, but I haven't run into
them yet.

STeVe

[mdsteele]

Steven Bethard wrote:

Here's the recipe I use::

[...]

There may be some special cases where this fails, but I haven't run into
them yet.

Wow, that's a really nice recipe; I didn't even know about the copy_reg
module. I'll have to start using that.

I did notice one failure mode, however--it doesn't work with methods
named __foo because im_func.__name__ contains the *unmangled* version
of the function name, so when you try to unpickle the method, the try
statement never succeeds and you get an UnboundLocalError on func.

The good news is that I think it can be fixed by mangling the name
manually in _pickle_method(), like so:

def _pickle_method(method):
func_name = method.im_func.__name__
obj = method.im_self
cls = method.im_class
if func_name.startswith('__') and not func_name.endswith('__'):
cls_name = cls.__name__.lstrip('_')
if cls_name: func_name = '_' + cls_name + func_name
return _unpickle_method, (func_name, obj, cls)