Why can't you pickle instancemethods ?

[Chris]

Why can pickle serialize references to functions, but not methods?

Pickling a function serializes the function name, but pickling a
staticmethod, classmethod, or instancemethod generates an error. In
these cases, pickle knows the instance or class, and the method, so
what's the problem? Pickle doesn't serialize code objects, so why can't
it serialize the name as it does for functions? Is this one of those
features that's feasible, but not useful, so no one's ever gotten
around to implementing it?

Regards,
Chris

>>import pickle

def somefunc():

.... return 1
....

>>class Functions(objec t):

.... @staticmethod
.... def somefunc():
.... return 1
....

>>class Foo(object):

.... pass
....

>>f = Foo()
f.value = somefunc
print pickle.dumps(f)

ccopy_reg
_reconstructor
p0
(c__main__
Foo
p1
c__builtin__
object
p2
Ntp3
Rp4
(dp5
S'value'
p6
c__main__
somefunc
p7
sb.

>>>
f.value = Functions.somef unc
print pickle.dumps(f)

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "C:\Program Files\Python24\ lib\pickle.py", line 1386, in dumps
Pickler(file, protocol, bin).dump(obj)
File "C:\Program Files\Python24\ lib\pickle.py", line 231, in dump
self.save(obj)
File "C:\Program Files\Python24\ lib\pickle.py", line 338, in save
self.save_reduc e(obj=obj, *rv)
File "C:\Program Files\Python24\ lib\pickle.py", line 433, in
save_reduce
save(state)
File "C:\Program Files\Python24\ lib\pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Program Files\Python24\ lib\pickle.py", line 663, in
save_dict
self._batch_set items(obj.iteri tems())
File "C:\Program Files\Python24\ lib\pickle.py", line 677, in
_batch_setitems
save(v)
File "C:\Program Files\Python24\ lib\pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Program Files\Python24\ lib\pickle.py", line 765, in
save_global
raise PicklingError(
pickle.Pickling Error: Can't pickle <function somefunc at 0x009EC5F0>:
it's not the same object as __
main__.somefunc

[mdsteele]

Chris wrote:

Why can pickle serialize references to functions, but not methods?

Pickling a function serializes the function name, but pickling a
staticmethod, classmethod, or instancemethod generates an error. In
these cases, pickle knows the instance or class, and the method, so
what's the problem? Pickle doesn't serialize code objects, so why can't
it serialize the name as it does for functions? Is this one of those
features that's feasible, but not useful, so no one's ever gotten
around to implementing it?

I have often wondered this myself. I'm convinced that it would in fact
be useful -- more than once I've written a program that has lots of
objects with function pointers, and where it was inconvenient that the
method pointers could not be pickled. One compromise that I have used
before is to write a class such as:

class InstanceMethodS et(object):
def __init__(self,m ethods):
self.methods = set(methods)
def __getstate__(se lf):
return [(method.im_self , method.im_func. func_name)
for method in self.method]
def __setstate__(se lf,state):
self.methods = set(getattr(obj ,name) for obj,name in state)

Obviously, this particular example is crude and not terribly robust,
but it seems to do the job -- it effectively lets you pickle a set of
instance method pointers. I don't know of any reason why instance
methods (or class or static methods) couldn't be pickled directly,
unless perhaps there exists some kind of pathological corner case that
would create Badness?

-Matt

[Gabriel Genellina]

At Friday 20/10/2006 18:33, Chris wrote:

>Why can pickle serialize references to functions, but not methods?

Because all references must be globally accessible.

>Pickling a function serializes the function name, but pickling a
staticmethod , classmethod, or instancemethod generates an error. In
these cases, pickle knows the instance or class, and the method, so
what's the problem? Pickle doesn't serialize code objects, so why can't
it serialize the name as it does for functions? Is this one of those
features that's feasible, but not useful, so no one's ever gotten
around to implementing it?

Well, it's not so common to take a method from one class and glue it
into an *instance* of another class...
(Note that if you copy&paste a method from one *class* into another
*class*, pickle works OK, as long as in the unpickling environment
you rebuild your classes the same way)

For static methods there is no way you can retrieve a globally usable
name (like 'Functions.some func' in your example) - at least I don't
know how to do it, they appear to have lost any reference to the
defining class.
For class methods you can build such reference using im_self and
im_func.func_na me
For instance methods use im_class and im_func.func_na me
Then define your own __getstate__ and __setstate__.
--
Gabriel Genellina
Softlab SRL

_______________ _______________ _______________ _____
Correo Yahoo!
Espacio para todos tus mensajes, antivirus y antispam ¡gratis!
¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar

[Chris]

md******@gmail. com wrote:

Chris wrote:

Why can pickle serialize references to functions, but not methods?

Pickling a function serializes the function name, but pickling a
staticmethod, classmethod, or instancemethod generates an error. In
these cases, pickle knows the instance or class, and the method, so
what's the problem? Pickle doesn't serialize code objects, so why can't
it serialize the name as it does for functions? Is this one of those
features that's feasible, but not useful, so no one's ever gotten
around to implementing it?

I have often wondered this myself. I'm convinced that it would in fact
be useful -- more than once I've written a program that has lots of
objects with function pointers, and where it was inconvenient that the
method pointers could not be pickled. One compromise that I have used
before is to write a class such as:

class InstanceMethodS et(object):
def __init__(self,m ethods):
self.methods = set(methods)
def __getstate__(se lf):
return [(method.im_self , method.im_func. func_name)
for method in self.method]
def __setstate__(se lf,state):
self.methods = set(getattr(obj ,name) for obj,name in state)

Obviously, this particular example is crude and not terribly robust,
but it seems to do the job -- it effectively lets you pickle a set of
instance method pointers. I don't know of any reason why instance
methods (or class or static methods) couldn't be pickled directly,
unless perhaps there exists some kind of pathological corner case that
would create Badness?

-Matt

Thanks, that's quite clever. Although I think you'd still have to
explicitly specify im_self for staticmethods. I could imagine a similar
callable proxy that simply removes the direct method reference from the
equation:

class MethodProxy(obj ect):
def __init__(self, obj, method):
self.obj = obj
if isinstance(meth od, basestring):
self.methodName = method
else:
assert callable(method )
self.methodName = method.func_nam e
def __call__(self, *args, **kwargs):
return getattr(self.ob j, self.methodName )(*args, **kwargs)

picklableMethod = MethodProxy(som eObj, someObj.method)

[Steven Bethard]

Chris wrote:

Why can pickle serialize references to functions, but not methods?

Here's the recipe I use::

def _pickle_method( method):
func_name = method.im_func. __name__
obj = method.im_self
cls = method.im_class
return _unpickle_metho d, (func_name, obj, cls)

def _unpickle_metho d(func_name, obj, cls):
for cls in cls.mro():
try:
func = cls.__dict__[func_name]
except KeyError:
pass
else:
break
return func.__get__(ob j, cls)

import copy_reg
import types
copy_reg.pickle (types.MethodTy pe, _pickle_method, _unpickle_metho d)

There may be some special cases where this fails, but I haven't run into
them yet.

STeVe

[mdsteele]

Steven Bethard wrote:

Here's the recipe I use::

[...]

There may be some special cases where this fails, but I haven't run into
them yet.

Wow, that's a really nice recipe; I didn't even know about the copy_reg
module. I'll have to start using that.

I did notice one failure mode, however--it doesn't work with methods
named __foo because im_func.__name_ _ contains the *unmangled* version
of the function name, so when you try to unpickle the method, the try
statement never succeeds and you get an UnboundLocalErr or on func.

The good news is that I think it can be fixed by mangling the name
manually in _pickle_method( ), like so:

def _pickle_method( method):
func_name = method.im_func. __name__
obj = method.im_self
cls = method.im_class
if func_name.start swith('__') and not func_name.endsw ith('__'):
cls_name = cls.__name__.ls trip('_')
if cls_name: func_name = '_' + cls_name + func_name
return _unpickle_metho d, (func_name, obj, cls)