473,700 Members | 2,424 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Why can't you pickle instancemethods ?

Why can pickle serialize references to functions, but not methods?

Pickling a function serializes the function name, but pickling a
staticmethod, classmethod, or instancemethod generates an error. In
these cases, pickle knows the instance or class, and the method, so
what's the problem? Pickle doesn't serialize code objects, so why can't
it serialize the name as it does for functions? Is this one of those
features that's feasible, but not useful, so no one's ever gotten
around to implementing it?

Regards,
Chris
>>import pickle

def somefunc():
.... return 1
....
>>class Functions(objec t):
.... @staticmethod
.... def somefunc():
.... return 1
....
>>class Foo(object):
.... pass
....
>>f = Foo()
f.value = somefunc
print pickle.dumps(f)
ccopy_reg
_reconstructor
p0
(c__main__
Foo
p1
c__builtin__
object
p2
Ntp3
Rp4
(dp5
S'value'
p6
c__main__
somefunc
p7
sb.
>>>
f.value = Functions.somef unc
print pickle.dumps(f)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "C:\Program Files\Python24\ lib\pickle.py", line 1386, in dumps
Pickler(file, protocol, bin).dump(obj)
File "C:\Program Files\Python24\ lib\pickle.py", line 231, in dump
self.save(obj)
File "C:\Program Files\Python24\ lib\pickle.py", line 338, in save
self.save_reduc e(obj=obj, *rv)
File "C:\Program Files\Python24\ lib\pickle.py", line 433, in
save_reduce
save(state)
File "C:\Program Files\Python24\ lib\pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Program Files\Python24\ lib\pickle.py", line 663, in
save_dict
self._batch_set items(obj.iteri tems())
File "C:\Program Files\Python24\ lib\pickle.py", line 677, in
_batch_setitems
save(v)
File "C:\Program Files\Python24\ lib\pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Program Files\Python24\ lib\pickle.py", line 765, in
save_global
raise PicklingError(
pickle.Pickling Error: Can't pickle <function somefunc at 0x009EC5F0>:
it's not the same object as __
main__.somefunc

Oct 20 '06 #1
5 93093
Chris wrote:
Why can pickle serialize references to functions, but not methods?

Pickling a function serializes the function name, but pickling a
staticmethod, classmethod, or instancemethod generates an error. In
these cases, pickle knows the instance or class, and the method, so
what's the problem? Pickle doesn't serialize code objects, so why can't
it serialize the name as it does for functions? Is this one of those
features that's feasible, but not useful, so no one's ever gotten
around to implementing it?
I have often wondered this myself. I'm convinced that it would in fact
be useful -- more than once I've written a program that has lots of
objects with function pointers, and where it was inconvenient that the
method pointers could not be pickled. One compromise that I have used
before is to write a class such as:

class InstanceMethodS et(object):
def __init__(self,m ethods):
self.methods = set(methods)
def __getstate__(se lf):
return [(method.im_self , method.im_func. func_name)
for method in self.method]
def __setstate__(se lf,state):
self.methods = set(getattr(obj ,name) for obj,name in state)

Obviously, this particular example is crude and not terribly robust,
but it seems to do the job -- it effectively lets you pickle a set of
instance method pointers. I don't know of any reason why instance
methods (or class or static methods) couldn't be pickled directly,
unless perhaps there exists some kind of pathological corner case that
would create Badness?

-Matt

Oct 20 '06 #2
At Friday 20/10/2006 18:33, Chris wrote:
>Why can pickle serialize references to functions, but not methods?
Because all references must be globally accessible.
>Pickling a function serializes the function name, but pickling a
staticmethod , classmethod, or instancemethod generates an error. In
these cases, pickle knows the instance or class, and the method, so
what's the problem? Pickle doesn't serialize code objects, so why can't
it serialize the name as it does for functions? Is this one of those
features that's feasible, but not useful, so no one's ever gotten
around to implementing it?
Well, it's not so common to take a method from one class and glue it
into an *instance* of another class...
(Note that if you copy&paste a method from one *class* into another
*class*, pickle works OK, as long as in the unpickling environment
you rebuild your classes the same way)

For static methods there is no way you can retrieve a globally usable
name (like 'Functions.some func' in your example) - at least I don't
know how to do it, they appear to have lost any reference to the
defining class.
For class methods you can build such reference using im_self and
im_func.func_na me
For instance methods use im_class and im_func.func_na me
Then define your own __getstate__ and __setstate__.
--
Gabriel Genellina
Softlab SRL

_______________ _______________ _______________ _____
Correo Yahoo!
Espacio para todos tus mensajes, antivirus y antispam ¡gratis!
¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar
Oct 20 '06 #3
md******@gmail. com wrote:
Chris wrote:
Why can pickle serialize references to functions, but not methods?

Pickling a function serializes the function name, but pickling a
staticmethod, classmethod, or instancemethod generates an error. In
these cases, pickle knows the instance or class, and the method, so
what's the problem? Pickle doesn't serialize code objects, so why can't
it serialize the name as it does for functions? Is this one of those
features that's feasible, but not useful, so no one's ever gotten
around to implementing it?

I have often wondered this myself. I'm convinced that it would in fact
be useful -- more than once I've written a program that has lots of
objects with function pointers, and where it was inconvenient that the
method pointers could not be pickled. One compromise that I have used
before is to write a class such as:

class InstanceMethodS et(object):
def __init__(self,m ethods):
self.methods = set(methods)
def __getstate__(se lf):
return [(method.im_self , method.im_func. func_name)
for method in self.method]
def __setstate__(se lf,state):
self.methods = set(getattr(obj ,name) for obj,name in state)

Obviously, this particular example is crude and not terribly robust,
but it seems to do the job -- it effectively lets you pickle a set of
instance method pointers. I don't know of any reason why instance
methods (or class or static methods) couldn't be pickled directly,
unless perhaps there exists some kind of pathological corner case that
would create Badness?

-Matt
Thanks, that's quite clever. Although I think you'd still have to
explicitly specify im_self for staticmethods. I could imagine a similar
callable proxy that simply removes the direct method reference from the
equation:

class MethodProxy(obj ect):
def __init__(self, obj, method):
self.obj = obj
if isinstance(meth od, basestring):
self.methodName = method
else:
assert callable(method )
self.methodName = method.func_nam e
def __call__(self, *args, **kwargs):
return getattr(self.ob j, self.methodName )(*args, **kwargs)

picklableMethod = MethodProxy(som eObj, someObj.method)

Oct 20 '06 #4
Chris wrote:
Why can pickle serialize references to functions, but not methods?
Here's the recipe I use::

def _pickle_method( method):
func_name = method.im_func. __name__
obj = method.im_self
cls = method.im_class
return _unpickle_metho d, (func_name, obj, cls)

def _unpickle_metho d(func_name, obj, cls):
for cls in cls.mro():
try:
func = cls.__dict__[func_name]
except KeyError:
pass
else:
break
return func.__get__(ob j, cls)

import copy_reg
import types
copy_reg.pickle (types.MethodTy pe, _pickle_method, _unpickle_metho d)

There may be some special cases where this fails, but I haven't run into
them yet.

STeVe
Oct 21 '06 #5
Steven Bethard wrote:
Here's the recipe I use::

[...]

There may be some special cases where this fails, but I haven't run into
them yet.
Wow, that's a really nice recipe; I didn't even know about the copy_reg
module. I'll have to start using that.

I did notice one failure mode, however--it doesn't work with methods
named __foo because im_func.__name_ _ contains the *unmangled* version
of the function name, so when you try to unpickle the method, the try
statement never succeeds and you get an UnboundLocalErr or on func.

The good news is that I think it can be fixed by mangling the name
manually in _pickle_method( ), like so:

def _pickle_method( method):
func_name = method.im_func. __name__
obj = method.im_self
cls = method.im_class
if func_name.start swith('__') and not func_name.endsw ith('__'):
cls_name = cls.__name__.ls trip('_')
if cls_name: func_name = '_' + cls_name + func_name
return _unpickle_metho d, (func_name, obj, cls)

Nov 11 '06 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
2931
by: Tracy Ruggles | last post by:
>>> sys.platform, sys.version ('freebsd4', '2.2.3 (#2, Nov 17 2003, 17:03:14) \n]') I've poured through pickle.py (line 525) and see where the PicklingError is being raised, but I can't figure out what conditions would create the error. It's testing a class' identity with the object's class that's being pickled. Somewhere along the line, the original class and the class of the instance being pickled have parted ways.
2
9843
by: Jane Austine | last post by:
Hi. class A: def __init__(self,tick): if tick: self.foo=self.bar else: self.foo=self.bur def bar(self): print 'bar'
0
1781
by: Mike P. | last post by:
Hi all, I'm working on a simulation (can be considered a game) in Python where I want to be able to dump the simulation state to a file and be able to load it up later. I have used the standard Python pickle module and it works fine pickling/unpickling from files. However, I want to be able to use a third party tool like an XML editor (or other custom tool) to setup the initial state of the simulation, so I have been playing around...
6
12340
by: Jim Lewis | last post by:
Pickling an instance of a class, gives "can't pickle instancemethod objects". What does this mean? How do I find the class method creating the problem?
10
4439
by: crystalattice | last post by:
I'm creating an RPG for experience and practice. I've finished a character creation module and I'm trying to figure out how to get the file I/O to work. I've read through the python newsgroup and it appears that shelve probably isn't the best option for various reasons. This lead me to try messing w/ pickle, but I can't figure out how to use it with classes. I've found many examples of using pickle w/ non-OOP code but nothing that...
3
6096
by: fizilla | last post by:
Hello all! I have the following weird problem and since I am new to Python I somehow cannot figure out an elegant solution. The problem reduces to the following question: How to pickle a collections.defaultdict object that has set the default_factory property? For Example (from the IDLE console): >>> words = collections.defaultdict(lambda: 1) >>> f = file("temp","w")
2
3537
by: Michele Simionato | last post by:
Can somebody explain what's happening with the following script? $ echo example.py import pickle class Example(object): def __init__(self, obj, registry): self._obj = obj self._registry = registry
10
7769
by: est | last post by:
>>import md5 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Python25\lib\pickle.py", line 1366, in dumps Pickler(file, protocol).dump(obj) File "C:\Python25\lib\pickle.py", line 224, in dump self.save(obj) File "C:\Python25\lib\pickle.py", line 306, in save rv = reduce(self.proto) File "C:\Python25\lib\copy_reg.py", line 69, in _reduce_ex
1
6346
by: IceMan85 | last post by:
Hi to all, I have spent the whole morning trying, with no success to pickle an object that I have created. The error that I get is : Can't pickle 'SRE_Match' object: <_sre.SRE_Match object at 0x2a969c0ad0> the complete stack is the following : Traceback (most recent call last): File "manager.py", line 305, in ? commandLineExec (log, parser) File "manager.py", line 229, in commandLineExec
0
8647
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9217
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8926
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7813
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6563
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5903
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4404
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4659
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
2392
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.