__reduce__(1) vs __reduce__(2)

[Kirill Simonov]

Could someone explain why __reduce__(2) works for files while
__reduce__(1) doesn't?

f = file('/etc/passwd')
f.__reduce__(1) Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib/python2.4/copy_reg.py", line 69, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle file objects f.__reduce__(2)

(<function __newobj__ at 0xb7e6317c>, (<type 'file'>,), None, None, None)

What is a correct procedure of getting state and restoring Python objects?

--
xi

[Ziga Seilnacht]

Kirill Simonov wrote:

Could someone explain why __reduce__(2) works for files while
__reduce__(1) doesn't?
I think it is a bug. Both should raise an error.
__reduce__ and __reduce_ex__ are part of the pickle protocol.
Files are not meant to be pickable, since they are already
persistent. With protocol 0 and 1, an error is raised when
you try to pickle a builtin object that doesn't have a custom
__reduce__() method. However, protocol 2 changed that, and
now almost all objects can be pickled, but not necessarily
unpickled to their original state. Files are one such example:
you can pickle them, but the unpickled object won't be useful;
you will get uninitialised file object. Basically, the result
is equivalent to the following:
unpickled = file.__new__(file)

[snipped] What is a correct procedure of getting state and restoring Python objects?
You should never call __reduce__() with the protocol argument.
It works only for objects that don't override the default
__reduce__() method, and is possible only because
object.__reduce_ex__ and object.__reduce__ represent the same
underlying function. For example, sets override this method,
and don't expect the protocol argument:

s = set([1,2,3])
s.__reduce__(2)

Traceback (most recent call last):
...
TypeError: __reduce__() takes no arguments (1 given)

The correct way is to try to call obj.__reduce_ex__(protocol)
first, and fall back to obj.__reduce__() if that method is not
available. Example:

if hasattr(obj, '__reduce_ex__'):
state_tuple = obj.__reduce_ex__(2)
else:
state_tuple = obj.__reduce__()

For more details check the pickle protocol documentation:
http://docs.python.org/lib/pickle-protocol.html

as well as Extensions to the pickle protocol PEP:
http://www.python.org/dev/peps/pep-0307/
--
xi

Ziga