473,320 Members | 1,961 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

AssertionError in pickle's memoize function

Hi,

under python 2.2, the pickle/unpickle sequence incorrectly restores
a larger data structure I have.

Under Python 2.3, these structures now give an explicit exception from
Pickle.memoize():
assert id(obj) not in self.memo

I'm shrinking the offending data structure down to find the problem
and provide an easily reproducible example,
but maybe someone on the list could tell me under what general
conditions this assertion is expected to fail.

Thanks,
Michael
Jul 18 '05 #1
3 3993
[Michael Hohn]
under python 2.2, the pickle/unpickle sequence incorrectly restores
a larger data structure I have.

Under Python 2.3, these structures now give an explicit exception from
Pickle.memoize():
assert id(obj) not in self.memo

I'm shrinking the offending data structure down to find the problem
and provide an easily reproducible example,
but maybe someone on the list could tell me under what general
conditions this assertion is expected to fail.


Assertions are never expected to fail, so "something impossible
happened" when they do fail.

See whether your Python has Lib/pickletools.py. There's an enormous
amount of info about pickles in that (for example, it will tell you
what "memo" means).

May help to try cPickle instead of pickle. Since they're distinct
implementations, they have different bugs. cPickle can be much faster
than pickle, but it's a lot easier to understand pickle.py.
Jul 18 '05 #2
Tim Peters <ti********@gmail.com> writes:
[Michael Hohn]
under python 2.2, the pickle/unpickle sequence incorrectly restores
a larger data structure I have.

Under Python 2.3, these structures now give an explicit exception from
Pickle.memoize():
assert id(obj) not in self.memo

I'm shrinking the offending data structure down to find the problem
and provide an easily reproducible example,
but maybe someone on the list could tell me under what general
conditions this assertion is expected to fail.


Assertions are never expected to fail, so "something impossible
happened" when they do fail.

See whether your Python has Lib/pickletools.py. There's an enormous
amount of info about pickles in that (for example, it will tell you
what "memo" means).

May help to try cPickle instead of pickle. Since they're distinct
implementations, they have different bugs. cPickle can be much faster
than pickle, but it's a lot easier to understand pickle.py.


Here is a code sample that shows the problem I ran into:

test.py:
=================================
import pickle

class aList(list):
def __init__(self, arg):
# w/o this call, pickle works...
list.__init__(self, arg)
pass

A = aList([1,2])
B = aList([A, 3])

the_data = {'a': A, 'b': B}
A._stored_by = the_data

pickle.dumps([the_data, B]) # ok
pickle.dumps([B, the_data]) # fails

=================================
Outputs under:

Python 2.3 (#1, Sep 13 2003, 00:49:11)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin
9 scarlet::~:0> python test.py
Traceback (most recent call last):
File "test.py", line 16, in ?
pickle.dumps([B, the_data]) # fails
File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 1386, in dumps
Pickler(file, protocol, bin).dump(obj)
File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 231, in dump
self.save(obj)
File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 614, in save_list
self._batch_appends(iter(obj))
File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 629, in _batch_appends
save(x)
File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 338, in save
self.save_reduce(obj=obj, *rv)
File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 419, in save_reduce
self.memoize(obj)
File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 251, in memoize
assert id(obj) not in self.memo
AssertionError

with the same problem under python on linux:

Python 2.3 (#1, Jul 31 2003, 14:19:24)
[GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-113)] on linux2
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/tmp/python-286703ll", line 1, in ?
pickle.dumps([B, the_data]) # fails
File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 1386, in dumps
Pickler(file, protocol, bin).dump(obj)
File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 231, in dump
self.save(obj)
File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 614, in save_list
self._batch_appends(iter(obj))
File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 629, in _batch_appends
save(x)
File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 338, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 419, in save_reduce
self.memoize(obj)
File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 251, in memoize
assert id(obj) not in self.memo
AssertionError

Jul 18 '05 #3
[Followups to python-dev, please.]

[Michael Hohn]
under python 2.2, the pickle/unpickle sequence incorrectly restores
a larger data structure I have.

Under Python 2.3, these structures now give an explicit exception from
Pickle.memoize():
assert id(obj) not in self.memo

[Tim Peters] Assertions are never expected to fail, so "something impossible
happened" when they do fail.

[Michael Hohn] Here is a code sample that shows the problem I ran into:


Summary for the OP: This is a bug in Python. Using cPickle won't help,
but if you don't subclass builtin container types others than list,
dict, and tuple, using pickle protocol 2 should work. The rest of this
message is for python-dev.
The simplest breaking case is:

t = type('t', (list,), {})
obj = t()
obj.append(obj)
pickle.dumps(obj)
[infinite recursion]

The subclass causes save_reduce to be used instead of save_list. For
proto < 2, copy_reg._reduce_ex returns (_reconstructor, list(a)), and
the args--list(a)--cycle back through obj. Initially it looks like this
should be okay, but the args are saved before obj is memoized, and obj
can't be memoized until REDUCE can be executed with the args--and
there's the cycle. It is even more obviously impossible from the
unpickler's perspective because it has to call _reconstructor([obj]) to
create obj!

There are two separate problems:

1. Any __reduce__ implementation that returns args that cycle back
through the object it tried to reduce hasn't done its job. As
described above, _reduce_ex is one such implementation. reduce_2
avoids this by using the listitems and dictitems parameters. Since
that's a pickler-side feature it can be used in _reduce_ex too. The
basetype(obj) hook (documented in PEP 307) would remain for
immutable bases; it doesn't work for containers, but user-defined
containers already have to implement their own reduce functions.
POC patch: http://www.trit.org/~dima/home/reduce_ex.diff

At least the set and deque types also have this problem.

2. The pickle implementations don't detect reduction cycles. Pickling
an instance of this obviously broken class causes an infinite
recursion:

class evil(object):
def __reduce__(self):
return evil, (self,)

It's easy to detect this case. POC patch for the pickle module:
http://www.trit.org/~dima/home/redcycle.diff

BTW, the failed assert the OP is seeing happens when the cycle goes
through another object:

t = type('t', (list,), {})
obj = t()
d = {'obj': obj}
obj.append(d)
pickle.dumps(obj)
[AssertionError]

cPickle has the same problem, but it lacks the assert, so it writes
garbage instead:

new = cPickle.loads(cPickle.dumps(obj))
new[0]['obj'] is new -> False # wrong
obj[0]['obj'] is obj -> True # right

This makes the reduction cycle check (#2 above) more than just cosmetic
since if cPickle had that assert (it should) it would've been a crash.
Right now it's garbage output instead, which is arguably worse.

Formally complete versions of the above patches will be on SF tomorrow
unless someone suggests better alternatives.

Dima.
Jul 18 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Chris Reedy | last post by:
I would like to memoize (I think I've got that right) a collection of functions for a project I'm working. <Aside for those not familiar with memoizing functions:> Given: def foo(a,b,c):...
1
by: ted kelly | last post by:
It seems I can not deepcopy or pickle an object with an attribute that points to any function that is not a built in. I thought it should be ok as long as the function is defined in the top level...
11
by: thattommyhallll | last post by:
im plugging away at the problems at http://www.mathschallenge.net/index.php?section=project im trying to use them as a motivator to get into advanced topics in python. one thing that Structure...
5
by: Chris | last post by:
Why can pickle serialize references to functions, but not methods? Pickling a function serializes the function name, but pickling a staticmethod, classmethod, or instancemethod generates an...
9
by: andrewfelch | last post by:
Hello all, I'm using the metaclass trick for automatic reloading of class member functions, found at: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164 My problem is that if I 1)...
8
by: Victor Kryukov | last post by:
Hello list, I've found the following strange behavior of cPickle. Do you think it's a bug, or is it by design? Best regards, Victor. from pickle import dumps from cPickle import dumps as...
3
by: fizilla | last post by:
Hello all! I have the following weird problem and since I am new to Python I somehow cannot figure out an elegant solution. The problem reduces to the following question: How to pickle a...
2
by: ankitks.mital | last post by:
I saw example of memoize function...here is snippet def memoize(fn, slot): def memoized_fn(obj, *args): if hasattr(obj, slot): return getattr(obj, slot) else: val = fn(obj, *args)...
1
by: srinivasan srinivas | last post by:
No. It does not work. def make_staticmethod(inst, methodname):     return getattr(inst, methodname) def pickle_function(method):     return make_staticmethod, (method.im_self,...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.