By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,345 Members | 1,758 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,345 IT Pros & Developers. It's quick & easy.

pickling extension class

P: n/a
Hi all,

I have a problem pickling an extension class. As written in the
Extending/Embedding Manual, I
provided a function __reduce__ that returns the appropreate tuple. This
seams to work fine,
but I still cannot pickle because of the following error:
from model import hyper
g = hyper.PeriodicGrid(4,4,1)
g.__reduce__() (<type 'hyper.PeriodicGrid'>,(4.,4.,1.)) import pickle
pickle.dump(g,file("test","w")) Traceback (most recent call last):
File "pickle_test.py", line 5, in ?
pickle.dump(g,file("test","w"))
File "/sw/lib/python2.4/pickle.py", line 1382, in dump
Pickler(file, protocol, bin).dump(obj)
File "/sw/lib/python2.4/pickle.py", line 231, in dump
self.save(obj)
File "/sw/lib/python2.4/pickle.py", line 338, in save
self.save_reduce(obj=obj, *rv)
File "/sw/lib/python2.4/pickle.py", line 414, in save_reduce
save(func)
File "/sw/lib/python2.4/pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/sw/lib/python2.4/pickle.py", line 760, in save_global
raise PicklingError(
pickle.PicklingError: Can't pickle <type 'hyper.PeriodicGrid'>: it's
not found as hyper.PeriodicGrid dir(hyper) ['Dir', 'Neighbors', 'PeriodicGrid', 'PeriodicPos', '__doc__',
'__file__', '__name__', 'refcount'] hyper.PeriodicGrid

<type 'hyper.PeriodicGrid'>

So pickle complains about the class PeriodicGrid not being found in the
module hyper, but a dir()
proves that python can find it. Has anyone an idea what's going wrong
here?

Any help appreceated,

- harold -

--
What is mind? -- Doesn't matter.
What is matter? -- Never mind!
--

Jul 18 '05 #1
Share this Question
Share on Google+
4 Replies


P: n/a
harold fellermann <ha***************@upf.edu> wrote:
File "/sw/lib/python2.4/pickle.py", line 760, in save_global
raise PicklingError(
pickle.PicklingError: Can't pickle <type 'hyper.PeriodicGrid'>: it's
not found as hyper.PeriodicGrid
>>> dir(hyper) ['Dir', 'Neighbors', 'PeriodicGrid', 'PeriodicPos', '__doc__',
'__file__', '__name__', 'refcount'] >>> hyper.PeriodicGrid

<type 'hyper.PeriodicGrid'>

So pickle complains about the class PeriodicGrid not being found in the
module hyper, but a dir()
proves that python can find it. Has anyone an idea what's going wrong
here?


These symptomps are pretty weird -- let's try to pin things down a bit
more. The relevant few lines of pickle.py are:

try:
__import__(module)
mod = sys.modules[module]
klass = getattr(mod, name)
except (ImportError, KeyError, AttributeError):
raise PicklingError(

so, could you please edit your pickle.py to provide VASTLY more info,
say:

try:
print 'Here it goes...:'
_xx = __import__(module)
print ' __import__ says: %r' % (_xx,)
mod = sys.modules[module]
print ' in sys.modules: %r' % (mod,)
klass = getattr(mod, name)
print ' klass is: %r' % (klass,)
except (ImportError, KeyError, AttributeError), _xx:
print ' OOPS, error (%s): %s' % (_xx.__class__, _xx)
raise PicklingError(

and let us know exactly what his modified pickle.py outputs...?
Thanks,

Alex
Jul 18 '05 #2

P: n/a

On 18.01.2005, at 20:31, Alex Martelli wrote:
harold fellermann <ha***************@upf.edu> wrote:
File "/sw/lib/python2.4/pickle.py", line 760, in save_global
raise PicklingError(
pickle.PicklingError: Can't pickle <type 'hyper.PeriodicGrid'>: it's
not found as hyper.PeriodicGrid
> dir(hyper) ['Dir', 'Neighbors', 'PeriodicGrid', 'PeriodicPos', '__doc__',
'__file__', '__name__', 'refcount']
> hyper.PeriodicGrid

<type 'hyper.PeriodicGrid'>

So pickle complains about the class PeriodicGrid not being found in
the
module hyper, but a dir()
proves that python can find it. Has anyone an idea what's going wrong
here?


These symptomps are pretty weird -- let's try to pin things down a bit
more. The relevant few lines of pickle.py are:

try:
__import__(module)
mod = sys.modules[module]
klass = getattr(mod, name)
except (ImportError, KeyError, AttributeError):
raise PicklingError(

so, could you please edit your pickle.py to provide VASTLY more info,


[...]
and let us know exactly what his modified pickle.py outputs...?

Here it goes...:
OOPS, error (exceptions.ImportError): No module named hyper
Traceback (most recent call last):
File "pickle_test.py", line 5, in ?
pickle.dump(g,file("test","w"))
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 1387,
in dump
Pickler(file, protocol, bin).dump(obj)
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 231,
in dump
self.save(obj)
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 338,
in save
self.save_reduce(obj=obj, *rv)
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 414,
in save_reduce
save(func)
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 293,
in save
f(self, obj) # Call unbound method with explicit self
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 765,
in save_global
raise PicklingError(
pickle.PicklingError: Can't pickle <type 'hyper.PeriodicGrid'>: it's
not found as hyper.PeriodicGrid
I have noticed that the error does not occur, when the imported module
('hyper') is in the same directory as the script that pickles. When it
is imported from a subpackage (like in the code
I sent you) it goes wrong.

- harold -

--
Reality is for people who lack imagination.

Jul 18 '05 #3

P: n/a
harold fellermann <ha***************@upf.edu> wrote:
...
Here it goes...:
OOPS, error (exceptions.ImportError): No module named hyper
So, the __import__ in pickle fails -- indeed, __import__('foo') when
'foo' ``is imported from a subpackage'' is _supposed_ to fail, as
'hyper' is not the name you SHOULD be importing. For example,

__import__('email.Encoders')

is fine, but just

__import__('Encoders')

fails with ImportError -- there IS no toplevel module by that name!

I have noticed that the error does not occur, when the imported module
('hyper') is in the same directory as the script that pickles. When it
is imported from a subpackage (like in the code
I sent you) it goes wrong.


If you can't fix the modulename given by your type, you can perhaps
kludge things up by (e.g.)

import the.real.hyper
sys.modules['hyper']=the.real.hyper

before you pickle; but I suspect UNpickling would fail in that case.

Using the standard library copy_reg module to register your way to
pickle and recover instances of your type might work better.
Alex
Jul 18 '05 #4

P: n/a
harold fellermann <ha***************@upf.edu> writes:
Hi all,

I have a problem pickling an extension class. As written in the
Extending/Embedding Manual, I
provided a function __reduce__ that returns the appropreate tuple.
This seams to work fine,
but I still cannot pickle because of the following error:
>>> from model import hyper
>>> g = hyper.PeriodicGrid(4,4,1)
>>> g.__reduce__() (<type 'hyper.PeriodicGrid'>,(4.,4.,1.)) >>> import pickle
>>> pickle.dump(g,file("test","w")) Traceback (most recent call last):
File "pickle_test.py", line 5, in ?
pickle.dump(g,file("test","w"))
File "/sw/lib/python2.4/pickle.py", line 1382, in dump
Pickler(file, protocol, bin).dump(obj)
File "/sw/lib/python2.4/pickle.py", line 231, in dump
self.save(obj)
File "/sw/lib/python2.4/pickle.py", line 338, in save
self.save_reduce(obj=obj, *rv)
File "/sw/lib/python2.4/pickle.py", line 414, in save_reduce
save(func)
File "/sw/lib/python2.4/pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/sw/lib/python2.4/pickle.py", line 760, in save_global
raise PicklingError(
pickle.PicklingError: Can't pickle <type 'hyper.PeriodicGrid'>: it's
not found as hyper.PeriodicGrid >>> dir(hyper) ['Dir', 'Neighbors', 'PeriodicGrid', 'PeriodicPos', '__doc__',
'__file__', '__name__', 'refcount'] >>> hyper.PeriodicGrid

<type 'hyper.PeriodicGrid'>

^^^^^

I think that's your error. The extension type is declared to be
hyper.PeriodicGrid, where it actually is model.hyper.PeriodicGrid
(because hyper is in the model package).

Pickle stores g.__class__.__module__ (which is "hyper") and
g.__class__.__name__ (="PeriodicGrid") to find the class object for
reimporting, and on unpickling, tries to do __import__("hyper"), which
fails.

The tp_name slot of your extension type should be "model.hyper.PeriodicGrid".

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)physics(dot)mcmaster(dot)ca
Jul 18 '05 #5

This discussion thread is closed

Replies have been disabled for this discussion.