473,846 Members | 1,866 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Moving class used in pickle

I have a class derived from string that is used in a pickle. In the new
version of my program, I moved the module containing the definition of the
class. Now the unpickle fails because it doesn't find the module. I was
thinking that I could make the unpickle work by putting a copy of the
module in the original location and then redefine the class by sticking a
__setstate__ in the class thusly:

def __setstate__(se lf, state):
self.__dict__.u pdate(state)
self.__class__ = NewClassName

My plan was to specify the new location of the module in NewClassName.
However, when I do this I get the message "'class' object layout differs
from 'class'". I don't think that they do as the new module is a copy of
the old one. I suspect that I am not allowed to make the class assignment
because my class is derived from string. What is the best way to update
the pickle? The only thought I have is to read all the data with the old
class module, store the data in some nonpickle format, and then, with
another program, read the nonpickle-format file and rewrite the pickle with
the class module in the new location.
--
Jeffrey Barish

May 15 '07 #1
3 2326
Jeffrey Barish <je*********@ea rthlink.netwrit es:
I have a class derived from string that is used in a pickle. In the new
version of my program, I moved the module containing the definition of the
class. Now the unpickle fails because it doesn't find the module. I was
thinking that I could make the unpickle work by putting a copy of the
module in the original location and then redefine the class by sticking a
__setstate__ in the class thusly:

def __setstate__(se lf, state):
self.__dict__.u pdate(state)
self.__class__ = NewClassName

My plan was to specify the new location of the module in NewClassName.
However, when I do this I get the message "'class' object layout differs
from 'class'". I don't think that they do as the new module is a copy of
the old one. I suspect that I am not allowed to make the class assignment
because my class is derived from string. What is the best way to update
the pickle? The only thought I have is to read all the data with the old
class module, store the data in some nonpickle format, and then, with
another program, read the nonpickle-format file and rewrite the pickle with
the class module in the new location.
You can fiddle with the file class used reading the pickled file. I.e.
the "read()" method could replace each instance of "foo.myclas s" by
"greatnewmodule .mynewclass" could bring you back in the game.

Porting some applications of my from 32 to 64 bit i discovered that my
Numeric int arrays really had to be int32. So i opened the (binary)
pickled files and replaced the occurences of "'l'" by "'i'". Later we
replaced Numeric by numpy. I used the editor approach again to replace
"Numeric" string by "numpy" or "numpy.oldnumer ic". A collegue of mine
wrote a small reader class implementing the approach working on the
input stream.

Regards
Berthold
--
May 21 '07 #2
Jeffrey Barish wrote:
I have a class derived from string that is used in a pickle. In the new
version of my program, I moved the module containing the definition of the
class. Now the unpickle fails because it doesn't find the module. I was
thinking that I could make the unpickle work by putting a copy of the
module in the original location and then redefine the class by sticking a
__setstate__ in the class thusly:

def __setstate__(se lf, state):
self.__dict__.u pdate(state)
self.__class__ = NewClassName

My plan was to specify the new location of the module in NewClassName.
However, when I do this I get the message "'class' object layout differs
from 'class'". I don't think that they do as the new module is a copy of
the old one. I suspect that I am not allowed to make the class assignment
because my class is derived from string. What is the best way to update
the pickle? The only thought I have is to read all the data with the old
class module, store the data in some nonpickle format, and then, with
another program, read the nonpickle-format file and rewrite the pickle
with the class module in the new location.
You could overwrite Unpickler.find_ class():

import pickle
from cStringIO import StringIO

class AliasUnpickler( pickle.Unpickle r):
def __init__(self, aliases, *args, **kw):
pickle.Unpickle r.__init__(self , *args, **kw)
self.aliases = aliases
def find_class(self , module, name):
module, name = self.aliases.ge t((module, name), (module, name))
return pickle.Unpickle r.find_class(se lf, module, name)

def loads(aliases, str):
file = StringIO(str)
return AliasUnpickler( aliases, file).load()

if __name__ == "__main__":
import before, after
data = before.A()
print data.__class__, data
dump = pickle.dumps(da ta)
data = loads({("before ", "A"): ("after", "B")}, dump)
print data.__class__, data

In the example the aliases dictionary maps (module, classname) pairs to
(module, classname) pairs. Of course this only works when the class layout
wasn't changed.

Peter
May 21 '07 #3
En Mon, 21 May 2007 16:24:55 -0300, Berthold Höllmann
<bh***@despamme d.comescribió:
Jeffrey Barish <je*********@ea rthlink.netwrit es:
>I have a class derived from string that is used in a pickle. In the new
version of my program, I moved the module containing the definition of
the class. Now the unpickle fails because it doesn't find the module.
I

You can fiddle with the file class used reading the pickled file. I.e.
the "read()" method could replace each instance of "foo.myclas s" by
"greatnewmodule .mynewclass" could bring you back in the game.
There is a hook in the pickle module (load_global or find_global) that you
can override instead, and it's exactly for this usage, see:
http://docs.python.org/lib/pickle-sub.html

--
Gabriel Genellina

May 22 '07 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
4736
by: Marc | last post by:
Hi all, I was using Tkinter.IntVar() to store values from a large list of parts that I pulled from a list. This is the code to initialize the instances: def initVariables(self): self.e = IntVar() for part, list in info.masterList.items():
1
1321
by: Thomas Aanensen | last post by:
Hi all. I'm sending a class over a socket connection, and I want to use this class at the new location. The problem is that I get a "module blah does not exist" error. Is it possible for me to use this class without importing the module? How? Any other suggestions?
4
1794
by: harold fellermann | last post by:
Hi all, I have a problem pickling an extension class. As written in the Extending/Embedding Manual, I provided a function __reduce__ that returns the appropreate tuple. This seams to work fine, but I still cannot pickle because of the following error: >>> from model import hyper >>> g = hyper.PeriodicGrid(4,4,1)
3
2436
by: syd | last post by:
Hello all, In my project, I have container classes holding lists of item classes. For example, a container class myLibrary might hold a list of item classes myNation and associated variables like myNation.name='USA' and myNation.continent='North America'. Bottom line, I was hoping to use this structure to marshal the classes to xml.
6
1436
by: Kent Johnson | last post by:
Is there a way to persist a class definition (not a class instance, the actual class) so it can be restored later? A naive approach using pickle doesn't work: >>> import pickle >>> class Foo(object): ... def show(self): ... print "I'm a Foo" ... >>> p = pickle.dumps(Foo) >>> p 'c__main__\nFoo\np0\n.'
6
1243
by: Marco Lierfeld | last post by:
Hello there, I want to save an instance of a class containing a dictionary with the pickle-module. The class looks like this: class subproject: configuration = {} build_steps = # some functions
2
5510
by: Peter Bengtsson | last post by:
Hi, I'm trying to pickle an object instance of a class that is like a dict but with a __getattr__ and I'm getting pickling errors. This works but is not good enough. $ python2.4 .... pass .... {'age': 40, 'name': 'Zahid'} {'age': 40, 'name': 'Zahid'}
0
935
by: Brian L. Troutwine | last post by:
I've a need to pickle arbitrary class hierarchies, which, luckily, can be made to conform to the pickle protocol. At the moment, however, I'm having a rather hard time discovering which classes in a heirarchy cannot be pickles. For instance, say class A has class B in it's __dict__ and let class B have a file handler in its __dict__. When I call cPickle.dumps(A) UnpickleableError will be raised when B's file handler is reached, but the...
4
7325
by: PBlitz | last post by:
I want to save the content of a custom made OrderedDict. (I got the OrderedDict from http://aspn.activestate.com/ASPN/Python) The pickle modul seemed to be the solution for the job. At first I tried to pickle the "nacked" OrderedDict derived from dict, but the pickle routine saves only a empty dict. A secont try to "replace" the __dict__ with the __getstate__ and __setstate__ methods also faild. How can I save the content (the items of the...
0
9879
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
10976
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10640
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
10330
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9477
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7877
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
7050
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5714
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
2
4111
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.