problem doing unpickle in an exec statement

[Danny Shevitz]

Howdy,

In my app I need to exec user text that defines a function. I want this
function to unpickle an object. Pickle breaks because it is looking for
the object definition that isn't in the calling namespace.

I have mocked up a simple example that shows the problem. Run this
first code (from create_pickle.py) to create the pickle.

create_pickle.py: (run this first)

#############################################
import cPickle

# the pickle file name
file_name = 'd:\\temp\\test1.pickle'

# define a class
class Tree(object):
pass
def main():
# instantiate
t = Tree()

# create the sweet pickle
fp = open(file_name, 'wb')
cPickle.dump(t, fp)
fp.close()

# try to unpickle directly
fp = open(file_name, 'rb')
result = cPickle.load(fp)
fp.close()
print "unpickling directly works just fine, result = ", result

if __name__=='__main__':
main()
#############################################

run this second:

exec_pickle.py
#############################################
# this file shows a problem with sweet pickle in an exec statement

# the pickle file name
file_name = 'd:\\temp\\test1.pickle'

# code to be turned into a function
code_text = '''
def include():
print "this works!"
'''

# a function for creating functions
def create_fun(code_text):
clean_dict = {}
exec code_text in clean_dict
return clean_dict['include']

# include_fun is a bona fide function
include_fun = create_fun(code_text)

# this works
include_fun()
# now try to load the pickle in an exec statement
code_text = '''
def include(file_name):
print "processing file_name: ", file_name
import cPickle
fp = open(file_name, "rb")
result = cPickle.load(fp)
fp.close()
print "result = ", result
'''

# create the new include_fun
include_fun = create_fun(code_text)

# run it
include_fun(file_name)

#############################################

Can anyone enlighten me what I need to do to exec_pickle.py
to get this to work?

thanks,
Danny

[castironpi]

On Jul 23, 6:01*pm, Danny Shevitz <[e-mail]:

Howdy,

In my app I need to exec user text that defines a function. I want this
function to unpickle an object. Pickle breaks because it is looking for
the object definition that isn't in the calling namespace.

I have mocked up a simple example that shows the problem. Run this
first code (from create_pickle.py) to create the pickle.

create_pickle.py: (run this first)

#############################################
import cPickle

# the pickle file name
file_name = 'd:\\temp\\test1.pickle'

# define a class
class Tree(object):
* * * * pass

def main():
* * * * # instantiate *
* * * * t = Tree()

* * * * # create the sweet pickle
* * * * fp = open(file_name, 'wb')
* * * * cPickle.dump(t, fp)
* * * * fp.close()

* * * * # try to unpickle directly
* * * * fp = open(file_name, 'rb')
* * * * result = cPickle.load(fp)
* * * * fp.close()
* * * * print "unpickling directly works just fine, result = ",result

if __name__=='__main__':
* * * * main()
#############################################

run this second:

exec_pickle.py
#############################################
# this file shows a problem with sweet pickle in an exec statement

# the pickle file name
file_name = 'd:\\temp\\test1.pickle'

# code to be turned into a function
code_text = '''
def include():
* print "this works!"
'''

# a function for creating functions
def create_fun(code_text):
* * * * clean_dict = {}
* * * * exec code_text in clean_dict
* * * * return clean_dict['include']

# include_fun is a bona fide function
include_fun = create_fun(code_text)

# this works
include_fun()

# now try to load the pickle in an exec statement
code_text = '''
def include(file_name):
* print "processing file_name: ", file_name
* import cPickle
* fp = open(file_name, "rb")
* result = cPickle.load(fp)
* fp.close()
* print "result = ", result
'''

# create the new include_fun
include_fun = create_fun(code_text)

# run it
include_fun(file_name)

#############################################

Can anyone enlighten me what I need to do to exec_pickle.py
to get this to work?

thanks,
Danny

Hi,

It works if you paste

# define a class
class Tree(object):
pass

into exec_pickle.py. There are inherent dilemmas in pickling
instances of non-primitive types, such as definition location, and
change in definition across versions. How does 'class Tree' fit in to
your design module?

[Peter Otten]

Danny Shevitz wrote:

Howdy,

In my app I need to exec user text that defines a function. I want this
function to unpickle an object. Pickle breaks because it is looking for
the object definition that isn't in the calling namespace.

I have mocked up a simple example that shows the problem. Run this
first code (from create_pickle.py) to create the pickle.

create_pickle.py: (run this first)

#############################################
import cPickle

# the pickle file name
file_name = 'd:\\temp\\test1.pickle'

# define a class
class Tree(object):
pass
def main():
# instantiate
t = Tree()

# create the sweet pickle
fp = open(file_name, 'wb')
cPickle.dump(t, fp)
fp.close()

# try to unpickle directly
fp = open(file_name, 'rb')
result = cPickle.load(fp)
fp.close()
print "unpickling directly works just fine, result = ", result

if __name__=='__main__':
main()
#############################################

run this second:

exec_pickle.py
#############################################
# this file shows a problem with sweet pickle in an exec statement

# the pickle file name
file_name = 'd:\\temp\\test1.pickle'

# code to be turned into a function
code_text = '''
def include():
print "this works!"
'''

# a function for creating functions
def create_fun(code_text):
clean_dict = {}
exec code_text in clean_dict
return clean_dict['include']

# include_fun is a bona fide function
include_fun = create_fun(code_text)

# this works
include_fun()
# now try to load the pickle in an exec statement
code_text = '''
def include(file_name):
print "processing file_name: ", file_name
import cPickle
fp = open(file_name, "rb")
result = cPickle.load(fp)
fp.close()
print "result = ", result
'''

# create the new include_fun
include_fun = create_fun(code_text)

# run it
include_fun(file_name)

#############################################

Can anyone enlighten me what I need to do to exec_pickle.py
to get this to work?

Pickling saves the name of the module and the class (and of course the
instance data). Because you put the class in your main script the module
name is __main__, and when you unpickle later pickle imports __main__ which
unfortunately is now a different script that doesn't contain the/a Tree
class. To solve that problem put the Tree class in a separate module, say
tree.py, that can be imported by both create_pickle.py and exec_pickle.py.
You only need an explicit

from tree import Tree

in create_pickle.

Peter