bug in copy.deepcopy or in getattr or in my understanding?

[Emin]

Dear experts,

I got some unexpected behavior in getattr and copy.deepcopy (see
transcript below). I'm not sure if this is actually a bug in
copy.deepcopy or if I'm doing something too magical with getattr.
Comments would be appreciated.

Thanks,
-Emin

######### Transcript follows ##################
Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

>># THE FOLLOWING BREAKS
class a:

.... def foo(self):
.... print 'hi'
....

>>class b(a):

.... def __init__(self):
.... self.y = getattr(self,'foo')
....

>>c = b()
import copy
copy.deepcopy(c)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "c:\Python25\lib\copy.py", line 162, in deepcopy
y = copier(x, memo)
File "c:\Python25\lib\copy.py", line 291, in _deepcopy_inst
state = deepcopy(state, memo)
File "c:\Python25\lib\copy.py", line 162, in deepcopy
y = copier(x, memo)
File "c:\Python25\lib\copy.py", line 254, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "c:\Python25\lib\copy.py", line 189, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "c:\Python25\lib\copy.py", line 322, in _reconstruct
y = callable(*args)
File "c:\Python25\lib\copy_reg.py", line 92, in __newobj__
return cls.__new__(cls, *args)
TypeError: instancemethod expected at least 2 arguments, got 0

>># THE FOLLOWING WORKS
class b(a):

.... def __init__(self):
.... self.x = self.__class__.__bases__[0].__dict__['foo']
....

>>c=b()
copy.deepcopy(c)

<__main__.b instance at 0x00EADE18>

[Gabriel Genellina]

At Thursday 4/1/2007 17:26, Emin wrote:

>I got some unexpected behavior in getattr and copy.deepcopy (see
transcript below). I'm not sure if this is actually a bug in
copy.deepcopy or if I'm doing something too magical with getattr.
Comments would be appreciated.

Both examples are different. #1 stores a *bound* method into an
instance attribute. Bound methods carry a reference to "self", this
creates a cyclic reference that may cause problems to the garbage
collector (and confuses deepcopy, apparently).
#2 uses and *unbound* method and it's the usual way.

>class a:

... def foo(self):
... print 'hi'
...

>class b(a): #1

... def __init__(self):
... self.y = getattr(self,'foo')

>class b(a): #2

... def __init__(self):
... self.x = self.__class__.__bases__[0].__dict__['foo']

For #2 you can simply say:

class b(a):
x = a.foo

If you have to choose at runtime (a simplified version of your own code):

class b(a):
def __init__(self):
name = select_method_to_use(..., default="foo")
self.x = getattr(a, name)

You *know* your bases because you wrote them in the class statement
(or use super() instead of playing with __bases__); and getattr works
fine here so you don't need to mess with the __dict__ details.

(Note that #1 was *almost* right, you had to replace self by the base class)
--
Gabriel Genellina
Softlab SRL


__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

[Emin]

Dear Gabriel,

Thank you for your reply. As you guessed, I want to be able to select
the method at runtime as in your final example, but when I tried your
suggestion I got the same error (see below). I think the problem is
that getattr is donig something different than in my example where I
explicitly get it from the dict (see the very end of the transcript
below):
--------------------------- Transcript Follows
----------------------------------------------------------

Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

>># The following shows that using getattr to grab a method is
# incompatible with copy.deepcopy
class a:

.... def foo(self): pass
....

>>class b(a):

.... def __init__(self):
.... self.x = getattr(a,'foo')
....

>>import copy
c=b()
copy.deepcopy(c)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "c:\Python25\lib\copy.py", line 162, in deepcopy
y = copier(x, memo)
File "c:\Python25\lib\copy.py", line 291, in _deepcopy_inst
state = deepcopy(state, memo)
File "c:\Python25\lib\copy.py", line 162, in deepcopy
y = copier(x, memo)
File "c:\Python25\lib\copy.py", line 254, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "c:\Python25\lib\copy.py", line 189, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "c:\Python25\lib\copy.py", line 322, in _reconstruct
y = callable(*args)
File "c:\Python25\lib\copy_reg.py", line 92, in __newobj__
return cls.__new__(cls, *args)
TypeError: instancemethod expected at least 2 arguments, got 0

>>>
# The following shows that getattr is doing something different
# than looking in the __dict__ of base classes
b.__bases__[0].__dict__['foo']

<function foo at 0x009F0CF0>

>>getattr(a,'foo')

<unbound method a.foo>

>>>

On Jan 4, 10:08 pm, Gabriel Genellina <[e-mail]:

At Thursday 4/1/2007 17:26, Emin wrote:

I got some unexpected behavior in getattr and copy.deepcopy (see
transcript below). I'm not sure if this is actually a bug in
copy.deepcopy or if I'm doing something too magical with getattr.
Comments would be appreciated.Both examples are different. #1 stores a *bound* method into an

instance attribute. Bound methods carry a reference to "self", this
creates a cyclic reference that may cause problems to the garbage
collector (and confuses deepcopy, apparently).
#2 uses and *unbound* method and it's the usual way.

>>class a:

... def foo(self):
... print 'hi'
...

>>class b(a): #1

... def __init__(self):
... self.y = getattr(self,'foo')

>>class b(a): #2

... def __init__(self):
... self.x = self.__class__.__bases__[0].__dict__['foo']For #2 you can simply say:

class b(a):
x = a.foo

If you have to choose at runtime (a simplified version of your own code):

class b(a):
def __init__(self):
name = select_method_to_use(..., default="foo")
self.x = getattr(a, name)

You *know* your bases because you wrote them in the class statement
(or use super() instead of playing with __bases__); and getattr works
fine here so you don't need to mess with the __dict__ details.

(Note that #1 was *almost* right, you had to replace self by the base class)

--
Gabriel Genellina
Softlab SRL

__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!http://www.yahoo.com.ar/respuestas

[Peter Otten]

Emin wrote:

Thank you for your reply. As you guessed, I want to be able to select
the method at runtime as in your final example, but when I tried your
suggestion I got the same error (see below). I think the problem is
that getattr is donig something different than in my example where I
explicitly get it from the dict (see the very end of the transcript
below):

This has nothing to do with getattr(). You currently can deep-copy
functions, but neither bound nor unbound methods:

>>import copy
def check(obj):

.... try:
.... copy.deepcopy(obj)
.... except:
.... return "FAILED"
.... return "OK"
....

>>def function(*args): pass

....

>>class A:

.... method = function
....

>>check(function), check(A.method), check(A().method)

('OK', 'FAILED', 'FAILED')

Whether this a bug or a sensible limitation I don't know.

Peter

[Gabriel Genellina]

At Friday 5/1/2007 11:01, Emin wrote:

>Thank you for your reply. As you guessed, I want to be able to select
the method at runtime as in your final example, but when I tried your
suggestion I got the same error (see below). I think the problem is
that getattr is donig something different than in my example where I
explicitly get it from the dict (see the very end of the transcript
below):

I've modified it as follows:
- using new.instancemethod to create the dynamic method
- copy uses __getstate__ to determine what is needed to copy/save.
Assuming that self.x is derived from other attributes, it's not
actually needed in the saved state, and can be omited.
- __setstate__ does the inverse: after reconstructing the object
state, sets self.x to the right value

=== cut ===
import new

class A:
def foo(self, arg):
print "in foo, self=%r arg=%r" % (self, arg)

class B(A):
def __init__(self):
self.update_dynamic_methods()

def update_dynamic_methods(self):
"Should assign the dynamic methods based on other attributes"
self.x = new.instancemethod(getattr(A,"foo"),self,B)

def __getstate__(self):
state = self.__dict__.copy()
if 'x' in state: del state['x']
return state

def __setstate__(self, state):
self.__dict__.update(state)
self.update_dynamic_methods()

b=B()
b.something = 'something'
b.foo(123)
b.x(123)

import copy
b2 = copy.deepcopy(b)
assert b2.something=='something'
b2.foo(123)
b2.x(123)
=== cut ===

--
Gabriel Genellina
Softlab SRL


__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas