deepcopy chokes with TypeError on dynamically assigned instance method

[‘5ÛHH575-UAZWKVVP-7H2H48V3]

(see end of message for example code)

When an instance has a dynamically assigned instance method, deepcopy
throws a TypeError with the message "TypeError: instancemethod
expected at least 2 arguments, got 0". Tested with Python 2.3.4 on
OpenBSD and Python 2.4 on Win98; same results. Is this a bug in
deepcopy, how I dynamically assign the instance method or something
else? (See example code for how I did it.)

If you're curious as to why the deep copy and dynamic assign are
necessary or have implementation suggestions (or alternatives), I bet
you'd like some details. The TypeError cropped up while coding a
'Point' class representing cartesian coordinates. I needed to
overload an 'origin' method as both a class method and an instance
method (if Python were more Perlesque... ;-) ). 'origin' returns a
point representing an origin. The class method requires an argument
for the dimension of the origin, while the instance method uses the
dimension of an instance (ortus overloading). As the instance
reference isn't bound when the class 'origin' method is defined,
there's no way to use a default argument. I instead dynamically
assign an instance method to the 'origin' attribute of the instance.

As for the deepcopy, scalars aren't necessarily of a built-in numeric
type, though I generally expect them to be numeric (you could use
lists or strings as scalars, but not many methods would still be
usable). Point is the base clase for Vector, and I want (e.g.)
vectors of vectors so I can eventually extend Vector to Matrix and
Tensor. The Point constructor has a single argument: a sequence of
scalars (which, as noted, can be sequences). In practice, the
sequence itself will be a tuple, a list, a Point or descendant of
Point (so that Point can act as a copy constructor). To prevent a
copied point from aliasing elements of a different Point, I used
deepcopy. When the TypeError struck, I switched to a generator, which
works as long as every constructor functions as a copy constructor
(not necessarily true, apparently, of lists, but true of Points). I
could also implement copy-on-write semantics for coordinates or
implement __deepcopy__ for Point (which will probably be the final
solution).
example code:

from copy import copy,deepcopy
import new

class Foo(list):
"Foo"

def __init__(self, l=[]):
list.__init__(self, deepcopy(l))
# using generator rather than deepcopy produces no errors.
#list.__init__(self, [copy(el) for el in l])
# 'copy(el)' in generator results in a deepcopy of sequence
# as long as each object encountered uses a copy constructor
# (which I expect of numeric types) and doesn't implement
# a shallow __copy__. Alternative is to use type(el)(el):
#list.__init__(self, [type(el)(el) for el in l])
def bar(self):
return 'bar'
self.bar=new.instancemethod(bar, self, self.__class__)
# also causes deepcopy to choke:
#self.bar = self._bar

def _bar(self):
return 'bar'

#deepcopy has no problem with this
bar = _bar

def __repr__(self):
return self.__class__.__name__+'(['\
+','.join(map(str, self))+'])'

# causes deepcopy to throw a TypeError
Foo(Foo('foo'))

[Nick Coghlan]

5ÛHH575-UAZWKVVP-7H2H48V3 wrote:

class Foo(list):
"Foo"

def __init__(self, l=[]):

Change this too:
def __init__(self, l=None):
if l is None: l = []
And see if your problem goes away.

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

[Kanenas]

On Thu, 10 Feb 2005 23:50:09 +1000, Nick Coghlan
<nc******@iinet.net.au> wrote:

def __init__(self, l=[]):

Change this too:
def __init__(self, l=None):
if l is None: l = []

Same error. The only ways of not getting the TypeError I've found are
not to call deepcopy or not assign an instancemethod to an instance
attribute (assigning an instancemethod to a class attribute is fine).

[Kanenas]

On Thu, 10 Feb 2005 00:54:04 -0800, Kanenas <kanenas @t comcast d.t
net> wrote:

When an instance has a dynamically assigned instance method, deepcopy
throws a TypeError with the message "TypeError: instancemethod
expected at least 2 arguments, got 0".

I forgot to mention that the TypeError is thrown only when
constructing an instance from another instance in the same class, e.g.
Foo('bar') is fine but Foo(Foo('bar')) will fail.

[Nick Coghlan]

Kanenas wrote:

On Thu, 10 Feb 2005 00:54:04 -0800, Kanenas <kanenas @t comcast d.t
net> wrote:

When an instance has a dynamically assigned instance method, deepcopy
throws a TypeError with the message "TypeError: instancemethod
expected at least 2 arguments, got 0".

I forgot to mention that the TypeError is thrown only when
constructing an instance from another instance in the same class, e.g.
Foo('bar') is fine but Foo(Foo('bar')) will fail.

Interesting. The problem appears to be that bound methods are not copyable:

x = Foo('bar')
x Foo([b,a,r]) x.bar <bound method Foo.bar of Foo([b,a,r])> copy(x.bar) Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "C:\Python24\lib\copy.py", line 95, in copy
return _reconstruct(x, rv, 0)
File "C:\Python24\lib\copy.py", line 320, in _reconstruct
y = callable(*args)
File "C:\Python24\lib\copy_reg.py", line 92, in __newobj__
return cls.__new__(cls, *args)
TypeError: instancemethod expected at least 2 arguments, got 0 x.__repr__ <bound method Foo.__repr__ of Foo([b,a,r])> copy(x.__repr__)

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "C:\Python24\lib\copy.py", line 95, in copy
return _reconstruct(x, rv, 0)
File "C:\Python24\lib\copy.py", line 320, in _reconstruct
y = callable(*args)
File "C:\Python24\lib\copy_reg.py", line 92, in __newobj__
return cls.__new__(cls, *args)
TypeError: instancemethod expected at least 2 arguments, got 0

Normally, the methods live in the class dictionary, so they don't cause a
problem with copying the instance.

It turns out this exception actually makes sense, since you *don't* want to copy
these atributes to the new instance. If you actually copied them, they'd be
bound to the *old* instance, rather than the new one.

So I expect you'll need to provide a __deepcopy__ in order to correctly generate
the instancemethods bound to the new instance.

I also realised that the reason the use of a mutable default is OK here is
because you're deepcopying it.

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

[@t comcast d.t net]

On Fri, 11 Feb 2005 21:22:35 +1000, Nick Coghlan <nc******@iinet.net.au>
wrote:

Interesting. The problem appears to be that bound methods are not copyable: Curiosity got the better of me and I started diggin about in copy.py.
Turns out that return values of reductors for functions don't include a
code object or global dict, and reductors for instancemethods don't
return a function or instance, hence the complaint that instancemethod
(or function, as below) didn't get enough arguments.

f = Foo('foo')
fbrc,fbargs = f.bar.__reduce_ex__(2)[:2]
fbargs (<type 'instancemethod'>,) fbrc(*fbargs) Traceback (most recent call last):
File "<pyshell#2>", line 1, in -toplevel-
fbrc(*fbargs)
File "C:\DEVEL\PYTHON\2.4\lib\copy_reg.py", line 92, in __newobj__
return cls.__new__(cls, *args)
TypeError: instancemethod expected at least 2 arguments, got 0 fbrc(fbargs[0], f.bar, f, type(f)) <bound method Foo.bar of Foo([f,o,o])> def bar(a, b): return (a,b) info = bar.__reduce_ex__(2)
rc,args = info[:2]
args (<type 'function'>,) rc(*args) Traceback (most recent call last):
File "<pyshell#9>", line 1, in -toplevel-
rc(*args)
File "C:\DEVEL\PYTHON\2.4\lib\copy_reg.py", line 92, in __newobj__
return cls.__new__(cls, *args)
TypeError: function() takes at least 2 arguments (0 given) baz=rc(args[0], bar.func_code, bar.func_globals)
baz(1,2)

(1,2)

Note 'args' is not something like :
(<type 'function'>, <function bar at 0xXXXXXXXX>, { ... })

As an aside, what is the tuple returned by a reductor called? What are
its components called?
Normally, the methods live in the class dictionary, so they don't cause a
problem with copying the instance.
But a dynamically assigned instance method lives in the intance
dictionary, making deepcopy choke when it deepcopies the instance state.
That makes sense now.
It turns out this exception actually makes sense, since you *don't* want to copythese atributes to the new instance. If you actually copied them, they'd bebound to the *old* instance, rather than the new one.
True. It wouldn't cause a problem within my __init__, since the
attribute is reassigned after the deepcopy, though should anyone else
deepcopy an instance... Definitely better that the deepcopy throws the
TypeError. But why shouldn't we be able to copy a non-method function?
So I expect you'll need to provide a __deepcopy__ in order to correctly generatethe instancemethods bound to the new instance.

That's what I've decided, too.

[Nick Coghlan]

"@t comcast d.t net <"@bag.python.org wrote:

As an aside, what is the tuple returned by a reductor called? What are
its components called?
Ya got me. Alex might have a name for it :)

>Normally, the methods live in the class dictionary, so they don't cause a
>problem with copying the instance.
>

But a dynamically assigned instance method lives in the intance
dictionary, making deepcopy choke when it deepcopies the instance state.
That makes sense now.

Yup.
True. It wouldn't cause a problem within my __init__, since the
attribute is reassigned after the deepcopy, though should anyone else
deepcopy an instance... Definitely better that the deepcopy throws the
TypeError. But why shouldn't we be able to copy a non-method function?

I honestly don't know, although I'm hard-pressed to see how doing so would ever
be *useful*. Duplicating *parts* of a function would seem to make sense (to
build a new function which is similar, but not identical), but duplicating the
whole thing seems rather pointless. Although I guess you could just pickle it
and then unpickle the result to make a copy :)

Again, Alex Martelli or someone else more familiar with the guts of copy.py than
I am might be able to give a better answer if they happen to see the question.

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

[Kanenas]

On Sun, 13 Feb 2005 12:01:42 +1000, Nick Coghlan
<nc******@iinet.net.au> wrote:

True. It wouldn't cause a problem within my __init__, since the
attribute is reassigned after the deepcopy, though should anyone else
deepcopy an instance... Definitely better that the deepcopy throws the
TypeError. But why shouldn't we be able to copy a non-method function?

I honestly don't know, although I'm hard-pressed to see how doing so would ever
be *useful*. Duplicating *parts* of a function would seem to make sense (to
build a new function which is similar, but not identical), but duplicating the
whole thing seems rather pointless. Although I guess you could just pickle it
and then unpickle the result to make a copy :)

It's not so much that copying a function is useful as it would be nice
if copy and deepcopy didn't fail on plain functions, as there are
cases where objects would need to store references to such functions.
Imagine classes implementing operations on functions such as numeric
differentiation and integration (while symbolic differentiation &
integration would probably be better for production code, the example
still stands). As it is, a little extra work would be needed to
support deepcopying such classes. And it's always nice to avoid work.

Thanks for your help and feedback on this.