Problem using copy.copy with my own class

[Jeffrey Barish]

(Pdb) myclass
MyClass( 0, 0, 'A string', 123.45)
(Pdb) copy.copy(myclass)
*** TypeError: TypeError('__new__() takes at least 4 arguments (2 given)',)

I see 4 arguments (actually, 5 because Python is passing cls invisibly to
__new__). Does anyone have an idea what is going on here?

I have not defined __new__ in MyClass above. I can make the problem go away
on one platform by defining __new__ as

return MyClass(self, self.arg1, self.arg2, self.arg3)

but I understand that return value to be the default, so I don't see why
defining that method makes a difference. On another platform, that
definition causes a different problem (I seem to be getting None as the
return value of the copy in some cases).

By the way, I have simplified somewhat the code in the explanation. In case
it might matter, know that there are actually two classes that exhibit this
problem. In one, there is actually a fourth argument in the __new__ method
that has a default value (the float above), which is why the error message
says that __new__ expects *at least* 4 arguments. In the other, the last
argument is a parg collector.

I have also had trouble pickling these classes. I surmise that pickle uses
copy.
--
Jeffrey Barish

[Marc 'BlackJack' Rintsch]

On Tue, 22 Apr 2008 11:13:43 -0600, Jeffrey Barish wrote:

By the way, I have simplified somewhat the code in the explanation.

Please simplify the code to a minimal example that still has the problem
and *show it to us*. It's hard to spot errors in code that nobody except
you knows.

[Jeffrey Barish]

Marc 'BlackJack' Rintsch wrote:

Please simplify the code to a minimal example that still has the problem
and *show it to us*. It's hard to spot errors in code that nobody except
you knows.

Here it is:

import copy

class Test(int):
def __new__(cls, arg1, arg2):
return int.__new__(cls, arg1)

def __init__(self, arg1, arg2):
self.arg2 = arg2

if __name__ == '__main__':
t = Test(0, 0)
t_copy = copy.copy(t)

Traceback (most recent call last):
File "copytest.py", line 12, in <module>
t_copy = copy.copy(t)
File "/usr/lib/python2.5/copy.py", line 95, in copy
return _reconstruct(x, rv, 0)
File "/usr/lib/python2.5/copy.py", line 322, in _reconstruct
y = callable(*args)
File "/usr/lib/python2.5/copy_reg.py", line 92, in __newobj__
return cls.__new__(cls, *args)
TypeError: __new__() takes exactly 3 arguments (2 given)
--
Jeffrey Barish

[Michael Torrie]

Jeffrey Barish wrote:

Marc 'BlackJack' Rintsch wrote:

>Please simplify the code to a minimal example that still has the problem
and *show it to us*. It's hard to spot errors in code that nobody except
you knows.

Here it is:

import copy

class Test(int):
def __new__(cls, arg1, arg2):

^^^^^^^
The exception is saying that copy.copy is not providing this 3rd
argument. I might be a bit dense, but what would arg2 be for? Isn't
arg1 supposed to be the object instance you are copying from? If so,
arg2 is superfluous.

[Gabriel Genellina]

En Thu, 24 Apr 2008 00:32:37 -0300, Michael Torrie <to*****@gmail.com>
escribió:

Jeffrey Barish wrote:

>Marc 'BlackJack' Rintsch wrote:

>>Please simplify the code to a minimal example that still has the
problem
and *show it to us*. It's hard to spot errors in code that nobody
except
you knows.

Here it is:

import copy

class Test(int):
def __new__(cls, arg1, arg2):

^^^^^^^
The exception is saying that copy.copy is not providing this 3rd
argument. I might be a bit dense, but what would arg2 be for? Isn't
arg1 supposed to be the object instance you are copying from? If so,
arg2 is superfluous.

I agree. In case arg2 were really meaningful, use __getnewargs__ (see )

import copy

class Test(int):
def __new__(cls, arg1, arg2):
return int.__new__(cls, arg1)
def __init__(self, arg1, arg2):
self.arg2 = arg2
def __getnewargs__(self):
return int(self), self.arg2

pyt = Test(3, 4)
pyprint type(t), t, t.arg2
<class '__main__.Test'3 4
pyt_copy = copy.copy(t)
pyprint type(t_copy), t_copy, t_copy.arg2
<class '__main__.Test'3 4

But note that subclassing int (and many other builtin types) doesn't work
as expected unless you redefine most operations:

pyx = t + t_copy
pyprint type(x), x
<type 'int'6
pyt += t_copy
pyprint type(t), t
<type 'int'6

--
Gabriel Genellina

[Gabriel Genellina]

En Thu, 24 Apr 2008 01:21:15 -0300, Gabriel Genellina
<ga*******@yahoo.com.arescribió:

I agree. In case arg2 were really meaningful, use __getnewargs__ (see )

I forget to include the link: <http://docs.python.org/lib/pickle-inst.html>

--
Gabriel Genellina

[George Sakkis]

On Apr 23, 9:48*pm, Jeffrey Barish <[e-mail]:

>
Here it is:

import copy

class Test(int):
* * def __new__(cls, arg1, arg2):
* * * * return int.__new__(cls, arg1)

* * def __init__(self, arg1, arg2):
* * * * self.arg2 = arg2

if __name__ == '__main__':
* * t = Test(0, 0)
* * t_copy = copy.copy(t)

First off, inheriting from a basic builtin type such as int and
changing its constructor's signature is not typical; you should
rethink your design unless you know what you're doing.

One way to make this work is to define the special __copy__ method
[1], specifying explicitly how to create a copy of a Test instance:

class Test(int):
...
def __copy__(self):
return Test(int(self), self.arg2)

The copy.copy() function looks for this special method and invokes it
if it's defined. Normally (i.e. for pure Python classes that don't
subclass a builtin other than object) copy.copy() is smart enough to
know how to create a copy without an explicit __copy__ method, so in
general you don't have to define it for every class that has to be
copyable.

Traceback (most recent call last):
* File "copytest.py", line 12, in <module>
* * t_copy = copy.copy(t)
* File "/usr/lib/python2.5/copy.py", line 95, in copy
* * return _reconstruct(x, rv, 0)
* File "/usr/lib/python2.5/copy.py", line 322, in _reconstruct
* * y = callable(*args)
* File "/usr/lib/python2.5/copy_reg.py", line 92, in __newobj__
* * return cls.__new__(cls, *args)
TypeError: __new__() takes exactly 3 arguments (2 given)

The traceback is not obvious indeed. It turns out it involves calling
the arcane __reduce_ex__ special method [2] defined for int, which
returns a tuple of 5 items; the second is the tuple
(<class '__main__.Test'>, 0) and these are the arguments passed to
Test.__new__. So another way of fixing it is keep Test.__new__
compatible with int.__new__ by making optional all arguments after the
first:

class Test(int):
def __new__(cls, arg1, arg2=None):
return int.__new__(cls, arg1)

# don't need to define __copy__ now

from copy import copy
t = Test(0, 0)
assert copy(t) == t

As a sidenote, your class works fine without changing anything when
pickling/unpickling instead of copying, although pickle calls
__reduce_ex__ too:

from pickle import dumps,loads
t = Test(0, 0)
assert loads(dumps(t)) == t

Perhaps someone more knowledgeable can explain the subtle differences
between pickling and copying here.

George

[1] http://docs.python.org/lib/module-copy.html
[2] http://docs.python.org/lib/node320.html

[Jeffrey Barish]

George Sakkis wrote:

First off, inheriting from a basic builtin type such as int and
changing its constructor's signature is not typical; you should
rethink your design unless you know what you're doing.

Nah, I would never claim to know what I'm doing. However, I have to say
that I have been finding this technique very useful. When I started
developing this program, I used an int. Then I discovered that I needed to
have a string associated with the int. By subclassing int to add a string,
I managed to make the change transparent to the code I had already written.
Only the new code that needed the associated string knew that it was
available. In another case, I subclassed str so that I could have a long
form for a string (e.g., a full name attached to the surname). Are these
applications of subclassing bad form? What is the motivation for your
warning?

One way to make this work is to define the special __copy__ method
[1], specifying explicitly how to create a copy of a Test instance:

Normally (i.e. for pure Python classes that don't
subclass a builtin other than object) copy.copy() is smart enough to
know how to create a copy without an explicit __copy__ method, so in
general you don't have to define it for every class that has to be
copyable.

Yes, I noted in my original posting (which seems to have fallen off this
thread) that the __copy__method solved the problem (at least on one
platform). However, I was wondering why it was necessary when what I was
defining was supposedly the default action. Thanks for your explanation.

The traceback is not obvious indeed. It turns out it involves calling
the arcane __reduce_ex__ special method [2] defined for int, which
returns a tuple of 5 items; the second is the tuple
(<class '__main__.Test'>, 0) and these are the arguments passed to
Test.__new__. So another way of fixing it is keep Test.__new__
compatible with int.__new__ by making optional all arguments after the
first:

This suggestion is very interesting. It seems to be an alternative to the
solution suggested by Gabriel. The reference that Gabriel provided
includes the statement:

Instances of a new-style type C are created using

obj = C.__new__(C, *args)

where args is the result of calling __getnewargs__() on the original
object; if there is no __getnewargs__(), an empty tuple is assumed.

Gabriel's solution using __getnewargs__ assures that args receives a
non-empty tuple. Your solution renders the empty tuple impotent by
specifying default values. Correct me if I am wrong.

I would be interested in a translation into English of the following
statement from the same reference that Gabriel provided:

Implementing this method [i.e., __getnewargs__] is needed if the
type establishes some internal invariants when the instance is
created, or if the memory allocation is affected by the values
passed to the __new__() method for the type (as it is for tuples
and strings).

What is an "internal invariant"? How do I know when the memory allocation
is affected? Does my Test class affect the memory allocation?

As a sidenote, your class works fine without changing anything when
pickling/unpickling instead of copying, although pickle calls
__reduce_ex__ too:

from pickle import dumps,loads
t = Test(0, 0)
assert loads(dumps(t)) == t

Perhaps someone more knowledgeable can explain the subtle differences
between pickling and copying here.

I have a situation in the full program where pickling seems to be failing in
the same manner as copy, but I have not been able yet to reproduce the
problem in a simple test program.

Thanks to all for your comments.
--
Jeffrey Barish