modifying __new__ of list subclass

[Ken Schutte]

Hi,

I'm been trying to create some custom classes derived from some of
python's built-in types, like int and list, etc. I've run into some
trouble, which I could explain with a couple simple examples. Lets say
I want an int-derived class that is initilized to one greater than what
it's constructor is given:

class myint(int):
def __new__(cls, intIn):
newint = int(intIn+1)
return int.__new__(cls, newint)

print myint(3), myint(10)
Okay, seems to do what I want. Now, lets say I want a list class that
creates a list of strings, but appends "_" to each element. I try the
same thing:
class mylist(list):
def __new__(cls, listIn):
newlist = list()
for i in listIn:
newlist.append(str(i) + "_")
print "newlist: ", newlist
return list.__new__(cls, newlist)

print mylist(("a","b","c"))

Doesn't seem to work, but that print statement shows that the newlist is
what I want... Maybe what I return from __new__ is overwritten in
__init__? Could someone enlighten me as to why - and why this is
different than the int case?

Thanks,
Ken

[Steven Bethard]

Ken Schutte wrote:

I want an int-derived class that is initilized to one greater than what
it's constructor is given:

class myint(int):
def __new__(cls, intIn):
newint = int(intIn+1)
return int.__new__(cls, newint)

Or simply:

class myint(int):
def __new__(cls, int_in):
return int.__new__(cls, int_in + 1)

Now, lets say I want a list class that
creates a list of strings, but appends "_" to each element. I try the
same thing:

class mylist(list):
def __new__(cls, listIn):
newlist = list()
for i in listIn:
newlist.append(str(i) + "_")
print "newlist: ", newlist
return list.__new__(cls, newlist)

The __new__ method is for immutable types. So things like str and int
do their initialization in __new__. But for regular mutable types, you
should do your initialization in __init__::

class mylist(list):
def __init__(self, list_in):
for item in list_in:
self.append(str(item) + '_')

STeve

[Ken Schutte]

Steven Bethard wrote:

>
The __new__ method is for immutable types. So things like str and int
do their initialization in __new__. But for regular mutable types, you
should do your initialization in __init__::

I see... So, is there a use for __new__ in mutable types? From my
list-derirved class, it was obviously being called, but it's return
value is totally ignored?

Thanks for the reply.

[Alex Martelli]

Ken Schutte <ks******@csail.mit.eduwrote:

Steven Bethard wrote:

The __new__ method is for immutable types. So things like str and int
do their initialization in __new__. But for regular mutable types, you
should do your initialization in __init__::

I see... So, is there a use for __new__ in mutable types? From my
list-derirved class, it was obviously being called, but it's return
value is totally ignored?

Wrong: the return value of __new__ is most definitely NOT "totally
ignored", since it's what gets passed as the first argument of __init__
(as long as it's an instance of the type in question). Easy to check
for yourself, e.g.:

>>class ha(list):

.... def __new__(cls, *a):
.... x = list.__new__(cls, *a)
.... x.foo = 23
.... return x
....

>>z = ha()
z.foo

23

>>>

as you can see, the "totally ignored" hypothesis is easily disproved.

Of course, there's no particular reason why class ha would _want_ to set
the .foo attribute in __new__ rather than __init__, so that doesn't yet
answer your other question about "is there a use". That answer is a
resounding "yes", but the uses may be subtler than you're considering:
for example, you may use the subtype as a general-purpose "factory", so
that instantiating the subtype may return objects that are not in fact
instances of the subtype (that bypasses the __init__ call); or, the
overriding of __new__ may go together with the overriding of __init__
(so that the latter doesn't blast the object's state) for such purposes
as singletons or more generally types with a finite "pool" of instances.
Alex

[Steven Bethard]

Ken Schutte wrote:

Steven Bethard wrote:

>>
The __new__ method is for immutable types. So things like str and int
do their initialization in __new__. But for regular mutable types,
you should do your initialization in __init__::

I see... So, is there a use for __new__ in mutable types? From my
list-derirved class, it was obviously being called, but it's return
value is totally ignored?

Not ignored, it's just having it's __init__ method called after your
__new__ method.

It might help for a moment to consider what happens when you call a
class object, e.g.::

c = C()

Just like any other object, when Python sees the ``()``, it looks for a
__call__ method on the object. Now classes are instances of the
``type`` type, which has a call method that looks something like::

def __call__(cls, *args, **kwargs):
result = cls.__new__(cls, *args, **kwargs)
if isinstance(result, cls):
result.__init__(*args, **kwargs)
return result

What's happening in your list case is that list.__init__ clears the list::

>>l = [1, 2, 3]
l.__init__()
l

[]

So even though your __new__ method returns the object you want, the
__init__ method is clearing out all the items you've added and then
re-adding them as it normally would. To prove this to yourself, take a
look at what happens when we override __init__::

>>class mylist(list):

... def __new__(cls, items):
... result = super(mylist, cls).__new__(cls)
... for item in items:
... result.append('%s_' % item)
... return result
...

>>mylist([1, 2, 3])

[1, 2, 3]

>>class mylist(list):

... def __new__(cls, items):
... result = super(mylist, cls).__new__(cls)
... for item in items:
... result.append('%s_' % item)
... return result
... def __init__(self, items):
... pass
...

>>mylist([1, 2, 3])

['1_', '2_', '3_']

Of course, I've made __new__ work above, but the simpler solution is
just to override __init__ since that's where all the work's being done
anyway.

See Alex Martelli's response to answer your question "So, is there a use
for __new__ in mutable types?". You'd probably only want to override
__new__ if you were going to use the class as a factory to produce a
bunch of different types of objects.

STeVe

[Ken Schutte]

Steven Bethard wrote:

So even though your __new__ method returns the object you want, the
__init__ method is clearing out all the items you've added and then
re-adding them as it normally would. To prove this to yourself, take a
look at what happens when we override __init__::

Okay, I see what's happening now. Steve and Alex - thanks for the great
explanations.

Ken