super, decorators and gettattribute

[Richard Szopa]

Hello all,

I am playing around w/ Python's object system and decorators and I
decided to write (as an exercise) a decorator that (if applied to a
method) would call the superclass' method of the same name before
doing anything (initially I wanted to do something like CLOS
[1] :before and :end methods, but that turned out to be too
difficult).

However, I cannot get it right (specially, get rid of the eval). I
suspect that I may be misunderstanding something happening between
super objects and __getattribute__ methods.

Here's my code:

def endmethod(fun):
"""Decorator to call a superclass' fun first.

If the classes child and parent are defined as below, it should
work like:

>>x = child()
x.foo()

I am parent's foo
I am child's foo.
"""
name = fun.__name__
def decorated(self, *args, **kwargs):
try:
super_object = super(self.__class__, self)

# now I want to achieve something equivalent to calling
# parent.foo(*args, **kwargs)
# if I wanted to limit it only to this example

# this doesn't work: in the example, it calls child's foo,
# entering in an eternal loop (instead of calling parent's
# foo, as I would expect).

# super_object.__getattribute__(name)(*args, **kwargs)

# this does work, but I feel it's ugly
eval('super_object.%s(*args, **kwargs)' % name)
except AttributeError:
pass # if parent doesn't implement fun, we don't care
# about it
return fun(self, *args, **kwargs) # hopefully none

decorated.__name__ = name
return decorated
class parent(object):
def foo(self):
print 'I am parent\'s foo'

class child(parent):
@endmethod
def foo(self):
print "I am foo\'s foo."

if __name__=='__main__':
x = child()
x.foo()

Can anybody tell me how to call a superclass method knowing its name?

Thanks in advance,
-- Richard

[1] http://en.wikipedia.org/wiki/Common_Lisp_Object_System

[Richard Szopa]

On Jan 12, 7:45 pm, Richard Szopa <[e-mail]:

doing anything (initially I wanted to do something like CLOS
[1] :before and :end methods, but that turned out to be too
difficult).

Erm, I meant :before and :after methods.

-- Richard

[Mike Meyer]

On Sat, 12 Jan 2008 10:45:25 -0800 (PST) Richard Szopa <ry***********@gmail.comwrote:

Hello all,

I am playing around w/ Python's object system and decorators and I
decided to write (as an exercise) a decorator that (if applied to a
method) would call the superclass' method of the same name before
doing anything (initially I wanted to do something like CLOS
[1] :before and :end methods, but that turned out to be too
difficult).

However, I cannot get it right (specially, get rid of the eval). I
suspect that I may be misunderstanding something happening between
super objects and __getattribute__ methods.

Here's my code:

def endmethod(fun):
"""Decorator to call a superclass' fun first.

If the classes child and parent are defined as below, it should
work like:

>>x = child()
>>x.foo()

I am parent's foo
I am child's foo.
"""
name = fun.__name__
def decorated(self, *args, **kwargs):
try:
super_object = super(self.__class__, self)

There's an apparently common bug here: you don't want to pass super
self.__class__, but the class that the method is bound to. The two
aren't the same, as an instance of a subclass will have the subclass
as self.__class__, and not the current class. So super will return the
current class or a subclass of it, meaning (since you invoked this
method from self) you'll wind up invoking this method recursively.
All of which means your decorator is probably going to have to take
the class as an argument.

# now I want to achieve something equivalent to calling
# parent.foo(*args, **kwargs)
# if I wanted to limit it only to this example

# this doesn't work: in the example, it calls child's foo,
# entering in an eternal loop (instead of calling parent's
# foo, as I would expect).

# super_object.__getattribute__(name)(*args, **kwargs)

# this does work, but I feel it's ugly
eval('super_object.%s(*args, **kwargs)' % name)
except AttributeError:
pass # if parent doesn't implement fun, we don't care
# about it
return fun(self, *args, **kwargs) # hopefully none

decorated.__name__ = name
return decorated
class parent(object):
def foo(self):
print 'I am parent\'s foo'

class child(parent):
@endmethod
def foo(self):
print "I am foo\'s foo."

if __name__=='__main__':
x = child()
x.foo()

Can anybody tell me how to call a superclass method knowing its name?

The same way you call any object's methods if you know it's name":

getattr(super_object, name)(*args, **kwargs)

The code seems to work the way you want:

>>x.foo()

I am parent's foo
I am foo's foo.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.

[Richard Szopa]

On Jan 12, 9:47 pm, Mike Meyer <[e-mail]>
wrote:

The same way you call any object's methods if you know it's name":

getattr(super_object, name)(*args, **kwargs)

Thanks a lot for your answer!

However, I am very surprised to learn that

super_object.__getattr__(name)(*args, **kwargs)

getattr(super_object, name)(*args, **kwargs)

are not equivalent. This is quite odd, at least when with len()
and .__len__, str() and .__str__. Do you maybe know what's the
rationale behind not following that convention by getattr?

Best regards,

-- Richard

[Steven D'Aprano]

On Sat, 12 Jan 2008 15:47:05 -0500, Mike Meyer wrote:

There's an apparently common bug here: you don't want to pass super
self.__class__, but the class that the method is bound to.

Given an instance method, is it possible to easily determine what class
it is defined in?

I thought the im_class attribute might do it, but it apparently just
points to self.__class__.

>>class Foo(object):

.... def foo(self):
.... pass
....

>>class Bar(Foo):

.... def bar(self):
.... pass
....

>>Bar().bar.im_class # expecting Bar

<class '__main__.Bar'>

>>Bar().foo.im_class # hoping for Foo

<class '__main__.Bar'>

--
Steven

[Marc 'BlackJack' Rintsch]

On Sat, 12 Jan 2008 14:23:52 -0800, Richard Szopa wrote:

However, I am very surprised to learn that

super_object.__getattr__(name)(*args, **kwargs)

getattr(super_object, name)(*args, **kwargs)

are not equivalent. This is quite odd, at least when with len()
and .__len__, str() and .__str__. Do you maybe know what's the
rationale behind not following that convention by getattr?

I think you are confusing `__getattr__` and `__getattribute__` here!
`getattr()` maps to `__getattr__()`, it's `__getattribute__` that's
different.

Ciao,
Marc 'BlackJack' Rintsch

[Richard Szopa]

On Jan 13, 8:59 am, Marc 'BlackJack' Rintsch <[e-mail]:

On Sat, 12 Jan 2008 14:23:52 -0800, Richard Szopa wrote:

However, I am very surprised to learn that

super_object.__getattr__(name)(*args, **kwargs)

getattr(super_object, name)(*args, **kwargs)

are not equivalent. This is quite odd, at least when with len()
and .__len__, str() and .__str__. Do you maybe know what's the
rationale behind not following that convention by getattr?

I think you are confusing `__getattr__` and `__getattribute__` here!
`getattr()` maps to `__getattr__()`, it's `__getattribute__` that's
different.

Well, in my code calling super_object.__getattr__(name)(*args,
**kwargs) and getattr(super_object, name)(*args, **kwargs) gives
*different* effects (namely, the latter works, while the former
doesn't). That kinda suggests that they don't map to each other :-).
And that makes me feel confused.

Cheers,

-- Richard

[thebjorn]

On Jan 13, 1:51 pm, Richard Szopa <[e-mail]:

On Jan 13, 8:59 am, Marc 'BlackJack' Rintsch <[e-mail]:

On Sat, 12 Jan 2008 14:23:52 -0800, Richard Szopa wrote:

However, I am very surprised to learn that

super_object.__getattr__(name)(*args, **kwargs)

getattr(super_object, name)(*args, **kwargs)

are not equivalent. This is quite odd, at least when with len()
and .__len__, str() and .__str__. Do you maybe know what's the
rationale behind not following that convention by getattr?

I think you are confusing `__getattr__` and `__getattribute__` here!
`getattr()` maps to `__getattr__()`, it's `__getattribute__` that's
different.

Well, in my code calling super_object.__getattr__(name)(*args,
**kwargs) and getattr(super_object, name)(*args, **kwargs) gives
*different* effects (namely, the latter works, while the former
doesn't). That kinda suggests that they don't map to each other :-).
And that makes me feel confused.

Cheers,

-- Richard

They do, except for when it comes to what super(..) returns. It isn't
really an object in the sense that they're presented in the tutorial,
but rather a sort of proxy to the methods in the ancestor classes of
the concrete object (self), relative to the current method's class. I
can't imagine that sentence would ease any confusion however, suffice
it to say that you have to call getattr(super(..), 'name') instead of
super(..).__getattr__('name') and you have to call super(..).__len__()
instead of len(super(..)) -- I can't imagine that lessens any
confusion either :-/

super(..) is designed to handle situations like this correctly

class Root(object):
n = 1

class Left(Root):
def foo(self):
print 'n =', self.n
print 'super n = ', super(Left, self).n

class Right(Root):
n = 2

class Leaf(Left,Right):
n = 3

x = Leaf()
x.foo()

the correct output is

n = 3
super n = 2
-- bjorn

[Richard Szopa]

On Jan 13, 3:31 pm, thebjorn <[e-mail]>
wrote:

They do, except for when it comes to what super(..) returns. It isn't
really an object in the sense that they're presented in the tutorial,
but rather a sort of proxy to the methods in the ancestor classes of
the concrete object (self), relative to the current method's class. I
can't imagine that sentence would ease any confusion however, suffice
it to say that you have to call getattr(super(..), 'name') instead of
super(..).__getattr__('name') and you have to call super(..).__len__()
instead of len(super(..)) -- I can't imagine that lessens any
confusion either :-/

Surprisingly, I think your first sentence *does* make something more
clear. Let me check if I understand it right: when we call a method on
super(Foo, self) it is as if we were calling call-next-method in
Common Lisp or Dylan (i.e. the method of the class on the right of Foo
in self.mro()). This however does not imply for super to have its dict
the same as the class on the right of Foo---it remains the same as
self's dict.

However, there's one piece that doesn't completely fit to the puzzle:
why does getattr work? The help says:

getattr(...)
getattr(object, name[, default]) -value

Get a named attribute from an object; getattr(x, 'y') is
equivalent to x.y.
When a default argument is given, it is returned when the
attribute doesn't
exist; without it, an exception is raised in that case.

Does it work on the basis that "getattr(x, 'y') is equivalent to x.y"?
What is then a "named attribute for an object" in Python? It seems not
to be equivalent to the value of the item whose name is 'y' in the
object's class __dict__...

Cheers,

-- Richard

[Michele Simionato]

On Jan 14, 1:41 pm, Richard Szopa <[e-mail]:

However, there's one piece that doesn't completely fit to the puzzle:
why does getattr work? The help says:

getattr(...)
getattr(object, name[, default]) -value

Get a named attribute from an object; getattr(x, 'y') is
equivalent to x.y.
When a default argument is given, it is returned when the
attribute doesn't
exist; without it, an exception is raised in that case.

Does it work on the basis that "getattr(x, 'y') is equivalent to x.y"?
What is then a "named attribute for an object" in Python? It seems not
to be equivalent to the value of the item whose name is 'y' in the
object's class __dict__...

Cheers,

-- Richard

I really need to publish this one day or another, since these
questions
about super keeps coming out:

http://www.phyast.pitt.edu/~micheles/python/super.html

[George Sakkis]

On Jan 12, 6:56 pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.auwrote:

On Sat, 12 Jan 2008 15:47:05 -0500, Mike Meyer wrote:

There's an apparently common bug here: you don't want to pass super
self.__class__, but the class that the method is bound to.

Given an instance method, is it possible to easily determine what class
it is defined in?

I thought the im_class attribute might do it, but it apparently just
points to self.__class__.

>class Foo(object):

... def foo(self):
... pass
...>>class Bar(Foo):

... def bar(self):
... pass
...>>Bar().bar.im_class # expecting Bar

<class '__main__.Bar'>>>Bar().foo.im_class # hoping for Foo

<class '__main__.Bar'>

Something like that seems to work for most cases:

from inspect import getmro

def getdef(obj,attr):
try: objattrs = obj.__dict__
except AttributeError:
objattrs = obj.__slots__
if attr in objattrs:
return obj
for cls in getmro(obj.__class__):
if attr in cls.__dict__:
return cls

>>getdef(Bar(), 'bar')

<class '__main__.Bar'>

>>getdef(Bar(), 'foo')

<class '__main__.Foo'>
It probably misses some edge cases but I can't think of any off the
top of my head.

George

[thebjorn]

On Jan 14, 1:41 pm, Richard Szopa <[e-mail]:

On Jan 13, 3:31 pm, thebjorn <[e-mail]>
wrote:

They do, except for when it comes to what super(..) returns. It isn't
really an object in the sense that they're presented in the tutorial,
but rather a sort of proxy to the methods in the ancestor classes of
the concrete object (self), relative to the current method's class. I
can't imagine that sentence would ease any confusion however, suffice
it to say that you have to call getattr(super(..), 'name') instead of
super(..).__getattr__('name') and you have to call super(..).__len__()
instead of len(super(..)) -- I can't imagine that lessens any
confusion either :-/

Surprisingly, I think your first sentence *does* make something more
clear. Let me check if I understand it right: when we call a method on
super(Foo, self) it is as if we were calling call-next-method in
Common Lisp or Dylan

I don't remember if CLOS was changed to use C3 Linearization also, but
the concept came from Dylan (http://www.webcom.com/haahr/dylan/
linearization-oopsla96.html) and that's what is implemented in Python.

[...]

However, there's one piece that doesn't completely fit to the puzzle:
why does getattr work? The help says:

getattr(...)
getattr(object, name[, default]) -value

Get a named attribute from an object; getattr(x, 'y') is
equivalent to x.y. When a default argument is given, it
is returned when the attribute doesn't exist; without it,
an exception is raised in that case.

Does it work on the basis that "getattr(x, 'y') is equivalent to x.y"?
What is then a "named attribute for an object" in Python? It seems not
to be equivalent to the value of the item whose name is 'y' in the
object's class __dict__...

Conceptually, x.y is always "get the y attribute of x" and the same as
getattr(x, 'y'). Depending on the type of x and y, and your
familiarity with Python internals, what actually happens during a
lookup might be surprising. In the vast majority of cases however, x.y
is equivalent to one of

x.__dict__['y']
type(x).__dict__['y']

but if you're a language geek like me, you might be excited that in
some cases it is

type(x).__dict__['y'].__get__(x, type(x))

which says you get the value of x.y by calling y and passing x as an
argument -- if you know CLOS you'll recognize that it's a primitive
multi-method call. (there are some other special cases too, although
not as exciting ;-)

Much more detail can be found in Raymond's paper on descriptors
(http://users.rcn.com/python/download/Descriptor.htm) and Michele's
paper on super (http://www.phyast.pitt.edu/~micheles/python/
super.html).

-- bjorn

[Ben Finney]

Michele Simionato <mi***************@gmail.comwrites:

I really need to publish this one day or another, since these
questions about super keeps coming out:

http://www.phyast.pitt.edu/~micheles/python/super.html

Yes, please. Your article has improved my understanding just from
skimming the main headings :-)

--
\ "Buy not what you want, but what you need; what you do not need |
`\ is expensive at a penny." -- Cato, 234-149 BC, Relique |
_o__) |
Ben Finney

[Richard Szopa]

On Jan 14, 11:05 pm, thebjorn <[e-mail]>
wrote:

I don't remember if CLOS was changed to use C3 Linearization also, but
the concept came from Dylan (http://www.webcom.com/haahr/dylan/
linearization-oopsla96.html) and that's what is implemented in Python.

The Common Lisp ANSI standard is from 1994, and the article you cite
is from 1996, which strongly suggests C3 linearization wasn't included
in CLOS. The most important difference between CLOS and C3
linearization AFAIK is that the latter enforces monotonicity, while
the former doesn't.

Of course, it shouldn't be very difficult to implement the C3 behavior
in Common Lisp using the de facto standard MetaObject Protocol.

(Nb. Dylan was such a nice language... It's a pity it is practically
dead right now.)

but if you're a language geek like me, you might be excited that in
some cases it is

type(x).__dict__['y'].__get__(x, type(x))

which says you get the value of x.y by calling y and passing x as an
argument -- if you know CLOS you'll recognize that it's a primitive
multi-method call. (there are some other special cases too, although
not as exciting ;-)

Yeah, I also feel the excitement, so probably I am a language geek
too ;-). However, this is still quite far away from full fledged
multimethods. (OTOH trying to get something more from these primitive
multimethods by abusing __get__ looks kind of tempting ;-))

Regards,

-- Richard

[Basilisk96]

On Jan 14, 7:53 am, Michele Simionato <[e-mail]>
wrote:

I really need to publish this one day or another, since these
questions
about super keeps coming out:

http://www.phyast.pitt.edu/~micheles/python/super.html

Please do. It is a very enlightening discussion, and I'm sure a bunch
of folks will benefit from it. And please update it (if necessary) to
the current Python version. At the time of that writing, 2.3 must have
been King, but oh my, how time flies :-)

Cheers,
-Basilisk96

[Michele Simionato]

On Jan 14, 11:47 pm, Richard Szopa <[e-mail]:

Could you tell me what are the pros and cons of the two approaches
(i.e. writing a decorator function and a decorator descriptor class)?

I prefer to use a class for introspection sake, since
there is no way to get information about an inner function
in a closure, whereas your users can introspect classes
pretty well.

super_object = super(self.__class__, self)

Notice that using super(self.__class__, self) is a common
mistake: the pitfalls with inheritance were discussed
very recently in this same newsgroup, you should be
able to find the post. What you need is

super(class_where_the_method_is_defined, self)

which is in general different from

super(self.__class__, self)

There is no clean way to determine the current class
in Python < 3.0 (Python 3.0 does it automatically);
if you want to see a hackish way involving
bytecode tricks see
http://groups.google.com/group/comp....a2da68961caeb6

(and notice that this is really not recommended).

Michele Simionato

[Rhamphoryncus]

On Jan 13, 5:51 am, Richard Szopa <[e-mail]:

On Jan 13, 8:59 am, Marc 'BlackJack' Rintsch <[e-mail]:

On Sat, 12 Jan 2008 14:23:52 -0800, Richard Szopa wrote:

However, I am very surprised to learn that

super_object.__getattr__(name)(*args, **kwargs)

getattr(super_object, name)(*args, **kwargs)

are not equivalent. This is quite odd, at least when with len()
and .__len__, str() and .__str__. Do you maybe know what's the
rationale behind not following that convention by getattr?

I think you are confusing `__getattr__` and `__getattribute__` here!
`getattr()` maps to `__getattr__()`, it's `__getattribute__` that's
different.

Well, in my code calling super_object.__getattr__(name)(*args,
**kwargs) and getattr(super_object, name)(*args, **kwargs) gives
*different* effects (namely, the latter works, while the former
doesn't). That kinda suggests that they don't map to each other :-).
And that makes me feel confused.

Don't think of them as mappings. Think of them as a way for a class
to hook into getattr's protocol, conveniently named similar to getattr
(__getattr__ and __getattribute__). getattr() may call several
methods, no methods at all, change the arguments, etc. Although len()
may seem simple, many others are not so simple.

--
Adam Olsen, aka Rhamphoryncus

[Helmut Jarausch]

Michele Simionato wrote:

I really need to publish this one day or another, since these
questions
about super keeps coming out:

http://www.phyast.pitt.edu/~micheles/python/super.html

Unfortunately the links [2], [3] and [4] are not given,

Helmut.
--
Helmut Jarausch

Lehrstuhl fuer Numerische Mathematik
RWTH - Aachen University
D 52056 Aachen, Germany

[Richard Szopa]

On Jan 15, 8:06 pm, Helmut Jarausch <[e-mail]:

Unfortunately the links [2], [3] and [4] are not given,

Luckily, there's Google :)

[2] Article about MRO: http://www.python.org/download/releases/2.3/mro/
[3] Descriptor HowTo: http://users.rcn.com/python/download/Descriptor.htm
[4] Articles about metaclasses (second one relevant to super):
* http://www.ibm.com/developerworks/li.../l-pymeta.html,
* http://www-128.ibm.com/developerwork...GX03&S_CMP=ART,
* http://www.ibm.com/developerworks/li...GX03&S_CMP=ART

Of course, it would be very nice if the article was updated to include
the links.

HTH,

-- Richard