Why doesn't __call__ lead to infinite recursion?

[Patrick Lioi]

def foo(): pass

foo is a function
foo is a callable object
foo has method __call__ defined

foo.__call__ is a function
foo.__call__ is a callable object
foo.__call__ has method __call__ defined

foo.__call__.__call__ is a function...
This seems to go on forever. How does calling foo() not lead to
an infinite loop while trying to execute?

[Aahz]

In article <bh**********@absinth.dialog.net.pl>,
bromden <br*****@gazeta.pl.no.spam> wrote:

foo is a function
foo is a callable object
foo has method __call__ defined

true
true
false

Wrong. See my followup.
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

This is Python. We don't care much about theory, except where it intersects
with useful practice. --Aahz

[Aahz]

In article <2a*************************@posting.google.com> ,
Patrick Lioi <pa*****@novaroot.com> wrote:

def foo(): pass

foo is a function
foo is a callable object
foo has method __call__ defined

foo.__call__ is a function
foo.__call__ is a callable object
foo.__call__ has method __call__ defined

You're mixing up the distinction between objects and types. If you do

print foo.__dict__

you'll see that __call__ isn't there. However, if you try

print type(foo).__dict__

you'll see __call__ there. When you do foo(), Python actually does
type(foo).__call__(foo). Because type(foo).__call__ is manipulating
foo, you don't get the circular reference.
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

This is Python. We don't care much about theory, except where it intersects
with useful practice. --Aahz

[John J. Lee]

aa**@pythoncraft.com (Aahz) writes:
[...]

you'll see __call__ there. When you do foo(), Python actually does
type(foo).__call__(foo). Because type(foo).__call__ is manipulating
foo, you don't get the circular reference.

Still seems weird that type(foo).__call__.__call__ (etc.) is defined.
John

[Andrew Dalke]

Aahz:

you'll see __call__ there. When you do foo(), Python actually does
type(foo).__call__(foo). Because type(foo).__call__ is manipulating
foo, you don't get the circular reference.

Not quite, but I don't understand how everything works so what I
say may also need corrections.

The call syntax "foo()" does two things. The first is to
get the 'thing' used for the call and the second is to actually
call it. The latter is not done recursively - if the returned
thing can't be called, the attempt at making the call fails.

If 'foo' is an instance, then the implementation code is
something like

thing_to_call = getattr(foo, "__call__")
if thing_to_call is not None:
DO_CALL(thing_to_call, args, kwargs)

The 'DO_CALL' is not a Python function, it's part of
how the implementation works.

The getattr implementation for an instance first tries to
find "__call__" in foo's instance __dict__. If that fails, it
looks in the parent class, and returns a bound method,
that is, a new 'thing' with references to the class method
and to the instance itself. The DO_CALL does the
actual call to this thing with the new arguments. The
bound method thing prepends the self parameter and
does the actual call to the underlying code.

Pretty complicated, and I don't think I was very clear
on that. Here's an example though to show that

foo() != foo.__class__.__call__(foo)

class SPAM: .... def __init__(self):
.... def callme(x):
.... print "Hello", x
.... self.__call__ = callme
.... def __call__(self, x):
.... print "Hej", x
.... spam = SPAM()
spam("world") Hello world
spam.__class__.__call__(spam, "world") Hej world getattr(spam, "__call__") <function callme at 0x014DA9B0> getattr(spam, "__init__") <bound method SPAM.__init__ of <__main__.SPAM instance at 0x013CF148>> getattr(SPAM, "__call__") <unbound method SPAM.__call__>
I'm also missing something because I don't know how
functions work. I thought it was always 'use
getattr(obj, "__call__") to get the thing to call then do
the call machinery on that thing", but it doesn't seem
to do that for functions.
def f(x): .... print "f(%s)" % x
.... def g(x): .... print "g(%s)" % x
.... f(5) f(5) f(7) f(7) f.__call__ = g.__call__
f(4) f(4) f.__call__ <method-wrapper object at 0x014AD6F0> f.__call__ <method-wrapper object at 0x014AD6F0> g.__call__ <method-wrapper object at 0x014A02D0> g.__call__ <method-wrapper object at 0x014AD5B0>
I expected the 'f(4)' call to return 'g(4)' since I replaced
the function's __call__ with g's __call__.

What's throwing me off is that g.__call__ returns
a new wrapper object each time, while once I
assigned f.__call__, it persistently stayed that way.
So there's some getattr shenanigans with functions
I don't understand. To make it worse, there's also
import types
types.FunctionType.__call__ <slot wrapper '__call__' of 'function' objects>

I'll leave the clarification to someone else.

Andrew
da***@dalkescientific.com

[Andrew Dalke]

Aahz:

No time to investigate further, but all your examples used classic
classes instead of new-style classes; I'm pretty sure that new-style
classes will more closely emulate the way functions work. There's also
the wrinkle I didn't mention that functions use a dict proxy IIRC.

Interesting. Very interesting.

class XYZ(object): .... def __init__(self):
.... def abc(x):
.... print "Hello", x
.... self.__call__ = abc
.... def __call__(self, x):
.... print "Yo", x
.... xyz = XYZ()
xyz("fred") Yo fred
getattr(xyz, "__call__") <function abc at 0x0168CB70>

I wonder if this will affect any of my code.

It does explain the observed differences better, since FunctionType
in 2.3 is derived from object while my class was not.

Andrew
da***@dalkescientific.com

[Michael Hudson]

"Andrew Dalke" <ad****@mindspring.com> writes:

Aahz:

No time to investigate further, but all your examples used classic
classes instead of new-style classes; I'm pretty sure that new-style
classes will more closely emulate the way functions work. There's also
the wrinkle I didn't mention that functions use a dict proxy IIRC.

Interesting. Very interesting.

Yes :-)

You have to have something like this when you do things like 'print
type(foo)'. This should call the *types* *bound* __str__ method, not
try to call the *instances* *unbound* __str__ method...

Cheers,
mwh

--
Its unmanageable complexity has spawned more fear-preventing tools
than any other language, but the solution _should_ have been to
create and use a language that does not overload the whole goddamn
human brain with irrelevant details. -- Erik Naggum, comp.lang.lisp

[Beni Cherniavsky]

In article <87************@pobox.com>, John J. Lee wrote:

aa**@pythoncraft.com (Aahz) writes:
[...]

you'll see __call__ there. When you do foo(), Python actually does
type(foo).__call__(foo). Because type(foo).__call__ is manipulating
foo, you don't get the circular reference.

Still seems weird that type(foo).__call__.__call__ (etc.) is defined.

[I'm afraid I sent this 2 or 3 already in private by mistake; resending in
public. I apologize, it's late...]

The point is that down there, sits the C level which doesn't go
through the Python definition. When you call ``foo()``, ``type(foo)``
is asked at the *C* level how to call it (read: the type struct is
accessed and the call-behavior slot, if non-NULL, is called by plain
C function call - which can't be intercepted in C so there is no futher
recursion).

It so happens that classes, when asked this, go back into the Python
level and look for the `__call__` attribute. They do this for all
operations, giving you the dynamism and flexibility we all love.

OTOH, functions, when asked this, simply execute their code objects with the
given arguments. The attribute `__call__` on function is a "proxy" attribute.
It is not used by function objects for the call, it only exposes the C-level
call machinery at Python level. The same convention is used for other
built-in types and operations, so that from Python you see Python-level and
C-level methods in the same way. You only have to be aware of the distinction
when working with bizzare extension types that don't respect this convention,
or asking questions like this one about how it all works... (Another one: why
__getattribute__ is not infinitely recusive? Same explanation.)

--
Beni Cherniavsky <cb**@tx.technion.ac.il>

Look, Mom, no viruses! [Hint: I use GNU/Linux]