__call__ considered harmful or indispensable?

[skip]

I don't personally use __call__ methods in my classes, but I have
encountered it every now and then here at work in code written by other
people. The other day I replaced __call__ with a more obvious method name,
so now instead of executing

obj(foo, bar, baz)

the code in question is

obj.execute(foo, bar, baz)

In this case there was a bug. Depending on inputs, sometimes obj
initialized to a class, sometimes an instance of that class. (I fixed that
too while I was at it.) The problem was that the use of __call__ obscured
the underlying bug by making the instance as well as the class callable.

In this particular case it was clearly unnecessary and just obfuscated the
code. I'm wondering, are there some general cases where __call__ methods of
a user-defined class are simply indispensable?

Thx,

Skip

[Neil Cerutti]

On 2007-08-02, sk**@pobox.com <sk**@pobox.comwrote:

I don't personally use __call__ methods in my classes, but I
have encountered it every now and then here at work in code
written by other people. The other day I replaced __call__
with a more obvious method name, so now instead of executing

obj(foo, bar, baz)

the code in question is

obj.execute(foo, bar, baz)

In this case there was a bug. Depending on inputs, sometimes
obj initialized to a class, sometimes an instance of that
class. (I fixed that too while I was at it.) The problem was
that the use of __call__ obscured the underlying bug by making
the instance as well as the class callable.

In this particular case it was clearly unnecessary and just
obfuscated the code. I'm wondering, are there some general
cases where __call__ methods of a user-defined class are simply
indispensable?

In C++ they are called functors, and they are very useful as
surrogate functions with state. I haven't seen them used in the
same way in Python much, because Python has more immediate
solutions to (most of) these problems.

Here's a (somewhat goofy) example:

class is_consecutive(object):
def __init__(self, seed, compare=operator.lt):
self.last_value = seed
self.compare = compare
def __call__(self, total, value):
if total is False:
return False
lv = self.last_value
self.last_value = value
return self.compare(lv, value):

a = range(15)

>>reduce(is_consecutive(0), a)

True

>>reduce(is_consecutive(0), a + [1,2])

False

It's been a while since I had to be in the STL mindset, so I
couldn't think of a better example.

--
Neil Cerutti
This is not a book to be put down lightly. It should be thrown with great
force. --Dorothy Parker

[Neil Cerutti]

On 2007-08-02, Neil Cerutti <ho*****@yahoo.comwrote:

On 2007-08-02, sk**@pobox.com <sk**@pobox.comwrote:

>I don't personally use __call__ methods in my classes, but I
have encountered it every now and then here at work in code
written by other people. The other day I replaced __call__
with a more obvious method name, so now instead of executing

obj(foo, bar, baz)

the code in question is

obj.execute(foo, bar, baz)

In this case there was a bug. Depending on inputs, sometimes
obj initialized to a class, sometimes an instance of that
class. (I fixed that too while I was at it.) The problem was
that the use of __call__ obscured the underlying bug by making
the instance as well as the class callable.

In this particular case it was clearly unnecessary and just
obfuscated the code. I'm wondering, are there some general
cases where __call__ methods of a user-defined class are simply
indispensable?

In C++ they are called functors, and they are very useful as
surrogate functions with state. I haven't seen them used in the
same way in Python much, because Python has more immediate
solutions to (most of) these problems.

Here's a (somewhat goofy) example:

class is_consecutive(object):
def __init__(self, seed, compare=operator.lt):

That was supposed to be operator.le, not lt. I forgot to repaste after
fixing that bug.

self.last_value = seed
self.compare = compare
def __call__(self, total, value):
if total is False:
return False
lv = self.last_value
self.last_value = value
return self.compare(lv, value):

a = range(15)

>>>reduce(is_consecutive(0), a)

True

>>>reduce(is_consecutive(0), a + [1,2])

False

It's been a while since I had to be in the STL mindset, so I
couldn't think of a better example.

--
Neil Cerutti
The pastor will preach his farewell message, after which the choir will sing,
"Break Forth Into Joy." --Church Bulletin Blooper

[Chris Mellon]

On 8/2/07, Neil Cerutti <ho*****@yahoo.comwrote:

On 2007-08-02, sk**@pobox.com <sk**@pobox.comwrote:

I don't personally use __call__ methods in my classes, but I
have encountered it every now and then here at work in code
written by other people. The other day I replaced __call__
with a more obvious method name, so now instead of executing

obj(foo, bar, baz)

the code in question is

obj.execute(foo, bar, baz)

In this case there was a bug. Depending on inputs, sometimes
obj initialized to a class, sometimes an instance of that
class. (I fixed that too while I was at it.) The problem was
that the use of __call__ obscured the underlying bug by making
the instance as well as the class callable.

In this particular case it was clearly unnecessary and just
obfuscated the code. I'm wondering, are there some general
cases where __call__ methods of a user-defined class are simply
indispensable?

In C++ they are called functors, and they are very useful as
surrogate functions with state. I haven't seen them used in the
same way in Python much, because Python has more immediate
solutions to (most of) these problems.

I don't think I've ever written one (in Python) except to demonstrate
how it worked. Pythons built in functions have all the functionality
that you'd have to use a functor for in C++.

[Diez B. Roggisch]

sk**@pobox.com schrieb:

I don't personally use __call__ methods in my classes, but I have
encountered it every now and then here at work in code written by other
people. The other day I replaced __call__ with a more obvious method name,
so now instead of executing

obj(foo, bar, baz)

the code in question is

obj.execute(foo, bar, baz)

In this case there was a bug. Depending on inputs, sometimes obj
initialized to a class, sometimes an instance of that class. (I fixed that
too while I was at it.) The problem was that the use of __call__ obscured
the underlying bug by making the instance as well as the class callable.

In this particular case it was clearly unnecessary and just obfuscated the
code. I'm wondering, are there some general cases where __call__ methods of
a user-defined class are simply indispensable?

Thanks to closures and bound methods, I don't think that there is a
case where you can't "fake" the missing __call__ at least when it comes
to call-backs. So in the end, you can always work around that. But if
you don't need to...

Se the gnosis multimethods for an implementation that makes sensible use
of the __call__-operator in my opinion.

diez

[Paul Rubin]

sk**@pobox.com writes:

In this particular case it was clearly unnecessary and just obfuscated the
code. I'm wondering, are there some general cases where __call__ methods of
a user-defined class are simply indispensable?

I don't know about "indispensable" but __call__ is convenient sometimes
and I do use it. I've wished that modules supported call, so I could
say

import foo
x = foo(3)

instead of having to say x=foo.foo(3) or something like that.

[Bruno Desthuilliers]

sk**@pobox.com a écrit :

I don't personally use __call__ methods in my classes, but I have
encountered it every now and then here at work in code written by other
people. The other day I replaced __call__ with a more obvious method name,
so now instead of executing

obj(foo, bar, baz)

the code in question is

obj.execute(foo, bar, baz)

In this case there was a bug. Depending on inputs, sometimes obj
initialized to a class, sometimes an instance of that class. (I fixed that
too while I was at it.) The problem was that the use of __call__ obscured
the underlying bug by making the instance as well as the class callable.

I don't quite get the point here. A concrete example would be welcome.

In this particular case it was clearly unnecessary and just obfuscated the
code. I'm wondering, are there some general cases where __call__ methods of
a user-defined class are simply indispensable?

"simply indispensable", I don't know. FWIW, most of the features of
Python are not "simply indispensable" !-)

Most of what you can do with a callable instance can be done with
closures (and is usually done so in FPLs), but given Python's OO nature,
it's sometimes clearer and simpler to use callable instances than
closures. One example that comes to mind is partial application that I
first saw implemented with closures in Django, then as a class in PEP
309's example. Another common use case is parameterized function
decorators, that are often cleaner (IMHO) when implemented as classes
with __call__ than with three levels of nested funcs. In both cases,
closures and callables implementations are functionally equivalent, and
this is transparent for user code, so it's mostly a matter of
readability and personal preferences...

As far as I'm concerned, I do use __call__ "every know and then".
Usually, because I had to pass a callable to some API and needed to
parameterize it (IOW, ad-hoc partial application).

[Bruno Desthuilliers]

Paul Rubin a écrit :

sk**@pobox.com writes:

>>In this particular case it was clearly unnecessary and just obfuscated the
code. I'm wondering, are there some general cases where __call__ methods of
a user-defined class are simply indispensable?

I don't know about "indispensable" but __call__ is convenient sometimes
and I do use it. I've wished that modules supported call, so I could
say

import foo
x = foo(3)

instead of having to say x=foo.foo(3) or something like that.

from foo import foo
x = foo(3)

Or did I miss the point ???

[Paul Rubin]

Bruno Desthuilliers <bd*****************@free.quelquepart.frwrites:

from foo import foo
x = foo(3)

Or did I miss the point ???

The foo module might define a bunch of additional functions that you
still want to be able to access in qualified form. For example it
would somewhat clean up the interface to the python random module if
you could say

import random
x = random() # get a random float between 0 and 1

while still having access to random.shuffle, random.choice, etc.

[infidel]

I find it useful in certain situations. In particular, I have used it
along with cx_Oracle to provide a python module that dynamically
mirrors a package of stored procedures in the database. Basically I
had class StoredProcedure(object) and each instance of this class
represented a particular stored procedure in the database. It just
made sense to use the __call__ method to make these instances
callable. Seemed silly to use some other method to actually call the
stored procedure in the database from such an instance.

[Paul Boddie]

Bruno Desthuilliers wrote:

>
Most of what you can do with a callable instance can be done with
closures (and is usually done so in FPLs), but given Python's OO nature,
it's sometimes clearer and simpler to use callable instances than
closures.

Indeed. I think __call__ has been neglected as closures have become
more widely used. In all cynicism, however, I'd argue that with
Python's very usable implementation of object-orientation, classes
with __call__ methods frequently make for clearer demonstrations of
solutions than closures do, despite requiring slightly more characters
of source code, and in contrast to the apparent fashion for using
closures for showing off more or less everything these days (despite
various pitfalls which may seem counter-intuitive to the beginner).

Paul

[Carl Banks]

On Aug 2, 4:27 pm, Paul Rubin <http://[e-mail]:

[e-mail] writes:

In this particular case it was clearly unnecessary and just obfuscated the
code. I'm wondering, are there some general cases where __call__ methods of
a user-defined class are simply indispensable?

I don't know about "indispensable" but __call__ is convenient sometimes
and I do use it. I've wished that modules supported call, so I could
say

import foo
x = foo(3)

instead of having to say x=foo.foo(3) or something like that.

I used to think that. Or, rather, I often thought that it'd be nice
if a function or class could be imported like a module.

These days I think it'd be enough if modules don't have the same name
as any (public) objects they define. No more glob.glob,
select.select, or time.time. It's ugly and potentially confusing.
And if I were Polynesian I would keep thinking I was seeing plurals.
Carl Banks

[Carl Banks]

On Aug 2, 2:30 pm, [e-mail] wrote:

I'm wondering, are there some general cases where __call__ methods of
a user-defined class are simply indispensable?

Indispensable's a strong word, but one thing that entails calling
syntax, and can't (reasonably) be done with closures is to override a
base class's __call__ method.

There is one notable type in Python that defines __call__ and that you
might want to subclass, namely type itself. So using __call__ is
indispensible for some metaclasses.

Here's my little example. I'm sure people could work out ways to do
it without __call__ or even a metaclass, but I doubt it'd be so
concise and out of the way as this:

class CachedResourceMetaclass(type):
def __init__(self,name,bases,clsdict):

super(CachedResourceMetaclass,self).__init__(name, bases,clsdict)
self._cache = {}
self._releasefunc = self.release
def decref(obj):
obj._refcount -= 1
if obj._refcount <= 0:
del self._cache[obj._key]
obj._releasefunc()
self.release = decref
def __call__(self,*args):
obj = self._cache.get(args)
if obj is not None:
obj._refcount += 1
else:
obj = super(CachedResourceMetaclass,self).__call__(*args )
obj._key = args
obj._refcount = 1
self._cache[args] = obj
return obj
Carl Banks

[Skip Montanaro]

In this case there was a bug. Depending on inputs, sometimes obj

initialized to a class, sometimes an instance of that class. (I fixed
that too while I was at it.) The problem was that the use of __call__
obscured the underlying bug by making the instance as well as the class
callable.

I don't quite get the point here. A concrete example would be welcome.

The bug went something like this:

obj = some.default_class
...
if some_other_rare_condition_met:
... several lines ...
obj = some.other_class()
...

Later,

x = obj()
x(...)

The bug was that in the rare condition branch obj should have been assigned
just the class. It shouldn't have been instantiated there. Having the
instance be callable (for no obvious reason that I could tell) misdirected
my attention because the x = obj() generally succeeded and I thought the
problem was with the call to x(...). If the instance wasn't callable it
would have been clear at "x = obj()" that I was trying to call an instance.

Thanks for the various bits of feedback. From the responses I've seen, it
seems that if one is tempted to use __call__ they should consider it before
just sprinkling it in their code. The places where people seem to use it
generally seem not to be in run-of-the-mill code. In the common case it
seems to me there are generally better ways to do things. (I did like the
StoredProcedure mirroring stuff someone posted. That does look kind of
cool.)

Skip

[Bruno Desthuilliers]

Skip Montanaro a écrit :

>>In this case there was a bug. Depending on inputs, sometimes obj
initialized to a class, sometimes an instance of that class. (I fixed
that too while I was at it.) The problem was that the use of __call__
obscured the underlying bug by making the instance as well as the class
callable.

>I don't quite get the point here. A concrete example would be welcome.

The bug went something like this:

obj = some.default_class
...
if some_other_rare_condition_met:
... several lines ...
obj = some.other_class()
...

Later,

x = obj()
x(...)

Ok, I see. A nasty one, indeed.

[Duncan Booth]

sk**@pobox.com wrote:

In this particular case it was clearly unnecessary and just obfuscated
the code. I'm wondering, are there some general cases where __call__
methods of a user-defined class are simply indispensable?

I don't know that you couldn't live without __call__, but it would make
some code harder to write and understand. The time you need __call__ is
when you need an object which is callable and which also has methods. e.g.
look at xmlrpclib:

server = ServerProxy('http://some.server.com/path')

server.method(x=5) retrieves the url http://some.server.com/path/method?x=5
server.method.submethod(x=5) retrieves the url
http://some.server.com/path/method/submethod?x=5
There is no need here to distinguish between a callable object and an
object used for traversal. I think without __call__, code to access web
servers would be less clean.

[Terry Reedy]

Skip Montanaro a écrit :

>>In this case there was a bug. Depending on inputs, sometimes obj
initialized to a class, sometimes an instance of that class. (I fixed
that too while I was at it.) The problem was that the use of __call__
obscured the underlying bug by making the instance as well as the class
callable.

>I don't quite get the point here. A concrete example would be welcome.

The bug went something like this:

obj = some.default_class
...
if some_other_rare_condition_met:
... several lines ...
obj = some.other_class()

Should that have been some.other_class (without the ()s?). It is in the
nature of Python's dynamic typing that mis-typing errors sometimes show up
later than one would wish. Hence the need for unit testing. Consider

i = condition and 1 or '2' # whoops, should have been 2, without the 's
....
k = j * i # j an int; whoops, bad i value passes silently
....
use of k as string raises exception

I do not think __call__ should be specifically blamed for this general
downside of Python's design.

Terry Jan Reedy

[Skip Montanaro]

Terry Reedy <tjreedy <atudel.eduwrites:

The bug went something like this:

obj = some.default_class
...
if some_other_rare_condition_met:
... several lines ...
obj = some.other_class()

Should that have been some.other_class (without the ()s?).

Either that or default value should have been an instance of
some.default_class. Didn't really matter in this case...

Skip