How about "pure virtual methods"?

[Noam Raphael]

Hello,

I thought about a new Python feature. Please tell me what you think
about it.

Say you want to write a base class with some unimplemented methods, that
subclasses must implement (or maybe even just declare an interface, with
no methods implemented). Right now, you don't really have a way to do
it. You can leave the methods with a "pass", or raise a
NotImplementedError, but even in the best solution that I know of,
there's now way to check if a subclass has implemented all the required
methods without running it and testing if it works. Another problem with
the existing solutions is that raising NotImplementedError usually means
"This method might be implemented some time", and not "you must
implement this method when you subclass me".

What I suggest is a new class, called notimplemented (you may suggest a
better name). It would get a function in its constructor, and would just
save a reference to it. The trick is that when a new type (a subclass of
the default type object) is created, It will go over all its members and
check to see if any of them is a notimplemented instance. If that is the
case, it would not allow an instantiation of itself.

What I want is that if I have this module:

======================

class BaseClass(object):
def __init__(self):
...

@notimplemented
def save_data(self, filename):
"""This method should save the internal state of the class to
a file named filename.
"""
pass

class RealClass(BaseClass):
def save_data(self, filename):
open(filename).write(self.data)

======================

then if I try to instantiate BaseClass I would get an exception, but
instantiating RealClass will be ok.
Well, what do you say?

Noam Raphael

[Erik Max Francis]

Noam Raphael wrote:

Well, what do you say?

Raising NotImplementedError in the methods that need to be overridden is
much more customary, more straightforward, and already works fine.

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
We learn from history that we do not learn from history.
-- Georg Friedrich Wilhelm Hegel

[Alex Martelli]

Noam Raphael <no***@remove.the.dot.myrea.lbox.com> wrote:

What I want is that if I have this module:

======================

class BaseClass(object):
def __init__(self):
...

@notimplemented
def save_data(self, filename):
"""This method should save the internal state of the class to
a file named filename.
"""
pass

class RealClass(BaseClass):
def save_data(self, filename):
open(filename).write(self.data)

======================

then if I try to instantiate BaseClass I would get an exception, but
instantiating RealClass will be ok.
Well, what do you say?

It's already easy to do as long as you don't insist that BaseClass must
subclass object (or allow a __metaclass__ assignment there). Basically,
what you describe is the job of a custom metaclass, because that's where
the __call__ method on classes is defined (classes being instances of
the metaclass)... much cleaner than tampering with the __new__ or
__init__ at class level.

At sketch level, you'd have, say:

def notimplemented(f):
# save some data about f somewhere if you wish, then, e.g.:
return notimplemented

import inspect

class Meta_for_NR(type):
def __init__(cls, cn, cb, cd):
super(Meta_for_NR, cls).__init__(cn, cb, cd)
abstract_methods = []
for n, v in inspect.getmembers(cls, inspect.ismethod):
if v is notimplemented: abstract_methods.append(n)
cls._abstract_methods = abstract_methods
def __call__(cls, *a, **k):
if cls._abstract_methods:
raise TypeError, ("Cannot instantiate abstract class %s."
" Abstract methods: %s." % (cls.__name__,
', '.join(cls._abstract_methods)))
return super(Meta_for_NR, cls).__call__(*a, **k)

class Base_for_NR: __metaclass__ = Meta_for_NR
Inheriting from Base_for_NR instead of inheriting from object should
give you just the behavior you desire (net of the fact that this code of
mine is untested, just a sketch -- but the key ideas shd be ok).
If you can implement this and get it into wide usage, it may well be
adopted in the standard library. Unless there is proof by experience
that a lot of people like this approach better than the de facto
standard, such adoption is very unlikely. But the great thing is that
it's EASY to implement your ideas and try to garner a following for
them, so you stand a chance to get them into the standard library if you
can prove they really work well and are well-liked!
Alex

[John Machin]

Alex Martelli wrote:

class Meta_for_NR(type):
def __init__(cls, cn, cb, cd):
super(Meta_for_NR, cls).__init__(cn, cb, cd)
abstract_methods = []
for n, v in inspect.getmembers(cls, inspect.ismethod):
if v is notimplemented: abstract_methods.append(n)

etc etc etc

I'm missing the point of the question and Alex's answer. Let's step
through this:

1. Programmer A designs a base class some (or all) of whose methods are
abstract i.e. not implemented in a do-something-useful fashion; they
may be only stubs which document their arguments.

2. Programmer B designs a sub-class, which *should* implement methods
which override those of the base class.

3. Programmer C writes an application which *should* create instances
of the sub-class.

4. Things that can go wrong are (a) C instantiates the base class (b) B
has not provided all the methods (c) B has provided methods but they
don't work properly (d) C's app doesn't invoke B's methods in the
approved manner or sequence or with meaningful arguments or ...

5. The standard approach in Python is to have the virtual methods raise
an exception if invoked. This of course doesn't help with problems c
and d. In an attempt to root out instances of problems c and d,
programmer C will unit-test the @#$% out of the app. This will quickly
discover any instances of problems a and b. One would hope that as well
B has unit-tested the @#$% out of the sub-class.

6. Alex's magnum opus appears to operate on problem a, and maybe on b
(it's over my head). It involves a fair chunk of mucking about -- for
what? Early warning, a few microseconds ahead of the invocation of a
method which will cause the stub in the base class to raise an
exception?

[Nick Coghlan]

John Machin wrote:

6. Alex's magnum opus appears to operate on problem a, and maybe on b
(it's over my head). It involves a fair chunk of mucking about -- for
what? Early warning, a few microseconds ahead of the invocation of a
method which will cause the stub in the base class to raise an
exception?

Alex eats metaclasses for breakfast though, so while it may look like a magnum
opus to mere mortals, it is in reality nothing more than an idle twiddle :)

Anyway, I see two major problems with the OP's idea:

1. It encourages laziness when subclassing
Subclassing necessarily involves intimate coupling of code. If you don't
understand the base class well enough to know which methods you are required to
implement, and haven't even included those methods in your unit tests. . . then
I'm willing to bet those missing methods are the least of your worries when it
comes to correct operation of your class.

2. It gives up some of the dynamism of Python
The OP's idea denies the inheritor the ability to partially implement the
defined interface. Sometimes, particularly for objects used for a very specific
purpose, it makes sense to only implement the portion of an interface that you
want to use. The standard method makes it easy to flag a portion of the
interface as off limits - just leave it raising a Not Implemented exception.
With the OP's idea in place, the inheritor is forced to implement the entirety
of the interface, even if they only need a fraction of it.

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

[Alex Martelli]

John Machin <sj******@lexicon.net> wrote:
...

6. Alex's magnum opus appears to operate on problem a, and maybe on b
(it's over my head). It involves a fair chunk of mucking about -- for
A dozen lines of code to write *ONCE* and put in your utilities module
is a "magnum opus" and "a fair chunk of mucking about"?! Oh my -- I
wonder what would count for you as parvus and incredibly tiny -- SIX
lines? THREE? ONE AND A HALF, for Pete's sake?!?!?!

If your problem with my code is simply that it uses parts of Python that
you're not familiar about, then I count this as a positive aspect of my
code, not a negative one: by providing motivation to study metaclasses
and inspect (by givin some small usefulness in very small space) it may
help you indirectly far more than directly.
what? Early warning, a few microseconds ahead of the invocation of a
method which will cause the stub in the base class to raise an
exception?

Exactly. Microseconds don't count, but stack levels do -- getting the
traceback end as close as possible to the real CAUSE of the problem,
rather than a few levels further down where the SYMPTOM appears, can
easily shave whole minutes off debugging time. Since the whole of what
you call "magnum opus" took about 4 minutes to write, if it acts this
way twice I'm gaining.

Indeed, this issue is important enough that it might be worth one's
while to ensure the exception comes at class definition rather than
instantiation time -- e.g. by saying that a class must explicitly expose
(not just inherit) a class attribute '__abstract__' that's True, or else
having abstract methods still present is an error. This obviously takes
a trivial mod to the (ha!) "magnum" opus, and makes errors even easier
to diagnose. Still, it wasn't among the OP's desiderata, so I didn't
change the semantics in this way.
Note that either or both changes work particularly well wrt UNIT tests,
tests of some component which sees the abstract base class as 'external'
and thus not subject to its unit tests. Say that 'external' component X
exposes ABC, the abstract base class, and many functions f, g, h, ...,
which take an ABC instance and do things with it. My own component Y
subclasses ABC with WCC, the wannabe concrete class, and passes an
instance of WCC to *SOME* of X.f, X.g, X.h -- SOME, not necessarily ALL,
because UNIT tests don't have to fully probe interfaces BETWEEN
components!!! (that's why they're UNIT tests, not INTEGRATION ones;-).

With Python's standard approach, no guarantee that WCC implements all
the methods it should. Maybe it did implement all methods needed at the
time it was written, then X release 1.1 added a couple abstract methods
to ABC and a couple new functions j and k using them, and Y knows
nothing about this upgrade. Y's unit tests are still gonna pass. And
somebody ELSE who's instantiating a Y.WCC and passing the instance to
X.k is gonna feel the pain -- not in their unit tests, mind you, this
has VERY MUCH become a problem of inter-component orchestration.

With the cost of 12 lines, ONCE, in a utility module, you ensure halfway
decent unit tests WILL expose all 'failed to implement needed methods'
errors, even as X's specs keep evolving. If you make any substantial
use of abstract classes and methods, the tradeoff is VERY favourable.
So, do _I_ use this? Nah: I think abstract classes with
explicitly-abstract "somebody's gotta implement this!" methods are
rarely a good idea and tend to lead to fat base classes and overuse of
inheritance. But if one disagrees, then I think that Python's suppport
for them with NotImplementedError is suboptimal, and some variation and
evolution along the lines of the tiny sketch I gave would be better.
Alex

[Fredrik Lundh]

Alex Martelli:

what? Early warning, a few microseconds ahead of the invocation of a
method which will cause the stub in the base class to raise an
exception?
Exactly. Microseconds don't count, but stack levels do -- getting the
traceback end as close as possible to the real CAUSE of the problem,
rather than a few levels further down where the SYMPTOM appears, can
easily shave whole minutes off debugging time.

minutes?

Traceback (most recent call last):
File "mymodule.py", line 9, in somefunction
someobj.bar()
... zero or more lines ...
File "somelibrary.py", line 3, in bar
raise NotImplementedError("must implement abstract method bar")
NotImplementedError: must implement abstract method bar

you have to look at the last line of a traceback to find the error message, and
the offending module/method is two lines above that.
So, do _I_ use this? Nah: I think abstract classes with
explicitly-abstract "somebody's gotta implement this!" methods are
rarely a good idea

yet you claim to be an expert on issues related to debugging NotImplemented-
Error exceptions?

</F>

[Alex Martelli]

Fredrik Lundh <fr*****@pythonware.com> wrote:

Alex Martelli:

what? Early warning, a few microseconds ahead of the invocation of a
method which will cause the stub in the base class to raise an
exception?

Exactly. Microseconds don't count, but stack levels do -- getting the
traceback end as close as possible to the real CAUSE of the problem,
rather than a few levels further down where the SYMPTOM appears, can
easily shave whole minutes off debugging time.

minutes?

Traceback (most recent call last):
File "mymodule.py", line 9, in somefunction
someobj.bar()
... zero or more lines ...
File "somelibrary.py", line 3, in bar
raise NotImplementedError("must implement abstract method bar")
NotImplementedError: must implement abstract method bar

you have to look at the last line of a traceback to find the error
message, and the offending module/method is two lines above that.

The offending module is the one which instantiates a class it shouldn't
instantiate -- a class which isn't made concrete by overriding all
methods which need to be. What is that module? What is that class?
The error is generally not the calling of 'bar' on someobj: someobj
arrives as a argument to somefunction from somewhere -- the caller of
somefunction is quite often innocent, in turn -- who BUILT someobj, by
instantiating which class? I think `minutes' is a very fair (not at all
pessimistic) assessment of the time it will generally take to find out.

With an exception at the time of instantiation, you shave those minutes
off. As I already explained: if it was acceptable to raise at the time
the class statement is executed, which doesn't make the class fully
concrete yet fails to explicitly assert __abstract__=True in classbody,
the distance between where the error is in the code, and where it's
diagnosed, could be further reduced in many cases; sometimes the error
is to mistakenly instantiate a class which is not yet meant to be fully
concrete, but equally well it could be to fail to override all needed
methods in a class which IS meant to be fully concrete. However, this
further refinement turns out to be generally of lesser impact wrt the
primary one: as long as the error msg at instantiation time fully
identifies what class is involved and why (because of what still
abstract methods) it's an error to instantiate it, then the productivity
plus of getting the error at class-statement time vs instantiation time
is generally no big deal -- the big deal is not having to wait until,
somewhere, somehow, a still-abstract method finally gets _called_, and
then having to trace back to how and where that class was instantiated.

So, do _I_ use this? Nah: I think abstract classes with
explicitly-abstract "somebody's gotta implement this!" methods are
rarely a good idea

yet you claim to be an expert on issues related to debugging NotImplemented-
Error exceptions?

Yes, the body of experience which essentially convinced me to avoid the
above idiom (because, as I say in the snippet I quote here, it's rarely
a good idea -- you snipped my summary of reasons why) essentially
overlaps with the body of experience which suggests to me that IF
somebody's keen to use the idiom widely anyway, they're going to be far
better off with the errors it's meant to catch causing errors to be
raised ASAP, as the custom metaclass so easily allows, rather than as
late as possible, as using 'raise NotImplementedError' leads to.

[[ Protocols (also unfortunately known as interfaces) and adaptation
work much better, essentially by not depending on inheritance. ]]

I really don't understand what's so difficult here, for you, as to make
it apparently so very hard for you to follow. The idiom (meant to catch
some wrong uses of inheritance) of 'abstract classes with certain
methods that need to get overridden' isn't all that helpful in Python
when all that happens is a NotImplementedError exception. If the base
class had simply omitted defining the method, as used to be the idiom,
you'd generally get an AttributeError instead -- slightly more generic
but happening basically at the same spot in the code, time difference in
finding the errors usually a few seconds. With a custom metaclass (and
optionally a decorator, though there are other ways, which we can
discuss in detail if you think such distinctions are oh so crucial) you
can easily make the idiom more helpful for the purpose of diagnosing
errors in inheritance &c -- i.e., the idiom's sole purpose.

Nevertheless, even when one has made the idiom as good as it can be, I
think it's still rarely a good idea. Protocols and adaptation take a
fresh approach, not as heavily influenced by C++ and Java limitations as
the idiom we're discussing here, and it's my contention that said
approach does a better job. One can perfectly well still use
inheritance for what inheritance does best (easy, simple, fast sharing
of some implementation) without depending on it for those protocol
compliance issues which were never its forte.
Alex

[Fredrik Lundh]

Alex Martelli wrote:

Traceback (most recent call last):
File "mymodule.py", line 9, in somefunction
someobj.bar()
... zero or more lines ...
File "somelibrary.py", line 3, in bar
raise NotImplementedError("must implement abstract method bar")
NotImplementedError: must implement abstract method bar

you have to look at the last line of a traceback to find the error
message, and the offending module/method is two lines above that.
The offending module is the one which instantiates a class it shouldn't
instantiate -- a class which isn't made concrete by overriding all
methods which need to be. What is that module?

in real life, it's mymodule.py or mylib.py, in code that you wrote very recently.
The error is generally not the calling of 'bar' on someobj: someobj
arrives as a argument to somefunction from somewhere -- the caller of
somefunction is quite often innocent, in turn -- who BUILT someobj, by
instantiating which class? I think `minutes' is a very fair (not at all
pessimistic) assessment of the time it will generally take to find out.

you can blather for years if you want to, but your arguments are just another variant
of the "you cannot use whitespace for indentation" stuff we've all seen from time to
time; it's trivial to come up with a large number of theoretical problems with any given
approach, but in real life, for real-life programmers, using real-life libraries designed
by real-life developers, it's simply not the problem you want it to be.

but not that I'm going to convince you, now that you've spent 15,000 characters on
this topic...

</F>

[Steve Holden]

Noam Raphael wrote:

Hello,

I thought about a new Python feature. Please tell me what you think
about it.

Say you want to write a base class with some unimplemented methods, that
subclasses must implement (or maybe even just declare an interface, with
no methods implemented). Right now, you don't really have a way to do
it. You can leave the methods with a "pass", or raise a
NotImplementedError, but even in the best solution that I know of,
there's now way to check if a subclass has implemented all the required
methods without running it and testing if it works. Another problem with
the existing solutions is that raising NotImplementedError usually means
"This method might be implemented some time", and not "you must
implement this method when you subclass me".

What I suggest is a new class, called notimplemented (you may suggest a
better name). It would get a function in its constructor, and would just
save a reference to it. The trick is that when a new type (a subclass of
the default type object) is created, It will go over all its members and
check to see if any of them is a notimplemented instance. If that is the
case, it would not allow an instantiation of itself.

What I want is that if I have this module:

======================

class BaseClass(object):
def __init__(self):
...

@notimplemented
def save_data(self, filename):
"""This method should save the internal state of the class to
a file named filename.
"""
pass

class RealClass(BaseClass):
def save_data(self, filename):
open(filename).write(self.data)

======================

then if I try to instantiate BaseClass I would get an exception, but
instantiating RealClass will be ok.
Well, what do you say?

Noam Raphael

OK, you've had many replies telling you why your suggestion isn't
particularly worthwhile, and a few telling you how you can already do
this in present-day Python.

Even if you can do it, how would you then implement a class hierarchy
where the ultimate base class had virtual methods, and you wanted to
derive from that class another class, to be used as a base class for
usable classes, which implemented only a subset of the virtual methods,
leaving the others to be implemented by the ultimate subclasses?

This use case makes your suggestion look impractical, methinks, since it
requires that the programmer do what you would apparently like to
specifically forbid.

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC [phone] [phone]

[Mike Meyer]

Noam Raphael <no***@remove.the.dot.myrea.lbox.com> writes:

What I suggest is a new class, called notimplemented (you may suggest
a better name). It would get a function in its constructor, and would
just save a reference to it. The trick is that when a new type (a
subclass of the default type object) is created, It will go over all
its members and check to see if any of them is a notimplemented
instance. If that is the case, it would not allow an instantiation of
itself.

+0

Python doesn't use classes for typing. As Alex Martelli puts it,
Python uses protocols. So the client expecting a concrete subclass of
your abstract class may get an instantiation of a class that doesn't
inherit from the abstract class at all.

Or maybe the subclass is only going to use a subset of the features of
the abstract class, and the author knows that sum deferred methods
won't be invoked. The correct behavior in this case would be to allow
the subclass to be instantiated, and then get a runtime error if one
of the features the author thought he could skip was actually called.

Finally, in a sufficiently complex class hierarchy, this still leaves
you wondering through the hierarchy trying to find the appropriate
parent class that tagged this method as unimplemented, and then
figuring out which class should have implemented it - as possibly a
parent of the class whose instantiation failed is the subclass that
should have made this method concrete.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Doug Holton]

Noam Raphael wrote:
even in the best solution that I know of,

there's now way to check if a subclass has implemented all the
required methods without running it and testing if it works.

I think there are some solutions like PyProtocols, see section 2.2 on
this page: http://www.python.org/cgi-bin/moinmoin/MetaClasses

But what you are wanting is used more in statically typed languages.
If you want interfaces, abstract classes and virtual methods in a
python-like language see boo: http://boo.codehaus.org/
It will catch errors like this at compile-time.

[Fredrik Lundh]

.... From: Doug ...

(I think you could create some kind of drinking game based on the number of
non-quoted lines between the "From"-line and the first line containing the word
"boo" in Doug's posts on comp.lang.python. A little like the game based on the
number of minutes between someone mentioning ElementTree or Anobind or
some other Python/XML toolkit in a blog post, and David Mertz adding a
comment about how his own tool can do the same thing faster and better...)

.... boo ...

(</F>)

[John Machin]

Alex Martelli wrote:

John Machin <sj******@lexicon.net> wrote:
...

6. Alex's magnum opus appears to operate on problem a, and maybe on b (it's over my head). It involves a fair chunk of mucking about --

for
A dozen lines of code to write *ONCE* and put in your utilities module is a "magnum opus" and "a fair chunk of mucking about"?! Oh my -- I
wonder what would count for you as parvus and incredibly tiny -- SIX
lines? THREE? ONE AND A HALF, for Pete's sake?!?!?!
"Magnum opus" was exaggerated politeness.

If your problem with my code is simply that it uses parts of Python that you're not familiar about, then I count this as a positive aspect of my code, not a negative one: by providing motivation to study metaclasses and inspect (by givin some small usefulness in very small space) it may help you indirectly far more than directly.

Thank you for the motivation. I now understand what it is attempting to
do. Unfortunately it doesn't succeed. Instead of:

if v is notimplemented: abstract_methods.append(n)

you need:
if v.im_func is notimplemented: abstract_methods.append(n)

[Alex Martelli]

Fredrik Lundh <fr*****@pythonware.com> wrote:
...

Alex Martelli wrote:

Traceback (most recent call last):
File "mymodule.py", line 9, in somefunction
someobj.bar()
... zero or more lines ...
File "somelibrary.py", line 3, in bar
raise NotImplementedError("must implement abstract method bar")
NotImplementedError: must implement abstract method bar

you have to look at the last line of a traceback to find the error
message, and the offending module/method is two lines above that.
The offending module is the one which instantiates a class it shouldn't
instantiate -- a class which isn't made concrete by overriding all
methods which need to be. What is that module?

in real life, it's mymodule.py or mylib.py, in code that you wrote very
recently.

In my experience, it ain't necessarily so. Maybe I happen to work more
than you do with, for example, people who've had indoctrination into
"the proper ways to do OOP" (meaning C++ or Java) -- indeed I've found
that wanting to make abstract classes with methods which raise
exceptions, rather than just omitting the methods (and thus getting an
AttributeError if anybody tries calling them on instances of subclasses
which lack them) has a good correlation with people with the mindset
typically coming from such doctrine.

Such people write modules which define abstract classes and also the
functions that receive instances of such abstract classes (sometimes
they even check for that with isinstance!) and call methods on them --
never, in practice, will the same 'mymodule.py' that calls someobj.bar()
be the one which created 'someobj'; rather 'someobj' will be an argument
to 'somefunction' -- so you need to continue the study up the stack of
calls. I'm not saying "it can't be done"! I'm saying it's quite likely
to take EXTRA MINUTES (that's ALL I'm claiming, remember: so, your
counterclaim is that such a search *hardly ever takes more than an extra
119 seconds*!!!) to find out where the should-be-concrete class is
defined and instantiated, starting from the first call to one method it
has wrongfully left un-overridden, compared to the case where the class
statement itself, or the instantiation, raise the exception.

As for the "code you've written recently": maybe, IF that abstract class
and supporting infrastructure never changes. If your application-level
code uses frameworks which change (never a pleasant situation, but it
does often happen), then it's quite likely the *framework* has changed
recently, adding a deuced new 'bar' method that now you're supposed to
override, too. Again, you must find out where you defined the subclass
which happens to be the class of 'someobj'. Again, I would NOT be
surprised if it took 121, even 122 seconds... or (gasp!) occasionally
even more!

The error is generally not the calling of 'bar' on someobj: someobj
arrives as a argument to somefunction from somewhere -- the caller of
somefunction is quite often innocent, in turn -- who BUILT someobj, by
instantiating which class? I think `minutes' is a very fair (not at all
pessimistic) assessment of the time it will generally take to find out.

you can blather for years if you want to, but your arguments are just
another variant of the "you cannot use whitespace for indentation" stuff
we've all seen from time to

I'm not saying you *cannot* do anything: I'm claiming that, if you do
things in a slightly different way, it will often save you 120 or more
seconds of time in finding out where you made a mistake.

If you think it's worth the bother of putting in 'empty' abstract
methods which raise NotImplementedException (rather than saving the time
in the first place by just omitting them), you must be convinced they
help productivity -- e.g. make it faster to find errors, compared to the
easier-to-obtain effect of getting an AttributeError. How? And how
comes a more precise indication of where the override was not done
(ideally by raising at the class statement, but instantiation time isn't
all that much worse) appears in your opinion to give NO further added
value, not even two minutes' worth?
time; it's trivial to come up with a large number of theoretical problems
with any given approach, but in real life, for real-life programmers,
using real-life libraries designed by real-life developers, it's simply
not the problem you want it to be.
I don't "want" anything in particular. I _have_ observed people
spending over 120 seconds finding out where the errant class was defined
and what methods it was missing -- with real-life everythings; just the
same amount of time they'd have spent without the silly
NotImplementedError to "help", just by reasoning from AttributeError.

but not that I'm going to convince you, now that you've spent 15,000
characters on this topic...

Characters are cheap. On the other hand, I can't see how you can
convince me that what I've observed (and common sense confirms) is an
illusion, that one can't waste two minutes finding exactly where the
errant class was defined. Since it takes about 4 minutes to code that
metaclass, if twice in a project's lifetime, 2 minutes each time are
saved, one breaks even; if three times, one's gaining. This doesn't
take into account the large amount of times wasted on this incredibly
silly thread, of course, but hey, I disclaim any responsibility for it:
I posted a sketch of a dozen lines of code to solve a real-life problem
somebody seemed to be having, and found myself unexpectedly under attack
for 'posting a magnum opus', 'blathering for years', and now even
arguing against using whitespace for indentation.

Definitely not the c.l.py I recalled; if that's what one can expect,
these days, for posting a dozen lines of HTH code -- amount of thanks or
any expression of gratitude a nice round zero, amount of flames and
direct attacks as high as I've gotten so far -- I guess I'm not going to
stay around all that long this time.
Alex

[Alex Martelli]

John Machin <sj******@lexicon.net> wrote:

Thank you for the motivation. I now understand what it is attempting to
do. Unfortunately it doesn't succeed. Instead of:

if v is notimplemented: abstract_methods.append(n)

you need:
if v.im_func is notimplemented: abstract_methods.append(n)

Right, thanks -- good thing I clearly posted it was a sketch and
untested;-). In practice it's no doubt better to mark the function as
being abstract (e.g. with a function attribute) and otherwise leave it
alone; people who are keen about abstract classes and methods often want
to be able to call such abstract methods explicitly (for dubious
reasons, but it _is_ a common practice in C++, for example -- make a
method pure virtual, and yet give it a body anyway, for the benefit of
subclasses' overriding-methods).
Alex

[Scott David Daniels]

John Machin wrote (of Alex Martelli's sketch):

Thank you for the motivation. I now understand what it is attempting to
do. Unfortunately it doesn't succeed. Instead of:
if v is notimplemented: abstract_methods.append(n)
you need:
if v.im_func is notimplemented: abstract_methods.append(n)

Here is an implementation from that sketch:

from inspect import getmembers

class MustImplement(type):
'''A metaclass to detect instantiation of abstract classes'''

def __init__(class_, name, bases, defined):
super(MustImplement, class_).__init__(name, bases, defined)
class_._abstract_methods = sorted(name
for name, value in getmembers(class_)
if value is NotImplemented)

def __call__(class_, *args, **kwargs):
if class_._abstract_methods:
raise NotImplementedError, 'Abstract class %s needs %s.' % (
class_.__name__, ', '.join(class_._abstract_methods))
return super(MustImplement, class_).__call__(*args, **kwargs)
class Abstract(object):
'''A class to stick anywhere in an inheritance chain'''
__metaclass__ = MustImplement
def notimplemented(method):
'''A decorator for those who prefer the parameters declared.'''
return NotImplemented
if __name__ == '__main__':
class SomeBase(dict):
__metaclass__ = MustImplement
needed = also_needed = NotImplemented
class Derived(SomeBase):
def needed(x): return 3
class Further(Derived):
def also_needed(x): return 1

class SomethingMissing(object):
@notimplemented
def function(): return 23

class UsesMissing(SomethingMissing, Abstract): pass

f = Further() # works
try:
d = Derived()
except NotImplementedError, e:
print 'Got expected exception:', e
else:
raise ValueError, 'Did not get expected exception'

s = SomethingMissing() # works
try:
u = UsesMissing()
except NotImplementedError, e:
print 'Got expected exception:', e
else:
raise ValueError, 'Did not get expected exception'
--Scott David Daniels
Sc***********@Acm.Org

[Steven Bethard]

Alex Martelli wrote:

Definitely not the c.l.py I recalled [snip] I guess I'm not going to stay around all that long this time.

That would be a sad loss for all of us out here who very much appreciate
your very deep knowledge of Python and your willingness to share it.

I do understand the feeling though; Fredrik Lundh jumped at me only a
few days ago when I said that I personally found list comprehensions
more readable than map. I certainly don't mind being advised or
corrected if I've written incorrect or even just suboptimal code, but
attacking a person's mental processes seems kind of needlessly
aggressive to me.

I don't really have a good solution; despite the unnecessarily vicious
comments, I don't feel like I can set my newsreader to ignore messages
from, for example, Fredrik, because his answers, when not attacks, are
often very insightful. If you find a good solution to this problem,
please let me know.

Steve

[Fredrik Lundh]

Steven Bethard wrote:

I do understand the feeling though; Fredrik Lundh jumped at me only a few days ago when I said
that I personally found list comprehensions more readable than map.
if you think that's what you said, you're clearly didn't read the post you
replied to very carefully. I read both posts before I made that comment.
If you find a good solution to this problem, please let me know.

well, since I'm not in the ego-stroking business, what if I promise never to
reply to posts by you, robert, and alex?

</F>

[Doug Holton]

Fredrik Lundh trolled:

(I think you could create some kind of drinking game based on the number of

....times the nasty trolls pounce on this list?

No, I think the idea is to actually address the content of someone's
question, politely and in the *holiday spirit*, not spirits.

[Doug Holton]

Fredrik Lundh wrote:

If you find a good solution to this problem, please let me know.

well, since I'm not in the ego-stroking business, what if I promise never to
reply to posts by you, robert, and alex?

That's not fair to the rest of us though :)

[Simon Wittber]

> I do understand the feeling though; Fredrik Lundh jumped at me only a

few days ago when I said that I personally found list comprehensions
more readable than map. I certainly don't mind being advised or
corrected if I've written incorrect or even just suboptimal code, but
attacking a person's mental processes seems kind of needlessly
aggressive to me.

We've just had a Python 2.4 release, and poor Mr Lundh has likely had
hundreds of user's clamouring for a binary, 2.4 compatible PIL
release.

That's enough to make any programmer grumpy.

Sw.

[Fredrik Lundh]

Simon Wittber wrote:

We've just had a Python 2.4 release, and poor Mr Lundh has likely had
hundreds of user's clamouring for a binary, 2.4 compatible PIL
release.

many hundreds, indeed. but that's not a problem at all; I have a webserver
that helps me with things like that.

</F>

[Doug Holton]

Steven Bethard wrote:

I don't really have a good solution; despite the unnecessarily vicious
comments, I don't feel like I can set my newsreader to ignore messages
from, for example, Fredrik, because his answers, when not attacks, are
often very insightful. If you find a good solution to this problem,
please let me know.

Thankyou for bringing it up. It looks like it may have a positive effect.

[Steven Bethard]

Doug Holton wrote:

Fredrik Lundh wrote:

well, since I'm not in the ego-stroking business, what if I promise
never to reply to posts by you, robert, and alex?

That's not fair to the rest of us though :)

That's not even fair to the non-rest of us. =) As I noted, "his answers
.... are often very insightful" -- it would be a pity to lose them.

Steve

[Doug Holton]

Steven Bethard wrote:

Doug Holton wrote:

Fredrik Lundh wrote:

well, since I'm not in the ego-stroking business, what if I promise
never to reply to posts by you, robert, and alex?

That's not fair to the rest of us though :)

That's not even fair to the non-rest of us. =) As I noted, "his answers
... are often very insightful" -- it would be a pity to lose them.

He was only acknowledging the problem to those 3 people who complained
about it. I was making the point that others do not like being trolled
either.

[Steven Bethard]

Doug Holton wrote:

He was only acknowledging the problem to those 3 people who complained
about it. I was making the point that others do not like being trolled
either.

Ahh, gotcha. Read your comment with the right intonation this time. =)

Steve

[Carl Banks]

Fredrik Lundh wrote:

(I think you could create some kind of drinking game based on the number of non-quoted lines between the "From"-line and the first line containing the word "boo" in Doug's posts on comp.lang.python. A little like the game based on the number of minutes between someone mentioning ElementTree or Anobind or some other Python/XML toolkit in a blog post, and David Mertz adding a comment about how his own tool can do the same thing faster and

better...)

Or until Peter Hansen mentions unit testing, or Fernando Perez mentions
IPython, or Michele Simionato (sp?) proposes a metaclass solution to a
problem, or Paul Rubin decries any aspect of Python that differs from
Lisp,
or Aahz asserts that the GIL isn't a significant problem in practice,
or....

all-in-fun-of-course-ly yr's,
--
CARL BANKS

[Doug Holton]

Doug Holton wrote:

Noam Raphael wrote:
even in the best solution that I know of,

there's now way to check if a subclass has implemented all the
required methods without running it and testing if it works.

I think there are some solutions like PyProtocols, see section 2.2 on
this page: http://www.python.org/cgi-bin/moinmoin/MetaClasses

Let me rephrase the rest of my answer.

This is something we could use in Python. You should add a feature
request to have this in Python 3.0, a.k.a. Python 3000:
http://www.python.org/cgi-bin/moinmoin/Python3.0

You might have simple code such as:

inteface ICallable:
def Call(args)

However, Python 3.0 is likely years away. If you want to know how to
run code like this and have real interfaces, abstract classes and
virtual methods today, consult Fredrik Lundh.

[Doug Holton]

Doug Holton wrote:

Steven Bethard wrote:

I don't really have a good solution; despite the unnecessarily vicious
comments, I don't feel like I can set my newsreader to ignore messages
from, for example, Fredrik, because his answers, when not attacks, are
often very insightful. If you find a good solution to this problem,
please let me know.

Thankyou for bringing it up. It looks like it may have a positive effect.

Unfortunately, I may have jumped the gun on that one. He does not even
acknowledge his behavior outside of the three instances he referred to.

[Aahz]

In article <Op********************@comcast.com>,
Doug Holton <in****@spam.here> wrote:

Unfortunately, I may have jumped the gun on that one. He does not even
acknowledge his behavior outside of the three instances he referred to.

From my POV, your use of a munged e-mail address makes you far more
antisocial than Fredrik.
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

"19. A language that doesn't affect the way you think about programming,
is not worth knowing." --Alan Perlis

[Noam Raphael]

Thank you all, especially Alex for your enlightening discussion, and
Scott for your implementation. I'm sorry that I can't be involved in a
daily manner - but I did read all of the posts in this thread. They
helped me understand the situation better, and convinced me that indeed
this feature is needed. Let's see if I can convince you too.

First, the actual situation in which I stood, which made me think, "I
would like to declare a method as not implemented, so that subclasses
would have to implement it."

I wrote a system in which objects had to interact between themselves. In
my design, all those objects had to implement a few methods for the
interaction to work. So I wrote a base class for all those objects, with
a few methods which the subclasses had to implement. I think it's good,
for *me*, to have an explicit list of what should be implemented, so
that when (in a function) I expect to get an object of this kind I know
what I may and may not do with it.

Then, I wrote the classes themselves. And I wrote unit tests for them.
(Ok, I lie. I didn't. But I should have!) Afterwards, I decided that I
needed all my objects of that kind to supply another method. So I added
another "raise NotImplementedError" method to the base class. But what
about the unit tests? They would have still reported a success - where
of course they shouldn't have; my classes, in this stage, didn't do what
they were expected to do. This problem might arise even when not
changing the interface at all - it's quite easy to write a class which,
by mistake, doesn't implement all the interface. Its successful unit
tests may check every single line of code of that class, but a complete
method was simply forgotten, and you wouldn't notice it until you try
the class in the larger framework (and, as I understand, the point of
unit testing is to test the class on its own, before integrating it).

Ok. This was the practical reason why this is needed. Please note that I
didn't use "isinstance" even once - all my functions used the
*interface* of the objects they got. I needed the extra checking for
myself - if someone wanted to implement a class that wouldn't inherit
from my base class, but would nevertheless implement the required
interface, he was free to do it, and it would have worked fine with the
framework I wrote.

Now for the "theoretical" reason why this is needed. My reasoning is
based on the existence of "isinstance" in Python. Well, what is the
purpose of isinstance? I claim that it doesn't test if an object *is* of
a given type. If that would have been its purpose, it would have checked
whether type(obj) == something. Rather, it checks whether an object is a
subclass of a given type. Why should we want such a function? A subclass
may do a completely different thing from what the original class did!
The answer is that a subclass is guaranteed to have the same *interface*
as the base class. And that's what matters.

So I conclude that a subclass, in Python, must implement the interface
of its parent class. Usually, this is obvious - there's no way for a
subclass not to implement the interface of its parent class, simply
because it can only override methods, but can't remove methods. But what
shall we do if the some methods in the base class consist *only* of an
interface? Can we implement only a part of the interface, and claim that
instances of that class are instances of the original class, in the
"isinstance" fashion? My answer is no. The whole point of "isinstance"
is to check whether an instance implements an interface. If it doesn't -
what is the meaning of the True that isinstance returns? So we should
simply not allow instances of such classes.

You might say that abstract classes at the base of the hierarchy are
"not Pythonic". But they are in Python already - the class basestring is
exactly that. It is an uninstantiable class, which is there only so that
you would be able to do isinstance(x, basestring). Classes with
"notimplemented" methods would behave in exactly the same way - you
wouldn't be able to instantiate them, just to subclass them (and to
check, using isinstance, whether they implement the required protocol,
which I agree that wouldn't be Pythonic, probably).

Ok. This is why I think this feature fits Python like a glove to a hand.
Please post your comments on this! I apologize now - I may not be able
to reply in the next few days. But I will read them at the end, and I
will try to answer.

Have a good day,
Noam

[Noam Raphael]

Steve Holden wrote:

Even if you can do it, how would you then implement a class hierarchy
where the ultimate base class had virtual methods, and you wanted to
derive from that class another class, to be used as a base class for
usable classes, which implemented only a subset of the virtual methods,
leaving the others to be implemented by the ultimate subclasses?

What I suggest is that only *instantiation* would be forbidden. You are
free to make a subclass which defines only some of the abstract methods,
and to subclass the subclass and define the rest. You would only be able
to make instances of the subclass of the subclass, but that's ok.

See Scott's implementation - the test at the end does exactly this.

I hope this helped,
Noam

[Jeff Shannon]

Noam Raphael wrote:

But what about the unit tests? They would have still reported a
success - where of course they shouldn't have; my classes, in this
stage, didn't do what they were expected to do. This problem might
arise even when not changing the interface at all - it's quite easy to
write a class which, by mistake, doesn't implement all the interface.
Its successful unit tests may check every single line of code of that
class, but a complete method was simply forgotten, and you wouldn't
notice it until you try the class in the larger framework (and, as I
understand, the point of unit testing is to test the class on its own,
before integrating it).

Except that unit tests should be written to the *specification*, not the
implementation. In other words, forgetting a complete method would
require that you forget to write the method, *and* that you failed to
translate the specification into unit tests *for that same method*.

In the context of changing an existing interface, a unit-testing
scenario would mean that, instead of installing a "pure virtual" method
on a base class, you'd change the unit-tests to follow the new
specification, and then write code that would pass the unit tests. If
you are subclassing from a common base, then you'd only need to change
the unit test for that common base class (presuming that all derived
classes would run those unit tests as well).

Jeff Shannon
Technician/Programmer
Credit International

[Noam Raphael]

My long post gives all the philosophy, but I'll give here the short answers.

Mike Meyer wrote:

+0

Python doesn't use classes for typing. As Alex Martelli puts it,
Python uses protocols. So the client expecting a concrete subclass of
your abstract class may get an instantiation of a class that doesn't
inherit from the abstract class at all.
That's right - this mechanism is useful mostly for he who implements
that class, to make sure that he implemented all that is needed to be
assigned the title "a subclass of that class".

Or maybe the subclass is only going to use a subset of the features of
the abstract class, and the author knows that sum deferred methods
won't be invoked. The correct behavior in this case would be to allow
the subclass to be instantiated, and then get a runtime error if one
of the features the author thought he could skip was actually called.
I disagree - my reasoning is that a subclass must implement the complete
interface of its base class (see my long post). The author may implement
a class which defines only a part of the interface, and give it to the
function, and it may work and be great. But it must not be called "an
instance of the abstract class".
Finally, in a sufficiently complex class hierarchy, this still leaves
you wondering through the hierarchy trying to find the appropriate
parent class that tagged this method as unimplemented, and then
figuring out which class should have implemented it - as possibly a
parent of the class whose instantiation failed is the subclass that
should have made this method concrete.
You are right - but I needed this for a class hierarchy of only two
levels (the base abstract class and the concrete subclasses), so there
were not many classes to blame for a missing method.
<mike

I hope this seems reasonable,
Noam

[Noam Raphael]

Jeff Shannon wrote:

Except that unit tests should be written to the *specification*, not the
implementation. In other words, forgetting a complete method would
require that you forget to write the method, *and* that you failed to
translate the specification into unit tests *for that same method*.
You are absolutely right - but when you are not that tidy, and don't
have a written specification apart from the code, it would be natural to
go over each method in the class definition, and write a test to check
if it does what it should. I'm not saying that it's the ideal way, but
it is not that bad, usually.
In the context of changing an existing interface, a unit-testing
scenario would mean that, instead of installing a "pure virtual" method
on a base class, you'd change the unit-tests to follow the new
specification, and then write code that would pass the unit tests. If
you are subclassing from a common base, then you'd only need to change
the unit test for that common base class (presuming that all derived
classes would run those unit tests as well).
The problem is that I couldn't write a general unit test, since the base
class wasn't instantiable - there wasn't even any point in making it
instantiable, since every subclass was constructed with different
argument definition. They were just required to provide some methods
(check whether they contain an object, for example) - I don't know how
to write a unit test for such a base class, or what does it mean. (Well,
it may mean: check whether all the required methods exist, but come on -
that's exactly the idea of my suggestion. There's no point in having to
write the list of required methods another time).
Jeff Shannon
Technician/Programmer
Credit International

Thanks for your comment. You're welcomed to reply if you don't agree.
Noam

[Noam Raphael]

Scott David Daniels wrote:

class Abstract(object):
'''A class to stick anywhere in an inheritance chain'''
__metaclass__ = MustImplement
def notimplemented(method):
'''A decorator for those who prefer the parameters declared.'''
return NotImplemented

I just wanted to say that I thought of notimplemented as a class, that
would save a reference to the functions it got in the constructor. In
that way pydoc and his friends would be able to find the arguments the
method was expected to get, and its documentation string.

But it's a great implementation.

Noam

Oh, and another thing - maybe "abstract" is a better name than
"notimplemented"? notimplemented might suggest a method which doesn't
have to be implemented - and raises NotImplementedError when it is
called. What do you think?

[Mike Meyer]

Noam Raphael <no***@remove.the.dot.myrea.lbox.com> writes:

The answer is that a subclass is guaranteed to have the same
*interface* as the base class. And that's what matters.

This is false. For instance:

class A(object):
def method(self, a):
print a

class B(A):
def method(self, a, b):
print a, b

B implements a different interface than A. Statically typed OO
languages either use multi-methods or disallow changing the signature
of an overridden method.

A tool to detect such cases would probably be almost as useful as the
tool you've proposed.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Fredrik Lundh]

Noam Raphael wrote:

Oh, and another thing - maybe "abstract" is a better name than "notimplemented"? notimplemented
might suggest a method which doesn't have to be implemented - and raises NotImplementedError when
it is called. What do you think?

what's the difference? no, really?

</F>

[Jeff Shannon]

Fredrik Lundh wrote:

Noam Raphael wrote:

Oh, and another thing - maybe "abstract" is a better name than "notimplemented"? notimplemented
might suggest a method which doesn't have to be implemented - and raises NotImplementedError when
it is called. What do you think?

what's the difference? no, really?

The difference would be that a derived class *must* override an
"abstract" method, but has the choice of leaving a "notimplemented"
method unimplemented -- one is obligate, the other optional.

Personally, all of this seems like more bother than it's worth to me.
I'm perfectly willing to "struggle" along without abstract base classes
enforcing an interface, and merely make do with unenforced, informal
protocols... but then, I'm hardly an expert in such matters. (Or any
other sort of matters, either.) ::shrug::

Jeff Shannon
Technician/Programmer
Credit International

[Noam Raphael]

Mike Meyer wrote:

Noam Raphael <no***@remove.the.dot.myrea.lbox.com> writes:

The answer is that a subclass is guaranteed to have the same
*interface* as the base class. And that's what matters.

This is false. For instance:

class A(object):
def method(self, a):
print a

class B(A):
def method(self, a, b):
print a, b

B implements a different interface than A. Statically typed OO
languages either use multi-methods or disallow changing the signature
of an overridden method.

A tool to detect such cases would probably be almost as useful as the
tool you've proposed.

<mike

I agree that such a tool would be very useful. In fact, I think it
exists - I'm sure pychecker checks for mistakes like that. I understand
that it checks for not implementing functions which just raise an
exception too, so you can say, "why all this mess? Run pychecker and
everything will be good." However, I think that there is a difference
between these two tests, which explains why one should be done by the
language itself and one should be done by an analysis tool.

The difference is that handling arguments, in Python, can be seen as a
part of the *implementation*, not the interface. The reason is that you
can write a method which simply gets a (*args, **kwargs), and raises a
TypeError if the number of args isn't two, and it would be completely
equivalent to a function which is defined using def f(a, b). Of course,
even in statically typed languages, you can't enforce an implementation
to do what it should (too bad - it would have made debugging so much
easier...)

So checking whether the argument list of a method of a subclass suits
the argument list of the original implementation is nice, but should be
left to external analysis tools, but checking whether a method is
defined at all can be done easily by the language itself.

Noam

[Alex Martelli]

Noam Raphael <no***@remove.the.dot.myrea.lbox.com> wrote:

However, I'm not sure if this solves my practical problem - testing
whether all abstract methods were implemented. I think that usually, you
can't write a test which checks whether an abstract method did what it
should have, since different implementations do different things. I
Still, there must be some _protocol_ about what those abstract methods
are all about... otherwise you _are_ just talking about an *interface*,
the way I use the terms (I didn't invent that usage, I just find it very
useful in practice). An interface is "just syntax" -- method names and
signatures; a protocol adds semantics and pragmatics (the distinction
between the last two may only be meaningful to somebody with a
background in linguistics, and it's not important here anyway).

If some method is a hook which may do anything including nothing then
it's most often more useful not to make it abstract (or absent) in the
base class, but rather to define it there as a no-op. This way, you
reduce the boilerplate in all subclasses: they only override the methods
they NEED to override. Besides no-op, this could also apply to other
semantics that were "a very likely possibility" -- but it's unusual for
any given semantics to be "very likely", except "first, do no harm";-)

When you design a protocol, _make it testable_. Say that you start with
the idea of the protocol having two methods:

def frambozzle(self):
''' must make the instance frambozzled '''
def unframbozzle(self):
''' must make the instance not-frambozzled '''

This protocol is not complete. Add a third method:

def isframbozzled(self):
''' return a true value iff the instance is frambozzled '''

NOW you have a testable protocol! Sorry for the toy-ness of the
example, but I don't want to post for hundreds of lines. Beyond the
direct advantages of being able to write much better tests, experience
teaches that a testable protocol tends to have extra resilience and
usefulness for perfectly reasonable uses that the protocol designer
might not have anticipated. E.g., with the testable protocol one can
write

def toggle_frambozzling(some_instance):
if some_instance.isframbozzled():
some_instance.unframbozzle()
else:
some_instance.frambozzle()

Note that we're only using semantics here, aka abstract semantics, not
pragmatics (just to illuminate the semi-relevant distinction I was using
above). Abstract semantics are those you can fully capture by axioms
related to methods that are part of the protocol itself, without for
example needing to refer to the external world, or other objects. You
get pragmatics when such references are there, e.g., informal specs of
the kind "the shipping costs of a frambozzled instance are expected to
not exceed that of an unframbozzled instance, and normally to be below;
on the other hand, frambozzled instances need only be marshalable on a
single platform and version of Python and of this framework, while
unframbozzled instances must ensure marshaling cross-platform on any
version of Python >= 2.3.4 and version of this framework >= 0.8.2". In
practice, only projects with relatively high formality tend to write
things down to this level, of course, but generally there is some less
formal pragmatic understanding of what it's all about.

For example, in a recent thread about hashing &c I pointed out that
adding a method such as "def __hash__(self): return 23" can ensure
semantically correct hashing of any class; OTOH, it DOES badly violate
pragmatics (and produces horrid performance!-), because hash values are
MEANT to "differentiate" among different instances (they don't _hafta_,
but they're expected to do SOME level of reasonable effort!-).

don't even know how you can test whether an abstract method was
implemented - should you run it and see if it raises a
NotImplementedError?
Makes sense to me, yes. assertRaises seems good;-).
But with what arguments?
If you're going to use your abstract base class for ANY purpose, you'd
better have SOME idea of the signature with which you can call those
methods which its subclasses must implement! Otherwise, it's all for
naught: you get zero polymorphism, period. Say that subclass A has
def whoa(self): ...
while subclass B has
def whoa(self, andmore): ...
then you can NEVER call whoa without knowing which subclass you have an
instance of -- no polymorphism! It's fine if a subclass wants to have a
WIDER signature (add OPTIONAL args to those specified in the base class,
even a *any if you wish), but there must be SOME commonality. Well
then, call according to that commonality, just like any USE of you
abstract base class will do.

This points out that, if you wish to do checks at 'class' statement time
which determine whether a class is instantiable, the signature of the
methods should be part of those checks. Fortunately, module inspect
makes it reasonably easy to get and check methods' signatures.
And even if you find a way
to test whether a method was implemented, I still think that the
duplication of code isn't very nice - you have both in your class
definition and in your test suite a section which says only "method
so-and-so should be implemented."
Then it may be best to ONLY have the test suite -- don't put the method
in the ABC at all. If you WANT the method in the ABC, for documentation
purposes, well then, that's not duplication of code, it's documentation,
which IS fine (just like it's quite OK to have some of the same info in
a Tutorial document, in a Reference one, AND in a class's docstring!).

If you don't want to have the duplication your unit tests become easier:
you just getattr from the class (don't even have to bother instantiating
it, ain't it great!), and check the result with inspect.
I think that making abstract methods a different object really makes
sense - they are just something else. Functions (and methods) define
what the computer should do. Abstract methods define what the
*programmer* should do.

Unit tests define what the programmer should do, to a large extent, much
better than other approaches do (until it comes to the integration of
components, but then, abstract methods ain't much use there either;
tests of other ilks may be preferable;-).
ITRW (in the real world) you MAY find yourself in situations where you
know unit-tests WILL be skimped on. Then you're in crisis mode from the
word go, and taking what little precautions you can (including custom
metaclasses that in practice won't be bypassed and can enforce a tiny
sliver of sanity in what will inevitably remain a crazy environment) may
be among the band-aids you can try. Besides, part of your investment in
building that infrastructure does get repaid in getting some errors
diagnosed earlier (though not as thoroughly and early as unit tests
might, but, we're starting from the sad RW assumption that _some_ deuced
coders won't do them, and we're just trying to reduce "totally dead" to
"mostly dead"...;-).
Alex

[Alex Martelli]

Mike Meyer <mw*@mired.org> wrote:

Noam Raphael <no***@remove.the.dot.myrea.lbox.com> writes:

The answer is that a subclass is guaranteed to have the same
*interface* as the base class. And that's what matters.

This is false. For instance:

class A(object):
def method(self, a):
print a

class B(A):
def method(self, a, b):
print a, b

B implements a different interface than A. Statically typed OO
languages either use multi-methods or disallow changing the signature
of an overridden method.

A tool to detect such cases would probably be almost as useful as the
tool you've proposed.

Agreed, and besides, the two tools can usefully be one. Look at the
inspect module for signature checking helpers...
Alex

[Mike Meyer]

Noam Raphael <no***@remove.the.dot.myrea.lbox.com> writes:

Noam Raphael <no***@remove.the.dot.myrea.lbox.com> writes:

Of course, even in statically typed languages, you can't enforce
an implementation to do what it should (too bad - it would have made
debugging so much easier...)

That's what DbC languages are for. You write the contracts first, then
the code to fullfill them. And get exceptions when the implementation
doesn't do what the contract claims it does.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Steve Holden]

Alex Martelli wrote:

[...]

Sorry for the toy-ness of the
example, but I don't want to post for hundreds of lines.

[...]

Can we take it this is some kind of early New Year's resolution?

happy-christmas-ly y'rs - steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC [phone] [phone]

[Noam Raphael]

Mike Meyer wrote:

That's what DbC languages are for. You write the contracts first, then
the code to fullfill them. And get exceptions when the implementation
doesn't do what the contract claims it does.

<mike

Can you give me a name of one of them? This is a very interesting thing
- I should learn one of those sometime. However, I'm pretty sure that
programming in them is hell, or at least, takes a very long time.

Noam

[Noam Raphael]

Thank you very much for this answer! I learned from you about unit
tests, and you convinced me that "testing oriented programming" is a
great way to program.

You made me understand that indeed, proper unit testing solves my
practical problem - how to make sure that all the methods which should
be implemented were implemented. However, I'm still convinced that this
feature should be added to Python, for what may be called "aesthetic
reasons" - I came to think that it fills a gap in Python's "logic", and
is not really an additional, optional feature. And, of course, there are
practical advantages to adding it.

The reason why this feature is missing, is that Python supports building
a class hierarchy. And, even in this dynamically-typed language, the
fact that B is a subclass of A means that B is supposed to implement the
interface of A. If you want to arrange in a class hierarchy a set of
classes, which all implement the same interface but don't have a common
concrete class, you reach the concept of an "abstract class", which
can't be instantiated. And the basestring class is exactly that.

The current Python doesn't really support this concept. You can write in
the __new__ of such a class something like "if cls == MyAbstractClass:
raise TypeError", but I consider this as a patch - for example, if you
have a subclass of this class which is abstract too, you'll have to
write this exception code again. Before introducing another problem, let
me quote Alex:

... If you WANT the method in the ABC, for documentation
purposes, well then, that's not duplication of code, it's documentation,
which IS fine (just like it's quite OK to have some of the same info in
a Tutorial document, in a Reference one, AND in a class's docstring!).

If you don't want to have the duplication your unit tests become easier:
you just getattr from the class (don't even have to bother instantiating
it, ain't it great!), and check the result with inspect.

That's actually right - listing a method which should be implemented by
subclasses, in the class definition is mainly a matter of
*documentation*. I like the idea that good documentation can be derived
from my documented code automatically, and even if I provide an external
documentation, the idea that the code should explain itself is a good
one. The problem is, that the current convention is not a good
documentation:

def frambozzle(self):
''' must make the instance frambozzled '''
raise NotImplementedError

The basic problem is that, if you take this basic structure, it already
means another thing: This is a method, which takes no arguments and
raises a NotImplementedError. This may mean, by convention, that this
method must be implemented by subclasses, but it may also mean that this
method *may* be implemented by subclasses. I claim that a declaration
that a method must be implemented by subclass is simply not a method,
and since Python's "logic" does lead to this kind of thing, it should
supply this object (I think it should be the class "abstract"). Two of
Python's principles are "explicit is better than implicit", and "there
should be (only?) one obvious way to do it". Well, I think that this:

@abstract
def frambozzle(self):
"""Must make the instance frambozzled"""
pass

is better than the previous example, and from

def frambozzle(self):
raise NotImplementedError, "You must implemented this method, and
it must make the instance frambozzled"

and from

def frambozzle(self):
"""Must make the instance frambozzled.

PURE VIRTUAL
"""
pass

and from maybe other possible conventions. Note also that making this
explicit will help you write your tests, even if Python would allow
instantiation of classes which contain abstract methods - you will be
able to simple test "assert not isinstance(MyClass.frambozzle, abstract)".
(I don't like the solution of not mentioning the method at all, which
makes the test equally simple, because it doesn't document what the
method should do in the class definition, and I do like in-code
documentation.)

To summarize, I think that this feature should be added to Python
because currently, there's no proper way to write some code which fits
the "Python way". As a bonus, it will help you find errors even when
your unit tests are not sufficient.

I plan to raise this issue in python-dev. If you have any additional
comments, please post them here. (I will probably be able to reply only
by the weekend.)

Have a good day,
Noam

[Mike Meyer]

Noam Raphael <no***@remove.the.dot.myrea.lbox.com> writes:

Mike Meyer wrote:

That's what DbC languages are for. You write the contracts first,
then
the code to fullfill them. And get exceptions when the implementation
doesn't do what the contract claims it does.
<mike

Can you give me a name of one of them? This is a very interesting
thing - I should learn one of those sometime. However, I'm pretty sure
that programming in them is hell, or at least, takes a very long time.

Eiffel. Google for SmartEiffel for a free portable implementation. If
you want an IDE/etc., look for EiffelStudio - though it's only
available in binary form. And the two implement different languages
(EiffelStudio implements things accepted as standard for OOSCv3,
whereas SmartEiffel implements things that are still under
consideration in their own variants, and fails to implement some
critical features), though the DbC parts are identical.

I find Eiffel fun to program in. SmartEiffel's libraries are as
flexible as Python's built in types. And debugging with all the
contract checking turned on causes exceptions for truly obscure bugs.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Mike Meyer]

Noam Raphael <no***@remove.the.dot.myrea.lbox.com> writes:

The current Python doesn't really support this concept. You can write
in the __new__ of such a class something like "if cls ==
MyAbstractClass: raise TypeError", but I consider this as a patch -
for example, if you have a subclass of this class which is abstract
too, you'll have to write this exception code again. Before
introducing another problem, let me quote Alex:

You can do *much* better than that. You can finger the class that the
object is an instance of. You don't need a new language construct to
do it, either:

class A: .. def virtualmethod(self):
.. raise NotImplementedError("virtualmethod not implemented in class %s" \
% self.__class__.__name__)
.. class B(A): pass
.. b = B()
b.virtualmethod() Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 3, in virtualmethod
NotImplementedError: virtualmethod not implemented in class B

I think that error message pinpoints the class of the instance that's
missing the virtual method, which is as good as your proposal would
do, adequately documents that this method needs to be implemented by
subclasses, and has the advantage that you can instantiate only part
of the interface to an abstract class if you only need part of it.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Alex Martelli]

Noam Raphael <no***@remove.the.dot.myrea.lbox.com> wrote:

That's what DbC languages are for. You write the contracts first, then
the code to fullfill them. And get exceptions when the implementation
doesn't do what the contract claims it does.

Can you give me a name of one of them? This is a very interesting thing
- I should learn one of those sometime. However, I'm pretty sure that
programming in them is hell, or at least, takes a very long time.

If you want to experiment with DbC within Python, I think you might want
to try <http://www.pypackage.org/packages/python-dbc> -- not sure it's
maintained and up-to-date with today's Python, but I suspect the answer
is "yes", and it would give you a chance to try DbC as a methodology,
quite separately from the issue of changing programming languages.
Alex