How about "pure virtual methods"?

[Noam Raphael]

Hello,

I thought about a new Python feature. Please tell me what you think
about it.

Say you want to write a base class with some unimplemented methods, that
subclasses must implement (or maybe even just declare an interface, with
no methods implemented). Right now, you don't really have a way to do
it. You can leave the methods with a "pass", or raise a
NotImplementedE rror, but even in the best solution that I know of,
there's now way to check if a subclass has implemented all the required
methods without running it and testing if it works. Another problem with
the existing solutions is that raising NotImplementedE rror usually means
"This method might be implemented some time", and not "you must
implement this method when you subclass me".

What I suggest is a new class, called notimplemented (you may suggest a
better name). It would get a function in its constructor, and would just
save a reference to it. The trick is that when a new type (a subclass of
the default type object) is created, It will go over all its members and
check to see if any of them is a notimplemented instance. If that is the
case, it would not allow an instantiation of itself.

What I want is that if I have this module:

=============== =======

class BaseClass(objec t):
def __init__(self):
...

@notimplemented
def save_data(self, filename):
"""This method should save the internal state of the class to
a file named filename.
"""
pass

class RealClass(BaseC lass):
def save_data(self, filename):
open(filename). write(self.data )

=============== =======

then if I try to instantiate BaseClass I would get an exception, but
instantiating RealClass will be ok.
Well, what do you say?

Noam Raphael

[Erik Max Francis]

Noam Raphael wrote:

Well, what do you say?

Raising NotImplementedE rror in the methods that need to be overridden is
much more customary, more straightforward , and already works fine.

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
We learn from history that we do not learn from history.
-- Georg Friedrich Wilhelm Hegel

[Alex Martelli]

Noam Raphael <no***@remove.t he.dot.myrea.lb ox.com> wrote:

What I want is that if I have this module:

=============== =======

class BaseClass(objec t):
def __init__(self):
...

@notimplemented
def save_data(self, filename):
"""This method should save the internal state of the class to
a file named filename.
"""
pass

class RealClass(BaseC lass):
def save_data(self, filename):
open(filename). write(self.data )

=============== =======

then if I try to instantiate BaseClass I would get an exception, but
instantiating RealClass will be ok.
Well, what do you say?

It's already easy to do as long as you don't insist that BaseClass must
subclass object (or allow a __metaclass__ assignment there). Basically,
what you describe is the job of a custom metaclass, because that's where
the __call__ method on classes is defined (classes being instances of
the metaclass)... much cleaner than tampering with the __new__ or
__init__ at class level.

At sketch level, you'd have, say:

def notimplemented( f):
# save some data about f somewhere if you wish, then, e.g.:
return notimplemented

import inspect

class Meta_for_NR(typ e):
def __init__(cls, cn, cb, cd):
super(Meta_for_ NR, cls).__init__(c n, cb, cd)
abstract_method s = []
for n, v in inspect.getmemb ers(cls, inspect.ismetho d):
if v is notimplemented: abstract_method s.append(n)
cls._abstract_m ethods = abstract_method s
def __call__(cls, *a, **k):
if cls._abstract_m ethods:
raise TypeError, ("Cannot instantiate abstract class %s."
" Abstract methods: %s." % (cls.__name__,
', '.join(cls._abs tract_methods)) )
return super(Meta_for_ NR, cls).__call__(* a, **k)

class Base_for_NR: __metaclass__ = Meta_for_NR
Inheriting from Base_for_NR instead of inheriting from object should
give you just the behavior you desire (net of the fact that this code of
mine is untested, just a sketch -- but the key ideas shd be ok).
If you can implement this and get it into wide usage, it may well be
adopted in the standard library. Unless there is proof by experience
that a lot of people like this approach better than the de facto
standard, such adoption is very unlikely. But the great thing is that
it's EASY to implement your ideas and try to garner a following for
them, so you stand a chance to get them into the standard library if you
can prove they really work well and are well-liked!
Alex

[John Machin]

Alex Martelli wrote:

class Meta_for_NR(typ e):
def __init__(cls, cn, cb, cd):
super(Meta_for_ NR, cls).__init__(c n, cb, cd)
abstract_method s = []
for n, v in inspect.getmemb ers(cls, inspect.ismetho d):
if v is notimplemented: abstract_method s.append(n)

etc etc etc

I'm missing the point of the question and Alex's answer. Let's step
through this:

1. Programmer A designs a base class some (or all) of whose methods are
abstract i.e. not implemented in a do-something-useful fashion; they
may be only stubs which document their arguments.

2. Programmer B designs a sub-class, which *should* implement methods
which override those of the base class.

3. Programmer C writes an application which *should* create instances
of the sub-class.

4. Things that can go wrong are (a) C instantiates the base class (b) B
has not provided all the methods (c) B has provided methods but they
don't work properly (d) C's app doesn't invoke B's methods in the
approved manner or sequence or with meaningful arguments or ...

5. The standard approach in Python is to have the virtual methods raise
an exception if invoked. This of course doesn't help with problems c
and d. In an attempt to root out instances of problems c and d,
programmer C will unit-test the @#$% out of the app. This will quickly
discover any instances of problems a and b. One would hope that as well
B has unit-tested the @#$% out of the sub-class.

6. Alex's magnum opus appears to operate on problem a, and maybe on b
(it's over my head). It involves a fair chunk of mucking about -- for
what? Early warning, a few microseconds ahead of the invocation of a
method which will cause the stub in the base class to raise an
exception?

[Nick Coghlan]

John Machin wrote:

6. Alex's magnum opus appears to operate on problem a, and maybe on b
(it's over my head). It involves a fair chunk of mucking about -- for
what? Early warning, a few microseconds ahead of the invocation of a
method which will cause the stub in the base class to raise an
exception?

Alex eats metaclasses for breakfast though, so while it may look like a magnum
opus to mere mortals, it is in reality nothing more than an idle twiddle :)

Anyway, I see two major problems with the OP's idea:

1. It encourages laziness when subclassing
Subclassing necessarily involves intimate coupling of code. If you don't
understand the base class well enough to know which methods you are required to
implement, and haven't even included those methods in your unit tests. . . then
I'm willing to bet those missing methods are the least of your worries when it
comes to correct operation of your class.

2. It gives up some of the dynamism of Python
The OP's idea denies the inheritor the ability to partially implement the
defined interface. Sometimes, particularly for objects used for a very specific
purpose, it makes sense to only implement the portion of an interface that you
want to use. The standard method makes it easy to flag a portion of the
interface as off limits - just leave it raising a Not Implemented exception.
With the OP's idea in place, the inheritor is forced to implement the entirety
of the interface, even if they only need a fraction of it.

Cheers,
Nick.

--
Nick Coghlan | nc******@email. com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

[Alex Martelli]

John Machin <sj******@lexic on.net> wrote:
...

6. Alex's magnum opus appears to operate on problem a, and maybe on b
(it's over my head). It involves a fair chunk of mucking about -- for
A dozen lines of code to write *ONCE* and put in your utilities module
is a "magnum opus" and "a fair chunk of mucking about"?! Oh my -- I
wonder what would count for you as parvus and incredibly tiny -- SIX
lines? THREE? ONE AND A HALF, for Pete's sake?!?!?!

If your problem with my code is simply that it uses parts of Python that
you're not familiar about, then I count this as a positive aspect of my
code, not a negative one: by providing motivation to study metaclasses
and inspect (by givin some small usefulness in very small space) it may
help you indirectly far more than directly.
what? Early warning, a few microseconds ahead of the invocation of a
method which will cause the stub in the base class to raise an
exception?

Exactly. Microseconds don't count, but stack levels do -- getting the
traceback end as close as possible to the real CAUSE of the problem,
rather than a few levels further down where the SYMPTOM appears, can
easily shave whole minutes off debugging time. Since the whole of what
you call "magnum opus" took about 4 minutes to write, if it acts this
way twice I'm gaining.

Indeed, this issue is important enough that it might be worth one's
while to ensure the exception comes at class definition rather than
instantiation time -- e.g. by saying that a class must explicitly expose
(not just inherit) a class attribute '__abstract__' that's True, or else
having abstract methods still present is an error. This obviously takes
a trivial mod to the (ha!) "magnum" opus, and makes errors even easier
to diagnose. Still, it wasn't among the OP's desiderata, so I didn't
change the semantics in this way.
Note that either or both changes work particularly well wrt UNIT tests,
tests of some component which sees the abstract base class as 'external'
and thus not subject to its unit tests. Say that 'external' component X
exposes ABC, the abstract base class, and many functions f, g, h, ...,
which take an ABC instance and do things with it. My own component Y
subclasses ABC with WCC, the wannabe concrete class, and passes an
instance of WCC to *SOME* of X.f, X.g, X.h -- SOME, not necessarily ALL,
because UNIT tests don't have to fully probe interfaces BETWEEN
components!!! (that's why they're UNIT tests, not INTEGRATION ones;-).

With Python's standard approach, no guarantee that WCC implements all
the methods it should. Maybe it did implement all methods needed at the
time it was written, then X release 1.1 added a couple abstract methods
to ABC and a couple new functions j and k using them, and Y knows
nothing about this upgrade. Y's unit tests are still gonna pass. And
somebody ELSE who's instantiating a Y.WCC and passing the instance to
X.k is gonna feel the pain -- not in their unit tests, mind you, this
has VERY MUCH become a problem of inter-component orchestration.

With the cost of 12 lines, ONCE, in a utility module, you ensure halfway
decent unit tests WILL expose all 'failed to implement needed methods'
errors, even as X's specs keep evolving. If you make any substantial
use of abstract classes and methods, the tradeoff is VERY favourable.
So, do _I_ use this? Nah: I think abstract classes with
explicitly-abstract "somebody's gotta implement this!" methods are
rarely a good idea and tend to lead to fat base classes and overuse of
inheritance. But if one disagrees, then I think that Python's suppport
for them with NotImplementedE rror is suboptimal, and some variation and
evolution along the lines of the tiny sketch I gave would be better.
Alex

[Fredrik Lundh]

Alex Martelli:

what? Early warning, a few microseconds ahead of the invocation of a
method which will cause the stub in the base class to raise an
exception?
Exactly. Microseconds don't count, but stack levels do -- getting the
traceback end as close as possible to the real CAUSE of the problem,
rather than a few levels further down where the SYMPTOM appears, can
easily shave whole minutes off debugging time.

minutes?

Traceback (most recent call last):
File "mymodule.p y", line 9, in somefunction
someobj.bar()
... zero or more lines ...
File "somelibrary.py ", line 3, in bar
raise NotImplementedE rror("must implement abstract method bar")
NotImplementedE rror: must implement abstract method bar

you have to look at the last line of a traceback to find the error message, and
the offending module/method is two lines above that.
So, do _I_ use this? Nah: I think abstract classes with
explicitly-abstract "somebody's gotta implement this!" methods are
rarely a good idea

yet you claim to be an expert on issues related to debugging NotImplemented-
Error exceptions?

</F>

[Alex Martelli]

Fredrik Lundh <fr*****@python ware.com> wrote:

Alex Martelli:

what? Early warning, a few microseconds ahead of the invocation of a
method which will cause the stub in the base class to raise an
exception?

Exactly. Microseconds don't count, but stack levels do -- getting the
traceback end as close as possible to the real CAUSE of the problem,
rather than a few levels further down where the SYMPTOM appears, can
easily shave whole minutes off debugging time.

minutes?

Traceback (most recent call last):
File "mymodule.p y", line 9, in somefunction
someobj.bar()
... zero or more lines ...
File "somelibrary.py ", line 3, in bar
raise NotImplementedE rror("must implement abstract method bar")
NotImplementedE rror: must implement abstract method bar

you have to look at the last line of a traceback to find the error
message, and the offending module/method is two lines above that.

The offending module is the one which instantiates a class it shouldn't
instantiate -- a class which isn't made concrete by overriding all
methods which need to be. What is that module? What is that class?
The error is generally not the calling of 'bar' on someobj: someobj
arrives as a argument to somefunction from somewhere -- the caller of
somefunction is quite often innocent, in turn -- who BUILT someobj, by
instantiating which class? I think `minutes' is a very fair (not at all
pessimistic) assessment of the time it will generally take to find out.

With an exception at the time of instantiation, you shave those minutes
off. As I already explained: if it was acceptable to raise at the time
the class statement is executed, which doesn't make the class fully
concrete yet fails to explicitly assert __abstract__=Tr ue in classbody,
the distance between where the error is in the code, and where it's
diagnosed, could be further reduced in many cases; sometimes the error
is to mistakenly instantiate a class which is not yet meant to be fully
concrete, but equally well it could be to fail to override all needed
methods in a class which IS meant to be fully concrete. However, this
further refinement turns out to be generally of lesser impact wrt the
primary one: as long as the error msg at instantiation time fully
identifies what class is involved and why (because of what still
abstract methods) it's an error to instantiate it, then the productivity
plus of getting the error at class-statement time vs instantiation time
is generally no big deal -- the big deal is not having to wait until,
somewhere, somehow, a still-abstract method finally gets _called_, and
then having to trace back to how and where that class was instantiated.

So, do _I_ use this? Nah: I think abstract classes with
explicitly-abstract "somebody's gotta implement this!" methods are
rarely a good idea

yet you claim to be an expert on issues related to debugging NotImplemented-
Error exceptions?

Yes, the body of experience which essentially convinced me to avoid the
above idiom (because, as I say in the snippet I quote here, it's rarely
a good idea -- you snipped my summary of reasons why) essentially
overlaps with the body of experience which suggests to me that IF
somebody's keen to use the idiom widely anyway, they're going to be far
better off with the errors it's meant to catch causing errors to be
raised ASAP, as the custom metaclass so easily allows, rather than as
late as possible, as using 'raise NotImplementedE rror' leads to.

[[ Protocols (also unfortunately known as interfaces) and adaptation
work much better, essentially by not depending on inheritance. ]]

I really don't understand what's so difficult here, for you, as to make
it apparently so very hard for you to follow. The idiom (meant to catch
some wrong uses of inheritance) of 'abstract classes with certain
methods that need to get overridden' isn't all that helpful in Python
when all that happens is a NotImplementedE rror exception. If the base
class had simply omitted defining the method, as used to be the idiom,
you'd generally get an AttributeError instead -- slightly more generic
but happening basically at the same spot in the code, time difference in
finding the errors usually a few seconds. With a custom metaclass (and
optionally a decorator, though there are other ways, which we can
discuss in detail if you think such distinctions are oh so crucial) you
can easily make the idiom more helpful for the purpose of diagnosing
errors in inheritance &c -- i.e., the idiom's sole purpose.

Nevertheless, even when one has made the idiom as good as it can be, I
think it's still rarely a good idea. Protocols and adaptation take a
fresh approach, not as heavily influenced by C++ and Java limitations as
the idiom we're discussing here, and it's my contention that said
approach does a better job. One can perfectly well still use
inheritance for what inheritance does best (easy, simple, fast sharing
of some implementation) without depending on it for those protocol
compliance issues which were never its forte.
Alex

[Fredrik Lundh]

Alex Martelli wrote:

Traceback (most recent call last):
File "mymodule.p y", line 9, in somefunction
someobj.bar()
... zero or more lines ...
File "somelibrary.py ", line 3, in bar
raise NotImplementedE rror("must implement abstract method bar")
NotImplementedE rror: must implement abstract method bar

you have to look at the last line of a traceback to find the error
message, and the offending module/method is two lines above that.
The offending module is the one which instantiates a class it shouldn't
instantiate -- a class which isn't made concrete by overriding all
methods which need to be. What is that module?

in real life, it's mymodule.py or mylib.py, in code that you wrote very recently.
The error is generally not the calling of 'bar' on someobj: someobj
arrives as a argument to somefunction from somewhere -- the caller of
somefunction is quite often innocent, in turn -- who BUILT someobj, by
instantiating which class? I think `minutes' is a very fair (not at all
pessimistic) assessment of the time it will generally take to find out.

you can blather for years if you want to, but your arguments are just another variant
of the "you cannot use whitespace for indentation" stuff we've all seen from time to
time; it's trivial to come up with a large number of theoretical problems with any given
approach, but in real life, for real-life programmers, using real-life libraries designed
by real-life developers, it's simply not the problem you want it to be.

but not that I'm going to convince you, now that you've spent 15,000 characters on
this topic...

</F>

[Steve Holden]

Noam Raphael wrote:

Hello,

I thought about a new Python feature. Please tell me what you think
about it.

Say you want to write a base class with some unimplemented methods, that
subclasses must implement (or maybe even just declare an interface, with
no methods implemented). Right now, you don't really have a way to do
it. You can leave the methods with a "pass", or raise a
NotImplementedE rror, but even in the best solution that I know of,
there's now way to check if a subclass has implemented all the required
methods without running it and testing if it works. Another problem with
the existing solutions is that raising NotImplementedE rror usually means
"This method might be implemented some time", and not "you must
implement this method when you subclass me".

What I suggest is a new class, called notimplemented (you may suggest a
better name). It would get a function in its constructor, and would just
save a reference to it. The trick is that when a new type (a subclass of
the default type object) is created, It will go over all its members and
check to see if any of them is a notimplemented instance. If that is the
case, it would not allow an instantiation of itself.

What I want is that if I have this module:

=============== =======

class BaseClass(objec t):
def __init__(self):
...

@notimplemented
def save_data(self, filename):
"""This method should save the internal state of the class to
a file named filename.
"""
pass

class RealClass(BaseC lass):
def save_data(self, filename):
open(filename). write(self.data )

=============== =======

then if I try to instantiate BaseClass I would get an exception, but
instantiating RealClass will be ok.
Well, what do you say?

Noam Raphael

OK, you've had many replies telling you why your suggestion isn't
particularly worthwhile, and a few telling you how you can already do
this in present-day Python.

Even if you can do it, how would you then implement a class hierarchy
where the ultimate base class had virtual methods, and you wanted to
derive from that class another class, to be used as a base class for
usable classes, which implemented only a subset of the virtual methods,
leaving the others to be implemented by the ultimate subclasses?

This use case makes your suggestion look impractical, methinks, since it
requires that the programmer do what you would apparently like to
specifically forbid.

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC [phone] [phone]