multiple inheritance of a dynamic list of classes?

[devicerandom]

Hi,

I am currently using the Cmd module for a mixed cli+gui application. I
am starting to refactor my code and it would be highly desirable if
many commands could be built as simple plugins.

My idea was:
- Load a list of plugin names (i.e. from the config file, or from the
plugins directory)
- Import all plugins found dynamically:
and this is easy, since I can do, for example:

PLUGIN_NAMES=['foo', 'bar']
PLUGIN_MODULES = map(__import__, PLUGIN_NAMES)
PLUGINS = [item.Commands for item in PLUGIN_MODULES]

Now, what I have to do is to define my command line class. This is
usually done by subclassing cmd.Cmd:

class MyCli(cmd.Cmd):
....

Now I want to add the commands defined in foo.Commands and
bar.Commands. foo.Commands contains the functions corresponding to the
new commands this way:
#foo.py
class Commands

def do_this(self,args):
...
def do_that(self,args):
...

I've seen I can do it by explicitely import them and using multiple
inheritance:

class MyCli(cmd.Cmd , foo.Commands, bar.Commands)
....

so that do_this and do_that are now methods of a Cmd command line.

Now:
- how can I instead have MyCli inherit from a dynamic list of modules?
- is there a better way than using multiple inheritance to plug-in
dynamically commands in a Cmd command line?

I hope it's all clear.

Thanks,
Massimo

[Peter Otten]

de**********@gmail.com wrote:

I am currently using the Cmd module for a mixed cli+gui application. I
am starting to refactor my code and it would be highly desirable if
many commands could be built as simple plugins.

My idea was:
- Load a list of plugin names (i.e. from the config file, or from the
plugins directory)
- Import all plugins found dynamically:
and this is easy, since I can do, for example:

PLUGIN_NAMES=['foo', 'bar']
PLUGIN_MODULES = map(__import__, PLUGIN_NAMES)
PLUGINS = [item.Commands for item in PLUGIN_MODULES]

Now, what I have to do is to define my command line class. This is
usually done by subclassing cmd.Cmd:

class MyCli(cmd.Cmd):
....

Now I want to add the commands defined in foo.Commands and
bar.Commands. foo.Commands contains the functions corresponding to the
new commands this way:
#foo.py
class Commands

def do_this(self,args):
...
def do_that(self,args):
...

I've seen I can do it by explicitely import them and using multiple
inheritance:

class MyCli(cmd.Cmd , foo.Commands, bar.Commands)
....

so that do_this and do_that are now methods of a Cmd command line.

Now:
- how can I instead have MyCli inherit from a dynamic list of modules?
- is there a better way than using multiple inheritance to plug-in
dynamically commands in a Cmd command line?

>>import cmd
class Foo:

.... def do_this(self, arg):
.... print "THIS", arg
....

>>class Bar:

.... def do_that(self, arg):
.... print "THAT", arg
....

>>plugins = [Foo, Bar]
def make_command_class(*bases):

.... return type(cmd.Cmd)("MyCli", bases + (cmd.Cmd,), {})
....

>>cmd = make_command_class(*plugins)()
cmd.cmdloop()

(Cmd) help

Undocumented commands:
======================
help that this

(Cmd) that one
THAT one
(Cmd)

Most of the above should be straight-forward. I used type(cmd.Cmd)(name,
bases, classdict) instead of just type(name, bases, classdict) because
cmd.Cmd is a classic class.

The thread "Partial 1.0 - Partial classes for Python" might also be of
interest.

Peter

[devicerandom]

Most of the above should be straight-forward. I used type(cmd.Cmd)(name,
bases, classdict) instead of just type(name, bases, classdict) because
cmd.Cmd is a classic class.

It seems it works. It is not so straight-forward to me because I don't
know about new-style types and classes very well, but I'm looking at
type() and it makes full sense. I'll also look into new style classes.

Thanks a lot. Anyway, just for the sake of asking: since I have no
formal CS education (I'm a biophysics ph.d. student), I wonder if
there is some kind of design pattern/coding tool that could work
better/help me along simple inheritance for coding plugin
architectures (inheritance probably works well in this case, but I'd
like to know what's in the swiss army knife).

Thanks again,

Massimo

[Michele Simionato]

On Feb 12, 4:48 pm, [e-mail] wrote:

- is there a better way than using multiple inheritance to plug-in
dynamically commands in a Cmd command line?

I had to solve the same problem recently, and I decided to avoid
multiple
inheritance by using delegation. My idea was to make the Cmd class a
wrapper
around an object with 'do_' methods. I give you the code as it is,
without
much explanations: just run it and give "help" on the command line.
You can cut the part about the 'do_input' method which I needed in
order
to pass lists of strings to the inner object.
Hope you can find some useful trick for solving your problem.

Michele Simionato

import sys, cmd, traceback, inspect
try: # Python 2.5
from functools import update_wrapper
except ImportError:
def update_wrapper(wrapper,
wrapped,
assigned=('__module__', '__name__', '__doc__'),
updated=('__dict__',)):
for attr in assigned:
setattr(wrapper, attr, getattr(wrapped, attr))
for attr in updated:
getattr(wrapper, attr).update(getattr(wrapped, attr))
return wrapper

def makewrapper(meth):
if inspect.ismethod(meth):
func = meth.im_func
elif inspect.isfunction(meth):
func = meth
else:
raise TypeError('%r must be a regular function or method' %
meth)
return update_wrapper(lambda self, arg: meth(arg), func)

# dispatch the input dictionary to self.innerobj
def do_input(self, arg):
if arg.split(): # passed some argument of type 'name <args>'
try:
name, args = arg.split(' ', 1)
except ValueError:
print 'Wrong format: use the format input <name<values>'
else:
self.inputdict[name] = args.split(self.inputsep)
else:
self.innerobj.input(self.inputdict)
self.inputdict = {}

def copy_interface_methods(innerobj, _CLI):
for attr in dir(innerobj):
if attr.startswith('do_'):
setattr(_CLI, attr, makewrapper(getattr(innerobj, attr)))
elif attr == 'input':
do_input.__doc__ = innerobj.input.__doc__
_CLI.do_input = do_input

class CLI(cmd.Cmd, object):
"""
Wraps an object with 'do_' methods with a command line interface.
"""
def __new__(cls, innerobj, completekey='tab', stdin=None,
stdout=None,
nonblocking=False):
class _CLI(cls):
prompt = 'CLI'
if stdin is not None:
use_rawinput = False
copy_interface_methods(innerobj, _CLI)
return super(CLI, cls).__new__(
_CLI, innerobj, completekey, stdin, stdout)

def __init__(self, innerobj, completekey='tab', stdin=None,
stdout=None):
self.innerobj = innerobj
if hasattr(self, 'do_input'):
self.inputdict = {}
self.inputsep = '|'
cmd.Cmd.__init__(self, completekey, stdin, stdout)

def onecmd(self, line): # enable comments
if not line.startswith('#'):
return super(CLI, self).onecmd(line)

def emptyline(self): # does not repeat the last command
pass

def do_EOF(self, arg):
"Called when you enter CTRL-D, it stops the command loop"
return 1

if __name__ == '__main__': # test
class PrintableObj(object):
def __init__(self, x, y):
self.x = x
self.y = y
def do_print(self, arg):
"example"
print self.x, self.y
def input(self, dict_of_lists):
'just print the entered value'
vars(self).update(dict_of_lists)
print dict_of_lists

cli = CLI(PrintableObj([], []))
cli.cmdloop()

[Kent Johnson]

de**********@gmail.com wrote:

Hi,

I am currently using the Cmd module for a mixed cli+gui application. I
am starting to refactor my code and it would be highly desirable if
many commands could be built as simple plugins.

My idea was:
- Load a list of plugin names (i.e. from the config file, or from the
plugins directory)
- Import all plugins found dynamically:
and this is easy, since I can do, for example:

PLUGIN_NAMES=['foo', 'bar']
PLUGIN_MODULES = map(__import__, PLUGIN_NAMES)
PLUGINS = [item.Commands for item in PLUGIN_MODULES]

Now, what I have to do is to define my command line class. This is
usually done by subclassing cmd.Cmd:

class MyCli(cmd.Cmd):
....

Now I want to add the commands defined in foo.Commands and
bar.Commands. foo.Commands contains the functions corresponding to the
new commands this way:
#foo.py
class Commands

def do_this(self,args):
...
def do_that(self,args):
...

I've seen I can do it by explicitely import them and using multiple
inheritance:

class MyCli(cmd.Cmd , foo.Commands, bar.Commands)
....

so that do_this and do_that are now methods of a Cmd command line.

Now:
- how can I instead have MyCli inherit from a dynamic list of modules?
- is there a better way than using multiple inheritance to plug-in
dynamically commands in a Cmd command line?

Your plugins could define plain functions with names starting with do_.
Then you can create an empty subclass of cmd.Cmd and just plug in the
imported commands:

In [1]: import cmd

In [3]: def do_this(self, arg): print 'This', arg
...:

In [4]: def do_that(self, arg): print 'That', arg
...:
In [8]: class MyCmd(cmd.Cmd): pass
...:

In [9]: MyCmd.do_this = do_this

In [10]: MyCmd.do_that = do_that

In [11]: c=MyCmd()

In [12]: c.cmdloop()
(Cmd) help

Undocumented commands:
======================
help that this

(Cmd) that
That

In your code you could use introspection to locate the plugin commands,
something like
PLUGIN_MODULES = map(__import__, PLUGIN_NAMES)
for module in PLUGIN_MODULES:
for name in dir(module):
if name.startswith('do_'):
setattr(MyCmd, name, getattr(module, name))

If the plugin module defines a list of commands then use that instead of
dir(module).

Kent

[devicerandom]

Thanks both for suggestions. I still think that using inheritance is
somehow cleanest in this case (I always hear the mantra "avoid
multiple inheritance!", but this is one of the cases it seems to make
a lot of sense to me), but it's nice food for thought/code anyway.

Other suggestions are always welcome, if there are!

Massimo

[Peter Otten]

de**********@gmail.com wrote:

Thanks both for suggestions. I still think that using inheritance is
somehow cleanest in this case (I always hear the mantra "avoid
multiple inheritance!", but this is one of the cases it seems to make
a lot of sense to me), but it's nice food for thought/code anyway.

"Avoid inheritance" would be almost as justified :-)

Problems that may arise with this case of multiple inheritance:

- If you need initializers, ensure that they are all invoked
- What would you do about name clashes? To avoid them your plugins need to
know about each other.
- State (instance attributes) is shared among all your plugins. Since you
call all base classes Commands, Python's double-underscore hack won't work.

Peter

[devicerandom]

On 13 Feb, 09:14, Peter Otten <[e-mail]:

[e-mail] wrote:

Thanks both for suggestions. I still think that using inheritance is
somehow cleanest in this case (I always hear the mantra "avoid
multiple inheritance!", but this is one of the cases it seems to make
a lot of sense to me), but it's nice food for thought/code anyway.

"Avoid inheritance" would be almost as justified :-)

Why?

Problems that may arise with this case of multiple inheritance:

- If you need initializers, ensure that they are all invoked

Yes, I figured it out. This should be easy in this case.

- What would you do about name clashes? To avoid them your plugins need to
know about each other.

Yes, I know, but I can't see any simple solution to this (if you can,
please share it with me!). The cmd module works by interpreting any
method starting with "do_" as a command, so "do_blah" becomes the
"blah" command. If two people write a "do_blah" command, and both
plugins are used, I see no easy way to solve the issue (apart
rewriting a cmd module).
Perhaps there can be some previous sanity check in each modules dict
to see if there are obvious namespace clashings, and in this case
issue a warning. I don't know.

- State (instance attributes) is shared among all your plugins. Since you
call all base classes Commands, Python's double-underscore hack won't work.

What double-underscore hack are you referring to? (sigh, my python
limits are all arising...) I can call all base classes
PluginNameCommand, however, this wouldn't break the thing (I'm still
at the very early planning stage) and would maybe work.

m.

[Peter Otten]

de**********@gmail.com wrote:

On 13 Feb, 09:14, Peter Otten <[e-mail]:

>[e-mail] wrote:

Thanks both for suggestions. I still think that using inheritance is
somehow cleanest in this case (I always hear the mantra "avoid
multiple inheritance!", but this is one of the cases it seems to make
a lot of sense to me), but it's nice food for thought/code anyway.

"Avoid inheritance" would be almost as justified :-)

Why?

Well, what problems ocurring with

class A: pass
class B: pass
class C(A, B): pass

could be avoided by writing

class A: pass
class B(A): pass
class C(B): pass

instead? Classes have to be designed for subclassing, so essentially you get
two interfaces, one for subclasses and one for client code instead of just
the latter. A more relevant mantra governing inheritance is "Flat is better
than nested".

>Problems that may arise with this case of multiple inheritance:

- If you need initializers, ensure that they are all invoked

Yes, I figured it out. This should be easy in this case.

>- What would you do about name clashes? To avoid them your plugins need
to know about each other.

Yes, I know, but I can't see any simple solution to this (if you can,
please share it with me!). The cmd module works by interpreting any
method starting with "do_" as a command, so "do_blah" becomes the
"blah" command. If two people write a "do_blah" command, and both
plugins are used, I see no easy way to solve the issue (apart
rewriting a cmd module).
Perhaps there can be some previous sanity check in each modules dict
to see if there are obvious namespace clashings, and in this case
issue a warning. I don't know.

>- State (instance attributes) is shared among all your plugins. Since you
call all base classes Commands, Python's double-underscore hack won't
work.

What double-underscore hack are you referring to? (sigh, my python
limits are all arising...) I can call all base classes
PluginNameCommand, however, this wouldn't break the thing (I'm still
at the very early planning stage) and would maybe work.

If you use attributes starting with two underscores inside a method, Python
transparently prepends them with the class name. This allows to you to use
the same variable name in two base classes and reduces coupling:

>>class A:

.... def set_a(self, v): self.__value = v
....

>>class B:

.... def set_b(self, v): self.__value = v
....

>>class C(A, B): pass

....

>>c = C()
c.set_a("alpha"); c.set_b("beta")
vars(c)

{'_A__value': 'alpha', '_B__value': 'beta'}

But if two classes with the same name use the "private" variable, the
mechanism fails:

>>OldA = A
class A:

.... def set_b(self, v): self.__value = v
....

>>class C(A, OldA): pass

....

>>c = C()
c.set_a("alpha"); c.set_b("beta")
vars(c)

{'_A__value': 'beta'}

Peter

[Neil Cerutti]

On 2007-02-13, Peter Otten <__*******@web.dewrote:

Well, what problems ocurring with

class A: pass
class B: pass
class C(A, B): pass

could be avoided by writing

class A: pass
class B(A): pass
class C(B): pass

instead?

With multiple inheritance, the choice of algorithm for Method
Resolution Ordering isn't obvious or trivial. As I understand it,
Python got it wrong for quite some time, allowing "bad"
hierarchies to compile, i.e., classes for which the properties of
local precedence ordering and monotonicity did not hold.

URL:http://www.python.org/download/releases/2.3/mro/

--
Neil Cerutti

[massimo s.]

On 13 Feb, 12:46, Peter Otten <[e-mail]:

Well, what problems ocurring with

class A: pass
class B: pass
class C(A, B): pass

could be avoided by writing

class A: pass
class B(A): pass
class C(B): pass

instead? Classes have to be designed for subclassing, so essentially you get
two interfaces, one for subclasses and one for client code instead of just
the latter. A more relevant mantra governing inheritance is "Flat is better
than nested".

I am truly getting lost. Are you saying that doing A-->B(A)--C(B) is
better than C(A,B)? And isn't the former thing nested? Or are you
saying that C(A,B) is better than A,B(A),C(B)? And in both cases:why?

And why "classes have to be designed for subclassing"? I often do
classes that are not thought to be subclassed.

If you use attributes starting with two underscores inside a method, Python
transparently prepends them with the class name. This allows to you to use
the same variable name in two base classes and reduces coupling:

Wow, I didn't know it. Thanks a lot.

But if two classes with the same name use the "private" variable, the
mechanism fails:

Of course.I'll remember it.

m.

[Bruno Desthuilliers]

de**********@gmail.com a écrit :

Hi,

(snip)

- is there a better way than using multiple inheritance to plug-in
dynamically commands in a Cmd command line?

Yes : use composition + delegation. Python makes it easy:

#foo.py
class Commands(object):
def do_this(self,args):
...
def do_that(self,args):
...
#bar.py
class Commands(object):
def do_what(self, args):
...
I've seen I can do it by explicitely import them and using multiple
inheritance:

class MyCli(cmd.Cmd):
def __init__(self, *plugins):
self.plugins = plugins
self._cache = {}

def __getattr__(self, name):
try:
return self._cache[name]
except KeyError:
for plugin in self.plugins:
attr = getattr(plugin, name, None)
if attr is not None:
self._cache[name] = attr
return attr
my_cli = MyCli(foo.Command(), bar.Command())

[Peter Otten]

Neil Cerutti wrote:

On 2007-02-13, Peter Otten <__*******@web.dewrote:

>Well, what problems ocurring with

class A: pass
class B: pass
class C(A, B): pass

could be avoided by writing

class A: pass
class B(A): pass
class C(B): pass

instead?

With multiple inheritance, the choice of algorithm for Method
Resolution Ordering isn't obvious or trivial. As I understand it,
Python got it wrong for quite some time, allowing "bad"
hierarchies to compile, i.e., classes for which the properties of
local precedence ordering and monotonicity did not hold.

URL:http://www.python.org/download/releases/2.3/mro/

Is it correct that as long as no classes occur twice in the hierarchy no
such ambiguities can arise?

Peter

[Peter Otten]

massimo s. wrote:

On 13 Feb, 12:46, Peter Otten <[e-mail]:

>Well, what problems ocurring with

class A: pass
class B: pass
class C(A, B): pass

could be avoided by writing

class A: pass
class B(A): pass
class C(B): pass

instead? Classes have to be designed for subclassing, so essentially you
get two interfaces, one for subclasses and one for client code instead of
just the latter. A more relevant mantra governing inheritance is "Flat is
better than nested".

I am truly getting lost. Are you saying that doing A-->B(A)--C(B) is
better than C(A,B)? And isn't the former thing nested? Or are you
saying that C(A,B) is better than A,B(A),C(B)? And in both cases:why?

Neither. I wanted to express that I don't buy the "mantra" you mentioned
above. Just because it uses only single inheritance code doesn't become
magically more robust. Use whatever works best to solve the actual problem.

And why "classes have to be designed for subclassing"? I often do
classes that are not thought to be subclassed.

That's fine. If (and only if) you expect a class to be subclassed you better
spend some thought on what methods should be overriden etc. This increases
the effort spent on the class significantly. Distinct classes are often
easier to write and maintain.

Hope-I'm-clear-this-time,
Peter

[Neil Cerutti]

On 2007-02-14, Peter Otten <__*******@web.dewrote:

Neil Cerutti wrote:

>On 2007-02-13, Peter Otten <__*******@web.dewrote:

>>Well, what problems ocurring with

class A: pass
class B: pass
class C(A, B): pass

could be avoided by writing

class A: pass
class B(A): pass
class C(B): pass

instead?

With multiple inheritance, the choice of algorithm for Method
Resolution Ordering isn't obvious or trivial. As I understand it,
Python got it wrong for quite some time, allowing "bad"
hierarchies to compile, i.e., classes for which the properties of
local precedence ordering and monotonicity did not hold.

URL:http://www.python.org/download/releases/2.3/mro/

Is it correct that as long as no classes occur twice in the
hierarchy no such ambiguities can arise?

As long as you forbid the diamond-shaped hierarchy, I think so.
Of course, since all new-style classes inherit from 'object' you
can't avoid it in practice, and a good algorithm for MRO had to
be stolen from Dylan. ;)

--
Neil Cerutti

[Michele Simionato]

On Feb 13, 9:14 am, Peter Otten <[e-mail]:

"Avoid inheritance" would be almost as justified :-)

Yep, I strongly agree. Inheritance is overrated, as Guido says.
For what concerns the debate of multiple vs single inheritance, yes it
is true
that multiple inheritance is worse, but even single inheritance can be
bad
enough, unfortunately :-(
The issue is that inheritance *does not scale*: i.e. in simple
situations
it works fine, but it has the tendence to becomes unmanageable very
easily
In other words, if you are inheriting just two or three methods it may
works, but when
you start having dozens of methods inherited from different sources,
your code will
become to look as spaghetti code.
This is why in general I (as many people here) suggest delegation over
inheritance.

Michele Simionato

[Neil Cerutti]

On 2007-02-15, Michele Simionato <mi***************@gmail.comwrote:

On Feb 13, 9:14 am, Peter Otten <[e-mail]:

>"Avoid inheritance" would be almost as justified :-)

In other words, if you are inheriting just two or three methods
it may works, but when you start having dozens of methods
inherited from different sources, your code will become to look
as spaghetti code. This is why in general I (as many people
here) suggest delegation over inheritance.

I consider inheritance in Python when I see that the class I'm
implementing contains some sort of status code that controls
behavior.

For example:

class Stream(object):
def __init__(self, readable, writable):
if readable and writable:
self.io_type = 'inout'
elif readable:
self.io_type = 'out'
else:
self.io_type = 'in'

That code sets me to thinking I'll get good mileage from:

class Stream(object):
...

class InStream(object):
...

class OutStream(object):
...

class InOutStream(object):
...

I always get myself into trouble when I try to design a class
hierarchy *before* I see something like that.

--
Neil Cerutti