473,320 Members | 1,991 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Is this a good use of __metaclass__?

Hi!

I need some input on my use of metaclasses since I'm not sure I'm using them in
a pythonic and graceful manner. I'm very grateful for any tips, pointers and
RTFMs I can get from you guys.

Below, you'll find some background info and an executable code example.

In the code example I have two ways of doing the same thing. The problem is
that the "Neat" version doesn't work, and the "Ugly" version that works gives
me the creeps.

The "Neat" version raises a TypeError when I try the multiple inheritance
(marked with comment in the code):

Traceback (most recent call last):
File "/bioinfo/yohell/pybox/gridjs/gridjs-2.0/test.py", line 132, in ?
class FullAPI(JobAPI, UserAPI, AdminAPI):
File "/bioinfo/yohell/pybox/gridjs/gridjs-2.0/test.py", line 43, in __new__
return type.__new__(cls,classname,bases,classdict)
TypeError: Error when calling the metaclass bases
metaclass conflict: the metaclass of a derived class must be a
(non-strict) subclass of the metaclasses of all its bases

In the "Ugly" version, I'm changing the metaclass in the global scope between
class definitions, and that gives me bad vibes.

What should I do? Is there a way to fix my "Neat" solution? Is my "Ugly"
solution in fact not so horrid as I think it is? Or should I rethink the whole
idea? Or maybe stick with decorating manually (or in BaseAPI.__init__)?

Sincere thanks for your time.

Cheers!
/Joel Hedlund


Background
##########
(feel free to skip this if you are in a hurry)
I'm writing an XMLRPC server that serves three types of clients (jobs, users
and admins). To do this I'm subclassing SimpleXMLRPCServer for all the
connection work, and I was planning on putting the entire XMLRPC API as public
methods of a class, and expose it to clients using .register_instance(). Each
session is initiated by a handshake where a challenge is presented to the
client, each method call must then be authenticated using certificates and
incremental digest response. Each client type must be authenticated
differently, and each type of client will also use a discrete set of methods.

At first, I was planning to use method decorators to do the validation, and
have a different decorator for each type of client validation, like so:

class FullAPI:
@valid_job_required
def do_routine_stuff(self, clientid, response, ...):
pass
@valid_user_required
def do_mundane_stuff(self, clientid, response, ...):
pass
@valid_admin_required
def do_scary_stuff(self, clientid, response, ...):
pass
...

There will be a lot of methods for each client type, so this class would become
monstrous. Therefore I started wondering if it weren't a better idea to put the
different client APIs in different classes and decorate them separately using
metaclasses, and finally bring the APIs together using multiple inheritance.
This is what I had in mind:

class BaseAPI(object):
pass

class JobApi(BaseAPI):
pass

class UserApi(BaseAPI):
pass

class AdminApi(BaseAPI):
pass

class FullApi(JobAPI, UserAPI, AdminAPI):
pass

Now, I'm having trouble implementing the metaclass bit in a nice and pythonic way.

Code example
############

test.py
================================================== =====================
# Base metaclass for decorating public methods:
from decorator import decorator

@decorator
def no_change(func, *pargs, **kwargs):
return func(*pargs, **kwargs)

class DecoratePublicMethods(type):
"""Equip all public methods with a given decorator.

Class data members:
decorator = no_change: <decorator>
The decorator that you wish to apply to public methods of the class
instances. The default does not change program behavior.
do_not_decorate = []: <iterable <str>>
Names of public methods that should not be decorated.
multiple_decoration = False: <bool>
If set to False, methods will not be decorated if they already
have been decorated by a prior metaclass.
decoration_tag = '__public_method_decorated__': <str>
Decorated public methods will be equipped with an attribute
with this name and a value of True.

"""

decorator = no_change
do_not_decorate = []
multiple_decoration = True
decoration_tag = '__public_method_decorated__'

def __new__(cls,classname,bases,classdict):
for attr,item in classdict.items():
if not callable(item):
continue
if attr in cls.do_not_decorate or attr.startswith('_'):
continue
if (not cls.multiple_decoration
and hasattr(classdict[attr], cls.decoration_tag)):
continue
classdict[attr] = cls.decorator(item)
setattr(classdict[attr], cls.decoration_tag, True)
return type.__new__(cls,classname,bases,classdict)
## Authentication stuff:
class AuthenticationError(Exception):
pass

import random

@decorator
def validate_job(func, self, id, response, *pargs, **kwargs):
"""Bogus authentiction routine"""
if random.randint(0,3) / 3:
raise AuthenticationError
return func(self, id, response, *pargs, **kwargs)

@decorator
def validate_user(func, self, id, response, *pargs, **kwargs):
"""Bogus authentiction routine"""
if random.randint(0,4) / 4:
raise AuthenticationError
return func(self, id, response, *pargs, **kwargs)

@decorator
def validate_admin(func, self, id, response, *pargs, **kwargs):
"""Bogus authentiction routine"""
if random.randint(0,1):
raise AuthenticationError
return func(self, id, response, *pargs, **kwargs)
## Ugly (?) way that works:
## -----------------------------------------------------------
#class BaseAPI(object):
# __metaclass__ = DecoratePublicMethods
#
#DecoratePublicMethods.decorator = validate_job
#
#class JobAPI(BaseAPI):
# def do_routine_stuff(self, clientid, response, foo):
# print "Routine stuff done."
#
#DecoratePublicMethods.decorator = validate_user
#
#class UserAPI(BaseAPI):
# def do_mundane_stuff(self, clientid, response, moo):
# print "Mundane stuff done."
#
#DecoratePublicMethods.decorator = validate_admin
#
#class AdminAPI(BaseAPI):
# def do_scary_stuff(self, clientid, response, moose):
# print "Scary stuff done."
#
#class FullAPI(JobAPI, UserAPI, AdminAPI):
# pass
#
#a = FullAPI()
## -----------------------------------------------------------

## Neat (?) way that doesn't work:
## -----------------------------------------------------------
class RequireJobValidation(DecoratePublicMethods):
decorator = validate_job

class RequireUserValidation(DecoratePublicMethods):
decorator = validate_user

class RequireAdminValidation(DecoratePublicMethods):
decorator = validate_admin

class BaseAPI(object):
pass

class JobAPI(BaseAPI):
__metaclass__ = RequireJobValidation
def do_routine_stuff(self, clientid, response, foo):
print "Routine stuff done."

class UserAPI(BaseAPI):
__metaclass__ = RequireUserValidation
def do_mundane_stuff(self, clientid, response, moo):
print "Mundane stuff done."

class AdminAPI(BaseAPI):
__metaclass__ = RequireAdminValidation
def do_scary_stuff(self, clientid, response, moose):
print "Scary stuff done."

print "OK up to here."

## FIXME: Illegal multiple inheritance.
class FullAPI(JobAPI, UserAPI, AdminAPI):
pass

b = FullAPI()
## -----------------------------------------------------------
================================================== =====================

Oh, and by the way - this example uses Michele Simionato's excellent decorator
module, available from here:
http://www.phyast.pitt.edu/~micheles.../decorator.zip

If you don't want to donwload it, for this example you can just substitute this:

@decorator
def foo(func, *pargs, **kwargs):
print "Code goes here"
return func(*pargs, **kwargs)

for this:

def foo(func):
def caller(*pargs, **kwargs):
print "Code goes here"
return func(*pargs, **kwargs)
return caller

The difference is that @decorator preserves function signatures and such for
decorated functions. Very neat.

Thanks again for your time.

/Joel
May 5 '06 #1
9 1576

"Joel Hedlund" <jo**********@gmail.com> wrote in message
news:e3**********@news.lysator.liu.se...
Below, you'll find some background info and an executable code example.

In the code example I have two ways of doing the same thing. The problem
is
that the "Neat" version doesn't work, and the "Ugly" version that works
gives
me the creeps.


To me, your Ugly version is easier to read and understand than the
so-called Neat version. You change an attribute of a class to change its
behavior without changing its identity. I can imagine ways to hide this,
but I would leave it simple and explicit.

Terry Jan Reedy

May 5 '06 #2
Joel Hedlund a écrit :
Hi!

I need some input on my use of metaclasses since I'm not sure I'm using
them in a pythonic and graceful manner. I'm very grateful for any tips,
pointers and RTFMs I can get from you guys.

Below, you'll find some background info and an executable code example.

In the code example I have two ways of doing the same thing. The problem
is that the "Neat" version doesn't work, and the "Ugly" version that
works gives me the creeps.

The "Neat" version raises a TypeError when I try the multiple
inheritance (marked with comment in the code):

Traceback (most recent call last):
File "/bioinfo/yohell/pybox/gridjs/gridjs-2.0/test.py", line 132, in ?
class FullAPI(JobAPI, UserAPI, AdminAPI):
File "/bioinfo/yohell/pybox/gridjs/gridjs-2.0/test.py", line 43, in
__new__
return type.__new__(cls,classname,bases,classdict)
TypeError: Error when calling the metaclass bases
metaclass conflict: the metaclass of a derived class must be a
(non-strict) subclass of the metaclasses of all its bases
http://aspn.activestate.com/ASPN/Coo.../Recipe/204197
In the "Ugly" version, I'm changing the metaclass in the global scope
between class definitions,
Yuck.
and that gives me bad vibes.

What should I do? Is there a way to fix my "Neat" solution? Is my "Ugly"
solution in fact not so horrid as I think it is? Or should I rethink the
whole idea? Or maybe stick with decorating manually (or in
BaseAPI.__init__)?


I'd go for 'manually decorating' anyway. Metaclasses can be really handy
for framework-like stuff, but for the use case you describe, I think the
explicit decorator option is much more, well, explicit - and also more
flexible - than metaclass black magic. This may also help you
distinguish 'published' API from implementation (which is what CherryPy
do). And finally, this may let you organize your code more freely - you
can mix methods needing different decorators in a same class.

Also, and FWIW, you may want to favor composition-delegation (which is a
piece of cake in Python, see below...) over inheritance. This is more
flexible. You would then have a 'server' class that just provides common
services and dispatch to specialized objects.

class Dispatcher(object):
def __init__(self, delegates):
self._delegates = delegates

def __getattr__(self, name):
for delegate in self._delegates:
try:
return getattr(delegate, name):
except AttributeError:
pass
else:
err = "object '%s' has no attribute '%s'" \
% (self.__class__.__name__, name)
raise AttributeError(err)

# note that you may want to reorganize your code differently
# if you go for explicit decorators
d = Dispatcher(JobApi, UserApi, AdminApi)
My 2 cents...
May 5 '06 #3
Hi!

Thank you for a quick and informative response!
I'd go for 'manually decorating' anyway. Metaclasses can be really handy
for framework-like stuff, but for the use case you describe, I think the
explicit decorator option is much more, well, explicit - and also more
flexible - than metaclass black magic.
Yes, point taken.
This may also help you distinguish 'published' API from implementation (which is what > CherryPy do)
Hmm... I'm not sure I understand how manually decorating would help me
do that? With SimpleXMLRPCServer.register_instance(obj) all public
methods of obj are published for XMLRPC, decorated or not. Off course,
I could write a @do_not_publish decorator, but that seems backwards to
me... I'm not familiar with how CherryPy works either, I'm sorry to
say.
And finally, this may let you organize your code more freely - you
can mix methods needing different decorators in a same class.
One can still do that, even if one would use a metaclass to set the
"bare necessities" decorators. All you have to do is add the extra ones
manually. I just meant to let the metaclass do the really, really
important ones for me (the validator for each API class). The ones that
I can't, won't, mustn't forget to add lest the Script Kiddies of the
Internet come brutalizing my data. :-)
You would then have a 'server' class that just provides common
services and dispatch to specialized objects.
Neat. It won't play nice with dir() or SimpleXMLRPCServer's
introspection functions though (system.listMethods(),
system.methodHelp()). That may be a showstopper, or do you know of any
fixes?
My 2 cents...


Thanks! Those were what I was hoping for, after all.

Thanks for your help!
/Joel

May 6 '06 #4
I played around with my old code before I saw your post, and I believe
I've found a solution that's a bit neater than what I had before. I
thought I could just as well post it if you're interested and have the
time. This one uses multiple inheritance, but it's legal and there's
only one metaclass.

When executed, this prints:
Validating admin.
Scary stuff done.

Cheers!
/Joel

Here ya go!
--------------------------------------------------------------------------
from decorator import decorator

# Metaclass for decorating public methods:
class DecoratePublicMethods(type):
"""Equip public methods of a class with a specified decorator.

Class data members:
decorator_attribute = '_public_method_decorator': <str>
If this attribute of the class exists and evaluates to True,
then
it is used to decorate all public methods of the class.
no_decoration_attribute = '_do_not_decorate': <str>
If this attribute of the class exists it should contain a list
of
names of public methods that should not be decorated.

"""

decorator_attribute = '_public_method_decorator'
no_decoration_attribute = '_do_not_decorate'

def __new__(cls, classname, bases, classdict):
decorator = classdict.get(cls.decorator_attribute, None)
if not decorator:
return type.__new__(cls,classname,bases,classdict)
do_not_decorate = classdict.get(cls.no_decoration_attribute,
[])
for attr,item in classdict.items():
if not callable(item):
continue
if attr in do_not_decorate or attr.startswith('_'):
continue
classdict[attr] = decorator(item)
return type.__new__(cls, classname, bases, classdict)

# Some decorators:
@decorator
def validate_job(func, self, id, response, *pargs, **kwargs):
"""Bogus authentiction routine"""
print "Validating job."
return func(self, id, response, *pargs, **kwargs)

@decorator
def validate_user(func, self, id, response, *pargs, **kwargs):
"""Bogus authentiction routine"""
print "Validating user."
return func(self, id, response, *pargs, **kwargs)

@decorator
def validate_admin(func, self, id, response, *pargs, **kwargs):
"""Bogus authentiction routine"""
print "Validating admin."
return func(self, id, response, *pargs, **kwargs)

# My API:
class BaseAPI(object):
__metaclass__ = DecoratePublicMethods

class JobValidatedAPI(BaseAPI):
_public_method_decorator = validate_job
def do_routine_stuff(self, clientid, response, foo):
print "Routine stuff done."

class UserValidatedAPI(BaseAPI):
_public_method_decorator = validate_user
def do_mundane_stuff(self, clientid, response, moo):
print "Mundane stuff done."

class AdminValidatedAPI(BaseAPI):
_public_method_decorator = validate_admin
def do_scary_stuff(self, clientid, response, moose):
print "Scary stuff done."

## FIXED: Multiple inheritance now legal.
class FullAPI(JobValidatedAPI, UserValidatedAPI, AdminValidatedAPI):
_public_method_decorator = None

# Now this works:
b = FullAPI()
b.do_scary_stuff('bofh', 2, 3)
--------------------------------------------------------------------------

May 6 '06 #5
Answering to the title of your post, no, this is not a good use of
metaclasses.
Your problem seems a textbook example of multiple dispatch, so I
suggest
you to look at PEAK with has an implementation of multimethods/generic
functions. Notice that Guido seems to be intentioned to add support for
generic functions in future versions of Python, so that solution would
likely have the
blessing of the BDFL ;)
The problem you have with your metaclass version, is the infamous
metaclass
conflict. It can be solved by hand or automatically
(http://aspn.activestate.com/ASPN/Coo.../Recipe/204197) but it
is best
to avoid it altogether. Just use PEAK or an hand made dispatcher, like
for
instance this one:
class SimpleDispatcher(object): # this is on purpose not object
oriented
"""A dispatcher is a callable object that looks in a "namespace"
for callable objects and calls them with the signature

``<dispatcher>(<callablename>, <dispatchtag>, <*args>, <**kw>)``

The "namespace" can be a module, a class, a dictionary, or anything
that responds to ``getattr`` or (alternatively) to ``__getitem__``.

Here is an example of usage:
call = SimpleDispatcher(globals()) def manager_showpage(): .... return 'Manager'
def member_showpage(): .... return 'Member'
def anonymous_showpage(): .... return 'Anonymous'
call('showpage', 'anonymous') 'Anonymous' call('showpage', 'manager') 'Manager' call('showpage', 'member')

'Member'
"""
def __init__(self, ns):
self._ns = ns
def __call__(self, funcname, classname, *args, **kw):
try:
func = getattr(self._ns, '%s_%s' % (classname, funcname))
except AttributeError:
func = self._ns['%s_%s' % (classname, funcname)]
return func(*args, **kw)

if __name__ == "__main__":
import doctest; doctest.testmod()

BTW, the usual advice holds here: if you can find an workable solution
not involving
metaclasses and decorators, don't use them.

Michele Simionato

May 8 '06 #6
Hi!

Thanks for taking the time to answer. I will definitely have a look at writing
dispatchers.
The problem you have with your metaclass version, is the infamous
metaclass conflict.


I think I solved the problem of conflicting metaclasses in this case and I
posted it as a reply to Bruno Desthuilliers in this thread. Do you also think
that's a bad use of __metaclass__? It's not that I'm hellbent on using
metaclasses - I'm just curious how people think they should be used.

Cheers!
/Joel
May 10 '06 #7
Jo**********@gmail.com wrote:
Hi!

Thank you for a quick and informative response!

I'd go for 'manually decorating' anyway. Metaclasses can be really handy
for framework-like stuff, but for the use case you describe, I think the
explicit decorator option is much more, well, explicit - and also more
flexible - than metaclass black magic.

Yes, point taken.

This may also help you distinguish 'published' API from implementation (which is what > CherryPy do)

Hmm... I'm not sure I understand how manually decorating would help me
do that? With SimpleXMLRPCServer.register_instance(obj) all public
methods of obj are published for XMLRPC, decorated or not.


So it's effectively useless (disclaimer : I've never used
SimpleXMLRPCServer).

(snip)
You would then have a 'server' class that just provides common
services and dispatch to specialized objects.


Neat. It won't play nice with dir() or SimpleXMLRPCServer's
introspection functions though (system.listMethods(),
system.methodHelp()). That may be a showstopper, or do you know of any
fixes?


Nope - well, at least not without digging into SimpleXMLRPCServer's
internals, but if it (-> the solution I suggested) makes things more
complicated, it's a bad idea anyway.

--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom.gro'.split('@')])"
May 10 '06 #8
Jo**********@gmail.com wrote:
I played around with my old code before I saw your post, and I believe
I've found a solution that's a bit neater than what I had before. I
thought I could just as well post it if you're interested and have the
time. This one uses multiple inheritance, but it's legal and there's
only one metaclass.

(snip code)

Seems quite clean.


--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom.gro'.split('@')])"
May 10 '06 #9
Joel Hedlund wrote:
It's not that I'm hellbent on using
metaclasses - I'm just curious how people think they should be used.


There are very few good use cases for metaclasses. *You should use a
metaclass
only when you want the additional metaclass-induced behavior to be
inherited
in the children classes.*

Here is a made up example (it assume you know the cmd.Cmd class in the
standard
library). Suppose you want to use a metaclasses to add aliases to the
"do_*"
methods of a subclass of cmd.Cmd. You can do it in this way:

import cmd

class make_oneletter_aliases_for_commands(type):
"""Typically used in Cmd classes."""
def __init__(cls, name, bases, dic):
for name,func in dic.iteritems():
if name.startswith("do_"):
firstletter = name[3]
setattr(cls, "do_" + firstletter, func)

class Cmd(cmd.Cmd, object): # make it a new-style class, so that super
works
__metaclass__ = make_oneletter_aliases_for_commands
def preloop(self):
"""Done once at the beginning."""
cmd.Cmd.preloop(self)
self.do_greetings("World!")

def do_greetings(self, arg):
"""Display a greeting message."""
print "Hello, %s!" % arg

def do_help(self, arg):
"""Print this help message."""
super(Cmd, self).do_help(arg)

def do_quit(self,arg):
"""Exit the command-loop."""
return 'quit' # anything != None will do

if __name__ == '__main__':
Cmd().cmdloop()

Here is how you use it:

$ python cmd_with_aliases.py
Hello, World!!
(Cmd) h

Documented commands (type help <topic>):
========================================
g greetings h help q quit

(Cmd) g World
Hello, World!
(Cmd) q

The metaclass has generated the methods 'do_h', 'do_g' and 'do_q' as
aliases for 'do_help', 'do_greetings', 'do_quit' respectively. You
could have
done the same without a metaclass, with a function modifying the class.
However every time you subclass Cmd (or a subclass of it), you would
have to invoke
the function again to generate the aliases corresponding to the new
methods in the subclass. The metaclass performs this step
automagically,
simplifying the life of your users.

If you don't care about giving magic abilities to the subclasses of
subclasses
you don't need a metaclass. For real use of metaclasses, see the Zope
framework code.

Michele Simionato

May 10 '06 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: David M. Wilson | last post by:
Hi there, I've been sitting on the __metaclass__ feature for about 2 weeks now, and haven't really got that far with it. It looks like a very powerful tool, and I think I have even already...
7
by: Michele Simionato | last post by:
So far, I have not installed Prothon, nor I have experience with Io, Self or other prototype-based languages. Still, from the discussion on the mailing list, I have got the strong impression that...
10
by: Paul Morrow | last post by:
Thinking about decorators, and looking at what we are already doing in our Python code, it seems that __metaclass__, __author__, __version__, etc. are all examples of decorators. So we already...
29
by: RAY | last post by:
Hi , my boss has asked I sit in on an interview this afternoon and that I create some interview questions on the person's experience. What is C++ used for and why would a company benefit from...
113
by: Bonj | last post by:
I was in need of an encryption algorithm to the following requirements: 1) Must be capable of encrypting strings to a byte array, and decyrpting back again to the same string 2) Must have the same...
17
by: Brett | last post by:
I'd like references on where to find some good (quality) icons to use for form and application icons (for the EXE). Most of the free stuff isn't that great looking and there isn't a good...
3
by: Mark Shroyer | last post by:
I guess this sort of falls under the "shameless plug" category, but here it is: Recently I used a custom metaclass in a Python program I've been working on, and I ended up doing a sort of write-up...
5
by: anne.nospam01 | last post by:
Dear fellow Pythonians, I just stumbled upon the following unexpected behavior: class TestType(type): def Foo(self): return 'TestType Foo' class Test(object): __metaclass__ = TestType def...
1
by: Wouter DW | last post by:
I read the article on http://www.python.org/download/releases/2.2/descrintro/#metaclasses and started using autoprop. But now I have a problem I can't seem to solve myself. class autoprop(type):...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.