Hi,
I think I have discovered a problem in context of
metaclasses and multiple inheritance in python 2.4,
which I could finally reduce to a simple example:
Look at following code:
class M_A (type) :
def __new__ (meta, name, bases, dict) :
print "M.__new__" , meta, name, bases
return super (M_A, meta).__new__ (meta, name, bases, dict)
class M_B (M_A) : pass
class A (object) : __metaclass__ = M_A
class B (object) : __metaclass__ = M_B
class D (A, B) : pass
One would expect that either
1) D inherits the metaclass M_A from A
2) or python raises the well known "Metatype conflict among bases"
error
Instead, if you run this code, you get the following output:
M.__new__ <class '__main__.M_A'> A (<type 'object'>,)
M.__new__ <class '__main__.M_B'> B (<type 'object'>,)
M.__new__ <class '__main__.M_A'> D (<class '__main__.A'>, <class
'__main__.B'>)
M.__new__ <class '__main__.M_B'> D (<class '__main__.A'>, <class
'__main__.B'>)
This means that when class D gets defined, the __new__ from M_A
get executed twice (first from M_A, then from M_B), which should not happen.
This suggests that either
1) cooperative supercalls do not work here
2) the metaclass of EACH of the bases of D does it's work independently
when D is defined.
Does anyone have a detailed explanation here ?
Is this problem already known ?
regards
chris 9 1654
Christian Eder wrote: Hi,
I think I have discovered a problem in context of metaclasses and multiple inheritance in python 2.4, which I could finally reduce to a simple example:
I don't know if this is a bug; but I will try to expain
what is happening; here is an example similar to yours: class M_A(type):
.... def __new__(meta, name, bases, dict):
.... print 'metaclass:', meta.__name__, 'class:', name
.... return super(M_A, meta).__new__(m eta, name, bases, dict)
.... class M_B(M_A):
.... pass
.... class A(object):
.... __metaclass__ = M_A
....
metaclass: M_A class: A class B(object):
.... __metaclass__ = M_B
....
metaclass: M_B class: B
So far everything is as expected.
class C(A, B):
.... __metaclass__ = M_B
....
metaclass: M_B class: C
If we explicitly declare that our derived class inherits
from the second base, which has a more derived metaclass,
everything is OK.
class D(A, B):
.... pass
....
metaclass: M_A class: D
metaclass: M_B class: D
Now this is where it gets interesting; what happens
is the following:
- Since D does not have a __metaclass__ attribute,
its type is determined from its bases.
- Since A is the first base, its type (M_A) is called;
unfortunately this is not the way metaclasses are
supposed to work; the most derived metaclass should
be selected.
- M_A's __new__ method calls the __new__ method of the
next class in MRO; that is, super(M_1, meta).__new__
is equal to type.__new__.
- In type.__new__, it is determined that M_A is not
the best type for D class; it should be actually M_B.
- Since type.__new__ was called with wrong metaclass
as the first argument, call the correct metaclass.
- This calls M_B.__new__, which again calls type.__new__,
but this time with M_B as the first argument, which
is correct.
As I said, I don't know if this is a bug or not,
but you can achieve what is expected if you do the
following in your __new__ method (warning, untested code):
from types import ClassType class AnyMeta(type):
.... """
.... Metaclass that follows type's behaviour in "metaclass
resolution".
....
.... Code is taken from Objects/typeobject.c and translated to
Python.
.... """
.... def __new__(meta, name, bases, dict):
.... winner = meta
.... for cls in bases:
.... candidate = type(cls)
.... if candidate is ClassType:
.... continue
.... if issubclass(winn er, candidate):
.... continue
.... if issubclass(cand idate, winner):
.... winner = candidate
.... continue
.... raise TypeError("meta class conflict: ...")
.... if winner is not meta and winner.__new__ !=
AnyMeta.__new__ :
.... return winner.__new__( winner, name, bases, dict)
.... # Do what you actually meant from here on
.... print 'metaclass:', winner.__name__ , 'class:', name
.... return super(AnyMeta, winner).__new__ (winner, name, bases,
dict)
.... class OtherMeta(AnyMe ta):
.... pass
.... class A(object):
.... __metaclass__ = AnyMeta
....
metaclass: AnyMeta class: A class B(object):
.... __metaclass__ = OtherMeta
....
metaclass: OtherMeta class: B class C(A, B):
.... pass
....
metaclass: OtherMeta class: C
Does anyone have a detailed explanation here ? Is this problem already known ?
regards chris
I hope that above explanation helps.
Ziga
Ziga Seilnacht wrote: - Since D does not have a __metaclass__ attribute, its type is determined from its bases. - Since A is the first base, its type (M_A) is called; unfortunately this is not the way metaclasses are supposed to work; the most derived metaclass should be selected. - M_A's __new__ method calls the __new__ method of the next class in MRO; that is, super(M_1, meta).__new__ is equal to type.__new__. - In type.__new__, it is determined that M_A is not the best type for D class; it should be actually M_B. - Since type.__new__ was called with wrong metaclass as the first argument, call the correct metaclass. - This calls M_B.__new__, which again calls type.__new__, but this time with M_B as the first argument, which is correct.
This is a very good explanation and it should go somewhere in the
standard docs.
I remember I spent a significant amount of time and effort to reach the
same conclusion
a while ago, and now I have already started to forget eveything a again
:-(
Anyway, I will bookmark this post for future reference ;)
Michele Simionato
Christian Eder wrote: Hi,
I think I have discovered a problem in context of metaclasses and multiple inheritance in python 2.4, which I could finally reduce to a simple example:
Look at following code:
class M_A (type) :
def __new__ (meta, name, bases, dict) : print "M.__new__" , meta, name, bases return super (M_A, meta).__new__ (meta, name, bases, dict)
class M_B (M_A) : pass
class A (object) : __metaclass__ = M_A
class B (object) : __metaclass__ = M_B
class D (A, B) : pass
One would expect that either 1) D inherits the metaclass M_A from A 2) or python raises the well known "Metatype conflict among bases" error
No, there is no conflict in this case: since M_B is a subclass of M_A,
the
metaclass of D is M_B. I don't think this is bug either: the fact is
that
type.__new__ works differently from object.__new__, so that it is
called twice in this case. Not sure if it could be avoided.
Speaking of the metatype conflict, I realized a while ago that it is
possible
to avoid it automatically even at the pure Python level, without
changing
the C code for type.__new__. However, the solution is too complicate.
According
to the Zen of Python "If the implementation is hard to explain, it's a
bad idea",
thus I have never used it.
Still, it is an interesting exercise if you are willing to risk the
melting of your brain,
so here is the code ;)
# noconflict.py
"""Deep, **DEEP** magic to remove metaclass conflicts.
``noconflict`` provides the ``safetype`` metaclass, the mother of
conflict-free
metaclasses. I you do
from noconflict import safetype as type
on top of your module, all your metaclasses will be conflict safe.
If you override ``__new__`` when you derive from ``safetype``,
you should do it cooperatively." ""
import inspect, types, __builtin__
try: set # python version >= 2.4
except NameError: # python version <= 2.3
from sets import Set as set
def skip_redundant( iterable, skipset=None):
"Redundant items are repeated items or items in the original
skipset."
if skipset is None: skipset = set()
for item in iterable:
if item not in skipset:
skipset.add(ite m)
yield item
memoized_metacl asses_map = {}
# utility function
def remove_redundan t(metaclasses):
skipset = set([types.ClassType])
for meta in metaclasses: # determines the metaclasses to be skipped
skipset.update( inspect.getmro( meta)[1:])
return tuple(skip_redu ndant(metaclass es, skipset))
############### ############### ############### ############### ######
## now the core of the module: two mutually recursive functions ##
############### ############### ############### ############### ######
def get_noconflict_ metaclass(bases , left_metas, right_metas):
"""Not intended to be used outside of this module, unless you know
what you are doing."""
# make tuple of needed metaclasses in specified priority order
metas = left_metas + tuple(map(type, bases)) + right_metas
needed_metas = remove_redundan t(metas)
# return existing confict-solving meta, if any
if needed_metas in memoized_metacl asses_map:
return memoized_metacl asses_map[needed_metas]
# nope: compute, memoize and return needed conflict-solving meta
elif not needed_metas: # wee, a trivial case, happy us
meta = type
elif len(needed_meta s) == 1: # another trivial case
meta = needed_metas[0]
# check for recursion, can happen i.e. for Zope ExtensionClasse s
elif needed_metas == bases:
raise TypeError("Inco mpatible root metatypes", needed_metas)
else: # gotta work ...
metaname = '_' + ''.join([m.__name__ for m in needed_metas])
meta = classmaker()(me taname, needed_metas, {})
memoized_metacl asses_map[needed_metas] = meta
return meta
def classmaker(left _metas=(), right_metas=()) :
def make_class(name , bases, adict):
metaclass = get_noconflict_ metaclass(bases , left_metas,
right_metas)
return metaclass(name, bases, adict)
return make_class
############### ############### ############### ############### #####
## and now a conflict-safe replacement for 'type' ##
############### ############### ############### ############### #####
__type__=__buil tin__.type # the aboriginal 'type'
# left available in case you decide to rebind __builtin__.typ e
class safetype(__type __):
"""Override s the ``__new__`` method of the ``type`` metaclass,
making the
generation of classes conflict-proof."""
def __new__(mcl, *args):
nargs = len(args)
if nargs == 1: # works as __builtin__.typ e
return __type__(args[0])
elif nargs == 3: # creates the class using the appropriate
metaclass
n, b, d = args # name, bases and dictionary
meta = get_noconflict_ metaclass(b, (mcl,), right_metas=())
if meta is mcl: # meta is trivial, dispatch to the default
__new__
return super(safetype, mcl).__new__(mc l, n, b, d)
else: # non-trivial metaclass, dispatch to the right
__new__
# (it will take a second round)
return super(mcl, meta).__new__(m eta, n, b, d)
else:
raise TypeError('%s() takes 1 or 3 arguments' %
mcl.__name__)
Michele Simionato
Ziga Seilnacht wrote: I hope that above explanation helps.
Thanks for your support.
I now understand what happens here,
but I'm not really happy with the situation.
Your solution is a nice workaround, but in a quite
huge and complex class framework with a lot a custom
metaclasses you don't want this code in each __new__
function. And in fact each __new__ which does not contain this
fix-code (and which is not completely side-effect free) might
break if someone adds additional classes deeps down in the
inheritance hierarchy (which is exactly what happened
for me). And this is clearly not what one should expect in
context of multiple inheritance and cooperative supercalls.
Raising a "metatype conflict among bases" error might be a
perfectly acceptable behavior here (though it would be better if
python resolves the conflict as your code does), but
double-executing code is not in my humble opinion.
Is this worth a bug-report on sourceforge ?
regards
chris
Michele Simionato wrote: Still, it is an interesting exercise if you are willing to risk the melting of your brain, so here is the code ;)
I tried your code and it fixes the double execution, but the __new__ is
executed in context of M_A instead of M_B which would be the more
specific type here.
Anyway, I think I found an acceptable solution for the problem.
The question is "What Do I need to fix metaclasses ?"
And the answer, of course, is "a meta-meta-class !"
The following code solves the problem.
It's basically Ziga's code except that I added one level of abstraction
(to avoid adding this kludge code to each custom metaclasses'es __new__):
class _fixed_type_ (type) :
def __call__ (meta, name, bases, dict) :
meta = meta._get_meta (bases, dict)
cls = meta.__new__ (meta, name, bases, dict)
meta.__init__ (cls, name, bases, dict)
return cls
# end def __call__
def _get_meta (meta, bases, dict) :
if "__metaclas s__" in dict :
return dict ["__metaclas s__"]
winner = meta
for b in bases :
cand = type (b)
if cand in (types.ClassTyp e, type) :
pass
elif issubclass (cand, winner) :
winner = cand
elif issubclass (winner, cand) :
pass
else :
raise TypeError ("Metatype conflict among bases")
return winner
# end def _get_meta
# end class _fixed_type_
class my_type (type) :
__metaclass__ = _fixed_type_ ### to fix metaclasses, we need
### meta-meta classes
# end class my_type
Then I made "my_type" the root of my metaclass hierarchy
(instead of "type") which solves all my problems.
After 5 years of Python, I still find it impressive how much
vodoo and mojo one can do here :-)
regards
chris
> After 5 years of Python, I still find it impressive how much vodoo and mojo one can do here :-)
True ;)
However, I should point out that I never use this stuff in production
code.
I have found out that for my typical usages metaclasses are too much:
a class decorator would be enough and much less fragile. At the
present, we
do not have class decorators, but we can nearly fake them with a very
neat
trick:
def thisclass(proc, *args, **kw):
""" Example: def register(cls): print 'registered'
... class C:
... thisclass(regis ter)
...
registered
"""
# basic idea stolen from zope.interface, which credits P.J. Eby
frame = sys._getframe(1 )
assert '__module__' in frame.f_locals # inside a class statement
def makecls(name, bases, dic):
try:
cls = type(name, bases, dic)
except TypeError, e:
if "can't have only classic bases" in str(e):
cls = type(name, bases + (object,), dic)
else: # other strange errors, such as __slots__ conflicts, etc
raise
del cls.__metaclass __
proc(cls, *args, **kw)
return cls
frame.f_locals["__metaclas s__"] = makecls
Figured you would like this one ;)
Michele Simionato
Michele Simionato wrote:
<snip>
There is a minor bug in your code: def thisclass(proc, *args, **kw): """ Example: >>> def register(cls): print 'registered' ... >>> class C: ... thisclass(regis ter) ... registered """ # basic idea stolen from zope.interface, which credits P.J. Eby frame = sys._getframe(1 ) assert '__module__' in frame.f_locals # <----------------------------------- here def makecls(name, bases, dic): try: cls = type(name, bases, dic) except TypeError, e: if "can't have only classic bases" in str(e): cls = type(name, bases + (object,), dic) else: # other strange errors, such as __slots__ conflicts, etc raise del cls.__metaclass __ proc(cls, *args, **kw) return cls frame.f_locals["__metaclas s__"] = makecls
Figured you would like this one ;)
Michele Simionato
See this example: import sys def in_class_statem ent1():
.... frame = sys._getframe(1 )
.... return '__module__' in frame.f_locals
.... def in_class_statem ent2():
.... frame = sys._getframe(1 )
.... return '__module__' in frame.f_locals and not \
.... '__module__' in frame.f_code.co _varnames
.... class A(object):
.... print in_class_statem ent1()
.... print in_class_statem ent2()
....
True
True def f():
.... __module__ = 1
.... print in_class_statem ent1()
.... print in_class_statem ent2()
.... f()
True
False
Well, I would not call it a bug, I would call it to cheat ;)
The assert is there I just wanted to prevent accidents, not to *really*
ensure
that 'thisclass' is called inside a class statement. Do you know of any
reliable method to enforce that restriction?
Michele Simionato
Ziga Seilnacht wrote: def in_class_statem ent2():
... frame = sys._getframe(1 ) ... return '__module__' in frame.f_locals and not \ ... '__module__' in frame.f_code.co _varnames
On second thought, to break this check is less easy than I expected, so
maybe it is reliable enough. BTW, if you are interested, you can check
the original code in zope.interface. advice. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Jp Calderone |
last post by:
Due to some bizarre constraints placed on me, I've written the following
metaclass:
import types
def remove(t, o):
return tuple()
class BizarreMetaclass(type):
def __new__(klass, name, bases, attrs):
|
by: Robin Becker |
last post by:
A colleague wanted to initialize his class __new__ and tried code resembling this
#######################1
class Metaclass (type):
def __init__(cls, name, bases, *args, **kwargs):
super(Metaclass, cls).__init__(cls, name, bases, *args, **kwargs)
print 'cls=',cls, cls.__new
cls.__new__ = staticmethod(cls.__new)
def __new(self,cls,*args):
|
by: Irmen de Jong |
last post by:
Hi,
I've developed the Metaclass below, because I needed a way
to make a bunch of classes thread-safe.
I didn't want to change every method of the class by adding
lock.aqcuire()..lock.release() around the existing code.
So I made a metaclass that essentially replaces every method
of a class with a 'wrapper' method, that does the locking,
invocation, unlocking.
Is this the right approach? It seems to work fine. But I have
|
by: Jacek Generowicz |
last post by:
I would like to write a metaclass which would allow me to overload
names in the definition of its instances, like this
class Foo(object):
__metaclass__ = OverloadingClass
att = 1
att = 3
|
by: zipher |
last post by:
After searching through comp.lang.python and the web regarding
metaclasses, I could not find an example for customing classes using
metaclass parameters.
I want to be able to create a class at runtime by calling some
function or 'meta-constructor' which returns a customized class and
sets a class attribute according a given parameter.
Ideally, I'd be able to do something like:
| |
by: ironfroggy |
last post by:
Hoping this isn't seeming too confusing, but I need to create a
metaclass and a class using that metaclass, such that one of the bases
of the metaclass is the class created with that metaclass. I can't
figure out a way to do this, even after trying to add the class as a
base after the classes have been created.
Is there some way I can get this to work properly?
|
by: Pedro Werneck |
last post by:
Hi
I have a class A, with metaclass M_A, and class B, subclass of A, with
metaclass M_B, subclass of M_A.
A class C, subclass of B must have M_B or a subclass of it as metaclass,
but what if I need to 'disable' the code in M_B on C ? The correct way
to do that seems to be with a M_C metaclass, subclass of M_B,
implementing but not calling parent class methods, or calling 'type'
methods.
|
by: Pedro Werneck |
last post by:
Hi all
I noticed something strange here while explaining decorators to someone.
Not any real use code, but I think it's worth mentioning.
When I access a class attribute, on a class with a custom metaclass with
a __getattribute__ method, the method is used when acessing some
attribute directly with the class object, but not when you do it from
the instance.
|
by: Dox33 |
last post by:
I ran into a very strange behaviour of raw_input().
I hope somebody can tell me how to fix this.
(Or is this a problem in the python source?)
I will explain the problem by using 3 examples. (Sorry, long email)
The first two examples are behaving normal, the thirth is
strange.......
I wrote the following flabbergasting code:
#-------------------------------------------------------------
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
| |
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |