Possible bug in "metaclass resolution order" ?

[Pedro Werneck]

Hi

I have a class A, with metaclass M_A, and class B, subclass of A, with
metaclass M_B, subclass of M_A.

A class C, subclass of B must have M_B or a subclass of it as metaclass,
but what if I need to 'disable' the code in M_B on C ? The correct way
to do that seems to be with a M_C metaclass, subclass of M_B,
implementing but not calling parent class methods, or calling 'type'
methods.

But if I try to do that using other metaclass, not related to M_B, I get a
"TypeError: metaclass conflict exception" as expected.

Python 2.4.1 (#1, Sep 16 2005, 17:47:47)
[GCC 3.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.

class M_A(type): pass .... class A: __metaclass__ = M_A .... class M_B(M_A): pass .... class B(A): __metaclass__ = M_B .... class M_C(type): pass .... class C(B): __metaclass__ = M_C ....
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: Error when calling the metaclass bases
metaclass conflict: the metaclass of a derived class must be a
(non-strict) subclass of the metaclasses of all its bases

The problem is, if I try to do the same thing with 'type' the
interpreter use M_B and I don't get an exception, warning or anything
else. In fact, the __metaclass__ attribute of the C class points to
'type' but the __class__ to M_B!
class C(B): __metaclass__ = type .... C.__metaclass__ <type 'type'> C.__class__ <class '__main__.M_B'> type(C)

<class '__main__.M_B'>
Since the explicit __metaclass__ attribute has priority over parent
classes, a case like this is an error and should raise an exception like
the metaclass conflict, right ?

Regards,

--
Pedro Werneck

[Simon Percivall]

Have you read the "Metaclasses" part of "Unifying types and classes in
Python 2.2"? (http://www.python.org/2.2.3/descrintro.html#metaclasses)

It discusses and explains the issues you seem to have.

[Pedro Werneck]

On 17 Sep 2005 02:04:39 -0700
"Simon Percivall" <pe*******@gmail.com> wrote:

Have you read the "Metaclasses" part of "Unifying types and classes in
Python 2.2"? (http://www.python.org/2.2.3/descrintro.html#metaclasses)

Yes, I read. Have you read and understood my message ? :)

A class B, subclass of class A, with a metaclass M_A should have M_A or
a subclass of it as metaclass. If you try with anything else, you get a
TypeError exception as expected. OK. But if you try with 'type', nothing
happens.

Python 2.4.1 (#1, Sep 16 2005, 17:47:47)
[GCC 3.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.

class M_A(type): pass .... class A: __metaclass__ = M_A .... class B(A): __metaclass__ = type .... B.__class__ <class '__main__.M_A'> B.__metaclass__

<type 'type'>
Regards,

--
Pedro Werneck

[Michele Simionato]

I think this is more of a documentation issue than of a bug.
It may seems strange at first, but __metaclass__ and __class__ may be
different.

For instance, if M is metaclass

class C(object): pass C.__metaclass__ = M

you get a class with metaclass hook equal to M, but C.__class__ is
still 'type'.

In you example, setting the __metaclass__ to 'type' does not change the
metaclass of
the created class, which is inherited from the base class. I suggest
you to file a documentation
bug. Unfortunately the basic documentation about metaclasses is a bit
lacking and you have
to discover many things by trial and errors.

Michele Simionato

[Pedro Werneck]

On 17 Sep 2005 08:51:50 -0700
"Michele Simionato" <mi***************@gmail.com> wrote:

Hi

I think this is more of a documentation issue than of a bug.
No... I don't think it's a documentation issue. What's the problem with
the documentation in this case ? Trying to use 'type' as a metaclass
with a subclass of another class using a custom metaclass is an error
and should raise the same exception the metaclass conflict does. In
fact, is the same error.
It may seems strange at first, but __metaclass__ and __class__ may be
different.

For instance, if M is metaclass

class C(object): pass C.__metaclass__ = M
you get a class with metaclass hook equal to M, but C.__class__ is
still 'type'.
But in this case, it's a __metaclass__ attribute defined after class
creation, after the call to the metaclass found in object.__class__, not
dict["__metaclass__"] which have priority over object.__class__. The
interpreter is not aware of this __metaclass__ attribute during class
creation, since it's not in the dict passed on the type(name, bases,
dict) call.
In you example, setting the __metaclass__ to 'type' does not change
the metaclass of the created class, which is inherited from the base
class.
Yes, but, as I said, dict["__metaclass__"] has priority over the base
class __class__. If I use a metaclass which is not a subclass of my base
classes metaclass I get a TypeError: metaclass conflict exception. If I
use 'type' which is also not a subclass of my base classes metaclass, I
was supposed to get the same exception, but the interpreter ignores
dict["__metaclass__"] and use the metaclass in base class __class__.

Seems like the problem is in Objects/typeobject.c, PyType_IsSubtype
(814-846) or the for loop in 1604-1621. Seems like PyType_IsSubtype is
returning true for a type being a subtype of it's own subtypes and it
never reaches the exception and return NULL. The problem seems to be the
third if statement and after it, when the winner and metatype are
exchanged if different.

It's more evident using 'type' because it's the base of all types, but
after looking the source code, I tested with this code:

Python 2.4.1 (#1, Sep 16 2005, 17:47:47)
[GCC 3.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
c>>> class M_A(type): pass
....

class A: __metaclass__ = M_A .... class M_B(M_A): pass .... class B(A): __metaclass__ = M_B .... class C(B): __metaclass__ = M_A .... C.__class__ <class '__main__.M_B'> C.__metaclass__ <class '__main__.M_A'>

Is this supposed to happen ? C's metaclass must be a subclass of M_B. If
I try with any other class not related to this hierarchy, I will get a
metaclass conflict error. But with M_A, the error is ignored, probably
because PyType_IsSubtype is returning 1 for M_A being a subtype of M_B
and the winner and metatype are exchanged later, so we end with M_B as
C's real type.

I suggest you to file a documentation bug. Unfortunately the basic
documentation about metaclasses is a bit lacking and you have to
discover many things by trial and errors.
I still think this is a bug, not a documentation issue.

Regards

--
Pedro Werneck

Michele Simionato

--
http://mail.python.org/mailman/listinfo/python-list

[Terry Reedy]

"Pedro Werneck" <pe***********@terra.com.br> wrote in message
news:20050917144655.2e7ec1b7.pe***********@terra.c om.br...

I still think this is a bug, not a documentation issue.

As an sometimes bug report reviewer, I appreciate your posting this issue
here to get a wider and quicker variety of responses than you might have on
SF. If, after any further responses, you still think you have discovered a
bug, do file a report on SourceForge. Since the examples and your
speculations on what is wrong are relatively lengthy, you might consider
putting a summary in the somewhat cramped report box and attaching a
separate 'metaclass.txt' file with a complete report. Do mention your post
here and any relavant responses. Since this is a relatively specialized
and advanced issue (one I can't comment on, for instance), please don't be
too impatient for responses.

Terry J. Reedy

[Pedro Werneck]

On Sat, 17 Sep 2005 15:07:01 -0400
"Terry Reedy" <tj*****@udel.edu> wrote:

If, after any further responses, you still think you have discovered a
bug, do file a report on SourceForge.

Thanks... report id 1294232, Error in metaclass search order

http://sourceforge.net/tracker/index...70&atid=105470
--
Pedro Werneck

[Michele Simionato]

Pedro Werneck wrote:

class M_A(type): pass ....

class A: __metaclass__ = M_A .... class M_B(M_A): pass .... class B(A): __metaclass__ = M_B .... class C(B): __metaclass__ = M_A .... C.__class__

<class '__main__.M_B'>
C.__metaclass__

<class '__main__.M_A'>

Is this supposed to happen ?

Yes, or at least I feel this is a reasonable behavior. You get the
stricted metaclass(M_B) and not the more generic one
(M_A), so you don't lose anything. Remember that given a class C, its
metaclass is given by C.__class__, not by C.__metaclass__,
despite the name. I wrote some code to automatically solve metaclass
conflicts (see for instance my Oxford lectures
http://www.phyast.pitt.edu/~micheles...d-lectures.zip) and it works
in the same way, it uses the strictest metaclass.
You argue that in this case an error should be raised, since "errors
should never pass silently". May be. You are free to post
the bug report and look at the opinions of the developers. I am happy
enough with the current behavior and I would just
update the docs.

Michele Simionato
Michele Simionato

[Pedro Werneck]

On 18 Sep 2005 00:39:31 -0700
"Michele Simionato" <mi***************@gmail.com> wrote:

Remember that given a class C, its metaclass is given by C.__class__,
not by > C.__metaclass__, despite the name.
Of course. Seems you think I'm arguing that C.__class__ and
__metaclass__ should always be the same. The metaclass is given by
C.__class__ after class creation. At the end of the 'class' statement,
searching for the 'winner' metatype, dict["__metaclass__"] is supposed
to have priority over B.__class__, and this over global __metaclass__.
Not only this is the behavior documented on GvR essay but is on the
source code I mentioned too.
You argue that in this case an error should be raised, since "errors
should never pass silently". May be.
Exactly... and the behaviour is inconsistent. I get an exception when
the metaclass is not related to the M_A-M_B hierarchy, like X below, and
the same error was supposed to be raised when using M_A or type, which
do not follow the same rule of being a "subclass of the metaclasses of
all its bases". In fact, I lost a lot of time trying to find an error
related to this in my code a few weeks ago.

class X(type): pass .... class C(B): __metaclass__ = X ....
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: Error when calling the metaclass bases
metaclass conflict: the metaclass of a derived class must be a
(non-strict) subclass of the metaclasses of all its bases class C(B): __metaclass__ = M_A .... C.__metaclass__ <class '__main__.M_A'> C.__class__

<class '__main__.M_B'>
You are free to post the bug report and look at the opinions of the
developers.

I posted a few hours ago.

Thank you.

--
Pedro Werneck

[Simon Percivall]

If you have read the document I referred you to, did you also read the
example where classes M1, M2, M3 and M4 were defined?

A quote from the discussion of that example:
"For class D, the explicit metaclass M1 is not a subclass of the base
metaclasses (M2, M3), but choosing M3 satisfies the constraint, so
D.__class__ is M3."

Isn't that exactly what you are doing?

[Pedro Werneck]

On 18 Sep 2005 10:33:11 -0700
"Simon Percivall" <pe*******@gmail.com> wrote:

Isn't that exactly what you are doing?

Yes, and that example has the same inconsistency.

class X(type): pass .... class D2(C3, C2): __metaclass__ = X

....
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: Error when calling the metaclass bases

For class D2, the explicit metaclass X is not a subclass of the base
metaclasses, just like M1 with D in the example, but with X you get a
metaclass conflict error, and with M1 doesn't. Choosing M3 satisfies the
constraint, but why it is choosen when you use M1 but not with X ?

That's exactly my point. If I wanted M3 or M2, I would be using an
explicit dict['__metaclass__'] = M3 or M2. Since I am trying to do it
with M1, which does not satisfy the constraint, it is an error, and
according to Python design concepts it was supposed to raise the same
exception.

I don't know if it was a deliberate decision to go silently for the base
class metaclass if the explicit metaclass does not satisfy the
constraint but is a base class of any of the base classes metaclass, but
if it was, the same should happen when the explicit metaclass has no
relation to the base classes metaclass, which seems a very strange
design decision, and not only from Python "explicity and simplicity"
standpoint.

If this is really not an error, just a documentation issue like Mr.
Simionato argues, the first rule for determining M a few lines above
your quote is much more complex and should be changed from:

* if dict['__metaclass__'] exists, it is used.

to something more like:

* if dict['__metaclass__'] exists and is equal to, or a subclass of,
each of the metaclasses of the bases, it is used; if it exists and is a
base class of any of the metaclasses of the bases, the first base class
metaclass is used. If it exists and doesn't satisfies any of these
constraints, TypeError is raised.

If I am right and this is a bug, althought I am not very familiar with
Python internals my speculations on what may be wrong are on the bug
report I filled yesterday. If it's not a bug and this was the intended
behaviour, I think it's not consistent with Python design concepts,
especially "errors should never pass silently" and "explicit is better
than implicit". If it's just a documentation issue, the first rule in
search order is much more complex than documented. If I am completely
wrong in all of this, maybe it's better stop wasting our time. :)
Thank you

--
Pedro Werneck

http://sourceforge.net/tracker/index...70&atid=105470

[Simon Percivall]

I definitely think that it's the intended behaviour: the example shows
how and why it works; and I definitely agree that it should be
documented better.

[Pedro Werneck]

On 18 Sep 2005 12:58:22 -0700
"Simon Percivall" <pe*******@gmail.com> wrote:

I definitely think that it's the intended behaviour: the example shows
how and why it works; and I definitely agree that it should be
documented better.

Yes. I finally understood the decision and now I agree it's not a bug,
it's the intended behaviour. If the explicit class is part of the
hierarchy but not a subclass of base classes metaclass, it will search
for the more specialized metaclass and use it, since it will have
anything the class need and conforms to the metaclass I set explicitly.

I will change the bug report and add some of the suggested
documentation.

Thanks
--
Pedro Werneck

[Bengt Richter]

On Sat, 17 Sep 2005 12:41:09 -0300, Pedro Werneck <pe***********@terra.com.br> wrote:

On 17 Sep 2005 02:04:39 -0700
"Simon Percivall" <pe*******@gmail.com> wrote:

Have you read the "Metaclasses" part of "Unifying types and classes in
Python 2.2"? (http://www.python.org/2.2.3/descrintro.html#metaclasses)

Yes, I read. Have you read and understood my message ? :)

A class B, subclass of class A, with a metaclass M_A should have M_A or
a subclass of it as metaclass. If you try with anything else, you get a
TypeError exception as expected. OK. But if you try with 'type', nothing
happens.

Python 2.4.1 (#1, Sep 16 2005, 17:47:47)
[GCC 3.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.

class M_A(type): pass... class A: __metaclass__ = M_A... class B(A): __metaclass__ = type... B.__class__<class '__main__.M_A'> B.__metaclass__<type 'type'>

FWIW, I think __metaclass__ can be any callable, but it seems to be the
first argument to type.__new__ that invokes the checking and
type(name, bases, cdict) seems to have the same effect as
type.__new__(type, name, bases, cdict). Maybe there has to be
a "real" class on the mro, and type isn't one, or it's a wild card ;-)

I haven't really grokked the error message's true meaning, not having
dealt with metaclass conflict before. Not ready today, sorry ;-)

class M_A(type): pass ... class A: __metaclass__ = M_A ... def foo(*args): print args; return 'silliness' ... def foo(cname, cbases, cdict): ... print 'cname, cbases:', cname, cbases
... print 'cdict:', cdict
... mt = type('M_B',(type,),{})
... print 'mt:', mt
... print 'mt.mro(mt):', mt.mro(mt)
... print 'mt.__new__:', mt.__new__
... something = mt.__new__(mt, cname, cbases, cdict)
... print 'something:', something
... return something
... class B(A): __metaclass__ = foo ...
cname, cbases: B (<class '__main__.A'>,)
cdict: {'__module__': '__main__', '__metaclass__': <function foo at 0x02EEBD84>}
mt: <class '__main__.M_B'>
mt.mro(mt): [<class '__main__.M_B'>, <type 'type'>, <type 'object'>]
mt.__new__: <built-in method __new__ of type object at 0x1E1BF670>
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 8, in foo
TypeError: Error when calling the metaclass bases
metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the
metaclasses of all its bases def bar(cname, cbases, cdict): ... print 'cname, cbases:', cname, cbases
... print 'cdict:', cdict
... mt = type
... print 'mt:', mt
... print 'mt.mro(mt):', mt.mro(mt)
... print 'mt.__new__:', mt.__new__
... something = mt.__new__(mt, cname, cbases, cdict)
... print 'something:', something
... return something
... class B(A): __metaclass__ = bar ...
cname, cbases: B (<class '__main__.A'>,)
cdict: {'__module__': '__main__', '__metaclass__': <function bar at 0x02EEBDF4>}
mt: <type 'type'>
mt.mro(mt): [<type 'type'>, <type 'object'>]
mt.__new__: <built-in method __new__ of type object at 0x1E1BF670>
something: <class '__main__.B'>
And the something returned, whatever it is, if no checking is triggered by normal use,
gets bound to the class name, e.g.,
class C(A): __metaclass__ = lambda *a:('silly', 'result') ... C

('silly', 'result')

Regards,
Bengt Richter

[Pedro Werneck]

On Mon, 19 Sep 2005 05:22:09 GMT
bo**@oz.net (Bengt Richter) wrote:

And the something returned, whatever it is, if no checking is
triggered by normal use, gets bound to the class name, e.g.,

Yes... this is the intended behaviour. In fact, the issue is already
solved and is really just a documentation problem.

Check the comments on the bug report here:

http://sourceforge.net/tracker/index...70&atid=105470

--
Pedro Werneck