New+old-style multiple inheritance

[stephenpas]

We are trying to monkey-patch a third-party library that mixes new and
old-style classes with multiple inheritance. In so doing we have
uncovered some unexpected behaviour:

<quote>
class Foo:
pass

class Bar(object):
pass

class Baz(Foo,Bar):
pass

# Monkey-patch Foo to add a special method
def my_nonzero(self ):
print "my_nonzero called"
return False
Foo.__nonzero__ = my_nonzero

b = Baz()

print "doing the test on Baz(Foo,Bar). Should return false"
if b:
print "true"
else:
print "false"
</quote>

Produces this output:

doing the test on Baz(Foo,Bar). Should return false
true

With some experimentation it is clear that this behaviour only occurs
when you combine new+old-style multiple inheritance, monkey-patching
and special methods. If Foo and Bar are either old or new-style it
works. calling b.__nonzero__() directly works. Defining __nonzero__
within Foo works.

I know this level of messing with python internals is a bit risky but
I'm wondering why the above code doesn't work.

Any ideas?

Cheers,
Stephen.

[Jonathan Gardner]

On Dec 18, 7:08 am, "stephen...@goo glemail.com"
<stephen...@goo glemail.comwrot e:

We are trying to monkey-patch a third-party library that mixes new and
old-style classes with multiple inheritance. In so doing we have
uncovered some unexpected behaviour:

<snip>

>
I know this level of messing with python internals is a bit risky but
I'm wondering why the above code doesn't work.

You've already discovered why--you're mixing old and new style
classes.

Monkey patching is definitely unpythonic. You must be a Ruby guy. Why
don't you try doing something else to get the behavior you want,
something more explicit?

[Gabriel Genellina]

On 18 dic, 12:08, "stephen...@goo glemail.com"
<stephen...@goo glemail.comwrot e:

We are trying to monkey-patch a third-party library that mixes new and
old-style classes with multiple inheritance. In so doing we have
uncovered some unexpected behaviour:

<quote>
class Foo:
pass

class Bar(object):
pass

class Baz(Foo,Bar):
pass

# Monkey-patch Foo to add a special method
def my_nonzero(self ):
print "my_nonzero called"
return False
Foo.__nonzero__ = my_nonzero

b = Baz()

print "doing the test on Baz(Foo,Bar). Should return false"
if b:
print "true"
else:
print "false"
</quote>

Produces this output:

doing the test on Baz(Foo,Bar). Should return false
true

With some experimentation it is clear that this behaviour only occurs
when you combine new+old-style multiple inheritance, monkey-patching
and special methods. If Foo and Bar are either old or new-style it
works. calling b.__nonzero__() directly works. Defining __nonzero__
within Foo works.

I know this level of messing with python internals is a bit risky but
I'm wondering why the above code doesn't work.

I think I can barely explain what happens here (but I may be
absolutely wrong! Please someone with greater knowledge of Python
innards correct whatever is wrong on my description!)

type objects contain "slots" (function pointers) corresponding to the
"magic" methods like __nonzero__ (this one is stored into the
nb_nonzero slot). new-style classes are types; old-style classes are
not (they're all instances of classobj).
The slots are populated when a new type is created (e.g., when
creating a new-style class) and are updated when a magic attribute is
set onto the type. By example, setting the __nonzero__ attribute on a
new-style class updates the nb_nonzero slot. old-style classes just
store the magic attribute in its __dict__. Note that if you patch Foo
*before* defining Baz, it works fine, because Baz sees the magic
attribute and can populate its nb_nonzero slot when the new type is
created.

When you define Baz, neither Foo nor Bar have a __nonzero__ at this
time, so the nb_nonzero slot on Baz is empty.
Later, when you alter Foo, Baz cannot notice it (Foo has no way to
notify Baz that something changed).

If Foo were a new-style class, things are different: new-style classes
maintain a list of subclasses, and the subclasses can then be notified
of changes. In particular, setting a magic attribute on a base class
notifies all its subclasses, and the corresponding slots are updated.

The problem seems to be exactly that: old-style base classes can't
notify its derived new-style classes when magic methods are added, so
the corresponding slots aren't updated. Always asking the base class
whether it has a magic method or not would slow down all method
lookups.

To force a slot update:

pyBaz.__nonzero __ = "xxx"
pydel Baz.__nonzero__
pybool(b)
my_nonzero called
False

(setting or deleting __nonzero__ triggers the slot update)

This is *why* it happens. How to avoid this... well, don't do that in
the first place :) Or try to patch the base class *before* the new-
style derived class is defined. Or replace the derived class with a
new version of itself (once the base was patched). Or, if you know all
the derived classes, force a slot update on them as above.

--
Gabriel Genellina

[Carl Banks]

On Dec 18, 10:08 am, "stephen...@goo glemail.com"
<stephen...@goo glemail.comwrot e:

We are trying to monkey-patch a third-party library that mixes new and
old-style classes with multiple inheritance.

New library? Geez, if people are dumb enough to do this, are you sure
you want your application to depend on their library?

Sometimes you have to work with code that's not up to your standards,
but come on.
Carl Banks

[Carl Banks]

On Dec 18, 2:09 pm, Jonathan Gardner
<jgardner.jonat hangardner....@ gmail.comwrote:

On Dec 18, 7:08 am, "stephen...@goo glemail.com"

<stephen...@goo glemail.comwrot e:

We are trying to monkey-patch a third-party library that mixes new and
old-style classes with multiple inheritance. In so doing we have
uncovered some unexpected behaviour:

<snip>

I know this level of messing with python internals is a bit risky but
I'm wondering why the above code doesn't work.

You've already discovered why--you're mixing old and new style
classes.

Monkey patching is definitely unpythonic. You must be a Ruby guy. Why
don't you try doing something else to get the behavior you want,
something more explicit?

Time and place.

A well-considered, timely monkey-patch can sometimes save all kinds of
workarounds and other headaches.
Carl Banks

[stephenpas]

On Dec 18, 8:25 pm, Carl Banks <pavlovevide... @gmail.comwrote :

On Dec 18, 2:09 pm, Jonathan Gardner

<jgardner.jonat hangardner....@ gmail.comwrote:

On Dec 18, 7:08 am, "stephen...@goo glemail.com"

<stephen...@goo glemail.comwrot e:

We are trying to monkey-patch a third-party library that mixes new and
old-style classes with multiple inheritance. In so doing we have
uncovered some unexpected behaviour:

<snip>

I know this level of messing with python internals is a bit risky but
I'm wondering why the above code doesn't work.

You've already discovered why--you're mixing old and new style
classes.

Monkey patching is definitely unpythonic. You must be a Ruby guy. Why
don't you try doing something else to get the behavior you want,
something more explicit?

Time and place.

A well-considered, timely monkey-patch can sometimes save all kinds of
workarounds and other headaches.

Carl Banks

Indeed, I chuckled at the idea I was a Ruby programmer. Of course
it's a bit of a hack but monkey-patching has been common place in the
Zope world for years (maybe Zope3 has rid itself of the problem).

Agreed, mixing new and old-style classes in multiple inheritance is
asking for trouble (although the Guido's essay on the new-style
classes suggests you can: http://www.python.org/download/relea.../#subclassing).
I won't name and shame the library involved but unfortunately we can't
always pick and choose the tools we work with unless we want to write
(or rewrite) everything ourselves.

Cheers,
Stephen.

[stephenpas]

Thank Gabriel,

That sounds exactly right and your work-around provides confirmation.
This code is an expedient solution for us so I expect we'll go with
forcing a slot update. Hopefully the library concerned is going to
fix the problem in a future release.

Cheers,
Stephen.

On Dec 18, 7:40 pm, Gabriel Genellina <[e-mail].a rwrote:

On 18 dic, 12:08, "stephen...@goo glemail.com"

<stephen...@goo glemail.comwrot e:

We are trying to monkey-patch a third-party library that mixes new and
old-style classes with multiple inheritance. In so doing we have
uncovered some unexpected behaviour:

<quote>
class Foo:
pass

class Bar(object):
pass

class Baz(Foo,Bar):
pass

# Monkey-patch Foo to add a special method
def my_nonzero(self ):
print "my_nonzero called"
return False
Foo.__nonzero__ = my_nonzero

b = Baz()

print "doing the test on Baz(Foo,Bar). Should return false"
if b:
print "true"
else:
print "false"
</quote>

Produces this output:

doing the test on Baz(Foo,Bar). Should return false
true

With some experimentation it is clear that this behaviour only occurs
when you combine new+old-style multiple inheritance, monkey-patching
and special methods. If Foo and Bar are either old or new-style it
works. calling b.__nonzero__() directly works. Defining __nonzero__
within Foo works.

I know this level of messing with python internals is a bit risky but
I'm wondering why the above code doesn't work.

I think I can barely explain what happens here (but I may be
absolutely wrong! Please someone with greater knowledge of Python
innards correct whatever is wrong on my description!)

type objects contain "slots" (function pointers) corresponding to the
"magic" methods like __nonzero__ (this one is stored into the
nb_nonzero slot). new-style classes are types; old-style classes are
not (they're all instances of classobj).
The slots are populated when a new type is created (e.g., when
creating a new-style class) and are updated when a magic attribute is
set onto the type. By example, setting the __nonzero__ attribute on a
new-style class updates the nb_nonzero slot. old-style classes just
store the magic attribute in its __dict__. Note that if you patch Foo
*before* defining Baz, it works fine, because Baz sees the magic
attribute and can populate its nb_nonzero slot when the new type is
created.

When you define Baz, neither Foo nor Bar have a __nonzero__ at this
time, so the nb_nonzero slot on Baz is empty.
Later, when you alter Foo, Baz cannot notice it (Foo has no way to
notify Baz that something changed).

If Foo were a new-style class, things are different: new-style classes
maintain a list of subclasses, and the subclasses can then be notified
of changes. In particular, setting a magic attribute on a base class
notifies all its subclasses, and the corresponding slots are updated.

The problem seems to be exactly that: old-style base classes can't
notify its derived new-style classes when magic methods are added, so
the corresponding slots aren't updated. Always asking the base class
whether it has a magic method or not would slow down all method
lookups.

To force a slot update:

pyBaz.__nonzero __ = "xxx"
pydel Baz.__nonzero__
pybool(b)
my_nonzero called
False

(setting or deleting __nonzero__ triggers the slot update)

This is *why* it happens. How to avoid this... well, don't do that in
the first place :) Or try to patch the base class *before* the new-
style derived class is defined. Or replace the derived class with a
new version of itself (once the base was patched). Or, if you know all
the derived classes, force a slot update on them as above.

--
Gabriel Genellina

[Terry Reedy]

<st********@goo glemail.comwrot e in message
news:a9******** *************** ***********@t1g 2000pra.googleg roups.com...
| Thank Gabriel,
|
| That sounds exactly right and your work-around provides confirmation.
| This code is an expedient solution for us so I expect we'll go with
| forcing a slot update. Hopefully the library concerned is going to
| fix the problem in a future release.

To work with 3.0, they will have to, or rather it will be done for them,
since there will only be new style classes.

[Michele Simionato]

On Dec 18, 9:25 pm, Carl Banks <pavlovevide... @gmail.comwrote :

A well-considered, timely monkey-patch can sometimes save all kinds of
workarounds and other headaches.

Carl Banks

+1

Michele Simionato

[George Sakkis]

On Dec 18, 3:16 pm, Carl Banks <pavlovevide... @gmail.comwrote :

On Dec 18, 10:08 am, "stephen...@goo glemail.com"

<stephen...@goo glemail.comwrot e:

We are trying to monkey-patch a third-party library that mixes new and
old-style classes with multiple inheritance.

New library? Geez, if people are dumb enough to do this, are you sure
you want your application to depend on their library?

Sometimes you have to work with code that's not up to your standards,
but come on.

Doing the armchair code reviewer without context is easy but my guess
would be that the old style library classes were written long before
the new style ones (perhaps even before the latter were introduced)
and/or written independently by different developers/groups, without
any plan to mix the two in the future. Also remember that up to 2.4
even the standard exception classes were old-style so it's not safe to
assume that you only have new style classes to worry about when the
very standard library includes lots of legacy code.

George