Magic methods in extension types

Jacek Generowicz

I am writing an extension type, and wish to add some magic methods
which are not catered for by the tp_<whatever> slots (eg tp_init ->
__init__) in PyTypeObject. My methods seem to work correctly when
invoked explicitly (eg obj.__iadd__(3)) but the method seems not to be
associated with the corresponding operator (ie obj += 3 does NOT
work).

What am I likely to be doing wrong ?

Jul 18 '05 #1

Michael Hudson

Jacek Generowicz <ja**************@cern.ch> writes:

I am writing an extension type, and wish to add some magic methods
which are not catered for by the tp_<whatever> slots (eg tp_init ->
__init__) in PyTypeObject. My methods seem to work correctly when
invoked explicitly (eg obj.__iadd__(3)) but the method seems not to be
associated with the corresponding operator (ie obj += 3 does NOT
work).

What am I likely to be doing wrong ?

Not finding the tp_as_number->nb_inplace_add field?

I think *all* magic methods correspond to slots in (or near) the type
object -- it's practically the definition of "magic method"!

Cheers,
mwh

--
I sense much distrust in you. Distrust leads to cynicism, cynicism
leads to bitterness, bitterness leads to the Awareness Of True
Reality which is referred to by those-who-lack-enlightenment as
"paranoia". I approve. -- David P. Murphy, alt.sysadmin.recovery

Jul 18 '05 #2

Jacek Generowicz

Michael Hudson <mw*@python.net> writes:

Not finding the tp_as_number->nb_inplace_add field?
( ... or tp_as_sequence ... )

I was afraid you (someone) was going to say that.
I think *all* magic methods correspond to slots in (or near) the type
object -- it's practically the definition of "magic method"!

Hmmm

class foo: ... def __iadd__(self,other):
... print "__iadd__ running"
... return self
... f = foo()
f += 2

__iadd__ running

I'd be surprised if I've added __iadd__ to a type object here, yet it
seems to work.

Python manages to map "+=" to the method called "__iadd__" in
user-defined classes, but not for extension types. What is the
essential difference that makes that mapping work in one case but not
in the other?

Jul 18 '05 #3

Michael Hudson

Jacek Generowicz <ja**************@cern.ch> writes:

Michael Hudson <mw*@python.net> writes:

Not finding the tp_as_number->nb_inplace_add field?
( ... or tp_as_sequence ... )

I was afraid you (someone) was going to say that.

Why?

I think *all* magic methods correspond to slots in (or near) the type
object -- it's practically the definition of "magic method"!

Hmmm

>>> class foo: ... def __iadd__(self,other):
... print "__iadd__ running"
... return self
... >>> f = foo()
>>> f += 2

__iadd__ running

I'd be surprised if I've added __iadd__ to a type object here, yet it
seems to work.

Well, the type of old-style classes has something in it's
tp_as_number->nb_inplace_add slot that looks in the class dictionary
for an __iadd__ method.

If you'd made that a new-style class, you would be surprised!
Python manages to map "+=" to the method called "__iadd__" in
user-defined classes, but not for extension types. What is the
essential difference that makes that mapping work in one case but
not in the other?

Well, for old-style classes a whole bunch of code like:

BINARY_INPLACE(instance_ior, "or", PyNumber_InPlaceOr)
BINARY_INPLACE(instance_ixor, "xor", PyNumber_InPlaceXor)
BINARY_INPLACE(instance_iand, "and", PyNumber_InPlaceAnd)
BINARY_INPLACE(instance_ilshift, "lshift", PyNumber_InPlaceLshift)
BINARY_INPLACE(instance_irshift, "rshift", PyNumber_InPlaceRshift)
BINARY_INPLACE(instance_iadd, "add", PyNumber_InPlaceAdd)
BINARY_INPLACE(instance_isub, "sub", PyNumber_InPlaceSubtract)
BINARY_INPLACE(instance_imul, "mul", PyNumber_InPlaceMultiply)
BINARY_INPLACE(instance_idiv, "div", PyNumber_InPlaceDivide)
BINARY_INPLACE(instance_imod, "mod", PyNumber_InPlaceRemainder)
BINARY_INPLACE(instance_ifloordiv, "floordiv", PyNumber_InPlaceFloorDivide)
BINARY_INPLACE(instance_itruediv, "truediv", PyNumber_InPlaceTrueDivide)

and for new-style classes much hair in typeobject.c:type_new and
therein called functions. Extension types don't go through type_new
and are expected to go the other way round, in a sense: define
something in tp_as_number->nb_inplace_add and a wrapper called
__iadd__ will be created for you.

Cheers,
mwh

--
Windows installation day one. Getting rid of the old windows
was easy - they fell apart quite happily, and certainly wont
be re-installable anywhere else. -- http://www.linux.org.uk/diary/
(not *that* sort of windows...)

Jul 18 '05 #4

Jacek Generowicz

Michael Hudson <mw*@python.net> writes:

Jacek Generowicz <ja**************@cern.ch> writes:

Michael Hudson <mw*@python.net> writes:

Not finding the tp_as_number->nb_inplace_add field?
( ... or tp_as_sequence ... )

I was afraid you (someone) was going to say that.

Why?

Because I'm too lazy to pollute my tiny extension type with a whole
thingy_as_number sturucture for the sake of just one maginc method :-)
Well, the type of old-style classes has something in it's
tp_as_number->nb_inplace_add slot that looks in the class dictionary
for an __iadd__ method.

If you'd made that a new-style class, you would be surprised!

Don't follow ... you mean if I had done this:

class foo(object): ... def __iadd__(self,other):
... print "__iadd__ running"
... return self
... f = foo()
f += 2

__iadd__ running

?
Python manages to map "+=" to the method called "__iadd__" in
user-defined classes, but not for extension types. What is the
essential difference that makes that mapping work in one case but
not in the other?

Well, for old-style classes a whole bunch of code like:

BINARY_INPLACE(instance_ior, "or", PyNumber_InPlaceOr)
BINARY_INPLACE(instance_ixor, "xor", PyNumber_InPlaceXor)
BINARY_INPLACE(instance_iand, "and", PyNumber_InPlaceAnd)
BINARY_INPLACE(instance_ilshift, "lshift", PyNumber_InPlaceLshift)
BINARY_INPLACE(instance_irshift, "rshift", PyNumber_InPlaceRshift)
BINARY_INPLACE(instance_iadd, "add", PyNumber_InPlaceAdd)
BINARY_INPLACE(instance_isub, "sub", PyNumber_InPlaceSubtract)
BINARY_INPLACE(instance_imul, "mul", PyNumber_InPlaceMultiply)
BINARY_INPLACE(instance_idiv, "div", PyNumber_InPlaceDivide)
BINARY_INPLACE(instance_imod, "mod", PyNumber_InPlaceRemainder)
BINARY_INPLACE(instance_ifloordiv, "floordiv", PyNumber_InPlaceFloorDivide)
BINARY_INPLACE(instance_itruediv, "truediv", PyNumber_InPlaceTrueDivide)

and for new-style classes much hair in typeobject.c:type_new and
therein called functions. Extension types don't go through type_new
and are expected to go the other way round, in a sense: define
something in tp_as_number->nb_inplace_add and a wrapper called
__iadd__ will be created for you.

Yuk.

You are, in summary, saying that by _far_ the simplest way of adding
__iadd__ to an extenion type is via tp_as_number->nb_inplace_add,
aren't you.

So be it.

Thanks.

Jul 18 '05 #5

Greg Ewing

Jacek Generowicz wrote:

You are, in summary, saying that by _far_ the simplest way of adding
__iadd__ to an extenion type is via tp_as_number->nb_inplace_add,
aren't you.

It's the *only* way to do it.

<plug shamelevel="0">

Unless you use Pyrex, of course:

http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/

</plug>

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg

Jul 18 '05 #6

Jacek Generowicz

Greg Ewing <gr**@cosc.canterbury.ac.nz> writes:

<plug shamelevel="0">

Unless you use Pyrex, of course:

http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/

</plug>

Hi Greg,

I've been wanting to pounce on Pyrex for ages. Unfortunately, it is
not politically correct for my current projects. I have vague hopes of
finding the time become sufficiently versed in Pyrex to be able to
make a convincing case for making it standard practice around here,
but, unfortunately, that time seems not to be getting any closer :-(

Jul 18 '05 #7

Michael Hudson

Jacek Generowicz <ja**************@cern.ch> writes:

Michael Hudson <mw*@python.net> writes:

Jacek Generowicz <ja**************@cern.ch> writes:

Michael Hudson <mw*@python.net> writes:

> Not finding the tp_as_number->nb_inplace_add field?

( ... or tp_as_sequence ... )

I was afraid you (someone) was going to say that.
Why?

Because I'm too lazy to pollute my tiny extension type with a whole
thingy_as_number sturucture for the sake of just one maginc method :-)

If __iadd__ is the *only* magic method you're adding, you deserve to
lose...

Well, the type of old-style classes has something in it's
tp_as_number->nb_inplace_add slot that looks in the class dictionary
for an __iadd__ method.

If you'd made that a new-style class, you would be surprised!

Don't follow ... you mean if I had done this:

>>> class foo(object): ... def __iadd__(self,other):
... print "__iadd__ running"
... return self
... >>> f = foo()
>>> f += 2 __iadd__ running

?

Yup: you've added something to a type object, something you'd said
you'd be surprised to hear.

Python manages to map "+=" to the method called "__iadd__" in
user-defined classes, but not for extension types. What is the
essential difference that makes that mapping work in one case but
not in the other?

Well, for old-style classes a whole bunch of code like:
[...]
and for new-style classes much hair in typeobject.c:type_new and
therein called functions. Extension types don't go through type_new
and are expected to go the other way round, in a sense: define
something in tp_as_number->nb_inplace_add and a wrapper called
__iadd__ will be created for you.

Yuk.

Why?
You are, in summary, saying that by _far_ the simplest way of adding
__iadd__ to an extenion type is via tp_as_number->nb_inplace_add,
aren't you.

Yes. I'm surprised at your apparent surprise, but then I've had my
head in the guts of Python's implementation for quite some time
now... implementing types in C is really quite different from
implementing them in Python, something which has only become
surprising since 2.2...

Cheers,
mwh

--
Gullible editorial staff continues to post links to any and all
articles that vaguely criticize Linux in any way.
-- Reason #4 for quitting slashdot today, from
http://www.cs.washington.edu/homes/k.../slashdot.html

Jul 18 '05 #8

Jacek Generowicz

Michael Hudson <mw*@python.net> writes:

Jacek Generowicz <ja**************@cern.ch> writes:

Michael Hudson <mw*@python.net> writes:

Jacek Generowicz <ja**************@cern.ch> writes:

> Michael Hudson <mw*@python.net> writes: and for new-style classes much hair in typeobject.c:type_new and
therein called functions. Extension types don't go through type_new
and are expected to go the other way round, in a sense: define
something in tp_as_number->nb_inplace_add and a wrapper called
__iadd__ will be created for you.
Yuk.

Why?

Partly because I misread (thinking that *I* had to write hairy stuff
for it to work), and partly because the dichotomy offends my sense of
symmetry, beauty, coherence, cleanlyness etc.

You are, in summary, saying that by _far_ the simplest way of adding
__iadd__ to an extenion type is via tp_as_number->nb_inplace_add,
aren't you.

Yes. I'm surprised at your apparent surprise, but then I've had my
head in the guts of Python's implementation for quite some time
now...

.... while I try to keep out of there. (I appreciate Python for its
ability to hide unimportant low-level details, not for it being a C
program; if I wanted to program in a low-level language, I wouldn't
need Python ... unless it would be to play with implementation details
of Python itself: you realize that I'm eagerly waiting for PyPy to
deliver :-)
implementing types in C is really quite different from
implementing them in Python, something which has only become
surprising since 2.2...

Yes, I was blindly hoping that type-class unification would have made
implementing extension types look more like implementing Python types.

Anyway, it's clear what I have to do. Thanks.

Jul 18 '05 #9