new-style classes multiplication error message isn't very informative

[Jon Guyer]

>>> This is a fake line to confuse the stupid top-posting filter at gmane

We have a rather complicated class that, under certain circumstances, knows
that it cannot perform various arithmetic operations, and so returns
NotImplemented. As a trivial example:

class my: ... def __mul__(self, other):
... return NotImplemented
... my() * my() Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: unsupported operand type(s) for *: 'instance' and 'instance'

This error message isn't hugely meaningful to many of our users (and in
complicated expressions, I'd certainly benefit from knowing exactly which
subclasses of 'my' are involved), but it beats the behavior with new-style
classes:
class my(object): ... def __mul__(self, other):
... return NotImplemented
... my() * my() Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: can't multiply sequence to non-int

After a lot of googling and a lot of pouring over abstract.c, I now
understand that object() is defined with a tp_as_sequence, and so the error
message is the result of the last-ditch effort to do sequence concatentation.

What if I don't want to permit sequence concatenation?
Is there a way to unset tp_as_sequence?
Should I be inheriting from a different class? We started inheriting from
object because we want a __new__ method.

The "'instance' and 'instance'" message would be OK, but even better is the
result of this completely degenerate class:
class my(object): ... pass
... class your(my): ... pass
... my() * your()

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: unsupported operand type(s) for *: 'my' and 'your'

That's an error message I can actually do something with. Is there any way
to get this behavior when I do have a __mul__ method and sometimes return
NotImplemented?

We're doing most of our development in Python 2.3, if it matters.

[Steven D'Aprano]

On Fri, 30 Dec 2005 03:47:30 +0000, Jon Guyer wrote:

This is a fake line to confuse the stupid top-posting filter at gmane
We have a rather complicated class that, under certain circumstances, knows
that it cannot perform various arithmetic operations, and so returns
NotImplemented. As a trivial example:
>>> class my: ... def __mul__(self, other):
... return NotImplemented
... >>> my() * my() Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: unsupported operand type(s) for *: 'instance' and 'instance'

This error message isn't hugely meaningful to many of our users (and in
complicated expressions, I'd certainly benefit from knowing exactly which
subclasses of 'my' are involved)

Why don't you raise the exception yourself?

(Note the difference between NotImplemented and NotImplementedE rror.)

class Parrot: .... def __mul__(self, other):
.... raise NotImplementedE rror("Can't multiply %s yet!" %
.... self.__class__. __name__)
.... x = Parrot()*2

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 3, in __mul__
NotImplementedE rror: Can't multiply Parrot yet!
I've always considered NotImplementedE rror to be for situations that I
just haven't got around to implementing *yet*. For operations which should
not be implemented because they aren't meaningful, I define my own
exceptions.
I'd never noticed the behaviour of Python where it takes a return value of
NotImplemented and raises a ValueError. Unless somebody can tell me why
this is justified, I consider this at best a wart. If I return something,
that's my return value! I don't see why arithmetic operations are special
enough to justify this special behaviour.
--
Steven.

[Jon Guyer]

Steven D'Aprano <steve <at> REMOVETHIScyber .com.au> writes:

On Fri, 30 Dec 2005 03:47:30 +0000, Jon Guyer wrote:

We have a rather complicated class that, under certain circumstances, knows
that it cannot perform various arithmetic operations, and so returns
NotImplemented. As a trivial example:
Why don't you raise the exception yourself?

(Note the difference between NotImplemented and NotImplementedE rror.)

Because although A may not know how to multiply B, B might know how to multiply A
I'd never noticed the behaviour of Python where it takes a return value of
NotImplemented and raises a ValueError. Unless somebody can tell me why
this is justified, I consider this at best a wart. If I return something,
that's my return value! I don't see why arithmetic operations are special
enough to justify this special behaviour.

Python Reference Manual, Section 3.2:

----

NotImplemented This type has a single value. There is a single object with
this value. This object is accessed through the built-in name
NotImplemented. Numeric methods and rich comparison methods may return this
value if they do not implement the operation for the operands provided. (The
interpreter will then try the reflected operation, or some other fallback,
depending on the operator.) Its truth value is true.

----

This is exactly the behavior we want. Our code paths are simpler and less
error prone if A and B don't both know how to multiply with each other, and
this seems to be exactly what NotImplemented and __mul__/__rmul__ are
designed for.

[ziga.seilnacht]

Jon Guyer wrote:

This is a fake line to confuse the stupid top-posting filter at gmane
We have a rather complicated class that, under certain circumstances, knows
that it cannot perform various arithmetic operations, and so returns
NotImplemented. As a trivial example:
>>> class my: ... def __mul__(self, other):
... return NotImplemented
... >>> my() * my() Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: unsupported operand type(s) for *: 'instance' and 'instance'

This error message isn't hugely meaningful to many of our users (and in
complicated expressions, I'd certainly benefit from knowing exactly which
subclasses of 'my' are involved), but it beats the behavior with new-style
classes:
>>> class my(object): ... def __mul__(self, other):
... return NotImplemented
... >>> my() * my() Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: can't multiply sequence to non-int

After a lot of googling and a lot of pouring over abstract.c, I now
understand that object() is defined with a tp_as_sequence, and so the error
message is the result of the last-ditch effort to do sequence concatentation.

What if I don't want to permit sequence concatenation?
Is there a way to unset tp_as_sequence?
Should I be inheriting from a different class? We started inheriting from
object because we want a __new__ method.

The "'instance' and 'instance'" message would be OK, but even better is the
result of this completely degenerate class:
>>> class my(object): ... pass
... >>> class your(my): ... pass
... >>> my() * your()

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: unsupported operand type(s) for *: 'my' and 'your'

That's an error message I can actually do something with. Is there any way
to get this behavior when I do have a __mul__ method and sometimes return
NotImplemented?

We're doing most of our development in Python 2.3, if it matters.

This is a bug in Python. See this thread:
http://mail.python.org/pipermail/pyt...er/059046.html

and this patch:
http://sourceforge.net/tracker/?grou...il&aid=1390657

for more details.

[google]

<ziga.seilnac ht <at> gmail.com> writes:

This is a bug in Python. See this thread:
http://mail.python.org/pipermail/pyt...er/059046.html
OK, thanks. This doesn't strike me as the same issue (but maybe it is).
We're not getting NotImplemented returned, we're getting a TypeError;
just not a good TypeError.
and this patch:
http://sourceforge.net/tracker/?grou...il&aid=1390657

for more details.

The patch certainly appears to be on topic, though. Thanks.

[Steven D'Aprano]

On Fri, 30 Dec 2005 13:23:30 +0000, Jon Guyer wrote:

Steven D'Aprano <steve <at> REMOVETHIScyber .com.au> writes:

On Fri, 30 Dec 2005 03:47:30 +0000, Jon Guyer wrote:

> We have a rather complicated class that, under certain circumstances, knows
> that it cannot perform various arithmetic operations, and so returns
> NotImplemented. As a trivial example:

Why don't you raise the exception yourself?

(Note the difference between NotImplemented and NotImplementedE rror.)

Because although A may not know how to multiply B, B might know how to multiply A

Given the way Python is now, that's fair enough. But see below:

I'd never noticed the behaviour of Python where it takes a return value
of NotImplemented and raises a ValueError. Unless somebody can tell me
why this is justified, I consider this at best a wart. If I return
something, that's my return value! I don't see why arithmetic
operations are special enough to justify this special behaviour.

Python Reference Manual, Section 3.2:

----

NotImplemented This type has a single value. There is a single object
with this value. This object is accessed through the built-in name
NotImplemented. Numeric methods and rich comparison methods may return
this value if they do not implement the operation for the operands
provided. (The interpreter will then try the reflected operation, or
some other fallback, depending on the operator.) Its truth value is
true.

----

This is exactly the behavior we want. Our code paths are simpler and
less error prone if A and B don't both know how to multiply with each
other, and this seems to be exactly what NotImplemented and
__mul__/__rmul__ are designed for.

And I argue that they *shouldn't be*. If my code returns some object,
Python shouldn't muck about with it. Making a special case behaviour for
arithmetic is poor design -- arithmetic isn't special enough to justify
the special behaviour.

If you need to trigger a special behaviour (such as "I don't know how to
do this multiplication, please try the other object") the correct way to
do it is with an exception, just as iterators use the StopIteration
exception to trigger special behaviour, or list iteration use IndexError.
Returning a magic value that is captured and handled specially is just
wrong.

Exceptions are more generous too: Python could use your exception's
message string when printing the traceback, instead of making its own.

All of the above is, of course, in my not-so-humble opinion, and none of
it solves your problem. Sorry.
--
Steven.