Python Mystery Theatre -- Episode 1: Exceptions

[Raymond Hettinger]

For your amusement and edification, I'm working on a
series of Python puzzles designed to highlight areas of
the language known only to those who have read the
docs more than once.

Each of the following sections contains a code snippet
that someone, somewhere might find a little mysterious.

Your goal is to sleuth through the code, identify what
was expected versus what really happened, and to
explain it briefly so that others will find it blindingly
obvious.

The mysteries are not difficult, but I would suprised
if most readers didn't learn something new along the way.

Of course, if you are a potential bot, then these mysteries
will be very difficult because all you will see is the code
operating as designed, implemented, documented and tested.

When you post your results, I would be interested in
knowing whether you already knew what was going
on or whether you had to resort to:

* reading other posts
* googling
* experimenting
* or worse, re-reading the docs
Raymond Hettinger
ACT I ---------------------------------------

s = list('abc')
try: .... result = s['a']
.... except IndexError, TypeError:
.... print 'Not found'
....

Traceback (most recent call last):
File "<pyshell#11>", line 2, in -toplevel-
result = s['a']
TypeError: list indices must be integers

ACT II -------------------------------------------- class MyMistake(Exception): .... pass
try: .... raise MyMistake, 'try, try again'
.... except MyMistake, msg:
.... print type(msg)
....

<type 'instance'>

ACT III -------------------------------------------- class Prohibited(Exception): .... def __init__(self):
.... print 'This class of should never get initialized'
.... raise Prohibited() This class of should never get initialized

Traceback (most recent call last):
File "<pyshell#40>", line 1, in -toplevel-
raise Prohibited()
Prohibited: <unprintable instance object> raise Prohibited This class of should never get initialized

Traceback (most recent call last):
File "<pyshell#41>", line 1, in -toplevel-
raise Prohibited
Prohibited: <unprintable instance object>

ACT IV ----------------------------------------------- module = 'Root'
try: .... raise module + 'Error'
.... except 'LeafError':
.... print 'Need leaves'
.... except 'RootError':
.... print 'Need soil'
.... except:
.... print 'Not sure what is needed'
....

Not sure what is needed

ACT V ----------------------------------------------- try:

.... raise KeyError('Cannot find key')
.... except LookupError, msg:
.... print 'Lookup:', msg
.... except OverflowError, msg:
.... print 'Overflow:', msg
.... except KeyError, msg:
.... print 'Key:', msg
Lookup: 'Cannot find key'

[Ben Finney]

On Sat, 12 Jul 2003 06:56:52 GMT, Raymond Hettinger wrote:

For your amusement and edification, I'm working on a series of Python
puzzles designed to highlight areas of the language known only to
those who have read the docs more than once.

Excellent stuff! Any plans to put these online, perhaps with collation
of the responses? How many episodes are currently planned?

I imagine that some of the "blindingly obvious" explanations would be
welcome in the docs :-)

--
\ "I wish a robot would get elected president. That way, when he |
`\ came to town, we could all take a shot at him and not feel too |
_o__) bad." -- Jack Handey |
http://bignose.squidly.org/ 9CFE12B0 791A4267 887F520C B7AC2E51 BD41714B

[Raymond Hettinger]

> Your goal is to sleuth through the code, identify what

was expected versus what really happened, and to
explain it briefly so that others will find it blindingly
obvious.

P.S. There's extra credit if you can also devine why Python
was designed/implemented with the demonstrated behaviors.
Raymond Hettinger

[Erik Max Francis]

Raymond Hettinger wrote:

(I've pasted in code I needed to use to illustrate or understand the
problems.)

ACT I ---------------------------------------

s = list('abc')
try: ... result = s['a']
... except IndexError, TypeError:
... print 'Not found'
...

Traceback (most recent call last):
File "<pyshell#11>", line 2, in -toplevel-
result = s['a']
TypeError: list indices must be integers
This one's easy. You didn't try to catch IndexErrors and TypeErrors,
you tried to catch an IndexError and, if caught, assign the exception
instance to TypeError. The except clause takes an exception class or
tuple of exception classes, and then an optional instance variable name,
and then some other optional things. So:

...
except (IndexError, TypeError):
...

will fix this.
ACT II -------------------------------------------- class MyMistake(Exception): ... pass
try: ... raise MyMistake, 'try, try again'
... except MyMistake, msg:
... print type(msg)
...

<type 'instance'>
I'm not sure what mystery you're trying to get at here. This is what
Python prints for all instances:

class C: .... pass
.... c = C()
type(c) <type 'instance'>

An exception is an instance of an exception class, so it looks like an
instance like any other. If you wanted the name of the class, then use
__class__:
c.__class__ <class __main__.C at 0x814f9e4> c.__class__.__name__ 'C'
ACT III --------------------------------------------

class Prohibited(Exception): ... def __init__(self):
... print 'This class of should never get initialized'
... raise Prohibited() This class of should never get initialized

Traceback (most recent call last):
File "<pyshell#40>", line 1, in -toplevel-
raise Prohibited()
Prohibited: <unprintable instance object> raise Prohibited This class of should never get initialized

Traceback (most recent call last):
File "<pyshell#41>", line 1, in -toplevel-
raise Prohibited
Prohibited: <unprintable instance object>
Is this some IDLE-specific thing? I don't see this at all:
class Prohibited(Exception): .... def __init__(self):
.... print 'This class should have never gotten initialized'
.... raise Prohibited() This class should have never gotten initialized
Traceback (most recent call last):
File "<stdin>", line 1, in ?
__main__.Prohibited>>> raise Prohibited This class should have never gotten initialized
Traceback (most recent call last):
File "<stdin>", line 1, in ?
__main__.Prohibited>>>

There is indeed no newline between the printed names of the class and
the following prompt, this is not a pasting error, which strongly
suggests to me that it's what you're trying to get at but is exhibiting
itself in a different way in the interactive interpreter vs. IDLE.
Undoubtedly it happens because Prohibited overrides Exception, and
Prohibited needs an __init__ method, but that method does not call
Exception.__init__. At the very least, fixing this for me makes the
output a little more sane:
class Proscribed(Exception): .... def __init__(self):
.... Exception.__init__(self)
.... print "That information is proscribed. Enter your DNA#."
.... raise Proscribed() That information is proscribed. Enter your DNA#.
Traceback (most recent call last):
File "<stdin>", line 1, in ?
__main__.Proscribed raise Proscribed That information is proscribed. Enter your DNA#.
Traceback (most recent call last):
File "<stdin>", line 1, in ?
__main__.Proscribed
ACT IV ----------------------------------------------- module = 'Root'
try: ... raise module + 'Error'
... except 'LeafError':
... print 'Need leaves'
... except 'RootError':
... print 'Need soil'
... except:
... print 'Not sure what is needed'
...

Not sure what is needed
You used string exceptions and so deserve punishment :-). In this case,
string exceptions in except clauses are tested by identity, not
equality, so building them with concatenation is unlikely to create a
string with the same ID:
s = "dataspace retrieval"
t = "dataspace " + "retrieval"
s is t 0 s == t 1

Equality and identity aren't the same thing, and this is one of the rare
cases in Python where identity really matters (usually, it's merely an
optimization implementation detail). Short answer: Don't use string
exceptions. Long answer: Seriously, don't use string exceptions.
ACT V ----------------------------------------------- try:

... raise KeyError('Cannot find key')
... except LookupError, msg:
... print 'Lookup:', msg
... except OverflowError, msg:
... print 'Overflow:', msg
... except KeyError, msg:
... print 'Key:', msg

Lookup: 'Cannot find key'

This means that KeyError is a subclass of LookupError (something I
wouldn't have known off the top of my head but which was easy to
verify):
issubclass(KeyError, LookupError)

1

Except clauses go in order and test to see whether the exception object
is an instance of the specified class or any of its subclasses. Since
KeyError is a subclass of LookupError, all KeyErrors are LookupErrors
too, and that's the except clause that gets executed. If you actually
_did_ want to distinguish between KeyErrors and LookupErrors, you can
put the KeyError clause first:

...
except KeyError, e:
...
except LookupError, e:
...
...

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
__ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
/ \ God said: "Let Newton be"; and all was light.
\__/ Alexander Pope

[Peter Hansen]

Erik Max Francis wrote:

> ACT II --------------------------------------------

>> class MyMistake(Exception):

... pass

>> try:

... raise MyMistake, 'try, try again'
... except MyMistake, msg:
... print type(msg)
...

<type 'instance'>

I'm not sure what mystery you're trying to get at here. This is what
Python prints for all instances:

[snip]

He's showing a case where a programmer thought he/she was using
a sort of "parallel form" with the "MyMistake, 'try, try again'"
part and the "MyMistake, msg" part.

The programmer expected print type(msg) to show "<type 'str'>".

This is probably an example of an error promoted by leaving the
redundant "raise Class,args" form of exception-raising in Python,
instead of having a single obvious way: "raise Class(args)" as
would be more Pythonic. ;-)

-Peter

[Jack Diederich]

On Sat, Jul 12, 2003 at 06:56:52AM +0000, Raymond Hettinger wrote:

Since you included the answers I was really unsuprised by what happened,
but maybe I'm wrong as to the why (no other posts, docs, or searching read
for my explanations).

ACT I ---------------------------------------

s = list('abc')
try: ... result = s['a']
... except IndexError, TypeError:
... print 'Not found'
...

Traceback (most recent call last):
File "<pyshell#11>", line 2, in -toplevel-
result = s['a']
TypeError: list indices must be integers
A TypeError was thrown, but writing this as
except (IndexError,), type_error:
explains why it wasn't caught.
ACT II -------------------------------------------- class MyMistake(Exception): ... pass
try: ... raise MyMistake, 'try, try again'
... except MyMistake, msg:
... print type(msg)
...

<type 'instance'>
type() of most any (new style?) object will print this
ACT III -------------------------------------------- class Prohibited(Exception): ... def __init__(self):
... print 'This class of should never get initialized'
... raise Prohibited() This class of should never get initialized

Traceback (most recent call last):
File "<pyshell#40>", line 1, in -toplevel-
raise Prohibited()
Prohibited: <unprintable instance object> raise Prohibited This class of should never get initialized

Traceback (most recent call last):
File "<pyshell#41>", line 1, in -toplevel-
raise Prohibited
Prohibited: <unprintable instance object>
This one is new to me, FWIW I write two kinds of exceptions,
one that is just defined as MyGuy(Excption):pass and another
where I define both __init__ and __repr__ to print what I want.
ACT IV ----------------------------------------------- module = 'Root'
try: ... raise module + 'Error'
... except 'LeafError':
... print 'Need leaves'
... except 'RootError':
... print 'Need soil'
... except:
... print 'Not sure what is needed'
...

Not sure what is needed
There is a reason string exceptions are deprecated ;)
the string 'RootError' is not a subclass of the string 'RootError'
and thus won't be caught.

ACT V ----------------------------------------------- try:

... raise KeyError('Cannot find key')
... except LookupError, msg:
... print 'Lookup:', msg
... except OverflowError, msg:
... print 'Overflow:', msg
... except KeyError, msg:
... print 'Key:', msg
Lookup: 'Cannot find key'

LookupError is the parent of KeyError and IndexError. I generally
catch the more specific list/dict exceptions depending on what I'm
trying to access.
The fact that I'm very certain about my answers increases the likelyhood
that I'm in fact wrong ;)

-jack

[I orignally had a long thing about the weird exceptions thrown by mmap here,
but decided A: no one cared, B: a patch would be better than bitching. But
really, how can a method with no arguments raise a ValueError?]

[Aahz]

In article <3F***************@alcyone.com>,
Erik Max Francis <ma*@alcyone.com> wrote:

Raymond Hettinger wrote:

ACT III --------------------------------------------

>>> class Prohibited(Exception):

... def __init__(self):
... print 'This class of should never get initialized'
...

>>> raise Prohibited()

This class of should never get initialized

Traceback (most recent call last):
File "<pyshell#40>", line 1, in -toplevel-
raise Prohibited()
Prohibited: <unprintable instance object>

>>> raise Prohibited

This class of should never get initialized

Traceback (most recent call last):
File "<pyshell#41>", line 1, in -toplevel-
raise Prohibited
Prohibited: <unprintable instance object>

Is this some IDLE-specific thing?

Nope, the point here is that

raise Prohibited

will always create a Prohibited() instance. (See also Peter's post
about One True Way for exceptions.)
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

"Not everything in life has a clue in front of it...." --JMS

[Steven Taschuk]

Quoth Erik Max Francis:
[...]

But, as I recall, PEP 317 was outright rejected, so it looks like this
will be with us for a long time.
It was indeed rejected, primarily on the grounds that its putative
benefit did not justify the cost of migration. In the end, even I
(the PEP author) agree with that assessment.

I still believe, however, that the implicit instantiation which
Raymond's Acts II and III illustrate is a wart, fully deserves
inclusion in Python Mystery Theatre, and, as a matter of style,
should usually be avoided. Of course, ...
I personally have never had a problem with the distinction, raise C, x
always seemed fairly clean to me even though really what you mean is
raise C(x).

.... opinions vary. Guido, for example, was not convinced by the
PEP's arguments that implicit instantiation is a Bad Thing. (Note
that even if he had been, the migration cost would still have sunk
the PEP.)

After being rejected, the PEP grew the section
<http://www.python.org/peps/pep-0317.html#summary-of-discussion>
which briefly discusses these points and others.

--
Steven Taschuk o- @
st******@telusplanet.net 7O )
" (

[John J. Lee]

Oops, still not used to Gnus. My followup to the 'What's new with
Gnosis' thread was supposed to be to this thread.
John

[John J. Lee]

aa**@pythoncraft.com (Aahz) writes:

In article <3F***************@alcyone.com>,
Erik Max Francis <ma*@alcyone.com> wrote:

[...]

Prohibited: <unprintable instance object>

Is this some IDLE-specific thing?

Nope, the point here is that

raise Prohibited

will always create a Prohibited() instance. (See also Peter's post
about One True Way for exceptions.)

Perhaps Erik was wondering, as I was, where that "<unprintable instance
object>" came from. On my machine, 2.3b1 doesn't print that in
response to Raymond's example code. Maybe it's from 2.3b2 (which I'm
downloading ATM), or IDLE, or something else?
John

[Erik Max Francis]

"John J. Lee" wrote:

Perhaps Erik was wondering, as I was, where that "<unprintable
instance
object>" came from.

Indeed. I was quite aware of what was happening, just not clear on why
his particular Python session said something that mine didn't.

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
__ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
/ \ Now I must follow them!
\__/ Beowulf, King of the Geats

[Jason Trowbridge]

Okay, without looking at anybody else's answers, using the docs, or
trying out the examples. This is straight from the head, cold.

Act I
's' is a list, and you can't lookup the resulting indice of the
sequence using the brackets. A s.index( 'a') would be valid.

As for why the exception isn't caught: There is a missing tuple there.
The following would work:
except (IndexError, TypeError):

The way it is currently given would only catch IndexError exceptions.
The exception instance would then be bound to the TypeError name.

Act II
The type of msg is always type instance, because it is a class
instance. The actual class is held under the msg.__class__ attribute.

Act III
I believe that you need to call Exception.__init__(self) in your
constructor. When the string representation of the exception is
printed, it use's Exception.__str__() (maybe Exception.__repr__(). I
forget which one the traceback shows). Since the instance hasn't been
initialized properly, that function can't print out its information.

Act IV
This has to do with string exceptions. I'm not sure how a specific
string exception can be caught. In any case, this is part of the
reason to stay away from string exceptions. Additionally, string
exceptions are scheduled to eventually disappear from the language.

Act V
Without looking at the docs, I'd say that KeyError is derived from the
LookupError class. Exceptions are supposed to be listed most-specific
to least specific, which translates to child classes before parent
classes here.

Hmm, time to look at the answers other people have given, and find how
badly off I am!

_ () () Jason Trowbridge | "There has been a coup. The Mac
( ' .~. Generic Programmer | users are liberating the printers."
\ = o = | --Schlake
---"`-`-'"---+ ra****@nmt.edu |

[Chris Reedy]

Ok. I'll give this a try. For reference, I fall into the class of users
who have read the docs more than once. (I also have been a college
professor at one point in my career.)

Chris

P.S. I've already read other peoples answers; but, I'll try not to let
that affect mine too much.

Raymond Hettinger wrote:

ACT I ---------------------------------------

s = list('abc')
try:
... result = s['a']
... except IndexError, TypeError:
... print 'Not found'
...

Traceback (most recent call last):
File "<pyshell#11>", line 2, in -toplevel-
result = s['a']
TypeError: list indices must be integers
I didn't have to think about this one. It comes up often enough on
c.l.py and I've been personally bitten by it as well.

The first question that struck me is why the user was trying to use a
string index on a list. Two possible answers: (1) This was included by
Raymond just to trigger the exception. (2) The individual is actually
confused about the differences between lists and dictionaries and was
expecting something like this to happen:

s = list('abc')
s['a']

0

In the latter case, I don't have any blindingly obvious comment except
to review the differences between lists and dictionaries.

The second question (the one I expect Raymond was really getting at) is
why the TypeError was not caught. The answer is that:

except IndexError, TypeError:

is syntactically the same as:

except IndexError, foo:

that is that the variable TypeError is created as a new local variable
which is assigned the exception that was raised, the same as what you
expected to happen when you used foo instead. The fix is:

except (IndexError, TypeError):

or maybe even to do:

except (IndexError, TypeError), foo:

to provide an additional visual clue as to exactly what is happening.
ACT II --------------------------------------------
class MyMistake(Exception):
... pass

try:
... raise MyMistake, 'try, try again'
... except MyMistake, msg:
... print type(msg)
...

<type 'instance'>
I learned something on this one. (I had to try this one to confirm my
suspicions.) The user is expecting this to print something like
'MyMistake', or maybe something like:

<class '__main__.MyMistake'>

The problem here is that Exception is an old-style class and type(x)
when x is an instance of an old-style class is always 'instance'. What
the user should do is:

print msg.__class__
ACT III --------------------------------------------
class Prohibited(Exception):
... def __init__(self):
... print 'This class of should never get initialized'
...
raise Prohibited()
This class of should never get initialized

Traceback (most recent call last):
File "<pyshell#40>", line 1, in -toplevel-
raise Prohibited()
Prohibited: <unprintable instance object>
raise Prohibited
This class of should never get initialized

Traceback (most recent call last):
File "<pyshell#41>", line 1, in -toplevel-
raise Prohibited
Prohibited: <unprintable instance object>
This one contains (at least) three issues that I could find.

1. The print statement 'This class should never get initialized',
appears to be an attempt to write an abstract class. Unfortunately, this
is not done properly. One problem here is that the Exception aspect of
prohibited is not initialized. This is what causes the '<unprintable
instance object>' behavior when instances of Prohibited are printed.

2. (After some experimenting on my part.) The phrase '<unprintable
instance object>' is produced when the __str__ method applied to an
exception when printing a traceback raises an exception. (I would assume
that this is required to avoid problems with recursive exceptions.)

3. (I already knew this one.) The fact that 'raise Prohibited()' and
'raise Prohibited' exhibit the same behavior is the result of the fact
that raising an instance of a class will raise that instance, raising a
class will cause an instance of that class to be constructed and then
raised.
ACT IV -----------------------------------------------
module = 'Root'
try:
... raise module + 'Error'
... except 'LeafError':
... print 'Need leaves'
... except 'RootError':
... print 'Need soil'
... except:
... print 'Not sure what is needed'
...

Not sure what is needed
This one is easy. (I knew this already from my second reading of the
documentation.) String exceptions are compared by object identity, that
is when 'RootError' is theException, rather than when 'RootError' == the
Exception, which is almost surely what the user was expecting. In
general when the string is constructed, like in this example, it becomes
very difficult no way to catch the exception.

If you want to throw string exceptions which are subsequently caught (I
can't think of a reason for doing this as opposed to defining a subclass
of Exception) you can try:

foo = 'My Error'
try:
...
raise foo
except foo:
print 'Foo caught'

which guarantees that the strings are identical.

Aside: (I wouldn't want to raise this to anyone who didn't already
understand the above.) This example also reveals that funny aspect of
the Python about the interpreter automatically interning strings that
look like variable names. Thus, in the example, the string 'RootError'
had to be constructed. If it was a literal, the example would have
behaved as "expected".
ACT V -----------------------------------------------
try:

... raise KeyError('Cannot find key')
... except LookupError, msg:
... print 'Lookup:', msg
... except OverflowError, msg:
... print 'Overflow:', msg
... except KeyError, msg:
... print 'Key:', msg
Lookup: 'Cannot find key'

(I had to confirm my guess on this one.) KeyError is a sub-class of
LookupError. So the except LookupError clause caught the exception
before the except KeyError clause was even checked. If you want to catch
both KeyError and LookupError in the same set of exceptions (which in my
mind is a questionable proposition), you would do:

except KeyError, msg:
...
except LookupError, msg:
...

Since the except clauses are processed serially, this would cause the
check for KeyError to occur before the one for LookupError.