Pythonic API design: detailed errors when you usually don't care

[Simon Willison]

Hi all,

I have an API design question. I'm writing a function that can either
succeed or fail. Most of the time the code calling the function won't
care about the reason for the failure, but very occasionally it will.

I can see a number of ways of doing this, but none of them feel
aesthetically pleasing:

1.

try:
do_something()
except HttpError:
# An HTTP error occurred
except ApplicationError:
# An application error occurred
else:
# It worked!

This does the job fine, but has a couple of problems. The first is that
I anticipate that most people using my function won't care about the
reason; they'll just want a True or False answer. Their ideal API would
look like this:

if do_something():
# It succeeded
else:
# It failed

The second is that the common path is success, which is hidden away in
the 'else' clause. This seems unintuitive.

2.

Put the method on an object, which stores the reason for a failure:

if obj.do_something():
# It succeeded
else:
# It failed; obj.get_error_reason() can be called if you want to know
why

This has an API that is closer to my ideal True/False, but requires me
to maintain error state inside an object. I'd rather not keep extra
state around if I don't absolutely have to.

3.

error = do_something()
if error:
# It failed
else:
# It succeeded

This is nice and simple but suffers from cognitive dissonance in that
the function returns True (or an object evaluating to True) for
failure.

4.

The preferred approach works like this:

if do_something():
# Succeeded
else:
# Failed

BUT this works too...

ok = do_something()
if ok:
# Succeeded
else:
# ok.reason has extra information
reason = ok.reason

This can be implemented by returning an object from do_something() that
has a __nonzero__ method that makes it evaluate to False. This solves
my problem almost perfectly, but has the disadvantage that it operates
counter to developer expectations (normally an object that evaluates to
False is 'empty').

I know I should probably just pick one of the above and run with it,
but I thought I'd ask here to see if I've missed a more elegant
solution.

Thanks,

Simon

[Sybren Stuvel]

Simon Willison enlightened us with:

try:
do_something()
except HttpError:
# An HTTP error occurred
except ApplicationError:
# An application error occurred
else:
# It worked!

This does the job fine, but has a couple of problems.

I anticipate that most people using my function won't care about the
reason; they'll just want a True or False answer.

Then they can use an except clause that catches the superclass of all
the possible exceptions your function raises.

Their ideal API would look like this:

if do_something():
# It succeeded
else:
# It failed

This is the C way of doing things. The problem with relying on a
return value, is that failure could go unnoticed if the value isn't
checked. This might introduce nasty bugs.

Sybren
--
Sybren Stüvel
Stüvel IT - http://www.stuvel.eu/

[Steve Holden]

Simon Willison wrote:

Hi all,

I have an API design question. I'm writing a function that can either
succeed or fail. Most of the time the code calling the function won't
care about the reason for the failure, but very occasionally it will.

I can see a number of ways of doing this, but none of them feel
aesthetically pleasing:

1.

try:
do_something()
except HttpError:
# An HTTP error occurred
except ApplicationError:
# An application error occurred
else:
# It worked!

This does the job fine, but has a couple of problems. The first is that
I anticipate that most people using my function won't care about the
reason; they'll just want a True or False answer. Their ideal API would
look like this:

if do_something():
# It succeeded
else:
# It failed

The second is that the common path is success, which is hidden away in
the 'else' clause. This seems unintuitive.

2.

Put the method on an object, which stores the reason for a failure:

if obj.do_something():
# It succeeded
else:
# It failed; obj.get_error_reason() can be called if you want to know
why

This has an API that is closer to my ideal True/False, but requires me
to maintain error state inside an object. I'd rather not keep extra
state around if I don't absolutely have to.

3.

error = do_something()
if error:
# It failed
else:
# It succeeded

This is nice and simple but suffers from cognitive dissonance in that
the function returns True (or an object evaluating to True) for
failure.

4.

The preferred approach works like this:

if do_something():
# Succeeded
else:
# Failed

BUT this works too...

ok = do_something()
if ok:
# Succeeded
else:
# ok.reason has extra information
reason = ok.reason

This can be implemented by returning an object from do_something() that
has a __nonzero__ method that makes it evaluate to False. This solves
my problem almost perfectly, but has the disadvantage that it operates
counter to developer expectations (normally an object that evaluates to
False is 'empty').

I know I should probably just pick one of the above and run with it,
but I thought I'd ask here to see if I've missed a more elegant
solution.

Thanks,

Simon

I should have thought the simplest spelling of your requirements would be

try:
do_something()
print "It worked"
except:
# it didn't

However there are good reasons why this pattern isn't often recommended:
it masks types of exception that you may not have anticipated.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[Carl Banks]

Simon Willison wrote:

I have an API design question. I'm writing a function that can either
succeed or fail. Most of the time the code calling the function won't
care about the reason for the failure, but very occasionally it will.

I can see a number of ways of doing this, but none of them feel
aesthetically pleasing:

[snip]

Whether it's aesthetically pleasing or not, raising an exception is the
way to do it.

You seem to care about your users. Keep this in mind: if your function
returns an error code, then it puts an burden on users to A. always
check for errors, and B. always check for errors right at the point
where the function is called.

Python has exceptions so that programmers can write code without
checking for an error in every other line like they would have to in C.
With exceptions, users are free to not bother checking for errors, if
program termination is acceptable, or to handle the exception somewhere
else. Please don't circumvent this on the misguided pretense of
"helping" the user.

If you really think most users would prefer to check the return value
for an error, then I'd recommend two versions of the function: the
regular, Pythonic version that raises exceptions, and the convenience
version that swallows the exception and returns a code. (Compare a[1]
vs. a.get(1).)

This is nice and simple but suffers from cognitive dissonance in that
the function returns True (or an object evaluating to True) for
failure.

BTW, this is true in a lot of places. Back in the days of DOS, zero
meant success, non-zero was an error code. Zero also signals success
for process exit codes on Unix. Also on Unix, negative values are
often considered errors, non-negative success. The idea that error ==
0 is not self-evident, and often not true.

If it helps, consider using string error codes, or defining some
constants to check against ("if error == NO_ERROR").
Carl Banks

[Steve Holden]

Steve Holden wrote:

Simon Willison wrote:

>>Hi all,

I have an API design question. I'm writing a function that can either
succeed or fail. Most of the time the code calling the function won't
care about the reason for the failure, but very occasionally it will.

I can see a number of ways of doing this, but none of them feel
aesthetically pleasing:

1.

try:
do_something()
except HttpError:
# An HTTP error occurred
except ApplicationError:
# An application error occurred
else:
# It worked!

This does the job fine, but has a couple of problems. The first is that
I anticipate that most people using my function won't care about the
reason; they'll just want a True or False answer. Their ideal API would
look like this:

if do_something():
# It succeeded
else:
# It failed

The second is that the common path is success, which is hidden away in
the 'else' clause. This seems unintuitive.

2.

Put the method on an object, which stores the reason for a failure:

if obj.do_something():
# It succeeded
else:
# It failed; obj.get_error_reason() can be called if you want to know
why

This has an API that is closer to my ideal True/False, but requires me
to maintain error state inside an object. I'd rather not keep extra
state around if I don't absolutely have to.

3.

error = do_something()
if error:
# It failed
else:
# It succeeded

This is nice and simple but suffers from cognitive dissonance in that
the function returns True (or an object evaluating to True) for
failure.

4.

The preferred approach works like this:

if do_something():
# Succeeded
else:
# Failed

BUT this works too...

ok = do_something()
if ok:
# Succeeded
else:
# ok.reason has extra information
reason = ok.reason

This can be implemented by returning an object from do_something() that
has a __nonzero__ method that makes it evaluate to False. This solves
my problem almost perfectly, but has the disadvantage that it operates
counter to developer expectations (normally an object that evaluates to
False is 'empty').

I know I should probably just pick one of the above and run with it,
but I thought I'd ask here to see if I've missed a more elegant
solution.

Thanks,

Simon

I should have thought the simplest spelling of your requirements would be

try:
do_something()
print "It worked"
except:
# it didn't

However there are good reasons why this pattern isn't often recommended:
it masks types of exception that you may not have anticipated.

The recipe for only catching known errors but giving them all the same
response would be

try:
do_something()
print "It worked!"
except (ThisError, ThatError, TheOtherError):
print "It failed!"

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[Paul Rubin]

Steve Holden <st***@holdenweb.comwrites:

I should have thought the simplest spelling of your requirements would be

try:
do_something()
print "It worked"
except:
# it didn't

This usage is generally disparaged because of the possibility of
exceptions having nothing to do with do_something, including
asynchronous exceptions like KeyboardInterrupt, that really should get
handed up to higher levels of the program.

An approach not mentioned is for do_something to arrange that its
"expected" exceptions would all belong to subclasses of some single
class, so you could catch all those exceptions at once:

try: do_something()
except SomethingException:
# it didn't work

or alternatively you could break out the individual subclasses:

try: do_something()
except SomethingHTTPException:
# it didn't work for this reason
except SomethingWhatsitException:
# or that one

Or alternatively you could just put everything in one exception
with the underlying exception in the returned arg:

try: do_something()
except SomethingException, e:
# it didn't work, e gives the reason
...

def do_something():
try:
do_something_1() # do the real something
except (HTTPError, ApplicationError, ...), e:
# pass detailed exception info up to the caller
raise SomethingException, (sys.exc_info(), e)

[Aahz]

In article <11**********************@e3g2000cwe.googlegroups. com>,
Simon Willison <sw*******@gmail.comwrote:

>
try:
do_something()
except HttpError:
# An HTTP error occurred
except ApplicationError:
# An application error occurred
else:
# It worked!

Use a similar but different idiom:

try:
do_something()
except DoSomethingError:
# handle error

IOW, your do_something() function maps all expected errors to
DoSomethingError (or at least to a series of related exceptions that are
all subclasses of DoSomethingError).
--
Aahz (aa**@pythoncraft.com) <* http://www.pythoncraft.com/

"LL YR VWL R BLNG T S" -- www.nancybuttons.com

[Kay Schluehr]

Simon Willison wrote:

Hi all,

I have an API design question. I'm writing a function that can either
succeed or fail. Most of the time the code calling the function won't
care about the reason for the failure, but very occasionally it will.

I can see a number of ways of doing this, but none of them feel
aesthetically pleasing:

1.

try:
do_something()
except HttpError:
# An HTTP error occurred
except ApplicationError:
# An application error occurred
else:
# It worked!

This does the job fine, but has a couple of problems. The first is that
I anticipate that most people using my function won't care about the
reason; they'll just want a True or False answer. Their ideal API would
look like this:

if do_something():
# It succeeded
else:
# It failed

The second is that the common path is success, which is hidden away in
the 'else' clause. This seems unintuitive.

2.

Put the method on an object, which stores the reason for a failure:

if obj.do_something():
# It succeeded
else:
# It failed; obj.get_error_reason() can be called if you want to know
why

This has an API that is closer to my ideal True/False, but requires me
to maintain error state inside an object. I'd rather not keep extra
state around if I don't absolutely have to.

3.

error = do_something()
if error:
# It failed
else:
# It succeeded

This is nice and simple but suffers from cognitive dissonance in that
the function returns True (or an object evaluating to True) for
failure.

4.

The preferred approach works like this:

if do_something():
# Succeeded
else:
# Failed

BUT this works too...

ok = do_something()
if ok:
# Succeeded
else:
# ok.reason has extra information
reason = ok.reason

This can be implemented by returning an object from do_something() that
has a __nonzero__ method that makes it evaluate to False. This solves
my problem almost perfectly, but has the disadvantage that it operates
counter to developer expectations (normally an object that evaluates to
False is 'empty').

I know I should probably just pick one of the above and run with it,
but I thought I'd ask here to see if I've missed a more elegant
solution.

Thanks,

Simon

As I see it this can become an architectural concern. If callers care
"occasionally" which precision or care is actually required? Is it just
that a developer wants to see a stack-trace for debugging purposes?
Here logging might be adaequate while otherwise if application logics
is dependent it might be helpfull to define a contract between caller
and callee. This could be a handler that is passed by the caller to
do_something() that must be aware of it by defining an optional
parameter in its interface. This is definitely framework overhead and
although I wouldn't qualify it as "ugly" it makes your program more
complex and dependent on more components. Finally you have to balance
the tradeoff. All other approaches I've seen here tried to tame the
effect of throwing exceptions that were only needed occasionally. It's
like dealing with unwanted checked exceptions in Java...

[John J. Lee]

"Simon Willison" <sw*******@gmail.comwrites:

Hi all,

I have an API design question. I'm writing a function that can either
succeed or fail. Most of the time the code calling the function won't
care about the reason for the failure, but very occasionally it will.

I can see a number of ways of doing this, but none of them feel
aesthetically pleasing:

1.

try:
do_something()
except HttpError:
# An HTTP error occurred
except ApplicationError:
# An application error occurred
else:
# It worked!

This does the job fine, but has a couple of problems. The first is that
I anticipate that most people using my function won't care about the
reason; they'll just want a True or False answer. Their ideal API would
look like this:

if do_something():
# It succeeded
else:
# It failed

The second is that the common path is success, which is hidden away in
the 'else' clause. This seems unintuitive.

It's hard to answer this without knowing any of the real details
you've hidden from us: I'm not sure I believe you that the code at the
call site necessarily needs to know whether the function call
succeeded at all. Perhaps the preferred usage might sometimes be:

do_something()
Exceptions raised may be caught (or not, if the application does not
find that necessary), elsewhere than at the call site -- they can be
caught further up the stack. You are of course aware of this fact,
but your discussion seems to imply that this crucial piece of
knowledge was not active in your mind when you wrote it.

I think Python users are right to "default to" raising exceptions.
There's a nice label for this practice:

http://c2.com/cgi/wiki?SamuraiPrinciple

"""
Samurai Principle

Return victorious, or not at all.
"""
:-)

(though I imagine that term is used with subtly different meanings by
different people...)
As others have suggested, often it is useful if most exceptions that
can be raised by a callable are instances of a single class. This can
be achieved through inheritance (make them all derive from one class)
or composition (stuff the various original exceptions into a wrapper
exception).
John