exception handling in complex Python programs

[eliben]

Python provides a quite good and feature-complete exception handling
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.

Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.

Now, what do I mean more specifically... When a program starts growing
large, I find myself a bit scared of all the exceptions that might be
thrown: Python's exceptions as a result of runtime-detection of errors
(Python's dynamic typing also comes into play here), exceptions from
libraries used by the code, and exceptions from my lower-level
classes.
Python doesn't allow to specify which exceptions are thrown (C++'s
feature adding 'throw' after a function/method declaration specifying
the exceptions that can be thrown), and this leaves me at loss - what
should be caught and where ? Which errors should be left to
propagate ?

I've tried looking around the Python blogosphere, but there doesn't
seem to be much concern with this topic.

Apologies for the not-too-coherent post, but I suspect you feel the
pain too and can understand my meaning.

Eli

P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?

P.P.S. There's a great post on conditions (Common Lisp's exceptions)
here:
http://dlweinreb.wordpress.com/2008/...-really-about/
Not really CL specific, and can apply to Python's exceptions.

[Rafe]

On Aug 20, 12:19 am, eliben <[e-mail]:

Python provides a quite good and feature-complete exception handling
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.

Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.

Now, what do I mean more specifically... When a program starts growing
large, I find myself a bit scared of all the exceptions that might be
thrown: Python's exceptions as a result of runtime-detection of errors
(Python's dynamic typing also comes into play here), exceptions from
libraries used by the code, and exceptions from my lower-level
classes.
Python doesn't allow to specify which exceptions are thrown (C++'s
feature adding 'throw' after a function/method declaration specifying
the exceptions that can be thrown), and this leaves me at loss - what
should be caught and where ? Which errors should be left to
propagate ?

I've tried looking around the Python blogosphere, but there doesn't
seem to be much concern with this topic.

Apologies for the not-too-coherent post, but I suspect you feel the
pain too and can understand my meaning.

Eli

P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?

P.P.S. There's a great post on conditions (Common Lisp's exceptions)
here:http://dlweinreb.wordpress.com/2008/...ns-exceptions-...
Not really CL specific, and can apply to Python's exceptions.

Maybe I am oversimplifying (and I am here to learn), but I catch all
exceptions which otherwise would be hard to understand as a user. In
other words, when a better error message is useful.

Again, this is probably too simple to help, but the only way to ignore
certain types of exceptions, as far as I know, is to catch them and
pass.
e.g. this ignores type errors...

try:
somethingBad()
except TypeError, err:
pass
except Exception, err:
raise TypeError(err)
I suppose you could write a decorator to do this if you want it at the
function level, but that seems a bit to broad. Shouldn't exceptions be
on a case-by-case basis to add protection and return information
exactly where it is needed?

- Rafe

[Chris Mellon]

On Tue, Aug 19, 2008 at 12:19 PM, eliben <el****@gmail.comwrote:

Python provides a quite good and feature-complete exception handling
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.

Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.

Lots of people seem to have this fear. They treat exceptions like they
would treat error codes, trying to handle any possible case around any
particular call.

This is the wrong thing to do, and it only leads to more fragile code.
There are only 2 reasonable things to do with an exception:
1) handle it, by which I mean catch the exception knowing what error
condition it signifies, and take an appropriate action to correct the
error and
2) pass it up so something else has a chance at it.

Catching an exception when you don't know exactly what to do to fix it
is an error. At best, it will make debugging a program harder (because
you're losing context information about the error) and at worst it
adds bugs to your program. The way Javas checked exceptions encourage
empty or otherwise useless exception handlers is a major problem with
them.

There's some fear about presenting exceptions to the end user. That's
a user interface issues, not a software quality or engineering issue,
and it's resolvable with top-level handlers that log tracebacks
somewhere a user can't see them if desired.

Now, what do I mean more specifically... When a program starts growing
large, I find myself a bit scared of all the exceptions that might be
thrown: Python's exceptions as a result of runtime-detection of errors
(Python's dynamic typing also comes into play here), exceptions from
libraries used by the code, and exceptions from my lower-level
classes.
Python doesn't allow to specify which exceptions are thrown (C++'s
feature adding 'throw' after a function/method declaration specifying
the exceptions that can be thrown), and this leaves me at loss - what
should be caught and where ? Which errors should be left to
propagate ?

You should catch anything that you can correct. If you don't have a
specific answer for a specific exception, don't catch it.

I've tried looking around the Python blogosphere, but there doesn't
seem to be much concern with this topic.

Apologies for the not-too-coherent post, but I suspect you feel the
pain too and can understand my meaning.

Eli

P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?

Same rules apply. The only sort-of exception (no pun intended) is that
sometimes you want to re-raise as a different type of exception. Make
sure that you preserve all of the original information (including the
original traceback) if you do this.

P.P.S. There's a great post on conditions (Common Lisp's exceptions)
here:
http://dlweinreb.wordpress.com/2008/...-really-about/
Not really CL specific, and can apply to Python's exceptions.
--
http://mail.python.org/mailman/listinfo/python-list

[Fredrik Lundh]

Rafe wrote:

Again, this is probably too simple to help, but the only way to ignore
certain types of exceptions, as far as I know, is to catch them and
pass.
e.g. this ignores type errors...

try:
somethingBad()
except TypeError, err:
pass
except Exception, err:
raise TypeError(err)

so what kind of code are you writing where *type errors* are not
considered programming errors? (catching them and proceeding is one
thing, but catching them and ignoring them?)

I'd be really worried if I found that in a piece of source code I had to
maintain.

</F>

[dbpokorny]

On Aug 19, 10:19*am, eliben <[e-mail]:

P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?

You want to look up Easier to Ask Forgivness than Permission (EAFP)
which is touted as the "canonical" error-handling paradigm for Python.
This would give rise to the following function:

def do_something(filename):
try:
f = open(filename)
except IOError:
return err("File %s not found" % filename)
...

where err is a function that generates an error object that your
application understands. I personally think this is sloppy because you
have to couple the exception type with the function --- between file()
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3 and an IOError is thrown in the other three cases <bashes
head against keyboard>. The alternative is

def do_something(filename):
if not os.access(filename,os.R_OK):
return err(...)
f = open(filename)
...

or, (and this last one I actually used for a web application)

def do_something(filename):
if not os.access(filename,os.R_OK):
raise MyApplicationsExceptionType("File not found...")
f = open(filename)
...

The last one has the advantage that you can write a request handler
like this

def handle_http_request(...):
func = specific_handler_func(...)
try:
response = func(...)
return response
except MyApplicationsExceptionType as exc: #3.0 syntax
return error_response(exc,...)

Exceptions you don't expect (i.e. bugs) will get handled by the web
app framework, but you get to handle your own exceptions. Raising your
own exception type can also be employed with the EAFP approach like
this:

def do_something(filename):
try:
f = open(filename)
except IOError:
raise MyApplicationsExceptionType("File %s not found" %
filename)
...

If you are writing a library (for instance using a file for persistent
storage), then the answer to your question is "don't catch the
exception." Clients will expect the usual exception to be thrown when
a bad file name is passed.

David

[Steven D'Aprano]

On Tue, 19 Aug 2008 11:07:39 -0700, db*******@gmail.com wrote:

def do_something(filename):
if not os.access(filename,os.R_OK):
return err(...)
f = open(filename)
...

You're running on a multitasking modern machine, right? What happens when
some other process deletes filename, or changes its permissions, in the
time after you check for access but before you actually open it?

This isn't just a theoretical risk. There's a whole class of errors and
security holes based on similar race conditions. I find it amusing that
you consider it "sloppy" to deal with errors raised when actually opening
a file, but then recommend a technique that has a well-known failure mode.

That's not to say that I never use such techniques myself. For quick and
dirty scripts, where I can tolerate the risk of some other process moving
a file behind my back, I've been known to do something similar.

--
Steven

[eliben]

""" between file()
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3 and an IOError is thrown in the other three cases <bashes
head against keyboard>.
"""

This is *exactly* my concern with Python exceptions. You just never
know what can be thrown at you.

You want to look up Easier to Ask Forgivness than Permission (EAFP)
which is touted as the "canonical" error-handling paradigm for Python.

Any (semi)complete guides on this canonical paradigm online ? I've
only found some references in maillist discussions.

* def do_something(filename):
* * if not os.access(filename,os.R_OK):
* * * return err(...)
* * f = open(filename)
* * ...

But does os.access cover absolutely all the errors that can happen
during open() ? What guarantees it, and how can I know without you
teaching me, just from the docs ?

[Marc 'BlackJack' Rintsch]

On Tue, 19 Aug 2008 22:24:45 -0700, eliben wrote:

>You want to look up Easier to Ask Forgivness than Permission (EAFP)
which is touted as the "canonical" error-handling paradigm for Python.

Any (semi)complete guides on this canonical paradigm online ? I've only
found some references in maillist discussions.

There's the glossary in the documentation:

http://docs.python.org/tut/node18.html

Look under 'duck-typing', 'EAFP', and 'LBYL'.

Ciao,
Marc 'BlackJack' Rintsch

[Steven D'Aprano]

On Tue, 19 Aug 2008 22:24:45 -0700, eliben wrote:

""" between file()
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3 and an IOError is thrown in the other three cases <bashes head
against keyboard>.
"""

I'm curious about the claim that open() will raise NameError in Python3.
I find it hard to credit that claim, but if it is correct, what's the
justification for that?

This is *exactly* my concern with Python exceptions. You just never know
what can be thrown at you.

It's true that documentation of exceptions is relatively weak in Python.
And some functions can raise a bewildering array of exceptions. See for
example this thread where somebody notes that urllib2.urlopen() can raise
any of six different exceptions:

http://mail.python.org/pipermail/bay...il/003187.html

And I've had it raise socket.error, which makes seven. And the
documentation only mentions one of those exceptions.

However, as Gregory Smith describes, some of those seven exceptions are
subclasses of others, so it is possible to reduce it down to three cases
-- and arguably one of those cases (ValueError) is a bug that needs
fixing, not an exception that needs catching.

That's probably as bad as it gets in Python, at least for the standard
library. Most functions don't raise arbitrary exceptions for sensible
data, and if you pass non-sensible data then you should treat the
exception as a bug in your code and fix it.

--
Steven

[dbpokorny]

On Aug 19, 4:12*pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.auwrote:

On Tue, 19 Aug 2008 11:07:39 -0700, [e-mail] wrote:

* def do_something(filename):
* * if not os.access(filename,os.R_OK):
* * * return err(...)
* * f = open(filename)
* * ...

You're running on a multitasking modern machine, right? What happens when
some other process deletes filename, or changes its permissions, in the
time after you check for access but before you actually open it?

This is a good point - if you want to use the correct way of opening
files, and
you don't want to worry about tracking down exception types, then we
can probably
agree that the following is the simplest, easiest-to-remember way:

def do_something(filename):
try:
f = open(filename)
except:
<handle exception>
...

Opening files is a special case where EAFP is the only correct
solution (AFAIK). I still liberally sprinkle LBYL-style "assert
isinstance(...)" and other similar assertions in routines. The point
is that EAFP conflicts with the interest of reporting errors as soon
as possible (on which much has been written see, for instance Ch. 8 -
Defensive Programming in Code Complete), but LBYL conflicts with
correctness when objects can be shared.

Also, look at the man page for access. I have found at least two (one
on my Linux box, another online) that essentially say "never use it."
I completely forgot about this in my last post...

David

[Bruno Desthuilliers]

db*******@gmail.com a écrit :

On Aug 19, 10:19 am, eliben <[e-mail]:

>P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?

You want to look up Easier to Ask Forgivness than Permission (EAFP)
which is touted as the "canonical" error-handling paradigm for Python.
This would give rise to the following function:

def do_something(filename):
try:
f = open(filename)
except IOError:
return err("File %s not found" % filename)
...

where err is a function that generates an error object that your
application understands.

Sorry but that's IMHO totally broken.

This "error object" is useless (heck, we *do* have exceptions, don't we
???), *returning* it ruins the whole point of structured exception
handling and take us back to infamous C error code checking (which are
almost never checked), and - icing on the cake - the error message is
very possibly wrong and misleading (IOError dont necessarily mean 'file
not found'). This kind of exception "handling" manages to be worse than
no exception handling at all.

I personally think this is sloppy because you
have to couple the exception type with the function --- between file()
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3

??? I suspect this has nothing to do with any error happening while
opening the file. NameError means the name doesn't exists in the current
namespace nor it's enclosing namespaces. Could it be possible that
open() has been removed from Py3k ?

and an IOError is thrown in the other three cases <bashes
head against keyboard>. The alternative is

def do_something(filename):
if not os.access(filename,os.R_OK):
return err(...)
f = open(filename)
...

This gets even worse. race condition... Things can change between the
call to os.access and the call to open. Well-known antipattern.

or, (and this last one I actually used for a web application)

def do_something(filename):
if not os.access(filename,os.R_OK):
raise MyApplicationsExceptionType("File not found...")

You loose all the useful information you'd have from an IOError raised
by a direct call to open...

f = open(filename)
...

.... IOError that you're still likely to see happen anyway.

The last one has the advantage that you can write a request handler
like this

def handle_http_request(...):
func = specific_handler_func(...)
try:
response = func(...)
return response
except MyApplicationsExceptionType as exc: #3.0 syntax
return error_response(exc,...)

If you want to raise a different exception type - which can indeed be a
sensible thing to do, depending on the context -, you can do it safely
and keep accurate informations:

def do_something(filename):
try:
f = open(filename)
except IOError, e
raise MyApplicationsExceptionType(e.msg)
# could even pass whole traceback etc
# etc...

Exceptions you don't expect (i.e. bugs)

An exception you don't expect is not necessarily a bug. Try unplugging
your lan cable while writing to a socket connected to another computer...

(snip)

If you are writing a library (for instance using a file for persistent
storage), then the answer to your question is "don't catch the
exception." Clients will expect the usual exception to be thrown when
a bad file name is passed.

Indeed.

[Bruno Desthuilliers]

eliben a écrit :

>
This is *exactly* my concern with Python exceptions. You just never
know what can be thrown at you.

This rarely happen to be a problem in real life. At least not in mine.

Exception that can be expected (ie : IOError when dealing with files)
are usually obvious and more or less documented - or easy to figure out
(like TypeError and ValueError when trying to build an int from an
arbitrary object, KeyError when working with dicts, AttributeError when
inspecting an object, etc) from concrete use.

IOW, it's usually easy to know which exceptions you're able to deal with
at the lower level.

Any other exception is either a programming error - which needs to be
fixed, not hidden - or nothing you can deal with at the lower level - in
which case just let it propagate until some other layer above deal with
it (eventually just logging the error, displaying a user-friendly
message, and crashing if nothing else is possible).

> def do_something(filename):
if not os.access(filename,os.R_OK):
return err(...)
f = open(filename)
...

But does os.access cover absolutely all the errors that can happen
during open() ? What guarantees it, and how can I know without you
teaching me, just from the docs ?

The above code is a perfect antipattern. It's useless (if you can't
access the file, you'll get an IOError when trying to open it anyway),
it's wrong (things may change between the call to os.access and the call
to open), and it defeats the whole point of exception handling (by
returning some kind of error object instead of using exception handling).

[Rafe]

On Aug 20, 12:47 am, Fredrik Lundh <[e-mail]:

Rafe wrote:

Again, this is probably too simple to help, but the only way to ignore
certain types of exceptions, as far as I know, is to catch them and
pass.
e.g. this ignores type errors...

try:
somethingBad()
except TypeError, err:
pass
except Exception, err:
raise TypeError(err)

so what kind of code are you writing where *type errors* are not
considered programming errors? (catching them and proceeding is one
thing, but catching them and ignoring them?)

I'd be really worried if I found that in a piece of source code I had to
maintain.

</F>

I'm not it was just the first exception that came to mind... It is
pretty rare that I would pass an exception in fact. Maybe as a last-
resort test in some cases.

- Rafe

[Steven D'Aprano]

On Wed, 20 Aug 2008 09:23:22 -0700, db*******@gmail.com wrote:

On Aug 19, 4:12Â*pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.auwrote:

>On Tue, 19 Aug 2008 11:07:39 -0700, [e-mail] wrote:

Â* def do_something(filename):
Â* Â* if not os.access(filename,os.R_OK):
Â* Â* Â* return err(...)
Â* Â* f = open(filename)
Â* Â* ...

You're running on a multitasking modern machine, right? What happens
when some other process deletes filename, or changes its permissions,
in the time after you check for access but before you actually open it?

This is a good point - if you want to use the correct way of opening
files, and
you don't want to worry about tracking down exception types, then we can
probably
agree that the following is the simplest, easiest-to-remember way:

def do_something(filename):
try:
f = open(filename)
except:
<handle exception>

No, we don't agree that that is the correct way of opening files. Simple
it might be, but correct it is not.

If you're using Python 2.6 or greater, then you should be using a with
block to handle file opening.

And regardless of which version of Python, you shouldn't use a bare
except. It will mask exceptions you *don't* want to catch, including
programming errors, typos and keyboard interrupts.

Opening files is a special case where EAFP is the only correct solution
(AFAIK). I still liberally sprinkle LBYL-style "assert isinstance(...)"

Oh goodie. Another programmer who goes out of his way to make it hard for
other programmers, by destroying duck-typing.

BTW, assertions aren't meant for checking data, because assertions can be
turned off. Outside of test frameworks (e.g. unit tests), assertions are
meant for verifying program logic:

def foo(x):
# This is bad, because it can be turned off at runtime,
# destroying your argument checking.
assert isinstance(x, int)
# And it raises the wrong sort of exception.

# This is better (but not as good as duck-typing).
if not isinstance(x, int):
raise TypeError('x not an int')
# And it raises the right sort of error.

y = some_function(x)
# y should now be between -1 and 1.
assert -1 < y < 1
do_something_with(y)

and other similar assertions in routines. The point is that EAFP
conflicts with the interest of reporting errors as soon as possible

Not necessarily. Tell me how this conflicts with reporting errors as soon
as possible:

def do_something(filename):
try:
f = open(filename)
except IOError, e:
report_exception(e) # use a GUI, log to a file, whatever...

How could you report the exception any earlier than immediately?


--
Steven

[Marc 'BlackJack' Rintsch]

On Wed, 20 Aug 2008 18:37:02 +0200, Bruno Desthuilliers wrote:

>I personally think this is sloppy because you have to couple the
exception type with the function --- between file() and open() in
Python 2 and 3, a NameError is thrown with open() in Python 3

??? I suspect this has nothing to do with any error happening while
opening the file. NameError means the name doesn't exists in the current
namespace nor it's enclosing namespaces. Could it be possible that
open() has been removed from Py3k ?

No it's moved/changed but there's still a name for it in the builtin
namespace. `file` on the other hand is gone:

Python 3.0b2 (r30b2:65080, Aug 20 2008, 20:41:17)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>open

<class 'io.OpenWrapper'>

>>file

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'file' is not defined

Ciao,
Marc 'BlackJack' Rintsch

[Gabriel Genellina]

En Wed, 20 Aug 2008 21:49:14 -0300, db*******@gmail.com <db*******@gmail.comescribió:

On Aug 20, 10:59 am, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.auwrote:

>Oh goodie. Another programmer who goes out of his way to make it hard for
other programmers, by destroying duck-typing.

Remember kids: personal attacks are cruise control for cool.

So this was a simplification - most of the asserts I've written don't
actually use isinstance, partly because typing isinstance takes too
long. The point is to create a barricade so that when something goes
wrong, you get an assertion error against the code you wrote, not an
exception against doing something like

print("blah blah %s" % message)

where message turns out to be None. This is simply a way to make
debugging a more pleasant experience (quite valuable IMHO since
debugging is inherently difficult and can be quite aggravating). Here
is a sampling:

assert statelt.tag == 'stat'
assert len(path) 0 and path[0] == '/'
assert self.__expr != None

So here asserts are used to made distinctions that are more fine-
grained than type.

I think you missed the point. All of those look like program logic verification, and that's fine. But using assert to check user-supplied data is wrong (here "user" may be another programmer if you're developing a library). Assertions may be turned off at runtime.

If you look at the history of the EAFP concept in Python, then you see
that it comes from Alex Martelli's Python in a Nutshell around pages
113-114.

Mmm, I think it's older than that.

I don't think the code examples make the case for EAFP very
well (not that I know what EAFP is in the first place, given that it
is barely explained.

Ok, so you don't know what it is, but dislike it anyway?

I interpret it as "wrap questionable stuff in try/
except blocks"), and in any case there is practically no support for
using EAFP as the dominant error-handling paradigm.

Uh? I think that using try/except IS the "dominant error-handling paradigm" and that's just EAFP.

Martelli's posts in support of EAFP are heavily skewed towards a
multithreaded scenario and avoiding race conditions. IMHO, letting
locking and race condition concerns dictate your error-handling
paradigm is a case of the tail wagging the dog, especially when there
are alternatives to this particular tar pit: pipes or a shared nothing
architecture.

There are race conditions since multiprogramming existed, around '70, and I'm afraid they'll stay for a long time...

--
Gabriel Genellina

[Richard Levasseur]

One common-place thing I've noticed in a lot of python code is that
every package or module has a main Error class, and all sub-types
inherit from that class. So you just catch mylib.Error, and you're
going to catch all the exceptions that package generates directly.

There seems to be a lot of concern about what exceptions a functions
might throw, and what to do when it happens. The way I see it, there
are only two types of exceptions: those you can recover from, and
those you can't.

The only ones -really- worth catching are those you can recover from.
There's a middle-ground type of 'cleanup', to free any resources, but
those generally go in a finally block, not an except block.

For the ones you can't handle, it doesn't matter if you catch them or
not. If you catch them, what do you do? Log an error message, then
rethrow it. You're still throwing an exception, so you haven't really
gained anything. You might repackage it and put additional
information in the exception so you can do something at a higher
level. What that is, I don't know. I don't think I've ever passed
information up in an exception that was of use to the program, and I'm
hard pressed to think of any information you could provide that could -
fix- the problem.

If you can derive recoverable information, then why rethrow? Thats
pretty much a recoverable situation, so there's not need to rethrow.

In java, there are checked exceptions, which are nice - they tell you
what a function might throw, so you know what to catch. I don't think
this improves the quality of anything, though. It just annoys the
developer. What they end up doing is writing an application-specific
exception class, and -everything- gets rethrown as that, and
everything begins to declare it throws AppError. Whats worse is that
you have heavily, heavily repackaged exceptions: SQLError -AppError
-MessageError -AppError -MessageError (yes, i've seen this
before).

That is almost completely useless. Sure, you could dig down to
SQLError, but how do you know to do that? If you knew how far you
should dig down, that means you know what the problem was, in which
case, you could have prevented it or aborted early. Whats worse, if
you have all these re-packaging catch blocks and they just log
something generic, which becomes common with all the catching going
on. "Couldn't do foo!", "Bar operation failed!", or "Couldn't fetch
filters from database" (why? We don't know, its catching an AppError
instead of something more specific), and then they rethrow the
exception. While trying to debug something not-during development,
those messages are completely useless, in fact, they're more than
useless. They're just more cruft to sift through in a log file.

Additionally, most root causes of an error are going to originate
where the input comes from the user. Handling anything at levels
deeper than that isn't going to gain you much. PrepareQuery threw an
error because of a missing field? Thats great. Where'd it come
from? There are 100 calls to PrepareQuery. There's only a few calls
to ReadUserInput(), and a single informative log message of "Query
failed, unknown field; fields=a, b, c" is much better than 100 lines
of traceback smattered with "Unknown field" and "Unable to prepare
query".

Finally, they give a false sense of security. "I'm catching
everything it could throw, so everything will be ok if an exception is
thrown!" I guess thats true. I guess. The only real advantage is
the whole program won't crash with the ever-helpful, single line of
"Segmentation fault." An improvement, but it doesn't prevent
anything.

In a complex system, an error can occur in any function at anytime.
Adding 'throws' to a method definition doesn't change that.

I guess my point is: everything Chris Mellon said was spot on.

[eliben]

On Aug 19, 7:19*pm, eliben <[e-mail]:

Python provides a quite good and feature-complete exception handling

<snip>

Thanks for the interesting discussion. Armed by the new information
and few online sources, I blogged a summary for myself on the topic of
robust exception handling in Python:

http://eli.thegreenplace.net/2008/08...tion-handling/

[Bruno Desthuilliers]

db*******@gmail.com a écrit :

On Aug 19, 4:12 pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.auwrote:

>On Tue, 19 Aug 2008 11:07:39 -0700, [e-mail] wrote:

>> def do_something(filename):
if not os.access(filename,os.R_OK):
return err(...)
f = open(filename)
...

You're running on a multitasking modern machine, right? What happens when
some other process deletes filename, or changes its permissions, in the
time after you check for access but before you actually open it?

This is a good point - if you want to use the correct way of opening
files, and
you don't want to worry about tracking down exception types, then we
can probably
agree that the following is the simplest, easiest-to-remember way:

def do_something(filename):
try:
f = open(filename)
except:
<handle exception>
...

Still not correct IMHO - bare except clauses are BAD. You want:

try:
f = open(filename)
except IOError, e:
<handle exception>

Opening files is a special case where EAFP is the only correct
solution (AFAIK). I still liberally sprinkle LBYL-style "assert
isinstance(...)"

Which defeats the whole point of dynamic typing...

and other similar assertions in routines.

The point
is that EAFP conflicts with the interest of reporting errors as soon
as possible (on which much has been written see, for instance Ch. 8 -
Defensive Programming in Code Complete),

Defensive programming makes sense in the context of a low-level language
like C where errors can lead to dramatic results. In high-level
languages like Python, the worse thing that an unhandled exception can
cause is an abrupt termination of the process and a nice traceback on
screen. In this context, defensive programming is mostly a waste of time
- if you can't *correctly* handle the exception where it happens, then
doing nothing is the better solution.

My 2 cents...

[Steven D'Aprano]

On Thu, 21 Aug 2008 00:34:21 -0700, eliben wrote:

On Aug 19, 7:19Â*pm, eliben <[e-mail]:

>Python provides a quite good and feature-complete exception handling

<snip>

Thanks for the interesting discussion. Armed by the new information and
few online sources, I blogged a summary for myself on the topic of
robust exception handling in Python:

http://eli.thegreenplace.net/2008/08...tion-handling/

Just a few random points. You say:

"Exceptions are better than returning error status codes. Some languages
(like Python) leave you with no choice as the whole language core and
standard libraries throw exceptions."

Of course you have a choice. Your function can return anything you want:

def mysqrt(x):
try:
return math.sqrt(x)
except ValueError:
return "Code 37"

I've written functions that return an object on success and None if the
function failed. In the context of what I was doing, that made more sense
than raising an exception.

Furthermore, the str.find() method returns -1 for not found instead of
raising an exception. There are probably other examples as well.

You also wrote:

"Exceptions exist for exceptional situations: unanticipated events that
are not a part of normal execution."

Exceptions can and often are anticipated. E.g. if you write code that
opens a URL, you better anticipate that the server might reject your
connection. You better expect to be asked for a cookie, or
authentication. If you check for robots.txt, you better expect that it
might not exist. That's all normal execution.

"When a programmer calls str.find('substring') he doesn’t expect an
exception to be thrown if the substring isn’t found."

But if he called str.index() then he does expect an exception to be
thrown, just like for list.index() and dict[key] can raise exceptions.
They are neither bugs nor unexpected.
"This is what he called find for. A better approach is to return a
special value like None or -1."

Sometimes, maybe. But usually not, because that just complicates the
calling code. You end up writing code that repeatedly checks that the
result isn't a special value before doing anything.

Often a better tactic is to write your code assuming that the result is
the unexceptional case, and then wrap it in a try...except block to catch
the exceptional cases.

"When used for flow-control, exceptions are like goto. There might be a
few esoteric cases in which they’re appropriate, but 99.99% of the time
they are not."

I strongly disagree. try...except is like break or continue. Yes, it
breaks the linear flow of control, but not in a wild, dangerous way like
goto.

It is possible to write bad code with exceptions, but you can write bad
code with anything.

--
Steven

[Bruno Desthuilliers]

eliben a écrit :

On Aug 19, 7:19 pm, eliben <[e-mail]:

>Python provides a quite good and feature-complete exception handling

<snip>

Thanks for the interesting discussion. Armed by the new information
and few online sources, I blogged a summary for myself on the topic of
robust exception handling in Python:

http://eli.thegreenplace.net/2008/08...tion-handling/

A couple comments (mostly python-specific, so I post them here):

"""
When used for flow-control, exceptions are like goto. There might be a
few esoteric cases in which they’re appropriate, but 99.99% of the time
they are not.
"""

Python itself uses exceptions for flow control in iterators.
"""
For some exceptions, like programming errors (e.g. IndexError,
TypeError, NameError etc.) exceptions are best left to the programmer /
user, because “handling” them will just hide real bugs.
"""

Depends on the context. There are cases where you expect these kind of
errors - like when dealing with program inputs, inspecting objects etc.
As a Q&D example:

while True:
raw_num = raw_input("enter a number")
try:
num = float(raw_num)
except TypeError, ValueError:
print "sorry, '%s' is not a valid number" % raw_num
else:
# ok
break

"""
This is also the reason why you should be extremely careful with except:
clauses that catch everything. These will not only catch the exceptions
you intended, but all of them.
"""

And remember that SysExit and KeyboardInterrupt *are* exceptions too...

"""
Document the exceptions thrown by your code
"""

If you mean "the exceptions *explicitely raised* by your code", then I
agree. But with any generic enough code, documenting any possible
exception that could be raised by lower layers, objects passed in as
arguments etc is just plain impossible. Like, if you have a function
that takes a file-like object as arg, you just cannot know in advance
what exceptions this object might raise.

My 2 cents.

[eliben]

http://eli.thegreenplace.net/2008/08...tion-handling/

>
Just a few random points. You say:

"Exceptions are better than returning error status codes. Some languages
(like Python) leave you with no choice as the whole language core and
standard libraries throw exceptions."

Of course you have a choice. Your function can return anything you want:

Of course. I didn't mean that the language prohibits returning error
codes, just that you can't use it without employing exception
handling. I've fixed the wording to make it clearer.

You also wrote:

"Exceptions exist for exceptional situations: unanticipated events that
are not a part of normal execution."

Exceptions can and often are anticipated. E.g. if you write code that
opens a URL, you better anticipate that the server might reject your
connection. You better expect to be asked for a cookie, or
authentication. If you check for robots.txt, you better expect that it
might not exist. That's all normal execution.

This is a point I'm not 100% in accord with. I still think that
exceptions are for exceptional situations. I've removed the word
"unanticipated" though, because it probably has no place in that
sentence. However, I think that if one of your valid execution paths
is w/o robots.txt, you should not use an exception to check whether
it's there. This indeed uses the "bad side" of exceptions, splitting
the exetution to two paths.
Check if robots.txt is there. If it is, open it. If you can't open it,
*that* is an exception, but if it's just not there, well it's part of
your application logic. I believe this isn't against EAFP.
I'm not sure I'm making the distinction clear here, it's a fine point.

>
"When a programmer calls str.find('substring') he doesn’t expect an
exception to be thrown if the substring isn’t found."

But if he called str.index() then he does expect an exception to be
thrown, just like for list.index() and dict[key] can raise exceptions.
They are neither bugs nor unexpected.

But why are there two versions that are the same except for the
behavior in case it wasn't found ? My wishful imagination is precisely
because of the reasons I've named. If you *know* it's there,
use .index() - then, if it fails, it's an exception, but if a part of
your logic is finding an item that might be missing, use a special
value because you want to keep the logic in a single path.

"When used for flow-control, exceptions are like goto. There might be a
few esoteric cases in which they’re appropriate, but 99.99% of the time
they are not."

I strongly disagree. try...except is like break or continue. Yes, it
breaks the linear flow of control, but not in a wild, dangerous way like
goto.

try...except can 'exit' to several 'catch points', unlike break/
continue. Furthermore, try...except can bring execution to another
hierarchy level if it's not caught where it's thrown, so it's much
more like goto in these senses. To find where the execution may go
you'll find yourself searhching for the exception name over your
source files, looking for the exception class name in some "except"
clause. Sounds like looking for a goto label.

P.S. Thanks a lot for taking the time to comment
Eli

[eliben]

On Aug 21, 12:40*pm, Bruno Desthuilliers <bruno.
[e-mail]:

eliben a écrit :On Aug 19, 7:19 pm, eliben <[e-mail]:

Python provides a quite good and feature-complete exception handling

<snip>

Thanks for the interesting discussion. Armed by the new information
and few online sources, I blogged a summary for myself on the topic of
robust exception handling in Python:

http://eli.thegreenplace.net/2008/08...tion-handling/

A couple comments (mostly python-specific, so I post them here):

Thanks for the feedback. My comments below:

"""
When used for flow-control, exceptions are like goto. There might be a
few esoteric cases in which they’re appropriate, but 99.99% of the time
they are not.
"""

Python itself uses exceptions for flow control in iterators.

Yep, I'm aware of StopIteration, but I'm not sure whether it's a good
or a bad feature. I'm a bit wary of the programming style it might
encourage in inexperienced programmers. When this behavior is hidden
inside the implementation of 'for', fair enough. But when you have to
catch exceptions just to walk over some iterable explicitly, I'm not
sure the designers of this Python feature made the correct choices
here.

"""
For some exceptions, like programming errors (e.g. IndexError,
TypeError, NameError etc.) exceptions are best left to the programmer /
user, because “handling” them will just hide real bugs.
"""

Depends on the context. There are cases where you expect these kind of
errors - like when dealing with program inputs, inspecting objects etc.
As a Q&D example:

while True:
* * *raw_num = raw_input("enter a number")
* * *try:
* * * * *num = float(raw_num)
* * *except TypeError, ValueError:
* * * * *print "sorry, '%s' is not a valid number" % raw_num
* * *else:
* * * * *# ok
* * * * *break

I agree.

"""
This is also the reason why you should be extremely careful with except:
clauses that catch everything. These will not only catch the exceptions
you intended, but all of them.
"""

And remember that SysExit and KeyboardInterrupt *are* exceptions too...

"""
Document the exceptions thrown by your code
"""

If you mean "the exceptions *explicitely raised* by your code", then I
agree. But with any generic enough code, documenting any possible
exception that could be raised by lower layers, objects passed in as
arguments etc is just plain impossible. Like, if you have a function
that takes a file-like object as arg, you just cannot know in advance
what exceptions this object might raise.

This is one of the main concerns with which I started this c.l.py
thread ! I think it's a pity that we have no way of anticipating and
constraining the exceptions thrown by our code, and that we should
strive to make it more explicit. The "function accepting a file" is a
case in point. You know what you do with this file, so why can't you
know what exceptions might be thrown ? If you're trying to open it,
IOError (and OSError ?), etc. Besides, as I noted in the article,
perhaps you want to hide some of inner-level exceptions in your own,
to keep encapsulation.

Eli

[Bruno Desthuilliers]

eliben a écrit :

On Aug 21, 12:40 pm, Bruno Desthuilliers <bruno.
[e-mail]:

>eliben a écrit :On Aug 19, 7:19 pm, eliben <[e-mail]:

(snip)

>>"""
Document the exceptions thrown by your code
"""

If you mean "the exceptions *explicitely raised* by your code", then I
agree. But with any generic enough code, documenting any possible
exception that could be raised by lower layers, objects passed in as
arguments etc is just plain impossible. Like, if you have a function
that takes a file-like object as arg, you just cannot know in advance
what exceptions this object might raise.

This is one of the main concerns with which I started this c.l.py
thread ! I think it's a pity that we have no way of anticipating and
constraining the exceptions thrown by our code,

Java's "checked exception" system has proven to be a total disaster.

and that we should
strive to make it more explicit. The "function accepting a file" is a
case in point. You know what you do with this file, so why can't you
know what exceptions might be thrown ?

Reread more carefully. I wrote "a *file-like* object", not "a file".
This is the whole point of duck typing. Given that any file-like object
will work ok with my function, I have *no way* to know what exceptions
this object may raise.

If you're trying to open it,

Trying to open a file *object* ? heck, it's supposed to be already
opened at this stage. And yes, it's a pretty common pattern in Python
(which is why I choose this example).

[magloca]

Bruno Desthuilliers @ Thursday 21 August 2008 17:31:

>>If you mean "the exceptions *explicitely raised* by your code", then
I agree. But with any generic enough code, documenting any possible
exception that could be raised by lower layers, objects passed in as
arguments etc is just plain impossible. Like, if you have a function
that takes a file-like object as arg, you just cannot know in
advance what exceptions this object might raise.

This is one of the main concerns with which I started this c.l.py
thread ! I think it's a pity that we have no way of anticipating and
constraining the exceptions thrown by our code,

Java's "checked exception" system has proven to be a total disaster.

Could you elaborate on that? I'm not disagreeing with you (or agreeing,
for that matter); I'd just really like to know what you mean by
a "total disaster."

m.

[dbpokorny]

On Aug 20, 10:13 pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.auwrote:

It might not be enjoyable to have a sarcastic remark directed your way,
but it isn't a personal attack. Just because a comment is about something
you do doesn't make it a personal attack. Personal attacks are about who
you are rather than what you do.

If you type in "Personal Attack" in Wikipedia (not an authoritative
source, I know) it takes you to the page on ad hominem arguments.
There you can find the following example of the fallacious ad hominem
argument:

Person A makes claim X
There is something objectionable about Person A
Therefore claim X is false

It is, ultimately, a matter of opinion, but "going out of one's way to
make it hard for other programmers" sounds objectionable to me. I
mean, I wouldn't want to work with anyone like that!

There's an apparent contradiction in your argument. You seem to be
arguing against EAFP and in favour of LBYL, but now you're suggesting
that you don't use type-checking. As near as I can tell, you don't do
type-checking, you don't do duck typing, you don't like catching
exceptions. So what do you actually do to deal with invalid data?

Here is an example from a Django web app: when there is a bug, a
generic Exception is thrown and Django catches it and reports a
beautifully formatted stack trace. When something must be reported to
the user, a MyAppException is thrown (not the real name). The HTTP
request handler for the application is wrapped in a single try:...
except MyAppException:.... the big idea is that there should be a
maximum of two try/except blocks on the stack at any particular point
in time [1]: at a high level (already mentioned) and for wrapping
primitive "execute" operations against Rpy and MySQLdb. In practice,
this doesn't always happen - there is one place where an EAFP-style
construct is used (the "operation" in this case is to generate some
HTML and cache it based on some source XML, but the source may have
"syntax errors", so if the HTML can't be generated, then cleanup is
performed and an error message returned).

So to summarize:
try/except blocks at boundaries between system components: good
try/except blocks within a single component: slightly concerning

I think I may have overstated the case against EAFP. There are
certainly cases where EAFP makes a lot of sense; I would object to
portraying EAFP as an alternative to defensive programming. [Side
note: defensive programming serves much the same purpose in Python as
it does in C, but I agree that in C there is extra motivation such as
avoiding buffer overruns. I think of defensive programming simply as
"taking proactive steps to reduce the expected time to debug a program
if a programming error should arise".]

By the way, if you're worried that isinstance() is too long to type, you
can do this:

isin = isinstance
isin(123, int)

Actually I'm holding out for type objects to grow __lt__ and __le__
methods so you can do something like

from abstract_base_classes import sequence
if type(my_obj) <= sequence:
...

This is borrowed from the <= notation for subgroups in math (there are
probably other cases too). I don't use abstract base classes, so I
don't even know if this is right, but hopefully you get the idea.

No no no, exceptions are not necessarily bugs!!! A bug is an exceptional
circumstance, but not all exceptional circumstances are bugs.

I tend to agree, but I have found that thinking about these issues
makes me question the wisdom of Python's built-ins throwing exceptions
in non-exceptional circumstances (for instance you try to open a file
that doesn't exist - IMHO this is about as exceptional as trying a no-
wait acquire on a busy lock, in other words it isn't exceptional at
all). As long as we are in fantasy realm, one could argue that open()
should return a pair like this:

f, open_ok = open(...)

where open_ok is a status object whose __nonzero__ (in 3.0 __bool__ is
used) is true on success and false on an error, and also has an error
code and error message field. The idea is from Django's get_or_create
method in the db API.

[1] That I have to think about. I don't particularly care about try/
except blocks in Django's, Rpy's, or MySQLdb's activation records.

David

[eliben]

Here is an example from a Django web app: when there is a bug, a

generic Exception is thrown and Django catches it and reports a
beautifully formatted stack trace. When something must be reported to
the user, a MyAppException is thrown (not the real name). The HTTP
request handler for the application is wrapped in a single try:...
except MyAppException:.... the big idea is that there should be a
maximum of two try/except blocks on the stack at any particular point
in time [1]: at a high level (already mentioned) and for wrapping
primitive "execute" operations against Rpy and MySQLdb.

This actually makes lots of sense as is exactly what Ned Batchelder
wrote here:
http://nedbatchelder.com/text/except...ainforest.html

* A-layer generates exceptions,
* B-layer can often ignore the whole issue, and
* C-layer decides what to do

It's worth a read.

[eliben]

On Aug 19, 7:34*pm, "Chris Mellon" <[e-mail]:

On Tue, Aug 19, 2008 at 12:19 PM, eliben <[e-mail]:

Python provides a quite good and feature-complete exception handling
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.

Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.

Lots of people seem to have this fear. They treat exceptions like they
would treat error codes, trying to handle any possible case around any
particular call.

This is the wrong thing to do, and it only leads to more fragile code.
There are only 2 reasonable things to do with an exception:
1) handle it, by which I mean catch the exception knowing what error
condition it signifies, and take an appropriate action to correct the
error and
2) pass it up so something else has a chance at it.

But by 'handling', do you also mean "rethrow with better
information" ?

I feel there's an inherent clash between two 'good practices' in
exception handling:
1) Using EAFP over LBYL
2) Hiding implementation details

Consider this code, which I wrote just yesterday:

elif type in ('LinearStartAddr', 'SegmentStartAddr'):
if len(data) != 4:
line_error('expecting a 4-byte data field for this record type,
got %s' % len(data))
self.data.start_address = unpack('>L', data)

This is part of a method in a class that parses a data file. I've
ended up using LBYL here, to hide an implementation detail. I could've
let the Exception from unpack propagate, but that doesn't make much
sense with "hiding implementation". So I'm throwing a more useful
exception myself.
Was wrapping the call to unpack with try/except that throws my
exception a better idea, in your opinion ? Because that makes the code
somewhat more convoluted.

Eli

[magloca]

Bruno Desthuilliers @ Thursday 21 August 2008 22:54:

magloca a écrit :

>Bruno Desthuilliers @ Thursday 21 August 2008 17:31:

>>>>If you mean "the exceptions *explicitely raised* by your code",
then I agree. But with any generic enough code, documenting any
possible exception that could be raised by lower layers, objects
passed in as arguments etc is just plain impossible. Like, if you
have a function that takes a file-like object as arg, you just
cannot know in advance what exceptions this object might raise.
>
This is one of the main concerns with which I started this c.l.py
thread ! I think it's a pity that we have no way of anticipating
and constraining the exceptions thrown by our code,
Java's "checked exception" system has proven to be a total disaster.

Could you elaborate on that? I'm not disagreeing with you (or
agreeing, for that matter); I'd just really like to know what you
mean by a "total disaster."

One of the most (in)famous Java coding pattern is the empty catchall
clause. Read Chris Mellon and Richard Levasseur posts in this thread
for more details - they already covered the whole point.

Thanks, I missed those. Having read them, I *am* agreeing with you.
Personally, I also dislike the predominance in the Java world of only
giving type information -- IllegalValueException,
ObjectRetrievalFailureException, and whatever. What was the illegal
value? What object couldn't be retrieved? How hard is it to use the
with-error-message version of the Exception constructor, and include
something that might actually be helpful in debugging?

m.

[Maric Michaud]

Le Thursday 21 August 2008 09:34:47 Bruno Desthuilliers, vous avez écrit*:

The point
is that EAFP conflicts with the interest of reporting errors as soon
as possible (on which much has been written see, for instance Ch. 8 -
Defensive Programming in Code Complete),

Defensive programming makes sense in the context of a low-level language
* *like C where errors can lead to dramatic results. In high-level
languages like Python, the worse thing that an unhandled exception can
cause is an abrupt termination of the process and a nice traceback on
screen.

... and leave your datas in inconsistent state. So, what C or any other
language could do worse to your application ?

In this context, defensive programming is mostly a waste of time
- if you can't *correctly* handle the exception where it happens, then
doing nothing is the better solution.

If I don't buy the argument I actually agree with the conclusion. Each
component of a program should try to manage only errors tied to their own
logic and let pass others up to the gui logic for rendering errors the good
way, persistence logic to rollback unwanted changes, and application logic to
continue execution the right way. This is hard to do in C because you have no
way to trap an error which happen randomly in the program, ie. a segfault
will interrupt the execution anyway.

--
_____________

Maric Michaud

[Lie]

On Aug 21, 12:59*am, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.auwrote:

On Wed, 20 Aug 2008 09:23:22 -0700, [e-mail] wrote:

On Aug 19, 4:12*pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.auwrote:

On Tue, 19 Aug 2008 11:07:39 -0700, [e-mail] wrote:
* def do_something(filename):
* * if not os.access(filename,os.R_OK):
* * * return err(...)
* * f = open(filename)
* * ...

You're running on a multitasking modern machine, right? What happens
when some other process deletes filename, or changes its permissions,
in the time after you check for access but before you actually open it?

This is a good point - if you want to use the correct way of opening
files, and
you don't want to worry about tracking down exception types, then we can
probably
agree that the following is the simplest, easiest-to-remember way:

* def do_something(filename):
* * try:
* * * f = open(filename)
* * except:
* * * <handle exception>

No, we don't agree that that is the correct way of opening files. Simple
it might be, but correct it is not.

If you're using Python 2.6 or greater, then you should be using a with
block to handle file opening.

And regardless of which version of Python, you shouldn't use a bare
except. It will mask exceptions you *don't* want to catch, including
programming errors, typos and keyboard interrupts.

Opening files is a special case where EAFP is the only correct solution
(AFAIK). I still liberally sprinkle LBYL-style "assert isinstance(...)"

Oh goodie. Another programmer who goes out of his way to make it hard for
other programmers, by destroying duck-typing.

BTW, assertions aren't meant for checking data, because assertions can be
turned off. Outside of test frameworks (e.g. unit tests), assertions are
meant for verifying program logic:

def foo(x):
* * # This is bad, because it can be turned off at runtime,
* * # destroying your argument checking.
* * assert isinstance(x, int)
* * # And it raises the wrong sort of exception.

* * # This is better (but not as good as duck-typing).
* * if not isinstance(x, int):
* * * * raise TypeError('x not an int')
* * * * # And it raises the right sort of error.

* * y = some_function(x)
* * # y should now be between -1 and 1.
* * assert -1 < y < 1
* * do_something_with(y)

and other similar assertions in routines. The point is that EAFP
conflicts with the interest of reporting errors as soon as possible

Not necessarily. Tell me how this conflicts with reporting errors as soon
as possible:

def do_something(filename):
* * try:
* * * * f = open(filename)
* * except IOError, e:
* * * * report_exception(e) *# use a GUI, log to a file, whatever...

How could you report the exception any earlier than immediately?

I'm sure different people would have different view on what
immediately means. The LBYL supporters define immediately as
immediately after a definite potential problem is detected (by ifs or
assertion), while the EAFP supporters define immediately as
immediately after a problem arises. No side is right or wrong, both
have weakness and strength, but python-style programs are encouraged
to use EAFP-style exception handling whenever feasible, but with the
spirit of the Zen: "Special cases aren't special enough to break the
rules, although practicality beats purity", if LBYL makes things much
easier in that case, and doing EAFP would just convolute the code,
then practicality beats purity.

--
Steven

[Bruno Desthuilliers]

Maric Michaud a écrit :

Le Thursday 21 August 2008 09:34:47 Bruno Desthuilliers, vous avez écrit :

>>The point
is that EAFP conflicts with the interest of reporting errors as soon
as possible (on which much has been written see, for instance Ch. 8 -
Defensive Programming in Code Complete),

Defensive programming makes sense in the context of a low-level language
like C where errors can lead to dramatic results. In high-level
languages like Python, the worse thing that an unhandled exception can
cause is an abrupt termination of the process and a nice traceback on
screen.

... and leave your datas in inconsistent state.

Not all applications persist data, so this is an application-specific
problem, to be solved at the application level - IOW, there's no
one-size-fits-all solution here. Anyway: transactions management is not
what I meant when talking about defensive programming.

As far as I'm concerned, the only place where the defensive approach
really makes sense whatever the language is when dealing with external
datas (user inputs etc).

So, what C or any other
language could do worse to your application ?

An error in a C program can do *way* worse than leave an application's
data in inconsistent state. See ART for more on this:
http://dialspace.dial.pipex.com/prod...een/gfd34/art/

>In this context, defensive programming is mostly a waste of time
- if you can't *correctly* handle the exception where it happens, then
doing nothing is the better solution.

If I don't buy the argument

cf above - maybe you buy it after all ?-)

I actually agree with the conclusion. Each
component of a program should try to manage only errors tied to their own
logic and let pass others up to the gui logic for rendering errors the good
way, persistence logic to rollback unwanted changes, and application logic to
continue execution the right way. This is hard to do in C because you have no
way to trap an error which happen randomly in the program, ie. a segfault
will interrupt the execution anyway.

Indeed.

[Lie]

On Aug 21, 2:34*pm, Bruno Desthuilliers <bruno.
[e-mail]:

[e-mail] a écrit :

On Aug 19, 4:12 pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.auwrote:

On Tue, 19 Aug 2008 11:07:39 -0700, [e-mail] wrote:
* def do_something(filename):
* * if not os.access(filename,os.R_OK):
* * * return err(...)
* * f = open(filename)
* * ...
You're running on a multitasking modern machine, right? What happens when
some other process deletes filename, or changes its permissions, in the
time after you check for access but before you actually open it?

This is a good point - if you want to use the correct way of opening
files, and
you don't want to worry about tracking down exception types, then we
can probably
agree that the following is the simplest, easiest-to-remember way:

* def do_something(filename):
* * try:
* * * f = open(filename)
* * except:
* * * <handle exception>
* * ...

Still not correct IMHO - bare except clauses are BAD. You want:

* * * try:
* * * * f = open(filename)
* * * except IOError, e:
* * * * <handle exception>

Opening files is a special case where EAFP is the only correct
solution (AFAIK). I still liberally sprinkle LBYL-style "assert
isinstance(...)"

Which defeats the whole point of dynamic typing...

and other similar assertions in routines.
The point
is that EAFP conflicts with the interest of reporting errors as soon
as possible (on which much has been written see, for instance Ch. 8 -
Defensive Programming in Code Complete),

Defensive programming makes sense in the context of a low-level language
* *like C where errors can lead to dramatic results. In high-level
languages like Python, the worse thing that an unhandled exception can
cause is an abrupt termination of the process and a nice traceback on
screen. In this context, defensive programming is mostly a waste of time
- if you can't *correctly* handle the exception where it happens, then
doing nothing is the better solution.

Ah... now I understand what the Zen is talking about when it said:
"Now is better then never, although never is often better than *right*
now." If you don't have all the necessary resources to fix an
exception right now, don't try to fix it, instead let it propagate,
and allow it to be handled in a level where there is enough
information how to fix it.

My 2 cents...

Steven D'Aprano says:

Exceptions can and often are anticipated. E.g. if you write code that
opens a URL, you better anticipate that the server might reject your
connection. You better expect to be asked for a cookie, or
authentication. If you check for robots.txt, you better expect that it
might not exist. That's all normal execution.

I think we should change except: into expect:, it would confuse less,
would it? It signifies that the program expects so and so kinds of
exceptional situations. The try: should also be changed to... perhaps
in:, block:, onthiscode:, etc (this paragraph is written with my taste
buds on the part of my face below my eye and above my jaw)

[Wojtek Walczak]

On Fri, 22 Aug 2008 06:43:58 -0700 (PDT), Lie wrote:

I think we should change except: into expect:, it would confuse less,
would it? It signifies that the program expects so and so kinds of
exceptional situations. The try: should also be changed to... perhaps
in:, block:, onthiscode:, etc (this paragraph is written with my taste
buds on the part of my face below my eye and above my jaw)

IMO it's not even worth considering. Breaking all the python
software that exists and moreover breaking the habits of programmists
to gain *nothing* is a waste of time of so many people, that
we should just forget it.
--
Regards,
Wojtek Walczak,
http://tosh.pl/gminick/

[Maric Michaud]

Le Friday 22 August 2008 15:03:21 Bruno Desthuilliers, vous avez écrit*:

Maric Michaud a écrit :

Le Thursday 21 August 2008 09:34:47 Bruno Desthuilliers, vous avez écrit :

>The point
is that EAFP conflicts with the interest of reporting errors as soon
as possible (on which much has been written see, for instance Ch. 8 -
Defensive Programming in Code Complete),

Defensive programming makes sense in the context of a low-level language
like C where errors can lead to dramatic results. In high-level
languages like Python, the worse thing that an unhandled exception can
cause is an abrupt termination of the process and a nice traceback on
screen.

... and leave your datas in inconsistent state.

Not all applications persist data, so this is an application-specific
problem, to be solved at the application level - IOW, there's no
one-size-fits-all solution here.

... or lose open connection, reset sessions, etc.. it doesn't really matter
what is lost when a program crash, if it can be restarted without changes it
is hardly what I'd qualify a "dramatic result". It doesn't depend on the
language what implications of an unhandled error are, It is always
application-specific.

Anyway: transactions management is not
what I meant when talking about defensive programming.

As far as I'm concerned, the only place where the defensive approach
really makes sense whatever the language is when dealing with external
datas (user inputs etc).

I agree, this is my whole point, "whatever the language is".

So, what C or any other
language could do worse to your application ?

An error in a C program can do *way* worse than leave an application's
data in inconsistent state. See ART for more on this:
http://dialspace.dial.pipex.com/prod...een/gfd34/art/

I didn't read "dramatic results" in that sense, but with the meaning of a
result that the program itself cannot handle.

If the whole system crash due to an unhandled error in a program, once
missile's dropped it doesn't really matter in which language it was written.
Reliability of a system, as high-availability of an application, is mostly a
problem beyond the scope of application level error checking.

In this context, defensive programming is mostly a waste of time
- if you can't *correctly* handle the exception where it happens, then
doing nothing is the better solution.

If I don't buy the argument

cf above - maybe you buy it after all ?-)

I actually agree with the conclusion. Each
component of a program should try to manage only errors tied to their own
logic and let pass others up to the gui logic for rendering errors the
good way, persistence logic to rollback unwanted changes, and application
logic to continue execution the right way. This is hard to do in C
because you have no way to trap an error which happen randomly in the
program, ie. a segfault will interrupt the execution anyway.

Indeed.
--
http://mail.python.org/mailman/listinfo/python-list

--
_____________

Maric Michaud

[Gabriel Genellina]

En Thu, 21 Aug 2008 14:48:45 -0300, magloca <ma*****@mailinater.com>
escribió:

Bruno Desthuilliers @ Thursday 21 August 2008 17:31:

>Java's "checked exception" system has proven to be a total disaster.

Could you elaborate on that? I'm not disagreeing with you (or agreeing,
for that matter); I'd just really like to know what you mean by
a "total disaster."

Please allow me to share a few links:
http://www.mindview.net/Etc/Discussi...ckedExceptions
http://radio.weblogs.com/0122027/sto...eAMistake.html
and see also a message from Richard Levasseur today in this very thread
explaining why checked exceptions are just an annoyance that buys you
nothing.

--
Gabriel Genellina