How to except the unexpected?

[Rene Pijlman]

One of the things I dislike about Java is the need to declare exceptions
as part of an interface or class definition. But perhaps Java got this
right...

I've writen an application that uses urllib2, urlparse, robotparser and
some other modules in the battery pack. One day my app failed with an
urllib2.HTTPError. So I catch that. But then I get a urllib2.URLError, so
I catch that too. The next day, it encounters a urllib2.HTTPError, then a
IOError, a socket.timeout, httplib.InvalidURL,...

How do you program robustly with these modules throwing all those
different (and sometimes undocumented) exceptions at you?

A catchall seems like a bad idea, since it also catches AttributeErrors
and other bugs in the program.

--
René Pijlman

[James Stroud]

Rene Pijlman wrote:

One of the things I dislike about Java is the need to declare exceptions
as part of an interface or class definition. But perhaps Java got this
right...

I've writen an application that uses urllib2, urlparse, robotparser and
some other modules in the battery pack. One day my app failed with an
urllib2.HTTPError. So I catch that. But then I get a urllib2.URLError, so
I catch that too. The next day, it encounters a urllib2.HTTPError, then a
IOError, a socket.timeout, httplib.InvalidURL,...

How do you program robustly with these modules throwing all those
different (and sometimes undocumented) exceptions at you?

A catchall seems like a bad idea, since it also catches AttributeErrors
and other bugs in the program.

The relevant lines of urllib2, for example, look as such:

class URLError(IOError):
class HTTPError(URLError, addinfourl):
class GopherError(URLError):

This suggests that catching URLError should have caught your HTTPError,
so you might have the chronology backwards above.

E.g.:

py> class BobError(Exception): pass
....
py> class CarolError(BobError): pass
....
py> try:
.... raise CarolError
.... except BobError:
.... print 'got it'
....
got it
Now,

% cat httplib.py | grep -e '^\s*class'

produces the following at one point in its output:

class HTTPException(Exception):
class NotConnected(HTTPException):
class InvalidURL(HTTPException):
class UnknownProtocol(HTTPException):
class UnknownTransferEncoding(HTTPException):
class UnimplementedFileMode(HTTPException):
class IncompleteRead(HTTPException):
class ImproperConnectionState(HTTPException):
class CannotSendRequest(ImproperConnectionState):
class CannotSendHeader(ImproperConnectionState):
class ResponseNotReady(ImproperConnectionState):
class BadStatusLine(HTTPException):

Which suggests that "try: except HTTPException:" will be specific enough
as a catchall for this module.

The following, then, should catch everything you mentioned except the
socket timeout:

try:
whatever()
except URLError, HTTPException:
alternative()

But it seems to me that working with the internet as you are doing is
fraught with peril anyway.

James

[Ben Caradoc-Davies]

James Stroud wrote:

except URLError, HTTPException:

Aieee! This catches only URLError and binds the name HTTPException to
the detail of that error. You must write

except (URLError, HTTPException):

to catch both.

--
Ben Caradoc-Davies <be*@wintersun.org>
http://wintersun.org/
"Those who deny freedom to others deserve it not for themselves."
- Abraham Lincoln

[James Stroud]

Ben Caradoc-Davies wrote:

James Stroud wrote:

except URLError, HTTPException:

Aieee! This catches only URLError and binds the name HTTPException to
the detail of that error. You must write

except (URLError, HTTPException):

to catch both.

Oops.

[Roy Smith]

In article <44***********************@per-qv1-newsreader-01.iinet.net.au>,
Ben Caradoc-Davies <be*@wintersun.org> wrote:

James Stroud wrote:

except URLError, HTTPException:

Aieee! This catches only URLError and binds the name HTTPException to
the detail of that error. You must write

except (URLError, HTTPException):

to catch both.

This exact issue came up just within the past week or so. I think that
qualifies it as a wart, but I think it's a double wart.

It's certainly a wart that the try statement syntax allows for such
ambiguity. But, I think it's also a wart in how the exceptions were
defined. I like to create a top-level exception class to encompass all the
possible errors in a given module, then subclass that. This way, if you
want to catch anything to goes wrong in a call, you can catch the top-level
exception class without having to enumerate them all.

[Peter Hansen]

Rene Pijlman wrote:

One of the things I dislike about Java is the need to declare exceptions
as part of an interface or class definition. But perhaps Java got this
right...

I've writen an application that uses urllib2, urlparse, robotparser and
some other modules in the battery pack. One day my app failed with an
urllib2.HTTPError. So I catch that. But then I get a urllib2.URLError, so
I catch that too. The next day, it encounters a urllib2.HTTPError, then a
IOError, a socket.timeout, httplib.InvalidURL,...

How do you program robustly with these modules throwing all those
different (and sometimes undocumented) exceptions at you?
I do it by not micromanaging things. Presumably if you plan to catch an
exception, you have a specific procedure in mind for handling the
problem. Maybe a retry, maybe an alternate way of attempting the same
thing? Look to the code that you are putting in those except:
statements (or that you think you want to put in them) to decide what to
do about this situation. If each type of exception will be handled in a
different manner, then you definitely want to identify each type by
looking at the source or the docs, or doing it empirically.

Most of the time there isn't a whole lot of real "handling" going on in
an exception handler, but merely something like logging and/or reporting
it onscreen in a cleaner fashion than a traceback, then failing anyway.
This is one reason Java does get it wrong: 95% of exceptions don't
need and shouldn't have special handling anyway.

Good code should probably have a very small set of real exception
handling cases, and one or two catchalls at a higher level to avoid
barfing a traceback at the user.
A catchall seems like a bad idea, since it also catches AttributeErrors
and other bugs in the program.

Generally speaking this won't be a problem if you have your catchalls at
a fairly high level and have proper unit tests for the lower level code
which is getting called. You are doing unit testing, aren't you? ;-)

-Peter

[Steven D'Aprano]

On Sat, 04 Mar 2006 00:10:17 +0100, Rene Pijlman wrote:

I've writen an application that uses urllib2, urlparse, robotparser and
some other modules in the battery pack. One day my app failed with an
urllib2.HTTPError. So I catch that. But then I get a urllib2.URLError, so
I catch that too. The next day, it encounters a urllib2.HTTPError, then a
IOError, a socket.timeout, httplib.InvalidURL,...

How do you program robustly with these modules throwing all those
different (and sometimes undocumented) exceptions at you?
How robust do you want to be? Do you want to take a leaf out of Firefox
and Windows XP by generating an error report and transmitting it back to
the program maintainer?
A catchall seems like a bad idea, since it also catches AttributeErrors
and other bugs in the program.

ExpectedErrors = (URLError, IOError)
ErrorsThatCantHappen = (LookupError, ArithmeticError, AssertionError)

try:
process_things()
except ExpectedErrors:
recover_from_error_gracefully()
except ErrorsThatCantHappen:
print "Congratulations! You have found a program bug!"
print "For a $327.68 reward, please send the following " \
"traceback to Professor Donald Knuth."
raise
except:
print "An unexpected error occurred."
print "This probably means the Internet is broken."
print "If the bug still occurs after fixing the Internet, " \
"it may be a program bug."
log_error()
sys.exit()

--
Steven.

[Paul Rubin]

Steven D'Aprano <st***@REMOVETHIScyber.com.au> writes:

try:
process_things()
except ExpectedErrors:
recover_from_error_gracefully()
except ErrorsThatCantHappen:
print "Congratulations! You have found a program bug!"
print "For a $327.68 reward, please send the following " \
"traceback to Professor Donald Knuth."
raise
except:
print "An unexpected error occurred."
print "This probably means the Internet is broken."

But this isn't good, it catches asynchronous exceptions like the user
hitting ctrl-C, which you might want to handle elsewhere. What you
want is a way to catch only actual exceptions raised from inside the
try block.

[Steven D'Aprano]

On Fri, 03 Mar 2006 21:10:22 -0800, Paul Rubin wrote:

Steven D'Aprano <st***@REMOVETHIScyber.com.au> writes:

try:
process_things()
except ExpectedErrors:
recover_from_error_gracefully()
except ErrorsThatCantHappen:
print "Congratulations! You have found a program bug!"
print "For a $327.68 reward, please send the following " \
"traceback to Professor Donald Knuth."
raise
except:
print "An unexpected error occurred."
print "This probably means the Internet is broken."

But this isn't good, it catches asynchronous exceptions like the user
hitting ctrl-C, which you might want to handle elsewhere. What you
want is a way to catch only actual exceptions raised from inside the
try block.

It will only catch the KeyboardInterrupt exception if the user actually
hits ctrl-C during the time the code running inside the try block is
executing. It certainly won't catch random ctrl-Cs happening at other
times.

The way to deal with it is to add another except clause to deal with the
KeyboardInterrupt, or to have recover_from_error_gracefully() deal with
it. The design pattern still works. I don't know if it has a fancy name,
but it is easy to describe:-

catch specific known errors that you can recover from, and recover from
them whatever way you like (including, possibly, re-raising the exception
and letting higher-level code deal with it);

then catch errors that cannot possibly happen unless there is a bug,
and treat them as a bug;

and lastly catch unexpected errors that you don't know how to handle and
die gracefully.

My code wasn't meant as production level code, nor was ExpectedErrors
meant as an exhaustive list. I thought that was too obvious to need
commenting on.

Oh, in case this also wasn't obvious, Donald Knuth won't really pay
$327.68 for bugs in your Python code. He only pays for bugs in his own
code. *wink*

--
Steven.

[Paul Rubin]

Steven D'Aprano <st***@REMOVETHIScyber.com.au> writes:

The way to deal with it is to add another except clause to deal with the
KeyboardInterrupt, or to have recover_from_error_gracefully() deal with
it.
I think adding another except clause for KeyboardInterrupt isn't good
because maybe in Python 2.6 or 2.6 or whatever there will be some
additional exceptions like that and your code will break. For example,
proposals have floated for years of adding ways for threads to raise
exceptions in other threads.

I put up a proposal for adding an AsynchronousException class to
contain all of these types of exceptions, so you can check for that.
Oh, in case this also wasn't obvious, Donald Knuth won't really pay
$327.68 for bugs in your Python code. He only pays for bugs in his own
code. *wink*

The solution to that one is obvious. We have to get Knuth using Python.
Anyone want to write a PEP? ;-)

[Rene Pijlman]

Roy Smith:

I like to create a top-level exception class to encompass all the
possible errors in a given module, then subclass that. This way, if you
want to catch anything to goes wrong in a call, you can catch the top-level
exception class without having to enumerate them all.

What do you propose to do with exceptions from modules called by the given
module?

--
René Pijlman

[Rene Pijlman]

James Stroud:

Which suggests that "try: except HTTPException:" will be specific enough
as a catchall for this module.

The following, then, should catch everything you mentioned except the
socket timeout:
Your conclusion may be (almost) right in this case. I just don't like this
approach. Basically this is reverse engineering the interface from the
source at the time of writing the app. Even if you get it right, it may
fail next week when someone added an exception to a module.
But it seems to me that working with the internet as you are doing is
fraught with peril anyway.

Why? It shouldn't be.

--
René Pijlman

[Rene Pijlman]

Peter Hansen:

Good code should probably have a very small set of real exception
handling cases, and one or two catchalls at a higher level to avoid
barfing a traceback at the user.

Good point.

A catchall seems like a bad idea, since it also catches AttributeErrors
and other bugs in the program.

Generally speaking this won't be a problem if you have your catchalls at
a fairly high level and have proper unit tests for the lower level code
which is getting called. You are doing unit testing, aren't you? ;-)

With low coverage, yes. But unit testing isn't the answer for this
particular problem. For example, yesterday my app was surprised by an
httplib.InvalidURL since I hadn't noticed this could be raised by
robotparser (this is undocumented). If that fact goes unnoticed when
writing the exception handling, it will also go unnoticed when designing
test cases. I probably wouldn't have thought of writing a test case with a
first url with some external domain (that triggers robots.txt-fetching)
that's deemed invalid by httplib.

--
René Pijlman

[Rene Pijlman]

Steven D'Aprano:

ExpectedErrors = (URLError, IOError)
ErrorsThatCantHappen =

try:
process_things()
except ExpectedErrors:
recover_from_error_gracefully()
except ErrorsThatCantHappen:
print "Congratulations! You have found a program bug!"
print "For a $327.68 reward, please send the following " \
"traceback to Professor Donald Knuth."
raise
except:
print "An unexpected error occurred."
print "This probably means the Internet is broken."
print "If the bug still occurs after fixing the Internet, " \
"it may be a program bug."
log_error()
sys.exit()

Yes, I think I'll do something like this. Perhaps combined with Peter's
advice to not micromanage, like so:

Reraise = (LookupError, ArithmeticError, AssertionError) # And then some

try:
process_things()
except Reraise:
raise
except:
log_error()

--
René Pijlman

[Rene Pijlman]

Paul Rubin <http://ph****@NOSPAM.invalid>:

We have to get Knuth using Python.

Perhaps a MIX emulator and running TeXDoctest on his books will convince
him..

--
René Pijlman

[Jorge Godoy]

Rene Pijlman <re********************@my.address.is.invalid> writes:

With low coverage, yes. But unit testing isn't the answer for this
particular problem. For example, yesterday my app was surprised by an
httplib.InvalidURL since I hadn't noticed this could be raised by
robotparser (this is undocumented). If that fact goes unnoticed when

It isn't undocumented in my module. From 'pydoc httplib':
CLASSES
exceptions.Exception
HTTPException
BadStatusLine
ImproperConnectionState
CannotSendHeader
CannotSendRequest
ResponseNotReady
IncompleteRead
InvalidURL <------- HERE
NotConnected
UnimplementedFileMode
UnknownProtocol
UnknownTransferEncoding
HTTPException
BadStatusLine
ImproperConnectionState
CannotSendHeader
CannotSendRequest
ResponseNotReady
IncompleteRead
InvalidURL
NotConnected
UnimplementedFileMode
UnknownProtocol
UnknownTransferEncoding
HTTP
HTTPConnection
HTTPSConnection
HTTPResponse
(Yes, it appears twice, don't ask me why...)

--
Jorge Godoy <go***@ieee.org>

"Quidquid latine dictum sit, altum sonatur."
- Qualquer coisa dita em latim soa profundo.
- Anything said in Latin sounds smart.

[Roy Smith]

Rene Pijlman <re********************@my.address.is.invalid> wrote:

A catchall seems like a bad idea, since it also catches AttributeErrors
and other bugs in the program.

All of the things like AttributeError are subclasses of StandardError. You
can catch those first, and then catch everything else. In theory, all
exceptions which represent problems with the external environment (rather
than programming mistakes) should derive from Exception, but not from
StandardError. In practice, some very old code may raise things which do
not derive from Exception, which complicates things somewhat.

--------------------------------------------------
#!/usr/bin/env python

import socket

try:
x = []
y = x[42]
except StandardError, foo:
print "Caught a StandardError: ", foo
except Exception, foo:
print "Caught something else: ", foo

try:
socket.socket (9999)
except StandardError, foo:
print "Caught a StandardError: ", foo
except Exception, foo:
print "Caught something else: ", foo

try:
raise "I'm a string pretending to be an exception"
except StandardError, foo:
print "Caught a StandardError: ", foo
except Exception, foo:
print "Caught something else: ", foo
--------------------------------------------------

Roy-Smiths-Computer:play$ ./ex.py
Caught a StandardError: list index out of range
Caught something else: (43, 'Protocol not supported')
Traceback (most recent call last):
File "./ex.py", line 21, in ?
raise "I'm a string pretending to be an exception"
I'm a string pretending to be an exception

[Rene Pijlman]

Jorge Godoy:

Rene Pijlman:

my app was surprised by an
httplib.InvalidURL since I hadn't noticed this could be raised by
robotparser (this is undocumented).

It isn't undocumented in my module. From 'pydoc httplib':

That's cheating: pydoc is reading the source :-)

What I meant was, I'm calling robotparser, and there's no mention of
exceptions on http://docs.python.org/lib/module-robotparser.html

--
René Pijlman

[Rene Pijlman]

Roy Smith:

In theory, all exceptions which represent problems with the external
environment (rather than programming mistakes) should derive from
Exception, but not from StandardError.

Are you sure?

"""
The class hierarchy for built-in exceptions is:

Exception
+-- StandardError
| +-- KeyboardInterrupt
| +-- ImportError
| +-- EnvironmentError
| | +-- IOError
"""
http://www.python.org/doc/current/li...xceptions.html

--
René Pijlman

[Roy Smith]

In article <8n********************************@4ax.com>,
Rene Pijlman <re********************@my.address.is.invalid> wrote:

Roy Smith:

In theory, all exceptions which represent problems with the external
environment (rather than programming mistakes) should derive from
Exception, but not from StandardError.

Are you sure?

"""
The class hierarchy for built-in exceptions is:

Exception
+-- StandardError
| +-- KeyboardInterrupt
| +-- ImportError
| +-- EnvironmentError
| | +-- IOError
"""
http://www.python.org/doc/current/li...xceptions.html

Hmmm, OK, I missed EnvironmentError. So, what you need to do is:

try:
whatever()
except EnvironmentError:
...
except StandardError:
...
except Exception:
...

or something like that.

I do agree with you that there is some value in Java's "must catch or
re-export all exceptions" semantics, and this would be one of those places
where it would be useful. In general, however, I've always found it to be
a major pain in the butt, to the point where I sometimes just punt and
declare all my methods to "throw Exception" (or whatever the correct syntax
is). Not to mention that with a dynamic language like Python, it's
probably impossible to implement.

I think the real problem here is that the on-line docs are incomplete
because they don't list all the exceptions that this module can raise. The
solution to that is to open a bug on sourceforge against the docs.

[Roman Susi]

Rene Pijlman wrote:

Paul Rubin <http://ph****@NOSPAM.invalid>:

We have to get Knuth using Python.

Perhaps a MIX emulator and running TeXDoctest on his books will convince
him..

Or maybe Python written for MIX...
-Roman

[Bruno Desthuilliers]

Rene Pijlman a écrit :

Jorge Godoy:

Rene Pijlman:

my app was surprised by an
httplib.InvalidURL since I hadn't noticed this could be raised by
robotparser (this is undocumented).
It isn't undocumented in my module. From 'pydoc httplib':

That's cheating: pydoc is reading the source :-)

Yes, and that's the Right Thing(tm) to do. Source code don't lie. Source
code don't get out of sync. So source code *is* the best documentation
(or at least the most accurate).
What I meant was, I'm calling robotparser, and there's no mention of
exceptions on http://docs.python.org/lib/module-robotparser.html

That's why reading the source is better. CQFD !-)

[James Stroud]

Rene Pijlman wrote:

Steven D'Aprano:

ExpectedErrors = (URLError, IOError)
ErrorsThatCantHappen =

try:
process_things()
except ExpectedErrors:
recover_from_error_gracefully()
except ErrorsThatCantHappen:
print "Congratulations! You have found a program bug!"
print "For a $327.68 reward, please send the following " \
"traceback to Professor Donald Knuth."
raise
except:
print "An unexpected error occurred."
print "This probably means the Internet is broken."
print "If the bug still occurs after fixing the Internet, " \
"it may be a program bug."
log_error()
sys.exit()

Yes, I think I'll do something like this. Perhaps combined with Peter's
advice to not micromanage, like so:

Reraise = (LookupError, ArithmeticError, AssertionError) # And then some

try:
process_things()
except Reraise:
raise
except:
log_error()

Why catch an error only to re-raise it?

This falls under http://c2.com/cgi/wiki?YouReallyArentGonnaNeedThis

[Alex Martelli]

James Stroud <js*****@ucla.edu> wrote:

Reraise = (LookupError, ArithmeticError, AssertionError) # And then some

try:
process_things()
except Reraise:
raise
except:
log_error()

Why catch an error only to re-raise it?

To avoid the following handler[s] -- a pretty common idiom.
Alex

[Paul Rubin]

James Stroud <js*****@ucla.edu> writes:

try:
process_things()
except Reraise:
raise
except:
log_error()

Why catch an error only to re-raise it?

This falls under http://c2.com/cgi/wiki?YouReallyArentGonnaNeedThis

No, without the reraise, the bare "except:" clause would capture the
exception incorrectly and log it as an error.

[James Stroud]

Rene Pijlman wrote:

James Stroud:

Which suggests that "try: except HTTPException:" will be specific enough
as a catchall for this module.

The following, then, should catch everything you mentioned except the
socket timeout:

Your conclusion may be (almost) right in this case. I just don't like this
approach. Basically this is reverse engineering the interface from the
source at the time of writing the app.

This is using the source as documentation, there is no law against that.
It's better than waiting for a closed source software mogul to
benevolently give you the API documentation, which may not be 100%
accurate itself.
Even if you get it right, it may
fail next week when someone added an exception to a module.
Given how all of the exceptions raised by both modules are either (1)
builtin or (2) defined in the module itself and given that all
exceptions defined in these modules inheret from a single class, my
money is on the fact that this approach should be forward compatible.

But it seems to me that working with the internet as you are doing is
fraught with peril anyway.

Why? It shouldn't be.

How things are and how things should be have always been 2 entirely
different things. See early books by Robert J. Ringer for a more
complete discussion.

[Paul Rubin]

James Stroud <js*****@ucla.edu> writes:

approach. Basically this is reverse engineering the interface from the
source at the time of writing the app.
This is using the source as documentation, there is no law against
that.

That's completely bogus. Undocumented interfaces in the library
source are allowed to change between Python versions and across Python
implementations. If you write your application according to the
documented interfaces in the Python manual, it should not break when
someone upgrades to the next Python release. You should not have to
depend on undocumented aspects of the Python implementation which
change out of sync with the application.
How things are and how things should be have always been 2 entirely
different things. See early books by Robert J. Ringer for a more
complete discussion.

That's like saying programs shouldn't have bugs, but they often do
anyway. It doesn't mean the bugs are a good thing or that they
shouldn't be fixed.

[James Stroud]

Paul Rubin wrote:

James Stroud <js*****@ucla.edu> writes:

approach. Basically this is reverse engineering the interface from the
source at the time of writing the app.

This is using the source as documentation, there is no law against
that.

That's completely bogus. Undocumented interfaces in the library
source are allowed to change between Python versions and across Python
implementations. If you write your application according to the
documented interfaces in the Python manual, it should not break when
someone upgrades to the next Python release. You should not have to
depend on undocumented aspects of the Python implementation which
change out of sync with the application.

My suggestion was to use some common sense about the source code and
apply it. I've wasted many hours programming to faulty documentation
when a quick peek at the source code would have saved some serious time.
How do you think I happened accross the technique?

I would wager that, in 10 years time, the urllib2 module will not raise
any exceptions that don't inheret from urllib2.URLError or a built-in
(i.e. documented) exception; unless, of course, someone made a change in
a deliberate attempt to win said wager ;o)

How things are and how things should be have always been 2 entirely
different things. See early books by Robert J. Ringer for a more
complete discussion.

That's like saying programs shouldn't have bugs, but they often do
anyway. It doesn't mean the bugs are a good thing or that they
shouldn't be fixed.

The original reference was to the internet, which seems vast in its
sources of unforseen peril. If some gracious soul aspires to fix every
bug in the internet, I invite him to be my guest.

James

[Paul Rubin]

James Stroud <js*****@ucla.edu> writes:

My suggestion was to use some common sense about the source code and
apply it.

The common sense to apply is that if the source code says one thing
and the documentation says another, then one of them is wrong and
should be updated. Usually it's the source code that's wrong, but in
either case, saying to ignore the documentation and follow the code,
rather than repairing the discrepancy somehow, is poor advice.

[Steven D'Aprano]

On Sat, 04 Mar 2006 13:19:29 -0800, James Stroud wrote:

Why catch an error only to re-raise it?

As a general tactic, you may want to catch an exception, print extra
debugging information, then raise the exception again, like this:

try:
x = somedata()
y = someotherdata()
z = process(x, y)
except Exception:
print "Error!"
print "Value of x: %s" % x
print "Value of y: %s" % y
raise
Although that's not the reason the Original Poster is doing this.

The OP is doing it because catching all exceptions masks bugs. There are
certain exceptions which should be allowed through, as they indicate a bug
in the OP's code. Normally the tactic is to catch only the exceptions you
are interested in, and let everything else through, but the OP needs to
catch all exceptions because there are rare exceptions which he can't
predict in advance.

--
Steven.

[Rene Pijlman]

Steven D'Aprano:

The OP is doing it because catching all exceptions masks bugs. There are
certain exceptions which should be allowed through, as they indicate a bug
in the OP's code. Normally the tactic is to catch only the exceptions you
are interested in, and let everything else through, but the OP needs to
catch all exceptions because there are rare exceptions which he can't
predict in advance.

To add to that explanation: this is in a multithreaded ZODB-application.
When an unexpected exception occurs and remains uncaught, a thread
terminates, causing the thread pool to wait forever since some thread is
not consuming its termination request from the queue, causing the app to
not terminate, causing the ZODB to remain locked, causing other apps to
fail, causing havoc on my server.

I don't mind this when it's caused by a bug in my code, since that creates
the sense of urgency required to fix the bug. But it's annoying when it
happens because of an I/O exception caused by some other guys bug on the
other side of the planet :-)

--
René Pijlman

[Scott David Daniels]

Rene Pijlman wrote:

Steven D'Aprano:

The OP is doing it because catching all exceptions masks bugs. There are
certain exceptions which should be allowed through, as they indicate a bug
in the OP's code. Normally the tactic is to catch only the exceptions you
are interested in, and let everything else through, but the OP needs to
catch all exceptions because there are rare exceptions which he can't
predict in advance.

... This is in a multithreaded ZODB-application....
When an unexpected exception occurs and remains uncaught, a thread
terminates, ....

At the base of the thread code, you could put

import sys, threading, logging

class MyThread(threading.Thread):
def run(self):
try:
threading.Thread.run(self) # or whatever
except:
etype, error, traceback = sys.exc_info()
logging.warning('Exception %s: %s seen at %s' %
(etype.__name__, error, _someformat_(traceback)))
_try_to_rescue_or remove_this_thread_

If the condition is infrequent enough. If not (if you run a real
risk of multiple threads accessing the log simultaneously), have
a queue of log messages that you feed to a single logging thread.

--
-Scott David Daniels
sc***********@acm.org

[plahey]

>Yes, and that's the Right Thing(tm) to do. Source code don't lie. Source

code don't get out of sync. So source code *is* the best documentation
(or at least the most accurate).
I could not disagree more strongly with this. Not just no, but hell
no!
Yes, and that's the Right Thing(tm) to do.
No, it is a horrible thing to do. But since the documentation of some
modules is just plain horrible we sometimes have no choice.
Source code don't lie. Source code don't get out of sync.
True but implementation details change from release to release.
So source code *is* the best documentation (or at least the most accurate).

No, source code is the *worst possible* documentation because it makes
no distinction between implementation detail and method contract. If
the implementer considers the implementation to be the documentation
then his/her refactoring options are significantly reduced. Typically
implementers are not aware of this and they refactor anyway, breaking
client code left and right.

The C++ FAQ has a nice discussion of this issue. Minimally acceptable
documentation consists of the following (I think this is language
independent):

PURPOSE: What does this method/function do
REQUIRE: pre-conditions - What must have happened before calling this
method (or restrictions on the domain of the inputs)
PROMISE: post-conditions - What can you expect upon return or what
exceptions can be thrown

I consider the above to be the minimal amount of documentation that is
acceptable. If you have less than that, I consider the method to be
undocumented. Needless to say, I consider lots of code that I see to
be undocumented.

If you don't have the above, you get the problems that OP was hitting
(or worse, see the C++ FAQ). I am not a huge fan of Java's ubiquitous
use of checked exceptions or even of static typing but it does help
supply some of the above documentation (although in a suboptimal way)
that must be supplied by hand in Python. This is the dirty little
secret of dynamically typed languages. It makes proper documentation
even more important because method signatures supply less information.