473,692 Members | 2,481 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

exception handling in complex Python programs

Python provides a quite good and feature-complete exception handling
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.

Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.

Now, what do I mean more specifically... When a program starts growing
large, I find myself a bit scared of all the exceptions that might be
thrown: Python's exceptions as a result of runtime-detection of errors
(Python's dynamic typing also comes into play here), exceptions from
libraries used by the code, and exceptions from my lower-level
classes.
Python doesn't allow to specify which exceptions are thrown (C++'s
feature adding 'throw' after a function/method declaration specifying
the exceptions that can be thrown), and this leaves me at loss - what
should be caught and where ? Which errors should be left to
propagate ?

I've tried looking around the Python blogosphere, but there doesn't
seem to be much concern with this topic.

Apologies for the not-too-coherent post, but I suspect you feel the
pain too and can understand my meaning.

Eli

P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?

P.P.S. There's a great post on conditions (Common Lisp's exceptions)
here:
http://dlweinreb.wordpress.com/2008/...-really-about/
Not really CL specific, and can apply to Python's exceptions.
Aug 19 '08 #1
35 3502
On Aug 20, 12:19 am, eliben <eli...@gmail.c omwrote:
Python provides a quite good and feature-complete exception handling
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.

Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.

Now, what do I mean more specifically... When a program starts growing
large, I find myself a bit scared of all the exceptions that might be
thrown: Python's exceptions as a result of runtime-detection of errors
(Python's dynamic typing also comes into play here), exceptions from
libraries used by the code, and exceptions from my lower-level
classes.
Python doesn't allow to specify which exceptions are thrown (C++'s
feature adding 'throw' after a function/method declaration specifying
the exceptions that can be thrown), and this leaves me at loss - what
should be caught and where ? Which errors should be left to
propagate ?

I've tried looking around the Python blogosphere, but there doesn't
seem to be much concern with this topic.

Apologies for the not-too-coherent post, but I suspect you feel the
pain too and can understand my meaning.

Eli

P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?

P.P.S. There's a great post on conditions (Common Lisp's exceptions)
here:http://dlweinreb.wordpress.com/2008/...ns-exceptions-...
Not really CL specific, and can apply to Python's exceptions.
Maybe I am oversimplifying (and I am here to learn), but I catch all
exceptions which otherwise would be hard to understand as a user. In
other words, when a better error message is useful.

Again, this is probably too simple to help, but the only way to ignore
certain types of exceptions, as far as I know, is to catch them and
pass.
e.g. this ignores type errors...

try:
somethingBad()
except TypeError, err:
pass
except Exception, err:
raise TypeError(err)
I suppose you could write a decorator to do this if you want it at the
function level, but that seems a bit to broad. Shouldn't exceptions be
on a case-by-case basis to add protection and return information
exactly where it is needed?

- Rafe
Aug 19 '08 #2
On Tue, Aug 19, 2008 at 12:19 PM, eliben <el****@gmail.c omwrote:
Python provides a quite good and feature-complete exception handling
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.

Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.
Lots of people seem to have this fear. They treat exceptions like they
would treat error codes, trying to handle any possible case around any
particular call.

This is the wrong thing to do, and it only leads to more fragile code.
There are only 2 reasonable things to do with an exception:
1) handle it, by which I mean catch the exception knowing what error
condition it signifies, and take an appropriate action to correct the
error and
2) pass it up so something else has a chance at it.

Catching an exception when you don't know exactly what to do to fix it
is an error. At best, it will make debugging a program harder (because
you're losing context information about the error) and at worst it
adds bugs to your program. The way Javas checked exceptions encourage
empty or otherwise useless exception handlers is a major problem with
them.

There's some fear about presenting exceptions to the end user. That's
a user interface issues, not a software quality or engineering issue,
and it's resolvable with top-level handlers that log tracebacks
somewhere a user can't see them if desired.

Now, what do I mean more specifically... When a program starts growing
large, I find myself a bit scared of all the exceptions that might be
thrown: Python's exceptions as a result of runtime-detection of errors
(Python's dynamic typing also comes into play here), exceptions from
libraries used by the code, and exceptions from my lower-level
classes.
Python doesn't allow to specify which exceptions are thrown (C++'s
feature adding 'throw' after a function/method declaration specifying
the exceptions that can be thrown), and this leaves me at loss - what
should be caught and where ? Which errors should be left to
propagate ?
You should catch anything that you can correct. If you don't have a
specific answer for a specific exception, don't catch it.
I've tried looking around the Python blogosphere, but there doesn't
seem to be much concern with this topic.

Apologies for the not-too-coherent post, but I suspect you feel the
pain too and can understand my meaning.

Eli

P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?
Same rules apply. The only sort-of exception (no pun intended) is that
sometimes you want to re-raise as a different type of exception. Make
sure that you preserve all of the original information (including the
original traceback) if you do this.
P.P.S. There's a great post on conditions (Common Lisp's exceptions)
here:
http://dlweinreb.wordpress.com/2008/...-really-about/
Not really CL specific, and can apply to Python's exceptions.
--
http://mail.python.org/mailman/listinfo/python-list
Aug 19 '08 #3
Rafe wrote:
Again, this is probably too simple to help, but the only way to ignore
certain types of exceptions, as far as I know, is to catch them and
pass.
e.g. this ignores type errors...

try:
somethingBad()
except TypeError, err:
pass
except Exception, err:
raise TypeError(err)
so what kind of code are you writing where *type errors* are not
considered programming errors? (catching them and proceeding is one
thing, but catching them and ignoring them?)

I'd be really worried if I found that in a piece of source code I had to
maintain.

</F>

Aug 19 '08 #4
On Aug 19, 10:19*am, eliben <eli...@gmail.c omwrote:
P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?
You want to look up Easier to Ask Forgivness than Permission (EAFP)
which is touted as the "canonical" error-handling paradigm for Python.
This would give rise to the following function:

def do_something(fi lename):
try:
f = open(filename)
except IOError:
return err("File %s not found" % filename)
...

where err is a function that generates an error object that your
application understands. I personally think this is sloppy because you
have to couple the exception type with the function --- between file()
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3 and an IOError is thrown in the other three cases <bashes
head against keyboard>. The alternative is

def do_something(fi lename):
if not os.access(filen ame,os.R_OK):
return err(...)
f = open(filename)
...

or, (and this last one I actually used for a web application)

def do_something(fi lename):
if not os.access(filen ame,os.R_OK):
raise MyApplicationsE xceptionType("F ile not found...")
f = open(filename)
...

The last one has the advantage that you can write a request handler
like this

def handle_http_req uest(...):
func = specific_handle r_func(...)
try:
response = func(...)
return response
except MyApplicationsE xceptionType as exc: #3.0 syntax
return error_response( exc,...)

Exceptions you don't expect (i.e. bugs) will get handled by the web
app framework, but you get to handle your own exceptions. Raising your
own exception type can also be employed with the EAFP approach like
this:

def do_something(fi lename):
try:
f = open(filename)
except IOError:
raise MyApplicationsE xceptionType("F ile %s not found" %
filename)
...

If you are writing a library (for instance using a file for persistent
storage), then the answer to your question is "don't catch the
exception." Clients will expect the usual exception to be thrown when
a bad file name is passed.

David
Aug 19 '08 #5
On Tue, 19 Aug 2008 11:07:39 -0700, db*******@gmail .com wrote:
def do_something(fi lename):
if not os.access(filen ame,os.R_OK):
return err(...)
f = open(filename)
...

You're running on a multitasking modern machine, right? What happens when
some other process deletes filename, or changes its permissions, in the
time after you check for access but before you actually open it?

This isn't just a theoretical risk. There's a whole class of errors and
security holes based on similar race conditions. I find it amusing that
you consider it "sloppy" to deal with errors raised when actually opening
a file, but then recommend a technique that has a well-known failure mode.

That's not to say that I never use such techniques myself. For quick and
dirty scripts, where I can tolerate the risk of some other process moving
a file behind my back, I've been known to do something similar.

--
Steven
Aug 19 '08 #6
""" between file()
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3 and an IOError is thrown in the other three cases <bashes
head against keyboard>.
"""

This is *exactly* my concern with Python exceptions. You just never
know what can be thrown at you.
You want to look up Easier to Ask Forgivness than Permission (EAFP)
which is touted as the "canonical" error-handling paradigm for Python.
Any (semi)complete guides on this canonical paradigm online ? I've
only found some references in maillist discussions.
* def do_something(fi lename):
* * if not os.access(filen ame,os.R_OK):
* * * return err(...)
* * f = open(filename)
* * ...
But does os.access cover absolutely all the errors that can happen
during open() ? What guarantees it, and how can I know without you
teaching me, just from the docs ?

Aug 20 '08 #7
On Tue, 19 Aug 2008 22:24:45 -0700, eliben wrote:
>You want to look up Easier to Ask Forgivness than Permission (EAFP)
which is touted as the "canonical" error-handling paradigm for Python.

Any (semi)complete guides on this canonical paradigm online ? I've only
found some references in maillist discussions.
There's the glossary in the documentation:

http://docs.python.org/tut/node18.html

Look under 'duck-typing', 'EAFP', and 'LBYL'.

Ciao,
Marc 'BlackJack' Rintsch
Aug 20 '08 #8
On Tue, 19 Aug 2008 22:24:45 -0700, eliben wrote:
""" between file()
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3 and an IOError is thrown in the other three cases <bashes head
against keyboard>.
"""
I'm curious about the claim that open() will raise NameError in Python3.
I find it hard to credit that claim, but if it is correct, what's the
justification for that?
This is *exactly* my concern with Python exceptions. You just never know
what can be thrown at you.

It's true that documentation of exceptions is relatively weak in Python.
And some functions can raise a bewildering array of exceptions. See for
example this thread where somebody notes that urllib2.urlopen () can raise
any of six different exceptions:

http://mail.python.org/pipermail/bay...il/003187.html

And I've had it raise socket.error, which makes seven. And the
documentation only mentions one of those exceptions.

However, as Gregory Smith describes, some of those seven exceptions are
subclasses of others, so it is possible to reduce it down to three cases
-- and arguably one of those cases (ValueError) is a bug that needs
fixing, not an exception that needs catching.

That's probably as bad as it gets in Python, at least for the standard
library. Most functions don't raise arbitrary exceptions for sensible
data, and if you pass non-sensible data then you should treat the
exception as a bug in your code and fix it.

--
Steven
Aug 20 '08 #9
On Aug 19, 4:12*pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com .auwrote:
On Tue, 19 Aug 2008 11:07:39 -0700, dbpoko...@gmail .com wrote:
* def do_something(fi lename):
* * if not os.access(filen ame,os.R_OK):
* * * return err(...)
* * f = open(filename)
* * ...

You're running on a multitasking modern machine, right? What happens when
some other process deletes filename, or changes its permissions, in the
time after you check for access but before you actually open it?
This is a good point - if you want to use the correct way of opening
files, and
you don't want to worry about tracking down exception types, then we
can probably
agree that the following is the simplest, easiest-to-remember way:

def do_something(fi lename):
try:
f = open(filename)
except:
<handle exception>
...

Opening files is a special case where EAFP is the only correct
solution (AFAIK). I still liberally sprinkle LBYL-style "assert
isinstance(...) " and other similar assertions in routines. The point
is that EAFP conflicts with the interest of reporting errors as soon
as possible (on which much has been written see, for instance Ch. 8 -
Defensive Programming in Code Complete), but LBYL conflicts with
correctness when objects can be shared.

Also, look at the man page for access. I have found at least two (one
on my Linux box, another online) that essentially say "never use it."
I completely forgot about this in my last post...

David
Aug 20 '08 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
5786
by: Babu | last post by:
Hi, I am a Perl newbie and have a doubt on Perl exception handling. My understanding regarding exception handling is execute a piece of code, if any exception occurs, handle the exception and proceed as if the error never occurred at all. So code would look like Eval { # do something }; if ($@)
28
12089
by: dcrespo | last post by:
Hi all, How can I get a raised exception from other thread that is in an imported module? For example: --------------- programA.py ---------------
44
4211
by: craig | last post by:
I am wondering if there are some best practices for determining a strategy for using try/catch blocks within an application. My current thoughts are: 1. The code the initiates any high-level user tasks should always be included in a try/catch block that actually handles any exceptions that occur (log the exception, display a message box, etc.). 2. Low-level operations that are used to carry out the high level tasks
0
748
by: Delaney, Timothy (Tim) | last post by:
Lie wrote: Well, I believe the original intent was more along the lines of adding features, etc to Python, but it's apropos here as well. Whilst the connotations are good, and I think create the right mindset, it ain't gonna happen.
0
8611
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9090
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8970
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8812
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8810
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
6462
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
4329
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
2984
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2242
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.