Python Code Auditing Tool

[Robey Holderith]

Does anybody know of a tool that can tell me all possible exceptions that
might occur in each line of code? What I'm hoping to find is something
like the following:

given all necessary python source and a given line ( my.py:40 ) it would
generate a list of possible exception classes sorted by function
(preferably in a tree).

Example:
------------------

my.py:40 | parsestring(genstring())

Possible Exceptions:

-def parsestring():
InvalidCharacterException
EmptyStringException
-class string, def split():
(All Exceptions that might occur directly in string.split() that are
not caught by parsestring())
(All functions called by string.split() and their exceptions and sub-
functions)
-def genstring():
SomeException
...

--------------------

This would be extremely useful for deciding how to write try: except
blocks and in figuring out what all possible errors that might occur would
be.

-Robey Holderith

[Paul Rubin]

Robey Holderith <ro***@flaminglunchbox.net> writes:

Does anybody know of a tool that can tell me all possible exceptions that
might occur in each line of code? What I'm hoping to find is something
like the following:

That is impossible. The parameter to the raise statement is a class
object, which can be anything. I.e. you could say:

class ex1: pass
class ex2: pass

if something(): my_ex = ex1
else: my_ex = ex2

raise my_ex # static tool can't know what exception gets raised here.

[Robey Holderith]

On Tue, 01 Feb 2005 21:52:28 -0800, Paul Rubin wrote:

Robey Holderith <ro***@flaminglunchbox.net> writes:

Does anybody know of a tool that can tell me all possible exceptions that
might occur in each line of code? What I'm hoping to find is something
like the following:

That is impossible. The parameter to the raise statement is a class
object, which can be anything. I.e. you could say:

class ex1: pass
class ex2: pass

if something(): my_ex = ex1
else: my_ex = ex2

raise my_ex # static tool can't know what exception gets raised here.

I suppose that I am willing to lessen my expectations from _all_ to most.
;-) Regarding your example I could also do:

if something():
def nothing(): return 0
else:
def nothing(): return 1

But this doesn't stop IDEs from attempting to do auto-completion. I'm not
trying to find hidden exceptions... just trying to easily get an idea of
what could go wrong on each line of code.

-Robey Holderith

[Diez B. Roggisch]

> I suppose that I am willing to lessen my expectations from _all_ to most.

;-) Regarding your example I could also do:

if something():
def nothing(): return 0
else:
def nothing(): return 1

But this doesn't stop IDEs from attempting to do auto-completion. I'm not
trying to find hidden exceptions... just trying to easily get an idea of
what could go wrong on each line of code.

There is AFAIK only one language that this can de accomplished - java, and
that's because of these checked exceptions of theirs. But checked
exceptions are considered harmful:

http://www.gcek.net/ref/books/sw/ooad/tip/#_Toc41169682

I totally agree with that - in java, I tend to throw SystemExceptions to rid
myself of endless try/catch clauses that obscure the real problem.

So - there is no way of knowing this. The only thing I can think of is to
keep some docs around that specify what exceptions to be expected, and that
tool of yours could try and see if it can identify a function/method by
name and notify you of the possible exceptions thrown. Might actually work
out quite well for the standardlib, if one does the work for annotating all
functions/methods properly.

--
Regards,

Diez B. Roggisch

[Neil Benn]

Diez B. Roggisch wrote:

I suppose that I am willing to lessen my expectations from _all_ to most.
;-) Regarding your example I could also do:

<

<snip>
There is AFAIK only one language that this can de accomplished - java, and
that's because of these checked exceptions of theirs. But checked
exceptions are considered harmful:

http://www.gcek.net/ref/books/sw/ooad/tip/#_Toc41169682

I totally agree with that - in java, I tend to throw SystemExceptions to rid
myself of endless try/catch clauses that obscure the real problem.

<snip>
Hello,

I'm afraid that the only reliable way to gather what exceptions are
raised is to read docs and/or come up with test cases. This has been a
bugbear of mine in Python as it's not common to find a nice :Exceptions:
IOError <desc>, IllegalArgumentError <desc> type of description in the docs.

However if you want an incomplete test, you could parse the code and
check for raises and retrieve the class name of the exception - however
this would be patchy at best. Therefore it would sort of negate the
point of doing the analysis in the first place.

Even in Java you cannot find every exception that will be
thrown, only 'checked' exceptions but this is a percentage of all the
exceptions (BTW why do you throw SystemException - it's a CORBA
exception! OK, it's a runtime exception but why not just simply extend
RuntimeException?). Also, if someone ever puts - catch (Exception e){}
in their code they deserve to be kneecapped, IMHO the fault is with
sloppy coding not with the supplied tools.

Unfortunately its docs and testing again, that's why we get paid (if
you're doing a job) or not paid (if you're doing it for fun!). Although
one language which comes closer is Eiffel which has require and ensure
clauses on every method (following Meyer's Programming by contract
philosophy).

Cheers,

Neil

--

Neil Benn
Senior Automation Engineer
Cenix BioScience
BioInnovations Zentrum
Tatzberg 46
D-01307
Dresden
Germany

Tel : [phone]
e-mail : be**@cenix-bioscience.com
Cenix Website : http://www.cenix-bioscience.com

[Diez B. Roggisch]

Hi,

I'm afraid that the only reliable way to gather what exceptions are
raised is to read docs and/or come up with test cases. This has been a
bugbear of mine in Python as it's not common to find a nice :Exceptions:
IOError <desc>, IllegalArgumentError <desc> type of description in the
docs.

However if you want an incomplete test, you could parse the code and
check for raises and retrieve the class name of the exception - however
this would be patchy at best. Therefore it would sort of negate the
point of doing the analysis in the first place.
I don't want that - the OP wants. I agree with you.
Even in Java you cannot find every exception that will be
thrown, only 'checked' exceptions but this is a percentage of all the
exceptions (BTW why do you throw SystemException - it's a CORBA
exception! OK, it's a runtime exception but why not just simply extend
RuntimeException?). Also, if someone ever puts - catch (Exception e){}
in their code they deserve to be kneecapped, IMHO the fault is with
sloppy coding not with the supplied tools.
Most probably I throw RuntimeException - that was out of my head, I luckily
I haven't been coding java too much lately :)
Unfortunately its docs and testing again, that's why we get paid (if
you're doing a job) or not paid (if you're doing it for fun!). Although
one language which comes closer is Eiffel which has require and ensure
clauses on every method (following Meyer's Programming by contract
philosophy).

Full ack again.

--
Regards,

Diez B. Roggisch

[System Administrator]

Does anybody know of a tool that can tell me all possible exceptions
that might occur in each line of code? What I'm hoping to find is
something like the following:

Paul> That is impossible. The parameter to the raise statement is a
Paul> class object, which can be anything.

Sure, but in all but the rarest of cases the first arg to raise is a
specific exception, probably one of the standard exceptions. In the Python
code in the distribution (ignoring the test directory where all sorts of
mischief is done to stress things), here are the most common words following
"raise" where "raise" is the first word on the line:

% find . -name '*.py' \ | egrep -v '\./test' \
| xargs egrep '^ *raise ' \
| awk '{print $3}' \
| sed -e 's/[(,].*//' \
| sort \
| uniq -c \
| sort -rn \
| head -15

246 ValueError
227 aetools.Error
216 Error
124 TypeError
101 error
75 RuntimeError
53 IOError
36 NotImplementedError
36 ImportError
36 EOFError
31 SyntaxError
23 KeyError
23 AttributeError
22 DistutilsPlatformError
21 UnicodeError

Without checking, my guess is that #5 ("error") is one of a handful of
exception classes defined at module scope (ftplib, anydbm, sre_constants,
poplib, among others all define such an exception class), and not a variable
that accepts multiple values as in your example.

In short, while not perfect, simply grepping for '^ *(class|def|raise) ' and
printing the first and second words of each output line would probably give
you a pretty good idea of what gets raised where.

Skip

[Roy Smith]

In article <ma***************************************@python. org>,
System Administrator <sk**@pobox.com> wrote:

>> Does anybody know of a tool that can tell me all possible exceptions
>> that might occur in each line of code? What I'm hoping to find is
>> something like the following:

Paul> That is impossible. The parameter to the raise statement is a
Paul> class object, which can be anything.

Sure, but in all but the rarest of cases the first arg to raise is a
specific exception, probably one of the standard exceptions. In the Python
code in the distribution (ignoring the test directory where all sorts of
mischief is done to stress things), here are the most common words following
"raise" where "raise" is the first word on the line:

% find . -name '*.py' \

> | egrep -v '\./test' \
> | xargs egrep '^ *raise ' \
> | awk '{print $3}' \
> | sed -e 's/[(,].*//' \
> | sort \
> | uniq -c \
> | sort -rn \
> | head -15

246 ValueError
227 aetools.Error
216 Error
124 TypeError
101 error
75 RuntimeError
53 IOError
36 NotImplementedError
36 ImportError
36 EOFError
31 SyntaxError
23 KeyError
23 AttributeError
22 DistutilsPlatformError
21 UnicodeError

It's kind of interesting (scarry?) that in roughly 20% of the cases
nothing more specific than Error is raised.

[Peter Hansen]

Roy Smith wrote:

Skip Montanaro wrote:

246 ValueError
227 aetools.Error
216 Error
124 TypeError
101 error
75 RuntimeError
53 IOError
36 NotImplementedError
36 ImportError
36 EOFError
31 SyntaxError
23 KeyError
23 AttributeError
22 DistutilsPlatformError
21 UnicodeError

It's kind of interesting (scarry?) that in roughly 20% of the cases
nothing more specific than Error is raised.

(In case someone reading doesn't know) there isn't actually
an "Error" in the standard set of exceptions. It seems
likely that pretty much all of those uses are actually
module-specific Errors, and as such they are probably much
more specific than the unadorned name might imply.

Also, when one is trying to pick an appropriate exception
to raise, it is often the case that none of the standard
exceptions seems appropriate. In those cases (although I
personally would prefer a different name than "Error")
there's often no good alternative to making your own
unique module-specific Exception subclass.

-Peter

[Skip Montanaro]

246 ValueError
227 aetools.Error
216 Error
124 TypeError
101 error
75 RuntimeError
53 IOError
36 NotImplementedError
36 ImportError
36 EOFError
31 SyntaxError
23 KeyError
23 AttributeError
22 DistutilsPlatformError
21 UnicodeError

Roy> It's kind of interesting (scarry?) that in roughly 20% of the cases
Roy> nothing more specific than Error is raised.

Not really. You might have code in a module named mod like this:

class Error(Exception): pass

then later:

def foo(...):
... blah blah blah ...
if condition:
raise Error, "hey dummy!"

The caller might look like:

import mod

...

try:
mod.foo()
except mod.Error, msg:
print msg

That said, the tendency for many newer modules seems to be to discriminate
exceptions based on type (look at urllib2 for example) while older modules
tended to have just a single exception they raised (look at ftplib). I
suspect that has something to do with whether the module was originally
written before or after Python introduced class exceptions.

Skip