mod_python exception catching, other repetitious per-page actions

[Karl A. Krueger]

I'm in the middle of refactoring a small mod_python Web application,
which uses the Publisher handler. This application is currently a
single main Python file (which loads several other files as modules)
with several function entry points for the different pages in the
application.

(The application is a tool to allow staff members here to request
firewall port openings for their servers. Thus there is a page (and
function) for "view my servers", "add a server", "add a service to a
server", and so forth. Each of these is a function in the published
file.)

One thing I would like to do is factor all of the top-level database
connecting, top-level exception handling, and error-page generation into
a single place. Whenever an exception occurs not trapped locally --
including an error connecting to the database -- I want to generate an
HTML error page instead of throwing the stacktrace at the user.

Currently this means that every function is one big try/except block
where the except block calls a TypesetErrorPage() function. Also
repeated are the function calls to set up the database backend
connection and to set up some per-user globals:

def mainpage(apache_request, some_cgi_arg):
try:
_connect_to_database()
# other repetitious stuff here
_do_user_specific_stuff(apache_request)
#
# specific code here
#
except Exception, err:
return _TypesetErrorPage(err)

def addserverpage(apache_request, some_cgi_arg, another_arg):
try:
_connect_to_database()
# other repetitious stuff here
_do_user_specific_stuff(apache_request)
#
# specific code here
#
except Exception, err:
return _TypesetErrorPage(err)

I'd like to factor this repetition out. However there does not seem to
be a straightforward way to do it -- the connect_to_database() could be
done at the module level (i.e. outside of any function), but this might
have problems if the Apache instance persists long enough between
requests for the database to time out.

One thing I have considered is to handle _all_ the pages through a
single function, which would look up the specific pages by name:

PAGES = { "mainpage" : mainpagefunc, "addserverpage" : addserverfunc }

def page(apache_request, which_page, *args):
try:
connect_to_database()
# other repetitious stuff here
do_user_specific_stuff(apache_request)
return PAGES[which_page](req, *args) # call the real page
except Exception, err:
return TypesetErrorPage(err)

.... but this has the problem that it sacrifices the argument name
matching which is a major useful feature of mod_python Publisher. I'm
not sure if keyword args are exactly the Right Thing either. It also
would make the URLs even longer than they already are, though I can
kludge that with Apache URL rewriting rules. (Bleah.)

I feel like I want a Lisp macro, but that's a topic I shouldn't even
invoke. Forget I said that. :)

Any thoughts?

--
Karl A. Krueger <kk******@example.edu>
Woods Hole Oceanographic Institution
Email address is spamtrapped. s/example/whoi/
"Outlook not so good." -- Magic 8-Ball Software Reviews

[Jeff Epler]

You could use a wrapper for each function, which at least keeps you
from needing the which_page argument..

def page_wrapper(callable):
def _wrapper(apache_request, *args):
try:
_connect_to_database()
_do_user_specific_stuff(apache_request)
return callable(*args)
except Exception, err:
return _TypesetErrorPage(err)
return _wrapper

def mainpage(apache_request, some_cgi_arg):
# specific code ehre
mainpage = page_wrapper(mainpage)
page_wrapper() could do extra magic with exec if you want it to have the
same signature as callable.. something like
def page_wrapper(callable):
d = {}
name = callable.__name__
argspec = getargspecstr(callable) # based on inspect.getargspec()
callspec = ... # argspec with default values removed
exec """def %s(%s): ... return callable(%s) ...""" \
% (name, argspec, callspec) in d
return d[name]
.... barf, eh?

Jeff
PS one of the currently discussed PEPs may let you write
def mainpage(apache_request, some_cgi_arg)[page_wrapper]: ...
instead of writing mainpage = page_wrapper(mainpage) after the function
body.

[Jacek Trzmiel]

"Karl A. Krueger" wrote:

I'm in the middle of refactoring a small mod_python Web application,
which uses the Publisher handler.
I have never used Publisher handler myself. Yes, I've read about it,
but I've decided to not use it after seeing that I will have to
explicitly tell which modules/functions I DON'T want external user to
be able to run - completely broken idea from security POV.
One thing I would like to do is factor all of the top-level database
connecting, top-level exception handling, and error-page generation into
a single place. Whenever an exception occurs not trapped locally --
including an error connecting to the database -- I want to generate an
HTML error page instead of throwing the stacktrace at the user. [...] One thing I have considered is to handle _all_ the pages through a
single function, which would look up the specific pages by name:
If you do this then you may as well drop Publisher handler completely.
One handle() function will work for you.
... but this has the problem that it sacrifices the argument name
matching which is a major useful feature of mod_python Publisher.
Not so big, as you need to validate arguments format anyway. With calls
like this:
pos = self.GetFsInt( 'posId' )
lastDate = self.GetFsDate( 'lastDate' )
you can get arguments from FieldStorage, ensure that arguments are
present and convert them to usable format (e.g. string=>datetime.date).
Some examples:

def RaiseMangled( self ):
raise apache.SERVER_RETURN, apache.HTTP_BAD_REQUEST

def RaiseNotFound( self ):
raise apache.SERVER_RETURN, apache.HTTP_NOT_FOUND

def GetFsString( self, key, default=None ):
if(default != None):
if self.fs.has_key(key):
value = self.fs[key]
else:
value = default
else:
value = self.fs[key]
if not isinstance( value, str ):
self.RaiseMangled()
return value

def GetFsInt( self, key ):
try:
return int( self.GetFsString(key,'') )
except ValueError:
self.RaiseMangled()

def GetFsDate( self, key ):
try:
return Url2Date( self.GetFsString(key,'') )
except ValueError:
self.RaiseMangled()

I've used one method in class to handle db connections, dispatch
requests and handle errors:

def handler( req ):
h = Handler()
return h.Handle( req )

class Handler:
[...]
def Handle(req):
try:
db = DbAccess.ConnectDb( dbName, host, user )
self.fs = util.FieldStorage( self.req, True )
[...]
self.req.add_common_vars()
env = req.subprocess_env
self.script = env['SCRIPT_NAME']
self.cmd = self.script[ rfind(self.script,'/')+1: ]

if self.cmd == '':
self.HandleMenu( db )
elif self.cmd == 'PosDailyFeed':
self.HandlePosDailyFeed( db )
elif self.cmd == 'PosDailySubmit':
self.HandlePosDailySubmit( db )
[...]
else:
self.RaiseNotFound()
[...]
except:
[...]

# handler for urls like this:
# /PosDailyFeed?posId=3&date=2004-03-22
def HandlePosDailyFeed( self, db ):
pos = self.GetFsInt( 'posId' )
date = self.GetFsDate( 'date' )
pb = self.CreatePageBuilder( db )
valueDict = db.GetPosWorkerData( date, pos )
BuildDailyFeedPage( pb, db, date, pos, valueDict,
String2Html(db.GetPosName(pos)) )
self.SendPage( pb )

I'm not sure if keyword args are exactly the Right Thing either. It also
would make the URLs even longer than they already are, though I can
kludge that with Apache URL rewriting rules. (Bleah.)

If you use straight handler() function instead of Publisher handler then
you have complete freedom how your URLs will look like.

Best regards,
Jacek.

[Jeff Epler]

One more idea -- I don't know mod_python, but is the Publisher class
something you could subclass, to add your own behavior there instead of
in each page function?

Jeff

[Karl A. Krueger]

Jeff Epler <je****@unpythonic.net> wrote:

page_wrapper() could do extra magic with exec if you want it to have the
same signature as callable.. something like
def page_wrapper(callable):
d = {}
name = callable.__name__
argspec = getargspecstr(callable) # based on inspect.getargspec()
callspec = ... # argspec with default values removed
exec """def %s(%s): ... return callable(%s) ...""" \
% (name, argspec, callspec) in d
return d[name]
... barf, eh?

Barf? I -did- say I kind of wanted a Lisp macro; exec is about as close
as Python can get, I suppose.

Anyway, this has given me useful thoughts. Of course, I may just end up
back with keyword arguments. :-/

(But probably -not- reimplementing my whole application in Lisp. I'm
not that good in Lisp yet.)

--
Karl A. Krueger <kk******@example.edu>
Woods Hole Oceanographic Institution
Email address is spamtrapped. s/example/whoi/
"Outlook not so good." -- Magic 8-Ball Software Reviews

[Karl A. Krueger]

Jacek Trzmiel <sc***@hot.pl> wrote:

"Karl A. Krueger" wrote:

I'm in the middle of refactoring a small mod_python Web application,
which uses the Publisher handler.

I have never used Publisher handler myself. Yes, I've read about it,
but I've decided to not use it after seeing that I will have to
explicitly tell which modules/functions I DON'T want external user to
be able to run - completely broken idea from security POV.

Actually, Publisher never exposes modules imported into yours. It
traverses your module for names and types before exposing anything -- it
won't expose any object whose name begins with an underscore, or any
module.

(Oddly enough, it does expose exceptions, but just their string
representation.)

If anyone reading this is concerned about Publisher security, these
audit functions might help:

def AuditModuleForPublisher(module):
import types
for name in module.__dict__:
object = module.__dict__[name]
if name.startswith("_"):
# name starts with an underscore, not exposed
pass
elif type(object) == types.ModuleType:
# object is a module, not exposed
pass
elif type(object) == types.FunctionType:
print ":: Exposed function:", name
elif isinstance(object, str) or isinstance(object, unicode):
if "passw" in name or "PASSW" in name:
print "!! Your %s password is %s." % (name, object)
else:
print ":: Exposed string:", name
else:
print ":: Exposed variable:", name

A nicer one:

def SymbolsExposedToPublisher(module):
return [ sym for sym in module.__dict__
if (not sym.startswith("_"))
and (type(module.__dict__[sym]) != type(module)) ]

By enumerating the list of functions and variables you *intend* to
expose, a unit test should not be too hard to derive from this.

One thing I have considered is to handle _all_ the pages through a
single function, which would look up the specific pages by name:

If you do this then you may as well drop Publisher handler completely.
One handle() function will work for you.

I agree. However, since I want the functionality of Publisher's
argument name mapping, that is not what I want to do.

--
Karl A. Krueger <kk******@example.edu>
Woods Hole Oceanographic Institution
Email address is spamtrapped. s/example/whoi/
"Outlook not so good." -- Magic 8-Ball Software Reviews

[Dave Benjamin]

In article <ma************************************@python.org >, Jeff Epler wrote:

One more idea -- I don't know mod_python, but is the Publisher class
something you could subclass, to add your own behavior there instead of
in each page function?

I suppose I should be embarrassed to say this, but I just made a copy and
modified it to suit my own needs. I don't think its design is particularly
extensible, but it's a small script that's easy to understand.

--
..:[ dave benjamin: ramen/[sp00] -:- spoomusic.com -:- ramenfest.com ]:.
: please talk to your son or daughter about parametric polymorphism. :