Block-structured resource handling via decorators

[John Perks and Sarah Mount]

When handling resources in Python, where the scope of the resource is
known, there seem to be two schools of thought:

(1) Explicit:
f = open(fname)
try:
# ...
finally:
f.close()

(2) Implicit: let the GC handle it.

I've come up with a third method that uses decorators to achieve a
useful mix between the two. The scope of the resource is clear when
reading the code (useful if the resource is only needed in part of a
function), while one does not have to write an explicit cleanup. A
couple of examples:

@withFile(fname, 'w')
def do(f):
# ... write stuff into file f ...

@withLock(aLock):
def do():
# ... whatever you needed to do once the lock was acquired,
# safe in the knowledge it will be released afterwards ...

(The name "do" is arbitrary; this method has the "mostly harmless"
side-effect of assigning None to a local variable with the function
name.)

I find it clear because I come from a C++/C#/Java background, and I
found C#'s using-blocks to very useful, compared to the explicit
finallys of Java. I know that Python's deterministic finalization sort
of achieves the same effect, but I had been led to believe there were
complications in the face of exceptions.

The implementation is easily extensible: a handler for a new type of
resource can be written in as a couple of lines. For the examples above:

class withFile(blockScopedResource):
init, cleanup = open, 'close'

It's so simple I was wondering why I haven't seen it before. Possibly:
it's a stupid idea and I just can't see why;
everyone knows about it except me;
it's counter-intuitive (that's not the way decorators were intended);
it's "writing C# in Python" or in some other way unPythonic;
I've actually had an idea that is both Original and non-Dumb.

If the last is the case, can someone let me know, and I'll put up the
code and explain how it all works. On the other hand, if there is
something wrong with it, please can someone tell me what it is?

Thanks

John Perks

[Neil Hodgson]

John Perks:

When handling resources in Python, where the scope of the resource is
known, there seem to be two schools of thought:
...

This is part of what PEP 343 addresses:
http://www.python.org/peps/pep-0343.html

Neil

[Mike Meyer]

"John Perks and Sarah Mount" <jo**********@estragon.freeserve.co.uk> writes:

When handling resources in Python, where the scope of the resource is
known, there seem to be two schools of thought:

(1) Explicit:
f = open(fname)
try:
# ...
finally:
f.close()

(2) Implicit: let the GC handle it.
The only cases I see the first school of thought is when the resource
in question is "scarce" in some way. For example, most OS's place a
limit on the number of open files a process can have, some rather
tight. CPython's garbage collector will close an open file when it
leaves scope. Jython's GC will close it when the file is collected,
but you have no idea of when that will be, and an "open" failing won't
trigger a GC. So in this case, the first form is less likely to fail
unexpectedly.
I've come up with a third method that uses decorators to achieve a
useful mix between the two. The scope of the resource is clear when
reading the code (useful if the resource is only needed in part of a
function), while one does not have to write an explicit cleanup. A
couple of examples:

@withFile(fname, 'w')
def do(f):
# ... write stuff into file f ...

@withLock(aLock):
def do():
# ... whatever you needed to do once the lock was acquired,
# safe in the knowledge it will be released afterwards ...

The implementation is easily extensible: a handler for a new type of
resource can be written in as a couple of lines. For the examples above:

class withFile(blockScopedResource):
init, cleanup = open, 'close'

It's so simple I was wondering why I haven't seen it before. Possibly:
it's a stupid idea and I just can't see why;
everyone knows about it except me;
it's counter-intuitive (that's not the way decorators were intended);
it's "writing C# in Python" or in some other way unPythonic;
I've actually had an idea that is both Original and non-Dumb.

Well, I'd say that using a string for cleanup and a function for init
is unpythonic. But the general idea seems to be a good one. Making it
easy to deal with resources that must be explicitly released is a good
thing. The question is whether having to turn your scope into a
function to do this is more trouble than it's worth.

I'd certainly be interested in seeing the implementation.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Paul Rubin]

Mike Meyer <mw*@mired.org> writes:

When handling resources in Python, where the scope of the resource is
known, there seem to be two schools of thought:
(1) Explicit: ...
(2) Implicit: let the GC handle it.

The only cases I see the first school of thought is when the resource
in question is "scarce" in some way. For example, most OS's place a
limit on the number of open files a process can have, some rather
tight. CPython's garbage collector will close an open file when it
leaves scope. Jython's GC will close it when the file is collected,
but you have no idea of when that will be, and an "open" failing won't
trigger a GC. So in this case, the first form is less likely to fail
unexpectedly.

I thought there was a Zen thing somewhere saying that Python is not
Perl. One of Perl's silliest tenets is that the implementation
defines the language. There are already 4 different Python
implementations (CPython, Jython, Pypy, IronPython) and probably more
on the way. We should take the view that the manual, not the
implementation, defines the language.

The manual doesn't specify anything about reference counting or GC
happening as soon as something goes out of scope. That is a GOOD
thing since requiring CPython-like behavior would eliminate many good
GC techniques, including some that are both more efficient and easier
for extension writers to deal with. Since reference counting is not
in the manual, it is not part of the language, it's subject to change
at any time in any implementation, and non-throwaway Python apps
should not depend on it.

On the other hand, the try/finally explicit release is cumbersome and
non-tasty.

Therefore, the correct solution is a language extension along the
lines of PEP 343. I haven't studied PEP 343 specifically enough to
say that I think every detail of it is the right thing, but at least
it is the right idea.

[Michele Simionato]

I will shamelessly point out my decorator module:
http://www.phyast.pitt.edu/~micheles.../decorator.zip

The documentation is here:
http://www.phyast.pitt.edu/~micheles...umentation.htm

There is an example of redirecting stdout which is
relevant to this thread.
HTH,

Michele Simionato

[John Perks and Sarah Mount]

> The only cases I see the first school of thought is when the resource

in question is "scarce" in some way.
By "resource" I meant anything with some sort of acquire/release
semantics. There may be plenty of threading.Locks available, but it's
still important that a given Lock is released when not needed.

For example, most OS's place a

class withFile(blockScopedResource):
init, cleanup = open, 'close'

Well, I'd say that using a string for cleanup and a function for init
is unpythonic.

I could have specified cleanup as lambda f:f.close(), but as I thought
it might be quite common to call a method on the resourse for cleanup,
if a string is specified a method of that name is used instead.
The question is whether having to turn your scope into a
function to do this is more trouble than it's worth.
Needing one slightly contrived-looking line (the def) vs a try-finally
block with explicit cleanup code? I know which I'd prefer, but for all I
know I could in a minority of 1 here.

I'd certainly be interested in seeing the implementation.

And so you shall...

I start with the base class. It does all the work, everything else is
just tweaks for convenience. Normally, then, you wouldn't need to bother
with all the __init__ params.

class blockScopedResource(object):
def __init__(self, init, cleanup,
initArgs, initKwargs, cleanupArgs, cleanupKwargs,
passResource, resourceIsFirstArg):

self.init = init # function to get resource
self.cleanup = cleanup # function to release resource
self.initArgs, self.initKwargs = initArgs, initKwargs
self.cleanupArgs, self.cleanupKwargs = cleanupArgs,
cleanupKwargs
self.passResource = passResource # whether resource is passed
into block
self.resourceIsFirstArg = resourceIsFirstArg # whether resource
is arg to init,
# rather than returned from it

def __call__(self, block):
resource = self.init(*self.initArgs, **self.initKwargs)
if self.resourceIsFirstArg:
resource = self.initArgs[0]

try:
if self.passResource:
block(resource)
else:
block()
finally:
self.cleanup(resource, *self.cleanupArgs,
**self.cleanupKwargs)

But this still won't do conveniently for files and locks, which are my
motivating examples.

The simpleResource class constructor gets its setup from attributes on
the type of the object being created, with sensible defaults being set
on simpleResource itself. As stated above, if a string is supplied as
init or cleanup, it is treated as a method name and that method is used
instead.

def stringToMethod(f):
# Getting the attribute from the class may have wrapped it into
# an unbound method; in this case, unwrap it
if isinstance(f, types.MethodType) and f.im_self is None:
f = f.im_func
if not isinstance(f, basestring): return f
def helper(resource, *args, **kwargs):
return getattr(resource, str(f))(*args, **kwargs)
return helper

class simpleResource(blockScopedResource):
def __init__(self, *initArgs, **initKwargs):
# get attributes off type
t = type(self)
blockScopedResource.__init__(self,
stringToMethod(t.init), stringToMethod(t.cleanup),
initArgs, initKwargs, t.cleanupArgs, t.cleanupKwargs,
t.passResource, t.resourceIsFirstArg)

# defaults supplied here
cleanupArgs, cleanupKwargs = (), {}
passResource = True
resourceIsFirstArg = False
Then useful implementations can be written by:

class withFile(simpleResource):
init, cleanup = open, 'close'

class withLock(simpleResource):
init, cleanup = 'acquire', 'release'
passResource = False
resourceIsFirstArg = True
And new ones can be created with a similar amount of effort.

Of course, one-liners can be done without using the decorator syntax:

withLock(aLock)(lambda:doSomething(withAnArg))

Gotcha: If you stack multiple resource-decorator it won't do what you
want:

# !!! DOESN'T WORK !!!
@withLock(aLock)
@withLock(anotherLock)
def do():
# ...

Either nest them explicitly (causing your code you drift ever further to
the right):
@withLock(aLock)
def do():
@withLock(anotherLock)
def do():
# ...

Or come up with a multiple-resource handler, which shouldn't be too
hard:

@withResources(withLock(aLock), withLock(anotherLock),
withFile('/dev/null'))

But I'll get round to that another day.