CGIs and file exclusion

[darkhorse]

Hi all,

While doing a quite big "set of programs" for a university subject I've
found myself in the middle of a problem someone surely has had before, so
I'm looking for some help.

At one point, I call a python cgi that pickle.load's a file, adds or deletes
a registry and dumps the result again in the file.
I'm worried that the cgi could be called simultaneously from two or more
different computers, thus most probably corrupting the files. I don't think
I can use a mutex as it's two different instances of the program and not
different threads, so I was thinking about locking the files between
programs, but I really don't know how to do that. It's not a problem if
there's no portable way of doing this, it's only going to be run on a linux
based computer.
Another solution I would accept is that the second called cgi detects that
other instance is running and displays a message saying to try again later.
Yes, not quite professional, but it'd do the job, after all this is just a
little detail I want to settle for a quite picky professor and not a "real
life" thing.
I think that's all the background you need, if someone can answer with
references on what should I look for or even better example code that would
be simply great.
Many thanks in advance.
DH

[Diez B. Roggisch]

I'd suggest using ZODB instead of directly pickling - then every cgi can
open the database with its own connection. ZODB will manage concurrency
issues.
--
Regards,

Diez B. Roggisch

[David M. Cooke]

"darkhorse" <no************ @terra.es> writes:

Hi all,

While doing a quite big "set of programs" for a university subject I've
found myself in the middle of a problem someone surely has had before, so
I'm looking for some help.

At one point, I call a python cgi that pickle.load's a file, adds or deletes
a registry and dumps the result again in the file.
I'm worried that the cgi could be called simultaneously from two or more
different computers, thus most probably corrupting the files. I don't think
I can use a mutex as it's two different instances of the program and not
different threads, so I was thinking about locking the files between
programs, but I really don't know how to do that. It's not a problem if
there's no portable way of doing this, it's only going to be run on a linux
based computer.
You want the fcntl module, and use either flock() or lockf().

import fcntl, cPickle
# open in update mode
fo = open('somefile. pickle', 'r+')
# acquire an exclusive lock; any other process trying to acquire this
# will block
fcntl.lockf(fo. fileno(), fcntl.LOCK_EX)
data = cPickle.load(fo )
.... do stuff to data ...
# go back to the beginning
fo.seek(0)
# write out the new pickle
cPickle.dump(da ta, fo)
# throw the rest of the (old) pickle away
fo.truncate()
# and close. This releases the lock.
fo.close()
Another solution I would accept is that the second called cgi detects that
other instance is running and displays a message saying to try again later.
Yes, not quite professional, but it'd do the job, after all this is just a
little detail I want to settle for a quite picky professor and not a "real
life" thing.

If you want to do something like that, use
fcntl.LOCK_EX | fcntl.LOCK_NB as the flags to fcntl.lockf(). If the
lock can't be acquired, an IOError will be raised, and you can print
out your message.

Note that lockf() does advisory locks; a process that just opens the
file and does something with it without calling lockf() *won't* be blocked.

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)phy sics(dot)mcmast er(dot)ca

[Tim Roberts]

"Diez B. Roggisch" <de*********@we b.de> wrote:

I'd suggest using ZODB instead of directly pickling - then every cgi can
open the database with its own connection. ZODB will manage concurrency
issues.

Holy moley, isn't that something like recommending a turbocharged,
12-cylinder, air-conditioned SUV as a cell phone charger?

Sure, Zope is the answer, but only if you phrase the question very, very
carefully.
--
- Tim Roberts, ti**@probo.com
Providenza & Boekelheide, Inc.

[Andrew Durdin]

On Thu, 04 Nov 2004 21:19:19 -0800, Tim Roberts <ti**@probo.com > wrote:

"Diez B. Roggisch" <de*********@we b.de> wrote:

I'd suggest using ZODB instead of directly pickling - then every cgi can
open the database with its own connection. ZODB will manage concurrency
issues.

Holy moley, isn't that something like recommending a turbocharged,
12-cylinder, air-conditioned SUV as a cell phone charger?

Sure, Zope is the answer, but only if you phrase the question very, very
carefully.

Note that he said ZODB, not Zope -- ZODB can be used on its own
without the rest of Zope -- see
http://www.zope.org/Wikis/ZODB/FrontPage

[darkhorse]

"David M. Cooke" <co**********@p hysics.mcmaster .ca> escribió en el mensaje
news:qn******** *****@arbutus.p hysics.mcmaster .ca...
[David wrotes about fcntl.lockf...]

Thank you all for your answers, seems a good solution for what I'm doing.
I'll try it this afternoon.

Thanks again
DH

[Michael Foord]

"darkhorse" <no************ @terra.es> wrote in message news:<2u******* ******@uni-berlin.de>...

Hi all,

While doing a quite big "set of programs" for a university subject I've
found myself in the middle of a problem someone surely has had before, so
I'm looking for some help.

At one point, I call a python cgi that pickle.load's a file, adds or deletes
a registry and dumps the result again in the file.
I'm worried that the cgi could be called simultaneously from two or more
different computers, thus most probably corrupting the files. I don't think
I can use a mutex as it's two different instances of the program and not
different threads, so I was thinking about locking the files between
programs, but I really don't know how to do that. It's not a problem if
there's no portable way of doing this, it's only going to be run on a linux
based computer.
Another solution I would accept is that the second called cgi detects that
other instance is running and displays a message saying to try again later.
Yes, not quite professional, but it'd do the job, after all this is just a
little detail I want to settle for a quite picky professor and not a "real
life" thing.
I think that's all the background you need, if someone can answer with
references on what should I look for or even better example code that would
be simply great.
Many thanks in advance.
DH

A simple solution that doesn't scale well is to create a file when the
access starts. You can check if the file exists and pause until the
other process deletes it - with a timeout in case the file gets keft
there due to an error.

Obviously not an industrial strength solution, but it does work...

import time
import os

def sleep(thelockfi le, sleepcycle=0.01 , MAXCOUNT=200):
"""Sleep until the lockfile has been removed or a certain number
of cycles have gone.
Defaults to a max 2 second delay.
"""
counter = 0
while os.path.exists( thelockfile):
time.sleep(slee pcycle)
counter += 1
if counter > MAXCOUNT: break

def createlock(thel ockfile):
"""Creates a lockfile from the path supplied."""
open(thelockfil e, 'w').close()

def releaselock(the lockfile):
"""Deletes the lockfile."""
if os.path.isfile( thelockfile):
os.remove(thelo ckfile)

The sleep function waits until the specified file dissapears - or it
times out.

Regards,

Fuzzy

http://www.voidspace.org.uk/atlantib...thonutils.html

[Michele Simionato]

"Diez B. Roggisch" <de*********@we b.de> wrote in message news:<cm******* ******@news.t-online.com>...

I'd suggest using ZODB instead of directly pickling - then every cgi can
open the database with its own connection. ZODB will manage concurrency
issues.

Ok, but then I guess you need an external long-living process keeping the
DB open for you. If the cgi script opens and close the database by itself,
the only thing the ZODB buys for you is raising

IOError: [Errno 11] Resource temporarily unavailable

(not unexpectedly). Here is what I tried:

$ cat concurrency.py
import ZODB
from ZODB.FileStorag e import FileStorage

def openzodb(dbname ):
db = ZODB.DB(FileSto rage(dbname + ".fs"))
conn = db.open()
return db, conn, conn.root()

if __name__ == "__main__":
print "Opening the db ..."
db, conn, root = openzodb("x")
print "Storing something ..."
root["somekey"] = "somedata"
get_transaction ().commit()
print "Closing the db ..."
conn.close(); db.close()

$ echo Makefile
default:
python concurrency.py&
python concurrency.py

$ make
python concurrency.py&
python concurrency.py
Opening the db ...
Opening the db ...
Traceback (most recent call last):
File "concurrency.py ", line 12, in ?
db, conn, root = openzodb("x")
File "concurrency.py ", line 6, in openzodb
db = ZODB.DB(FileSto rage(dbname + ".fs"))
File "/opt/zope/lib/python/ZODB/FileStorage.py" , line 232, in __init__
Storing something ...
self._lock_file = LockFile(file_n ame + '.lock')
File "/opt/zope/lib/python/ZODB/lock_file.py", line 62, in __init__
lock_file(self. _fp)
File "/opt/zope/lib/python/ZODB/lock_file.py", line 42, in lock_file
fcntl.flock(fil e.fileno(), _flags)
IOError: [Errno 11] Resource temporarily unavailable
Closing the db ...

BTW, it is clear from the traceback than internally ZODB uses fcntl
and probably a custom solution based on it would be simpler than
installing the ZODB (unless the OP has some reason to want it).
But I guess you had in mind something different, care to explain?

Michele Simionato

[Diez B. Roggisch]

> Ok, but then I guess you need an external long-living process keeping the

DB open for you. If the cgi script opens and close the database by itself,
the only thing the ZODB buys for you is raising
Oops, yes. I'm not sure if anything else than filestorage will help with
that, but of course my application is a "real" server.

BTW, it is clear from the traceback than internally ZODB uses fcntl
and probably a custom solution based on it would be simpler than
installing the ZODB (unless the OP has some reason to want it).
But I guess you had in mind something different, care to explain?

I've run into the same tracebacks when my application died and got stale so
starting it up again provoked that locking error.

But this exactly is the reason why I personally would refrain from using a
fcntl-approach. What happens if your cgi process freezes? As I never used
mod_python or similar "primitive" http frameworks, I don't know if
malfunctioning scripts are reliably killed after some time - and even if
they _are_, that would most probably be after at least 30seconds, if not
more.

So to paraphrase my suggestion: Instead of trying to reinvent the wheel of
serialized access to a critical resource from different processes, use some
database abstraction layer. As the OP wanted to use pickle, I guess he's
not interested in OR-mapping his stuff to a RDBMS, but wants to deal with
"pure" python objects - thats why I recommended zodb in the first place.

So maybe a small application server is in order - you could for exampl use
corba, xmlrpc or pyro to connect to that service from the cgi, keeping the
cgi-layer extremely simple.

--
Regards,

Diez B. Roggisch

[Mike Meyer]

fu******@gmail. com (Michael Foord) writes:

"darkhorse" <no************ @terra.es> wrote in message news:<2u******* ******@uni-berlin.de>...

Hi all,

While doing a quite big "set of programs" for a university subject I've
found myself in the middle of a problem someone surely has had before, so
I'm looking for some help.

At one point, I call a python cgi that pickle.load's a file, adds or deletes
a registry and dumps the result again in the file.
I'm worried that the cgi could be called simultaneously from two or more
different computers, thus most probably corrupting the files. I don't think
I can use a mutex as it's two different instances of the program and not
different threads, so I was thinking about locking the files between
programs, but I really don't know how to do that. It's not a problem if
there's no portable way of doing this, it's only going to be run on a linux
based computer.
Another solution I would accept is that the second called cgi detects that
other instance is running and displays a message saying to try again later.
Yes, not quite professional, but it'd do the job, after all this is just a
little detail I want to settle for a quite picky professor and not a "real
life" thing.
I think that's all the background you need, if someone can answer with
references on what should I look for or even better example code that would
be simply great.
Many thanks in advance.
DH
A simple solution that doesn't scale well is to create a file when the
access starts. You can check if the file exists and pause until the
other process deletes it - with a timeout in case the file gets keft
there due to an error.

Obviously not an industrial strength solution, but it does work...

To strengthen the solution, write the process id of the script
(available via os.getpid()) to the file. If the file doesn't vanish before
your timeout, you can check to see if the process is still around, and
kill it.

<mike
import time
import os

def sleep(thelockfi le, sleepcycle=0.01 , MAXCOUNT=200):
"""Sleep until the lockfile has been removed or a certain number
of cycles have gone.
Defaults to a max 2 second delay.
"""
counter = 0
while os.path.exists( thelockfile):
time.sleep(slee pcycle)
counter += 1
if counter > MAXCOUNT: f = file(thelockfil e)
pid = int(f.read())
f.close()
p = os.popen("/bin/ps -p %s" % pid)
l = p.read().split( '\n')
p.close()
if len(l) > 2:
os.kill(pid, 9)

def createlock(thel ockfile):
"""Creates a lockfile from the path supplied.""" f = file(thelockfil e, 'w')
f.write(str(os. getpid()))
f.close()
def releaselock(the lockfile):
"""Deletes the lockfile."""
if os.path.isfile( thelockfile):
os.remove(thelo ckfile)

The sleep function waits until the specified file dissapears - or it
times out.

Regards,

Fuzzy

http://www.voidspace.org.uk/atlantib...thonutils.html

--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.