How to lock files (the easiest/best way)?

[Elmo Mäntynen]

Is there something better than using fnctl? It seems a bit intimidating
with a quick look.

[Elmo Mäntynen]

On Sat, 15 Jul 2006 23:28:21 +0200, Sybren Stuvel wrote:

Elmo Mäntynen enlightened us with:

>Is there something better than using fnctl? It seems a bit
intimidating with a quick look.

Locking files is a complex business. What do you want to lock? Why?
Lock it with respect to what? It's easier to lock a file for local
use, compared to when the file also has to be locked from, say, use
via the network.

Sybren

Only locally. I want to be able to read/write to a single file from
multiple possibly parallel processes. Would 'touch lock' (or something
like that) work reliably (this just occured to me)?

[Elmo Mäntynen]

On Sat, 15 Jul 2006 23:52:10 +0200, Sybren Stuvel wrote:

Elmo Mäntynen enlightened us with:

>Only locally. I want to be able to read/write to a single file from
multiple possibly parallel processes. Would 'touch lock' (or
something like that) work reliably (this just occured to me)?

I use a lock directory for that, os.mkdir('/var/lock/somedir').
If you use a file, you need two steps:
1) Check whether the lock-file exists
2) Create the lock-file

This is not atomic. With a directory, creating it will fail if it
already exists. This means you can atomically check for the lock, and
if it doesn't exist already, you've immediately created it too.

Sybren

Thanks. Is that what atomic basically means?

[Gerhard Fiedler]

On 2006-07-15 18:52:10, Sybren Stuvel wrote:

Elmo Mäntynen enlightened us with:

>Only locally. I want to be able to read/write to a single file from
multiple possibly parallel processes. Would 'touch lock' (or
something like that) work reliably (this just occured to me)?

I use a lock directory for that, os.mkdir('/var/lock/somedir').
If you use a file, you need two steps:
1) Check whether the lock-file exists
2) Create the lock-file

cvsnt for example used to use lock files. Now it uses a lock server: a
server app that just sits there and allows different processes to acquire,
check for and release locks on files. More implementation work, probably,
but more efficient.

Gerhard

[Paul Rubin]

Sybren Stuvel <sy*******@YOUR thirdtower.com. imaginationwrit es:

I use a lock directory for that, os.mkdir('/var/lock/somedir').
If you use a file, you need two steps:
1) Check whether the lock-file exists
2) Create the lock-file
This is not atomic. With a directory, creating it will fail if it
already exists. This means you can atomically check for the lock, and
if it doesn't exist already, you've immediately created it too.

The classic way in Unix was to make a link:

1) link('some_othe r_file', 'lockfile')

This was atomic and would fail if lockfile already existed.

I'm not sure whether it can still be reliable, in current environments
that can involve remote file systems etc. But I'm not sure if the
mkdir approach works either.

Maybe you want to use something like shm or posh?

[Eric S. Johansson]

Elmo Mäntynen wrote:

Is there something better than using fnctl? It seems a bit intimidating
with a quick look.

try the portlocker wrapper from the active state cookbook. I have a
version which has been slightly updated for more modern pythons. I
really need to make my darcs repository visible so you could get it.

Alternatively, you can try the makedir trick. Creating directories are
atomic operations if it exists, you will get error message. If it
doesn't, you will get a directory and thereby capture the lock. You
delete the directory when you release the lock.

crude but effective. I used it in building a file based queue system.
You know, I really should learn how to build eggs because I have a whole
bunch of little pieces of software that would probably be useful to others.

---eric

[Jim Segrave]

In article <ma************ *************** ************@py thon.org>,
Eric S. Johansson <es*@harvee.org wrote:

>Elmo Mäntynen wrote:

>Is there something better than using fnctl? It seems a bit intimidating
with a quick look.

try the portlocker wrapper from the active state cookbook. I have a
version which has been slightly updated for more modern pythons. I
really need to make my darcs repository visible so you could get it.

Alternativel y, you can try the makedir trick. Creating directories are
atomic operations if it exists, you will get error message. If it
doesn't, you will get a directory and thereby capture the lock. You
delete the directory when you release the lock.

crude but effective. I used it in building a file based queue system.
You know, I really should learn how to build eggs because I have a whole
bunch of little pieces of software that would probably be useful to others.

Here's a more complete file locking scheme which uses a lockfile with
the O_CREAT and O_EXCL flags to make the creation atomic. If it gets
the lock, it writes its PID in readable form in the file. It also does
two other things:

If you know that no process should lock the file for more than a fixed
period of time, it will retry once/second as long as the lock file is
not older than that fixed period of time. If it is older, it will
report the problem, including the PID of the locking process and exit.

This caters for a process which has set the lock file and then
terminates without removing it (which can happen do to an application
or server crash).

------------------------------------------

import os
import errno
import sys
import time
import stat

# the maximum reasonable time for aprocesstobe
max_wait = 10

lockfile = "/tmp/mylock"

while True:
try:
fd = os.open(lockfil e, os.O_EXCL | os.O_RDWR | os.O_CREAT)
# we created the lockfile, so we're the owner
break
except OSError, e:
if e.errno != errno.EEXIST:
# should not occur
raise

try:
# the lock file exists, try to stat it to get its age
# and read it's contents to report the owner PID
f = open(lockfile, "r")
s = os.stat(lockfil e)
except OSError, e:
if e.errno != errno.EEXIST:
sys.exit("%s exists but stat() failed: %s" %
(lockfile, e.strerror))
# we didn't create the lockfile, so it did exist, but it's
# gone now. Just try again
continue

# we didn't create the lockfile and it's still there, check
# its age
now = int(time.time() )
if now - s[stat.ST_MTIME] max_wait:
pid = f.readline()
sys.exit("%s has been locked for more than " \
"%d seconds (PID %s)" % (lockfile, max_wait,
pid))

# it's not been locked too long, wait a while and retry
f.close()
time.sleep(1)

# if we get here. we have the lockfile. Convert the os.open file
# descriptor into a Python file object and record our PID in it

f = os.fdopen(fd, "w")
f.write("%d\n" % os.getpid())
f.close()


--
Jim Segrave (je*@jes-2.demon.nl)

[Jim Segrave]

In article <12************ *@corp.supernew s.com>,
Jim Segrave <je*@jes-2.demon.nlwrote :

except OSError, e:
if e.errno != errno.EEXIST:

this should read:
if e.errno != errno.ENOENT:

(it was left with EEXIST from testing this code by forcing an error,
as the code for this failure requires a very tight race condition to test)

sys.exit("%s exists but stat() failed: %s" %
(lockfile, e.strerror))
# we didn't create the lockfile, so it did exist, but it's

--
Jim Segrave (je*@jes-2.demon.nl)

[Donn Cave]

In article <pa************ *************** *@jippii.fi>,
Elmo Mäntynen <el****@jippii. fiwrote:

On Sat, 15 Jul 2006 23:52:10 +0200, Sybren Stuvel wrote:

Elmo Mäntynen enlightened us with:

Only locally. I want to be able to read/write to a single file from
multiple possibly parallel processes. Would 'touch lock' (or
something like that) work reliably (this just occured to me)?

I use a lock directory for that, os.mkdir('/var/lock/somedir').
If you use a file, you need two steps:
1) Check whether the lock-file exists
2) Create the lock-file

This is not atomic. With a directory, creating it will fail if it
already exists. This means you can atomically check for the lock, and
if it doesn't exist already, you've immediately created it too.

Sybren

Thanks. Is that what atomic basically means?

Yes, and also "race condition". That's why Jim Segrave's
example code uses O_EXCL with open(2).

Donn Cave, do**@u.washingt on.edu

[Fuzzyman]

Sybren Stuvel wrote:

Elmo Mäntynen enlightened us with:

Only locally. I want to be able to read/write to a single file from
multiple possibly parallel processes. Would 'touch lock' (or
something like that) work reliably (this just occured to me)?

I use a lock directory for that, os.mkdir('/var/lock/somedir').
If you use a file, you need two steps:
1) Check whether the lock-file exists
2) Create the lock-file

This is not atomic. With a directory, creating it will fail if it
already exists. This means you can atomically check for the lock, and
if it doesn't exist already, you've immediately created it too.

`Pathutils <http://www.voidspace.o rg.uk/python/pathutils.html> `_ has a
simple interface to a system like this.

Apparently not all platforms guarantee that attempts to create a
directory will fail if the directory already exists - so pathutils does
*slightly* more, but it's the same idea.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

>
Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa