How to lock files (the easiest/best way)?

[Elmo Mäntynen]

Is there something better than using fnctl? It seems a bit intimidating
with a quick look.

[Elmo Mäntynen]

On Sat, 15 Jul 2006 23:28:21 +0200, Sybren Stuvel wrote:

Elmo Mäntynen enlightened us with:

>Is there something better than using fnctl? It seems a bit
intimidating with a quick look.

Locking files is a complex business. What do you want to lock? Why?
Lock it with respect to what? It's easier to lock a file for local
use, compared to when the file also has to be locked from, say, use
via the network.

Sybren

Only locally. I want to be able to read/write to a single file from
multiple possibly parallel processes. Would 'touch lock' (or something
like that) work reliably (this just occured to me)?

[Elmo Mäntynen]

On Sat, 15 Jul 2006 23:52:10 +0200, Sybren Stuvel wrote:

Elmo Mäntynen enlightened us with:

>Only locally. I want to be able to read/write to a single file from
multiple possibly parallel processes. Would 'touch lock' (or
something like that) work reliably (this just occured to me)?

I use a lock directory for that, os.mkdir('/var/lock/somedir').
If you use a file, you need two steps:
1) Check whether the lock-file exists
2) Create the lock-file

This is not atomic. With a directory, creating it will fail if it
already exists. This means you can atomically check for the lock, and
if it doesn't exist already, you've immediately created it too.

Sybren

Thanks. Is that what atomic basically means?

[Gerhard Fiedler]

On 2006-07-15 18:52:10, Sybren Stuvel wrote:

Elmo Mäntynen enlightened us with:

>Only locally. I want to be able to read/write to a single file from
multiple possibly parallel processes. Would 'touch lock' (or
something like that) work reliably (this just occured to me)?

I use a lock directory for that, os.mkdir('/var/lock/somedir').
If you use a file, you need two steps:
1) Check whether the lock-file exists
2) Create the lock-file

cvsnt for example used to use lock files. Now it uses a lock server: a
server app that just sits there and allows different processes to acquire,
check for and release locks on files. More implementation work, probably,
but more efficient.

Gerhard

[Paul Rubin]

Sybren Stuvel <sy*******@YOURthirdtower.com.imaginationwrites:

I use a lock directory for that, os.mkdir('/var/lock/somedir').
If you use a file, you need two steps:
1) Check whether the lock-file exists
2) Create the lock-file
This is not atomic. With a directory, creating it will fail if it
already exists. This means you can atomically check for the lock, and
if it doesn't exist already, you've immediately created it too.

The classic way in Unix was to make a link:

1) link('some_other_file', 'lockfile')

This was atomic and would fail if lockfile already existed.

I'm not sure whether it can still be reliable, in current environments
that can involve remote file systems etc. But I'm not sure if the
mkdir approach works either.

Maybe you want to use something like shm or posh?

[Eric S. Johansson]

Elmo Mäntynen wrote:

Is there something better than using fnctl? It seems a bit intimidating
with a quick look.

try the portlocker wrapper from the active state cookbook. I have a
version which has been slightly updated for more modern pythons. I
really need to make my darcs repository visible so you could get it.

Alternatively, you can try the makedir trick. Creating directories are
atomic operations if it exists, you will get error message. If it
doesn't, you will get a directory and thereby capture the lock. You
delete the directory when you release the lock.

crude but effective. I used it in building a file based queue system.
You know, I really should learn how to build eggs because I have a whole
bunch of little pieces of software that would probably be useful to others.

---eric

[Jim Segrave]

In article <ma***************************************@python. org>,
Eric S. Johansson <es*@harvee.orgwrote:

>Elmo Mäntynen wrote:

>Is there something better than using fnctl? It seems a bit intimidating
with a quick look.

try the portlocker wrapper from the active state cookbook. I have a
version which has been slightly updated for more modern pythons. I
really need to make my darcs repository visible so you could get it.

Alternatively, you can try the makedir trick. Creating directories are
atomic operations if it exists, you will get error message. If it
doesn't, you will get a directory and thereby capture the lock. You
delete the directory when you release the lock.

crude but effective. I used it in building a file based queue system.
You know, I really should learn how to build eggs because I have a whole
bunch of little pieces of software that would probably be useful to others.

Here's a more complete file locking scheme which uses a lockfile with
the O_CREAT and O_EXCL flags to make the creation atomic. If it gets
the lock, it writes its PID in readable form in the file. It also does
two other things:

If you know that no process should lock the file for more than a fixed
period of time, it will retry once/second as long as the lock file is
not older than that fixed period of time. If it is older, it will
report the problem, including the PID of the locking process and exit.

This caters for a process which has set the lock file and then
terminates without removing it (which can happen do to an application
or server crash).

------------------------------------------

import os
import errno
import sys
import time
import stat

# the maximum reasonable time for aprocesstobe
max_wait = 10

lockfile = "/tmp/mylock"

while True:
try:
fd = os.open(lockfile, os.O_EXCL | os.O_RDWR | os.O_CREAT)
# we created the lockfile, so we're the owner
break
except OSError, e:
if e.errno != errno.EEXIST:
# should not occur
raise

try:
# the lock file exists, try to stat it to get its age
# and read it's contents to report the owner PID
f = open(lockfile, "r")
s = os.stat(lockfile)
except OSError, e:
if e.errno != errno.EEXIST:
sys.exit("%s exists but stat() failed: %s" %
(lockfile, e.strerror))
# we didn't create the lockfile, so it did exist, but it's
# gone now. Just try again
continue

# we didn't create the lockfile and it's still there, check
# its age
now = int(time.time())
if now - s[stat.ST_MTIME] max_wait:
pid = f.readline()
sys.exit("%s has been locked for more than " \
"%d seconds (PID %s)" % (lockfile, max_wait,
pid))

# it's not been locked too long, wait a while and retry
f.close()
time.sleep(1)

# if we get here. we have the lockfile. Convert the os.open file
# descriptor into a Python file object and record our PID in it

f = os.fdopen(fd, "w")
f.write("%d\n" % os.getpid())
f.close()


--
Jim Segrave (je*@jes-2.demon.nl)

[Jim Segrave]

In article <12*************@corp.supernews.com>,
Jim Segrave <je*@jes-2.demon.nlwrote:

except OSError, e:
if e.errno != errno.EEXIST:

this should read:
if e.errno != errno.ENOENT:

(it was left with EEXIST from testing this code by forcing an error,
as the code for this failure requires a very tight race condition to test)

sys.exit("%s exists but stat() failed: %s" %
(lockfile, e.strerror))
# we didn't create the lockfile, so it did exist, but it's

--
Jim Segrave (je*@jes-2.demon.nl)

[Donn Cave]

In article <pa****************************@jippii.fi>,
Elmo Mäntynen <el****@jippii.fiwrote:

On Sat, 15 Jul 2006 23:52:10 +0200, Sybren Stuvel wrote:

Elmo Mäntynen enlightened us with:

Only locally. I want to be able to read/write to a single file from
multiple possibly parallel processes. Would 'touch lock' (or
something like that) work reliably (this just occured to me)?

I use a lock directory for that, os.mkdir('/var/lock/somedir').
If you use a file, you need two steps:
1) Check whether the lock-file exists
2) Create the lock-file

This is not atomic. With a directory, creating it will fail if it
already exists. This means you can atomically check for the lock, and
if it doesn't exist already, you've immediately created it too.

Sybren

Thanks. Is that what atomic basically means?

Yes, and also "race condition". That's why Jim Segrave's
example code uses O_EXCL with open(2).

Donn Cave, do**@u.washington.edu

[Fuzzyman]

Sybren Stuvel wrote:

Elmo Mäntynen enlightened us with:

Only locally. I want to be able to read/write to a single file from
multiple possibly parallel processes. Would 'touch lock' (or
something like that) work reliably (this just occured to me)?

I use a lock directory for that, os.mkdir('/var/lock/somedir').
If you use a file, you need two steps:
1) Check whether the lock-file exists
2) Create the lock-file

This is not atomic. With a directory, creating it will fail if it
already exists. This means you can atomically check for the lock, and
if it doesn't exist already, you've immediately created it too.

`Pathutils <http://www.voidspace.org.uk/python/pathutils.html>`_ has a
simple interface to a system like this.

Apparently not all platforms guarantee that attempts to create a
directory will fail if the directory already exists - so pathutils does
*slightly* more, but it's the same idea.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

>
Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa

[pavlovevidence]

Elmo Mäntynen wrote:

Is there something better than using fnctl? It seems a bit intimidating
with a quick look.

Although fcntl is pretty opaque, it's quite easy to use if all you want
is a simple exclusive lock for all your data. The thing to keep in
mind is, you don't have to lock every file you want exclusive access
to. In fact, doing that provides no added security (since it's an
advisory lock and programs are free to ignore it). For simple cases,
locking one file and only accessing your data if you have that lock
works, and is very simple.

What I usually do is touch an empty file (say, "lockfile") in my data
directory. I don't access any files in that directory unless I have a
lock to the lockfile. This is done simply with (untested):

import fcntl

f = open("/path/to/data/directory/lockfile","r")
try:
fcntl.flock(f.fileno(),fcntl.LOCK_EX)
...access data freely here...
finally:
f.close()

Closing the file should release the lock (unless you have a truly
horrible operating system).
Carl Banks

[Carl J. Van Arsdall]

pa************@gmail.com wrote:

>

>ith a quick look.

f = open("/path/to/data/directory/lockfile","r")
try:
fcntl.flock(f.fileno(),fcntl.LOCK_EX)
...access data freely here...
finally:
f.close()

Closing the file should release the lock (unless you have a truly
horrible operating system).

I also find that fcntl has problems with NFS (or at least, *I* had
problems using the python fcntl module and nfs - could be that horrible
operating system, but doing things like that over nfs can be tricky).
-carl

--

Carl J. Van Arsdall
cv*********@mvista.com
Build and Release
MontaVista Software

[Carl Banks]

Carl J. Van Arsdall wrote:

pa************@gmail.com wrote:

f = open("/path/to/data/directory/lockfile","r")
try:
fcntl.flock(f.fileno(),fcntl.LOCK_EX)
...access data freely here...
finally:
f.close()

Closing the file should release the lock (unless you have a truly
horrible operating system).

I also find that fcntl has problems with NFS (or at least, *I* had
problems using the python fcntl module and nfs - could be that horrible
operating system, but doing things like that over nfs can be tricky).

Ah, that's a tough one.

Apparently, you could lock files on NFS if both the client OS and NFS
server are up to the task (i.e., sufficiently recent), but good luck
getting that to fly. And, with NFS, even some of the seemingly
foolproof methods like "lock directories" aren't necessarily going to
work. Cross your fingers and hope you have a solid NFS server and
well-behaved clients.
Carl Banks