Locking around

[Nikolaus Rath]

Hello,

I need to synchronize the access to a couple of hundred-thousand
files[1]. It seems to me that creating one lock object for each of the
files is a waste of resources, but I cannot use a global lock for all
of them either (since the locked operations go over the network, this
would make the whole application essentially single-threaded even
though most operations act on different files).

My idea is therefore to create and destroy per-file locks "on-demand"
and to protect the creation and destruction by a global lock
(self.global_lo ck). For that, I add a "usage counter"
(wlock.user_cou nt) to each lock, and destroy the lock when it reaches
zero. The number of currently active lock objects is stored in a dict:

def lock_s3key(s3ke y):

self.global_loc k.acquire()
try:

# If there is a lock object, use it
if self.key_lock.h as_key(s3key):
wlock = self.key_lock[s3key]
wlock.user_coun t += 1
lock = wlock.lock

# otherwise create a new lock object
else:
wlock = WrappedLock()
wlock.lock = threading.Lock( )
wlock.user_coun t = 1
self.key_lock[s3key] = wlock

finally:
self.global_loc k.release()

# Lock the key itself
lock.acquire()
and similarly

def unlock_s3key(s3 key):

# Lock dictionary of lock objects
self.global_loc k.acquire()
try:

# Get lock object
wlock = self.key_lock[s3key]

# Unlock key
wlock.lock.rele ase()

# We don't use the lock object any longer
wlock.user_coun t -= 1

# If no other thread uses the lock, dispose it
if wlock.user_coun t == 0:
del self.key_lock[s3key]
assert wlock.user_coun t >= 0

finally:
self.global_loc k.release()
WrappedLock is just an empty class that allows me to add the
additional user_count attribute.
My questions:

- Does that look like a proper solution, or does anyone have a better
one?

- Did I overlook any deadlock possibilities?
Best,
Nikolaus

[1] Actually, it's not really files (because in that case I could use
fcntl) but blobs stored on Amazon S3.
--
»It is not worth an intelligent man's time to be in the majority.
By definition, there are already enough people to do that.«
-J.H. Hardy

PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C

[Enrico]

"Brian Quinlan" <br***@sweetapp .comha scritto nel messaggio
news:ma******** *************** **************@ python.org...

Hey,

I'm trying to figure out how I can validate an XML file using a DTD that
isn't specified in the XML file.

I did it once using lxml. You can read from:
http://codespeak.net/lxml/validation.html

With this package is quite simple (code not tested):

from lxml import etree
dtd = etree.DTD('mydt d.dtd')
f = file('mydoc.xml ')
xml = etree.XML(f.rea d())
dtd.validate(xm l)

Enrico

[linuxnow]

On Aug 6, 8:33*pm, Nikolaus Rath <Nikol...@rath. orgwrote:

Tobiah <[e-mail] gwrites:

On Mon, 04 Aug 2008 15:30:51 +0200, Nikolaus Rath wrote:

Do you think you could use an SQL database on the network to
handle the locking?

Yeah, I could. It wouldn't even have to be over the network (I'm
synchronizing access from within the same program). But I think that
is even more resource-wasteful than my original idea.

You could use Amazon SQS to queue the requests, as you are talking
about files kept in S3 anyway, that way it would be fully distributed,

--
Pau