473,326 Members | 2,048 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

queue deadlock possibility

Hi -

i was just going through this thread: http://mail.python.org/
pipermail/python-list/2006-April/336948.html , where it is suggested
that the Lock instance used by Queue.Queue should be publically
configurable. I have identified another situation where a Queue can
be deadlocked, one which is also alleviated by configuring the type
of Lock used by the Queue (or just changing it to an RLock).

The scenario arises when the Queue is operated upon within the
__del__ method of an object; since __del__ can be called at somewhat
unpredictable times, I have observed that it is in fact possible, in
extremely rare circumstances, for put() to be called within a get()
or possibly vice versa; since both methods lock on the same
underlying mutex object which is an instance of threading.Lock, a
deadlock occurs.

The issue can be fixed by substituting a threading.RLock for the
threading.Lock object that Queue instantiates by default.

The scenario this has arisen within is a database connection pool,
which puts connections in a Queue, returns them via get() within a
wrapper object, and the wrapper object automatically returns the
connection to the Queue via put() within its __del__ method (an
explicit close() method is available as well). While I cant
reproduce it locally, one of my users experiences it regularly. I
had him install the "threadframe" module to trace it out, and it
reveals that all threads are hung within Queue on the acquiring of
the "not_empty" and "not_full" Conditionals, and the offending stack
trace within it looks like this:

File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 84, in
connect
File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 130, in
__init__
File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 102, in
get
File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 226, in
do_get
File "/usr/lib/python2.4/Queue.py", line 116, in get
raise Empty
File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 157, in
__del__
File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 163, in
_close
File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 99, in
return_conn
File "build/bdist.linux-i686/egg/sqlalchemy/pool.py", line 216, in
do_return_conn
File "/usr/lib/python2.4/Queue.py", line 71, in put
self.not_full.acquire()

this is a simplified version of the logic, the actual version is the
pool.py module in the SQLAlchemy package:

import Queue

pool = Queue.Queue(maxsize=10)

class ConnectionWrapper(object):
def __init__(self, connection):
self.connection = connection
def __del__(self):
pool.put_nowait(self)

# fill up the pool with 10 connections
for x in range(0,10):
pool.put_nowait(database.connect())

def connect():
return ConnectionWrapper(pool.get())

At the moment I am modifying the Queue's mutex to be a
threading.RLock to fix the problem; what does the community think of
either making the Queue's Lock instance public or changing it to an
RLock ?

- mike


Jun 26 '06 #1
0 1419

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: Duncan Grisby | last post by:
Hi, Does anyone know of a deadlock detector for Python? I don't think it would be too hard to hook into the threading module and instrument mutexes so they can be tested for deadlocks. I've...
9
by: phil | last post by:
And sorry I got ticked, frustrating week >And I could help more, being fairly experienced with >threading issues and race conditions and such, but >as I tried to indicate in the first place,...
8
by: Anita | last post by:
Hi All, Can multiple updates on one table using single query generate deadlock ? For example, at the same time, there are 2 users run 2 queries as follows : User1 runs : update tab1 set...
3
by: Nigel Robbins | last post by:
Hi There, I'm getting a deadlock when I have two clients running the following statement. DELETE FROM intermediate.file_os_details WHERE file_uid = ? AND obj_uid There is a compound index on...
15
by: Zeng | last post by:
Hi, The bigger my C# web-application gets, the more places I need to put in the tedious retrying block of code to make sure operations that can run into database deadlocks are re-run (retried)...
13
by: Jonathan Amsterdam | last post by:
I think there's a slight design flaw in the Queue class that makes it hard to avoid nested monitor deadlock. The problem is that the mutex used by the Queue is not easy to change. You can then...
4
by: Russell Warren | last post by:
I'm guessing no, since it skips down through any Lock semantics, but I'm wondering what the best way to clear a Queue is then. Esentially I want to do a "get all" and ignore what pops out, but I...
12
by: Paul Rubin | last post by:
I'd like to suggest adding a new operation Queue.finish() This puts a special sentinel object on the queue. The sentinel travels through the queue like any other object, however, when...
13
by: Alexandru Mosoi | last post by:
how is Queue intended to be used? I found the following code in python manual, but I don't understand how to stop consumers after all items have been produced. I tried different approaches but all...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.