thread.error: release unlocked lock

So you have this problem, and you decide to use threads to solve it...

System is Python 2.3.3 on Solaris 9.

I'm using the classic Python thread model. Main thread puts stuff into
a Queue.Queue() and worker threads get stuff out of it and do their
thing.
A bounded queue or an unbounded queue? Blocking get/put or
non-blocking get/put? About how many items are in the queue? Since
nobody has reported your symptom before, some detail or other is going
to be important. How do the standard test_thread, test_threading, and
test_queue tests fare on this box? Was this Python configured to use
pthreads or native Solaris threads?
On occaision, I get an exception in the main thread when it tries to
put something into the Queue.

File "/usr/local/bin/poller.py", line 47, in fragnap_it
work_queue.put( (foo, bar, baz))
File "/usr/local/lib/python2.3/Queue.py", line 106, in put
self.fsema.rele ase()
thread.error: release unlocked lock

Seems like a bad thing, and I'm not sure what can be done. Should I
catch the thread.error and retry the put()? Or just give up and
shutdown nicely? (Or quickly finish the twisted version of this code?)

Any idea on the underlying cause of this exception?

[John P.Speno]

I'm using the classic Python thread model. Main thread puts stuff into
a Queue.Queue() and worker threads get stuff out of it and do their
thing.

A bounded queue or an unbounded queue? Blocking get/put or
non-blocking get/put? About how many items are in the queue? Since
Thanks for asking. Here's what I can tell you.

Unbounded queues with non-blocking get/put. At most there are 1500 items
in the queue. There are 20 threads getting from the queue, and one
putting.

I did have bounded queues (max 40 items) and blocking puts but got the same
insane error then too.
nobody has reported your symptom before, some detail or other is going
to be important. How do the standard test_thread, test_threading, and
test_queue tests fare on this box? Was this Python configured to use
pthreads or native Solaris threads?

I'm using the classic Python thread model. Main thread puts stuff
into a Queue.Queue() and worker threads get stuff out of it and do
their thing.

[Tim Peters] A bounded queue or an unbounded queue? Blocking get/put or
non-blocking get/put? About how many items are in the queue?
Since nobody has reported your symptom before, some detail or
other is going to be important.
[John]

Thanks for asking. Here's what I can tell you.

Unbounded queues with non-blocking get/put.
That's odd -- non-blocking put doesn't really make sense with an
unbounded queue, esp. when only one thread does put(). It will work
anyway, it's just peculiar.

So if the queue is unbounded, the put() method is the only code
anywhere that acquires or releases fsema. That simplifies analysis,
but only in that it leads most directly to the conclusion that it's
impossible to get the error you're seeing. The candidates remaining
are things like compiler optimization bugs, buggy platform threads,
buggy C extension modules.

You should really file a bug report, on Python's SourceForge tracker
-- there's already too much detail to keep straight in
email/newsgroup.
At most there are 1500 items in the queue. There are 20 threads
getting from the queue, and one putting.

I did have bounded queues (max 40 items) and blocking puts but got
the same insane error then too.
Since nobody else has reported this problem, a successful conclusion
is going to require analyzing a small-as-possible failing test case.
I'm afraid that falls on you, unless someone else can reproduce your
symptom. Here's a start, modeling one plausible set of guesses for
exactly what all the above means:

"""
import threading
import time
from Queue import Queue, Empty
from random import random

class Worker(threadin g.Thread):
def run(self):
while True:
try:
k = q.get_nowait()
assert k == 1
except Empty:
pass
time.sleep(rand om()/10.0)

q = Queue()
ts = [Worker() for dummy in range(20)]
for t in ts:
t.start()

while True:
if q.qsize() > 1500:
print q.qsize()
time.sleep(3)
q.put_nowait(1)
time.sleep(0.00 1)
"""

I can apparently run that all day under 2.3.4 (or 2.4c1, which has an
entirely different Queue implementation) and not see anything odd.
Does it fail for you? If not, how does it differ from what your app
does? IOW, can you fiddle it so that it does fail?
How do the standard test_thread, test_threading, and
test_queue tests fare on this box? Was this Python configured to
use pthreads or native Solaris threads?

% grep -i thread pyconfig.h | grep -v ^/
#define HAVE_PTHREAD_H 1
#define HAVE_PTHREAD_SI GMASK 1
#define HAVE_THREAD_H 1
#define PTHREAD_SYSTEM_ SCHED_SUPPORTED 1
#define SIZEOF_PTHREAD_ T 4
#define WITH_THREAD 1

I think that means it is using native Solaris threads, assuming my
reading of thread.c is good.
I don't really know about Solaris config (another reason this would be
better in a bug report, where more people who care about bugs <wink>
would pay attention), but I'd *guess* it means you're using pthreads.
All of test_thread, test_threading, and test_queue all passed.

[John]

Unbounded queues with non-blocking get/put.

[Tim]
That's odd -- non-blocking put doesn't really make sense with an
unbounded queue, esp. when only one thread does put(). It will work
anyway, it's just peculiar.