I'm writing a cluster monitor, that collects information from a set of
machines and logs it to a database

In the interests of not hammering the db unnecessarily, I'm
considering the following
1. A series of independent "monitor" threads that collect information
over TCP from the cluster of machines, and write it to a queue
2. A "logger" thread that empties the queue every second or so and
inserts the collected information to the db via a single insert
statement

Reading up on python's built in Queue class, though, it seems oriented
towards "job queues", with a two-step dequeue operation (get() and
task_done()). I'm worried that this would make it too heavyweight for
my application. Is ther documentation somewhere on what exactly
task_done() does, and whether I can disable the tracking of a job once
it's removed from the queue? The python docs for the Queue module were
a bit light.

martin

Martin DeMello wrote:

Reading up on python's built in Queue class, though, it seems oriented
towards "job queues", with a two-step dequeue operation (get() and
task_done()). I'm worried that this would make it too heavyweight for
my application. Is ther documentation somewhere on what exactly
task_done() does, and whether I can disable the tracking of a job once
it's removed from the queue? The python docs for the Queue module were
a bit light.

"task_done" just decrements a counter (incremented by "put"). when the
counter reaches zero, the "join" call is unblocked.

</F>

On Sep 4, 12:41*pm, Fredrik Lundh <fred...@pythonware.comwrote:

"task_done" just decrements a counter (incremented by "put"). *when the
counter reaches zero, the "join" call is unblocked.

Thanks! Is there any standard python idiom to empty a queue into a
list? Or do I just call get() repeatedly and catch the exception when
it's done?

martin

On Sep 4, 2:51*pm, Martin DeMello <martindeme...@gmail.comwrote:

On Sep 4, 12:41*pm, Fredrik Lundh <fred...@pythonware.comwrote:
>

>
Thanks! Is there any standard python idiom to empty a queue into a
list? Or do I just call get() repeatedly and catch the exception when
it's done?
>
martin

Random access isn't supported by the defined interface. You can make
it more convenient, though.

import Queue

class IterQueue( Queue.Queue ):
def __iter__( self ):
return self
def next( self ):
if self.empty():
raise StopIteration
return self.get()

q= IterQueue()
q.put( 'a' )
q.put( 'b' )
q.put( 'c' )

print [ x for x in q ]

/Output:
['a', 'b', 'c']

On Sep 4, 1:04*pm, castironpi <castiro...@gmail.comwrote:

>
Random access isn't supported by the defined interface. *You can make
it more convenient, though.

Thanks. I wasn't looking for random access, just wondering what the
cleanest way to implement items = Queue.get_all() was. Your code
should work nicely for that.

martin

Martin DeMello wrote:

I'm writing a cluster monitor, that collects information from a set of
machines and logs it to a database
>
In the interests of not hammering the db unnecessarily, I'm
considering the following
1. A series of independent "monitor" threads that collect information
over TCP from the cluster of machines, and write it to a queue
2. A "logger" thread that empties the queue every second or so and
inserts the collected information to the db via a single insert
statement

why are you using a queue for this case, btw? why not just use a plain list

L = []
lock = threading.Lock()

and add stuff using append in the monitor threads

with lock:
L.append(item)

and regularily reset the list in the logger thread

with lock:
data = L[:]
L[:] = [] # clear the list
for item in data:
... insert into database ...

(list append and assignments to global variables are atomic in CPython,
so you can eliminate the lock by being a bit clever, but that's probably
better left for a non-premature optimization pass).

</F>

On Sep 4, 1:51*pm, Fredrik Lundh <fred...@pythonware.comwrote:

Martin DeMello wrote:

>

>
why are you using a queue for this case, btw? *why not just use a plainlist
>
* * *L = []
* * *lock = threading.Lock()

Good point - I thought of queue because it was self-locking, but
you're right, I can as well use a simple list and lock it myself.

martin

In article <mailman.504.1220561513.3487.python-list@python.org>,
Fredrik Lundh <fredrik@pythonware.comwrote:

>Martin DeMello wrote:

>
>why are you using a queue for this case, btw? why not just use a plain list
>
L = []
lock = threading.Lock()
>
>and add stuff using append in the monitor threads
>
with lock:
L.append(item)

Because using a queue requires less thinking. I certainly would use a
queue in this case instead of rolling my own.
--
Aahz (aahz@pythoncraft.com) <* http://www.pythoncraft.com/

"Argue for your limitations, and sure enough they're yours." --Richard Bach

Aahz wrote:

>
Because using a queue requires less thinking.

given that the whole reason for this thread was that Queue API didn't
fit the OP:s problem, that's a rather dubious statement.

(btw, I've always thought that Python was all about making it easy to
express the solution to a given problem in code, not to let you write
programs without using your brain. when did that change?)

</F>

Fredrik Lundh <fred...@pythonware.comwrote:

(btw, I've always thought that Python was all about making it easy to
express the solution to a given problem in code, not to let you write
programs without using your brain. *when did that change?)

The day Google App Engine was opened up to developers, I believe.