473,320 Members | 1,865 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Not fully understanding the role of Queue.task_done()

I'm writing a cluster monitor, that collects information from a set of
machines and logs it to a database

In the interests of not hammering the db unnecessarily, I'm
considering the following
1. A series of independent "monitor" threads that collect information
over TCP from the cluster of machines, and write it to a queue
2. A "logger" thread that empties the queue every second or so and
inserts the collected information to the db via a single insert
statement

Reading up on python's built in Queue class, though, it seems oriented
towards "job queues", with a two-step dequeue operation (get() and
task_done()). I'm worried that this would make it too heavyweight for
my application. Is ther documentation somewhere on what exactly
task_done() does, and whether I can disable the tracking of a job once
it's removed from the queue? The python docs for the Queue module were
a bit light.

martin
Sep 4 '08 #1
9 12718
Martin DeMello wrote:
Reading up on python's built in Queue class, though, it seems oriented
towards "job queues", with a two-step dequeue operation (get() and
task_done()). I'm worried that this would make it too heavyweight for
my application. Is ther documentation somewhere on what exactly
task_done() does, and whether I can disable the tracking of a job once
it's removed from the queue? The python docs for the Queue module were
a bit light.
"task_done" just decrements a counter (incremented by "put"). when the
counter reaches zero, the "join" call is unblocked.

</F>

Sep 4 '08 #2
On Sep 4, 12:41*pm, Fredrik Lundh <fred...@pythonware.comwrote:
"task_done" just decrements a counter (incremented by "put"). *when the
counter reaches zero, the "join" call is unblocked.
Thanks! Is there any standard python idiom to empty a queue into a
list? Or do I just call get() repeatedly and catch the exception when
it's done?

martin
Sep 4 '08 #3
On Sep 4, 2:51*pm, Martin DeMello <martindeme...@gmail.comwrote:
On Sep 4, 12:41*pm, Fredrik Lundh <fred...@pythonware.comwrote:
"task_done" just decrements a counter (incremented by "put"). *when the
counter reaches zero, the "join" call is unblocked.

Thanks! Is there any standard python idiom to empty a queue into a
list? Or do I just call get() repeatedly and catch the exception when
it's done?

martin
Random access isn't supported by the defined interface. You can make
it more convenient, though.

import Queue

class IterQueue( Queue.Queue ):
def __iter__( self ):
return self
def next( self ):
if self.empty():
raise StopIteration
return self.get()

q= IterQueue()
q.put( 'a' )
q.put( 'b' )
q.put( 'c' )

print [ x for x in q ]

/Output:
['a', 'b', 'c']
Sep 4 '08 #4
On Sep 4, 1:04*pm, castironpi <castiro...@gmail.comwrote:
>
Random access isn't supported by the defined interface. *You can make
it more convenient, though.
Thanks. I wasn't looking for random access, just wondering what the
cleanest way to implement items = Queue.get_all() was. Your code
should work nicely for that.

martin
Sep 4 '08 #5
Martin DeMello wrote:
I'm writing a cluster monitor, that collects information from a set of
machines and logs it to a database

In the interests of not hammering the db unnecessarily, I'm
considering the following
1. A series of independent "monitor" threads that collect information
over TCP from the cluster of machines, and write it to a queue
2. A "logger" thread that empties the queue every second or so and
inserts the collected information to the db via a single insert
statement
why are you using a queue for this case, btw? why not just use a plain list

L = []
lock = threading.Lock()

and add stuff using append in the monitor threads

with lock:
L.append(item)

and regularily reset the list in the logger thread

with lock:
data = L[:]
L[:] = [] # clear the list
for item in data:
... insert into database ...

(list append and assignments to global variables are atomic in CPython,
so you can eliminate the lock by being a bit clever, but that's probably
better left for a non-premature optimization pass).

</F>

Sep 4 '08 #6
On Sep 4, 1:51*pm, Fredrik Lundh <fred...@pythonware.comwrote:
Martin DeMello wrote:
I'm writing a cluster monitor, that collects information from a set of
machines and logs it to a database
In the interests of not hammering the db unnecessarily, I'm
considering the following
1. A series of independent "monitor" threads that collect information
over TCP from the cluster of machines, and write it to a queue
2. A "logger" thread that empties the queue every second or so and
inserts the collected information to the db via a single insert
statement

why are you using a queue for this case, btw? *why not just use a plainlist

* * *L = []
* * *lock = threading.Lock()
Good point - I thought of queue because it was self-locking, but
you're right, I can as well use a simple list and lock it myself.

martin
Sep 4 '08 #7
In article <ma*************************************@python.or g>,
Fredrik Lundh <fr*****@pythonware.comwrote:
>Martin DeMello wrote:
>>
In the interests of not hammering the db unnecessarily, I'm
considering the following
1. A series of independent "monitor" threads that collect information
over TCP from the cluster of machines, and write it to a queue
2. A "logger" thread that empties the queue every second or so and
inserts the collected information to the db via a single insert
statement

why are you using a queue for this case, btw? why not just use a plain list

L = []
lock = threading.Lock()

and add stuff using append in the monitor threads

with lock:
L.append(item)
Because using a queue requires less thinking. I certainly would use a
queue in this case instead of rolling my own.
--
Aahz (aa**@pythoncraft.com) <* http://www.pythoncraft.com/

"Argue for your limitations, and sure enough they're yours." --Richard Bach
Sep 7 '08 #8
Aahz wrote:
>why are you using a queue for this case, btw? why not just use a plain list

L = []
lock = threading.Lock()

and add stuff using append in the monitor threads

with lock:
L.append(item)

Because using a queue requires less thinking.
given that the whole reason for this thread was that Queue API didn't
fit the OP:s problem, that's a rather dubious statement.

(btw, I've always thought that Python was all about making it easy to
express the solution to a given problem in code, not to let you write
programs without using your brain. when did that change?)

</F>

Sep 7 '08 #9
Fredrik Lundh <fred...@pythonware.comwrote:
(btw, I've always thought that Python was all about making it easy to
express the solution to a given problem in code, not to let you write
programs without using your brain. *when did that change?)
The day Google App Engine was opened up to developers, I believe.
Sep 8 '08 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Ralf | last post by:
Hi ! When I insert a new record to a table "table_A" I want to Grant a Role "Role_X" to the User "User_Y". So I made a Trigger who should do this work, but it doesn't: When I write: "GRANT...
3
by: craig | last post by:
I am working on my first .NET development project that involves custom role-based security per the project requirements. This lead to a general design issue this week that really caused us some...
3
by: Kceiw | last post by:
Dear all, When I use #include "queue.h", I can't link it. The error message follows: Linking... G:\Projects\Datastructure\Queue\Debug\main.o(.text+0x136): In function `main':...
1
by: Erland | last post by:
Hi all, As per my understanding in order to load an assembly using Assembly.Load() you have to provide fully qualified name of the assembly you are trying to load e.g. Assembly...
4
by: Russell Warren | last post by:
I'm guessing no, since it skips down through any Lock semantics, but I'm wondering what the best way to clear a Queue is then. Esentially I want to do a "get all" and ignore what pops out, but I...
4
by: cybertoast | last post by:
i seem to have some misunderstanding about how roles work in sql server 2005. i see that i can add a role to a database (dbname->->properties->permissions->. THis allows me to add either users or...
13
by: Alexandru Mosoi | last post by:
how is Queue intended to be used? I found the following code in python manual, but I don't understand how to stop consumers after all items have been produced. I tried different approaches but all...
3
by: redbaron | last post by:
I run into problem with queue from multiprocessing. Even if I queue.qsize() != 0 queue.get() still blocks and queue.get_nowait() raises Emtpy error. I'm unable to cut my big part to small test...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.