Overwriting / reassigning instance / garbage collection?

[Ken Godee]

I'm using the python Queue module, but since I can not
find anyway to clear the Queue, I thought I would try
to reassign ie......

q1 = Queue.Queue()
q1.put('test1')
q1.put('test2')
q1.qsize()
2
q1 = Queue.Queue()
q1.qsize()
0

q1 now has a new address, so I'm
assuming the old q1 instance now has a
reference count of zero and python will garbage collect it?????

[Peter Hansen]

Ken Godee wrote:

I'm using the python Queue module, but since I can not
find anyway to clear the Queue, I thought I would try
to reassign ie......

q1 = Queue.Queue()
q1.put('test1')
q1.put('test2')
q1.qsize()
2
q1 = Queue.Queue()
q1.qsize()
0

q1 now has a new address, so I'm
assuming the old q1 instance now has a
reference count of zero and python will garbage collect it?????

Given that the only decent reason to use a Queue is to communicate
between separate threads, or at least between different parts of
the code, if you rebind the name "q1" to a new instance of Queue,
but you don't do anything with any other names that are bound to
it, it won't work right. And if you have no other names bound to
it, why do you have a Queue?

(Or are you just referencing the same name from both threads?
In that case, note that you might lose data when one thread is
putting stuff into the old Queue, but the other one is starting
to use the new Queue. Of course, since your purpose here is to
"clear" the Queue, I suppose that would actually work okay... hmm.)

-Peter

[Werner Schiendl]

Hi,

Ken Godee wrote:

q1 now has a new address, so I'm
assuming the old q1 instance now has a
reference count of zero and python will garbage collect it?????

of course, the old one will eventually be garbage collected.

The question is, what do you actually want to do?
hth

Werner

[Ken Godee]

>>q1 = Queue.Queue()

q1.put('test1')
q1.put('test2')
q1.qsize()
2
q1 = Queue.Queue()
q1.qsize()
0

q1 now has a new address, so I'm
assuming the old q1 instance now has a
reference count of zero and python will garbage collect it?????

Given that the only decent reason to use a Queue is to communicate
between separate threads, or at least between different parts of
the code, if you rebind the name "q1" to a new instance of Queue,
but you don't do anything with any other names that are bound to
it, it won't work right. And if you have no other names bound to
it, why do you have a Queue?

I know, I'm reaching.... I'll try a short explaination...

The code I'm working on has a registered call back in it
that is communicating with a module that is threaded and continuosly
sending event messages back to my function.

def myfunc(event):
print event.headers

manager.Event.register("*", myfunc)

Which of coarse for instance if I put a print statement in my call back
func. you would see the constaint flow of event messages on standard out.
Now later in my code I only want to try to catch a few of these event
messages that have to do with the commands I'm sending, so in an effort
to buffer(kinda), "Queue"

q1 = Queue.Queue()

def myfunc(event):
q1.put(event.headers)

manager.Event.register("*", myfunc)

paused, waiting to send command
commands sent.......

while q1.qsize > 0:
x = q1.get()
if x['Event'] == 'Call':
etc...
if x['Event'] == 'Hangup':
break etc....
.....
At this point the program is going to loop
and be waiting to send more commands.

I can filter out what I want, but since
this program remains active (still filling the queue)
I see the queue filling with alot of unneeded info and to as how big,
who knows.

So, what I was thinking was to set a default size for
the queue q1 = Queue.Queue(200) and when ready to
use, clear it.
Since there's no "clear" method, and the queue is only being
referenced by the single instance, I was toying with
just recreating, I don't know, but it seems like the first
reference is then wiped out and since the object would then
have no reference count, I would think python would gc the first one?

[Ken Godee]

Werner Schiendl wrote:

q1 now has a new address, so I'm
assuming the old q1 instance now has a
reference count of zero and python will garbage collect it?????

of course, the old one will eventually be garbage collected.

The question is, what do you actually want to do?

Actually, all I want to do is "clear" the queue.

I saw some other queue modules and they include
a method to clear the queue.
ie...

q1.clear()

but as far as I can tell the standard included Queue module
that comes with python lacks this method.

[David C. Fox]

Ken Godee wrote:

Werner Schiendl wrote:

q1 now has a new address, so I'm
assuming the old q1 instance now has a
reference count of zero and python will garbage collect it?????

of course, the old one will eventually be garbage collected.

The question is, what do you actually want to do?

Actually, all I want to do is "clear" the queue.

I saw some other queue modules and they include
a method to clear the queue.
ie...

q1.clear()

but as far as I can tell the standard included Queue module
that comes with python lacks this method.

A queue is a data structure with the restriction that elements are added
to it on one end and removed on the other (also known as first-in
first-out, because the first element added will be the first to be removed).

The Queue in the Queue module is not just a queue data structure - it is
designed to be safely shared between multiple program threads, with only
only one thread allowed to access it at one time. The idea is to ensure
that requests from 1 or more thread are handled in order, and none of
them are lost. Therefore, it usually doesn't make sense to clear the
queue (deliberately losing all elements)

Those other queue modules you saw were probably implementing only the
data structure, and not the multi-threaded aspects of Queue.Queue.

If you understand this, and *really* do want to discard all the elements
from a Queue.Queue, you can just do:

try:
while 1:
q1.get(0)
except Queue.Empty:
pass

However, you should be aware that any other parts of the program which
have references to q1 can always add more elements to q1, so it won't
necessarily stay empty. Also, if you do have a multi-threaded
application, and other threads are adding elements to the Queue faster
than the code above can remove them, the code above might never finish.

David

[Ken Godee]

>

If you understand this, and *really* do want to discard all the elements
from a Queue.Queue, you can just do:

try:
while 1:
q1.get(0)
except Queue.Empty:
pass

However, you should be aware that any other parts of the program which
have references to q1 can always add more elements to q1, so it won't
necessarily stay empty. Also, if you do have a multi-threaded
application, and other threads are adding elements to the Queue faster
than the code above can remove them, the code above might never finish.

David

yes, I understand this.
and that's why I'm wondering if de-referencing the instance
as below would work and that the old address space would
get garabage collected. This would then happen in a fraction
of the time that it might take the while statement to empty
the queue, if ever. I'm not building a heart monitor, just trying
to find a way to buffer the messages and then use them on a FIFO basis.
The Queue module is perfect for what I'm trying to do and even
takes care of the locking issues.
Well, heck, now that I'm thinking about it again, I'm not sure
how well the function that's puttting the messages into the queue
is going to like it if I try the recreate instance method, argggg.....
Might just have to use the while method anyway. Maybe pause sending
messages in some form while the queue empties.

q1 = Queue.Queue()
q1.put('test1')
q1.put('test2')
q1.qsize()
2
q1 = Queue.Queue()
q1.qsize()
0

[Peter Hansen]

Ken Godee wrote:

try:
while 1:
q1.get(0)
except Queue.Empty:
pass [snip]

yes, I understand this.
and that's why I'm wondering if de-referencing the instance
as below would work and that the old address space would
get garabage collected. This would then happen in a fraction
of the time that it might take the while statement to empty
the queue, if ever.

Ah, there's your problem. You're trying to optimize before you have
a working solution *or* actual measurements showing a performance
problem.

My strong suggestion is write a little clearQueue() routine as
above, and call that, and only worry about the whole thing if you
ever notice a serious performance issue. Most likely you won't.

-Peter

[Werner Schiendl]

Hi,

Ken Godee wrote:

Well, heck, now that I'm thinking about it again, I'm not sure
how well the function that's puttting the messages into the queue
is going to like it if I try the recreate instance method, argggg.....
Might just have to use the while method anyway. Maybe pause sending
messages in some form while the queue empties.

The only thing you need to pay attention too is that both ends use the
same variable to access the queue.

If you were to pass the queue as a parameter, your sender would continue
to write to the old one after you replaced it.

This is probably a case where a global variable is appropriate.
hth

Werner

[Ken Godee]

>

try:
while 1:
q1.get(0)
except Queue.Empty:
pass [snip] yes, I understand this.
and that's why I'm wondering if de-referencing the instance
as below would work and that the old address space would
get garabage collected. This would then happen in a fraction
of the time that it might take the while statement to empty
the queue, if ever.

Ah, there's your problem. You're trying to optimize before you have
a working solution *or* actual measurements showing a performance
problem.

You're right I don't have the actual measurements yet, but
when it goes live the queue will be receiving possibly 100's+
of events per minute, while my program may just sit there
waiting to send a command anywhere from minutes to hours.
Meanwhile the queue if not limited in size or zero'ed out just prior
to sending a command could grow to thousands+ of items.
Once the command is sent, I'm only interested in filtering out
the responses from the command sent. Multiple responses from the command
sent return almost instantly, mixed in with other responses from the
system.
Unfortunately the back end of this system doesn't allow one to only
receive responses from the user who sent it, but receives all responses
from all events on the system.

I experimented last night, with just limiting the size of the queue
and then just before issuing my command reassigning the instance of
the queue. ie....

q1 = Queue.Queue(200)

blah..
program running, q1 filled
q1.qsize()
200
(top of loop)
wait for command input
....getting ready to run my command
q1 = Queue.Queue(200)
q1.qsize()
0
run command
process q1
loop back

Whichs brings me back to original post....
I was worried about the inner workings of
how python would garbage collect the previous instance
of q1, since the first q1 vs 2nd q1 had dif. reference
addresses.
After monitoring the memory usage with the above
code it would appear python handles it quite well
and reuses the memory it already allocated.
Now, I've never had to do this type of
thing with a class instance before, so I
was trying to find out any problems.