Re: Producer-consumer threading problem

[Jean-Paul Calderone]

On Tue, 10 Jun 2008 22:46:37 -0700 (PDT), George Sakkis <ge***********@gmail.comwrote:

>On Jun 10, 11:47 pm, Larry Bates <[e-mail]`wrote:

>>
I had a little trouble understanding what exact problem it is that you are
trying to solve but I'm pretty sure that you can do it with one of two methods:

Ok, let me try again with a different example: I want to do what can
be easily done with 2.5 Queues using Queue.task_done()/Queue.join()
(see example at http://docs.python.org/lib/QueueObjects.html), but
instead of having to first put all items and then wait until all are
done, get each item as soon as it is done.

>1) Write the producer as a generator using yield method that yields a result
every time it is called (something like os.walk does). I guess you could yield
None if there wasn't anything to consume to prevent blocking.

Actually the way items are generated is not part of the problem; it
can be abstracted away as an arbitrary iterable input. As with all
iterables, "there are no more items" is communicated simply by a
StopIteration.

>2) Usw somethink like Twisted insted that uses callbacks instead to handle
multiple asynchronous calls to produce. You could have callbacks that don't do
anything if there is nothing to consume (sort of null objects I guess).

Twisted is interesting and very powerful but requires a different way
of thinking about the problem and designing a solution. More to the
point, callbacks often provide a less flexible and simple API than an
iterator that yields results (consumed items). For example, say that
you want to store the results to a dictionary. Using callbacks, you
would have to explicitly synchronize each access to the dictionary
since they may fire independently.

This isn't true. Access is synchronized automatically by virtue of the
fact that there is no pre-emptive multithreading in operation. This is
one of the many advantages of avoiding threads. :) There may be valid
arguments against callbacks, but this isn't one of them.

Jean-Paul

[George Sakkis]

On Jun 11, 10:13*am, Jean-Paul Calderone <[e-mail]:

On Tue, 10 Jun 2008 22:46:37 -0700 (PDT), George Sakkis <[e-mail]:

On Jun 10, 11:47 pm, Larry Bates <[e-mail]`wrote:

I had a little trouble understanding what exact problem it is that you are
trying to solve but I'm pretty sure that you can do it with one of two methods:

Ok, let me try again with a different example: I want to do what can
be easily done with 2.5 Queues using Queue.task_done()/Queue.join()
(see example athttp://docs.python.org/lib/QueueObjects.html), but
instead of *having to first put all items and then wait until all are
done, get each item as soon as it is done.

1) Write the producer as a generator using yield method that yields a result
every time it is called (something like os.walk does). *I guess you could yield
None if there wasn't anything to consume to prevent blocking.

Actually the way items are generated is not part of the problem; it
can be abstracted away as an arbitrary iterable input. As with all
iterables, "there are no more items" is communicated simply by a
StopIteration.

2) Usw somethink like Twisted insted that uses callbacks instead to handle
multiple asynchronous calls to produce. *You could have callbacks that don't do
anything if there is nothing to consume (sort of null objects I guess).

Twisted is interesting and very powerful but requires a different way
of thinking about the problem and designing a solution. More to the
point, callbacks often provide a less flexible and simple API than an
iterator that yields results (consumed items). For example, say that
you want to store the results to a dictionary. Using callbacks, you
would have to explicitly synchronize each access to the dictionary
since they may fire independently.

This isn't true. *Access is synchronized automatically by virtue of the
fact that there is no pre-emptive multithreading in operation. *This is
one of the many advantages of avoiding threads. :) *There may be valid
arguments against callbacks, but this isn't one of them.

Jean-Paul

Thanks for the correction; that's an important advantage for callbacks
in this case.

George