I'm attempting to turn some process than uses callback to return result
into a more user friendly generator. I'm hitting some road block so any
pointer would be appriciated.
Let say there is an existing method producer(x, cb). It calls the supplied
cb() every time when there is data available. The definititon of cb should
be:
def cb(data)
To turn it into generator, I tried:
def product_generator(x):
producer(x,cb)
def cb(data):
yield data
But that's not really the way Python works. 'yield' is not only a
imperative statement but it also the keyword to signify the containing
function as a generator function. So in this case product_generator is an
oridinary function and cb is a generator.
Note that producer() is an existing method that I cannot change. Otherwise
I might strip the cb and yield directly from the producer().
If python would define another keyword 'generator' to explicit distinguish
normal function from generator and decouple it from the 'yield' keyword,
it could perhaps be done that way:
generator product_generator(x):
producer(x,cb)
def cb(data):
yield data
Is this just an syntactic issue or am I hitting on some generator's
implementation issue? Is there a proper way to do this in Python 2.3?
Thanks for any pointer. 4 2306
Wai Yip Tung wrote: I'm attempting to turn some process than uses callback to return result into a more user friendly generator. I'm hitting some road block so any pointer would be appriciated.
I asked the same question a while back and there were no satisfying
suggestions. http://mail.python.org/pipermail/pyt...er/197726.html
At the very least you need threads, and when the generator is not fully
exhausted it's easy to end up with a thread waiting forever.
I finally dropped the idea, but if you are really determined and want to
hack something together, the following might (or might not, this is really
a shot in the dark) serve as a starting point: http://mail.python.org/pipermail/pyt...ly/173872.html
Peter
On Mon, 06 Sep 2004 23:24:17 +0200, Peter Otten <__*******@web.de> wrote: Wai Yip Tung wrote:
I'm attempting to turn some process than uses callback to return result into a more user friendly generator. I'm hitting some road block so any pointer would be appriciated.
I asked the same question a while back and there were no satisfying suggestions.
http://mail.python.org/pipermail/pyt...er/197726.html
At the very least you need threads, and when the generator is not fully exhausted it's easy to end up with a thread waiting forever.
I finally dropped the idea, but if you are really determined and want to hack something together, the following might (or might not, this is really a shot in the dark) serve as a starting point:
http://mail.python.org/pipermail/pyt...ly/173872.html
Not that familiar with python threads and queue, but here's a start: import threading import Queue q = Queue.Queue(3) # ridiculously short
def producer(n, cb):
... for i in xrange(n): cb(i)
... def product_generator(p, *a, **kw):
... tpq = threading.Thread(target=p, args=a, kwargs=kw)
... tpq.start()
... try:
... while True: yield q.get(True,5)
... except Queue.Empty:
... print 'No data for 5 seconds'
... for item in product_generator(producer, 8, q.put): print item
...
0
1
2
3
4
5
6
7
No data for 5 seconds
Not tested beyond what you see ;-)
Regards,
Bengt Richter
Bengt Richter wrote: Not that familiar with python threads and queue, but here's a start:
>>> import threading >>> import Queue >>> q = Queue.Queue(3) # ridiculously short
But already too long if the items yielded are somehow interdependant.
>>> >>> def producer(n, cb): ... for i in xrange(n): cb(i) ... >>> def product_generator(p, *a, **kw): ... tpq = threading.Thread(target=p, args=a, kwargs=kw) ... tpq.start() ... try: ... while True: yield q.get(True,5) ... except Queue.Empty: ... print 'No data for 5 seconds' ... >>> for item in product_generator(producer, 8, q.put): print item ... 0 1 2 3 4 5 6 7 No data for 5 seconds >>>
Not tested beyond what you see ;-)
The problem is _not_ to signal that there are no more items to be expected -
you can easily do that by wrapping the producer to put a special object
into the queue when it's done and test for that in the product_generator().
This would avoid the 5-second jet lag. It's a bit harder to avoid a
dangling thread when you have code like
for item in product_generator(...):
if pred(item): break
You could attack that with a long timeout for Queue.put(), too, but I'd
rather signal the generator that we're done (via a second Queue, maybe).
Unfortunately this imposes changes on the client code, thus defeating the
original goal of making generators and visitors/callbacks freely
interchangeable.
Peter
On Tue, 07 Sep 2004 10:15:55 +0200, Peter Otten <__*******@web.de> wrote: Bengt Richter wrote:
Not that familiar with python threads and queue, but here's a start:
>>> import threading >>> import Queue >>> q = Queue.Queue(3) # ridiculously short But already too long if the items yielded are somehow interdependant. >>> >>> def producer(n, cb): ... for i in xrange(n): cb(i) ... >>> def product_generator(p, *a, **kw): ... tpq = threading.Thread(target=p, args=a, kwargs=kw) ... tpq.start() ... try: ... while True: yield q.get(True,5) ... except Queue.Empty: ... print 'No data for 5 seconds' ... >>> for item in product_generator(producer, 8, q.put): print item ... 0 1 2 3 4 5 6 7 No data for 5 seconds >>>
Not tested beyond what you see ;-)
The problem is _not_ to signal that there are no more items to be expected - you can easily do that by wrapping the producer to put a special object into the queue when it's done and test for that in the product_generator(). This would avoid the 5-second jet lag. It's a bit harder to avoid a dangling thread when you have code like
for item in product_generator(...): if pred(item): break
You could attack that with a long timeout for Queue.put(), too, but I'd rather signal the generator that we're done (via a second Queue, maybe). Unfortunately this imposes changes on the client code, thus defeating the original goal of making generators and visitors/callbacks freely interchangeable.
Peter
Thanks for the pointers. Actually I'm looking for a solution that do not
use thread or to buffer all result in memory since one of the great thing
about generator is that it can achieve the process without using those
resource intensive techniques.
About the dangling thread, perhaps the destructor method __del__() can
help ( http://www.python.org/dev/doc/devel/...mization.html). It is
really nasty to end up with dangling threads.
Wai Yip This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Francis Avila |
last post by:
A little annoyed one day that I couldn't use the statefulness of
generators as "resumable functions", I came across Hettinger's PEP 288
(http://www.python.org/peps/pep-0288.html, still listed as...
|
by: Peter Otten |
last post by:
It's easy to write a function that wraps a generator and provides a
callback. E. g.:
import os, sys
def walk(path, visit):
""" Emulate os.path.walk() (simplified) using os.walk()"""
for dir,...
|
by: ma740988 |
last post by:
// file sltest.h
#ifndef SLTEST_H
#define SLTEST_H
class CallbackBase // herb shutters gotW source ..
{
public:
virtual void operator()() const { };
virtual ~CallbackBase() = 0;
};
|
by: Felix Kater |
last post by:
Hi,
in a given library I register callback functions with this function:
bool set_callback(int index, int (*callback_function)(long));
I need the callback function to also pass the index...
|
by: Robert |
last post by:
After failing on a yield/iterator-continuation problem in Python (see
below) I tried the Ruby (1.8.2) language first time on that construct:
The example tries to convert a block callback interface...
| |
by: Kirk McDonald |
last post by:
Let's say I have a function that takes a callback function as a
parameter, and uses it to describe an iteration:
def func(callback):
for i in :
callback(i)
For the sake of argument, assume...
|
by: DaTurk |
last post by:
Hi,
I'm coding up an application that has a native c++ layer,
asynchronously calling callback in a CLI layer.
We originally did this with static inline methods in the CLI layer,
but this...
|
by: Tim Spens |
last post by:
--- On Fri, 6/27/08, Tim Spens <t_spens@yahoo.comwrote:
I think I know where the problem is but I'm unsure how to fix it. When I call Register_Handler(...) from python via...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
| |
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated ...
|
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence...
| |