473,748 Members | 2,243 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Turning a callback function into a generator

Let's say I have a function that takes a callback function as a
parameter, and uses it to describe an iteration:

def func(callback):
for i in [1, 2, 3, 4, 5]:
callback(i)

For the sake of argument, assume the iteration is something more
interesting than this which relies on the callback mechanism. The
function is an existing interface, and I cannot change it.

I want to somehow, in some way, provide an iteration interface to this
function. Thoughts?

-Kirk McDonald
Jul 3 '06 #1
7 3594
Kirk McDonald wrote:
Let's say I have a function that takes a callback function as a
parameter, and uses it to describe an iteration:

def func(callback):
for i in [1, 2, 3, 4, 5]:
callback(i)

For the sake of argument, assume the iteration is something more
interesting than this which relies on the callback mechanism. The
function is an existing interface, and I cannot change it.

I want to somehow, in some way, provide an iteration interface to this
function. Thoughts?
I don't think there is a robust solution, see

http://groups.google.com/group/comp....6408134?&hl=en
aka http://mail.python.org/pipermail/pyt...er/197726.html

http://groups.google.com/group/comp....5d84694?&hl=en
aka http://mail.python.org/pipermail/pyt...er/239025.html

Peter

Jul 3 '06 #2

Peter Otten wrote:
Kirk McDonald wrote:
Let's say I have a function that takes a callback function as a
parameter, and uses it to describe an iteration:

def func(callback):
for i in [1, 2, 3, 4, 5]:
callback(i)
Which object is immutable? the callback or the function? If its the
callback then

def func(callback):
for i in numbers:
yield callback(i)

If the function is immutable, then its a bit harder. The callback has
to be able to do the processing. You can't use an iterator here
because call stack gets in the way. You could store the information
being passed to the callback in a list, then iterate over the list
afterwards. Or you could have the callback be able to handle all of
the work at once.

What do you intend to use this for? Python has a lot of options and
you may not be using the best one for the problem

Jul 3 '06 #3
In article <44******@nntp0 .pdx.net>,
Kirk McDonald <ki************ **@gmail.comwro te:
>I want to somehow, in some way, provide an iteration interface to this
function. Thoughts?
Run it in a separate thread/process?
Jul 3 '06 #4
cm************@ yaho.com wrote:
Peter Otten wrote:
>>Kirk McDonald wrote:

>>>Let's say I have a function that takes a callback function as a
parameter, and uses it to describe an iteration:

def func(callback):
for i in [1, 2, 3, 4, 5]:
callback(i)


Which object is immutable? the callback or the function? If its the
callback then

def func(callback):
for i in numbers:
yield callback(i)

If the function is immutable, then its a bit harder. The callback has
to be able to do the processing. You can't use an iterator here
because call stack gets in the way. You could store the information
being passed to the callback in a list, then iterate over the list
afterwards. Or you could have the callback be able to handle all of
the work at once.

What do you intend to use this for? Python has a lot of options and
you may not be using the best one for the problem
It is the function that is immutable.

I am writing a library for the D programming language that is not
totally unlike Boost.Python:

http://dsource.org/projects/pyd/wiki

It is still in the fairly early stages, although the basic function and
class wrapping do work.

This particular functionality is required to wrap D's basic iteration
protocol, the opApply function:

http://www.digitalmars.com/d/statement.html#foreach

opApply works using a callback, as described. Because D does not (yet)
have anything like Python's "yield", wrapping it with Python's iteration
interface is turning out to be dreadfully annoying.

-Kirk McDonald
Jul 3 '06 #5
Lawrence D'Oliveiro <ld*@geek-central.gen.new _zealandwrote:
In article <44******@nntp0 .pdx.net>,
Kirk McDonald <ki************ **@gmail.comwro te:
I want to somehow, in some way, provide an iteration interface to this
function. Thoughts?

Run it in a separate thread/process?
Sounds best to me. Specifically, given (e.g.) something like the OP's

def func(callback):
for i in [1, 2, 3, 4, 5]:
callback(i)

we might have a wrapper such as:

import thread
import Queue

def wrap_cb_into_ge n(func):
q = Queue.Queue(1) # maximum size of 1
def callback(item):
q.put(item)
all_done_sentin el = object()
def thread_skeleton ():
func(callback)
q.put(all_done_ sentinel)
thread.start_ne w_thread(thread _skeleton, ())
while True:
item = q.get()
if item is all_done_sentin el: break
yield item
Of course, there are lighter-weight options than a length-1 Queue for
the purpose of synchronizing these two threads, but I always tend to
prefer Queue (for its simplicity, generality, solidity) to other
synchronization structures and architectures, unless there is some very
strong reason to do otherwise in a specific case. Also, I normally use
module threading rather than the lower-level module thread -- here,
clearly, either one will do just fine.
Alex
Jul 3 '06 #6
Alex Martelli wrote:
Lawrence D'Oliveiro <ld*@geek-central.gen.new _zealandwrote:

>>In article <44******@nntp0 .pdx.net>,
Kirk McDonald <ki************ **@gmail.comwro te:

>>>I want to somehow, in some way, provide an iteration interface to this
function. Thoughts?

Run it in a separate thread/process?


Sounds best to me. Specifically, given (e.g.) something like the OP's

def func(callback):
for i in [1, 2, 3, 4, 5]:
callback(i)

we might have a wrapper such as:

import thread
import Queue

def wrap_cb_into_ge n(func):
q = Queue.Queue(1) # maximum size of 1
def callback(item):
q.put(item)
all_done_sentin el = object()
def thread_skeleton ():
func(callback)
q.put(all_done_ sentinel)
thread.start_ne w_thread(thread _skeleton, ())
while True:
item = q.get()
if item is all_done_sentin el: break
yield item
Of course, there are lighter-weight options than a length-1 Queue for
the purpose of synchronizing these two threads, but I always tend to
prefer Queue (for its simplicity, generality, solidity) to other
synchronization structures and architectures, unless there is some very
strong reason to do otherwise in a specific case. Also, I normally use
module threading rather than the lower-level module thread -- here,
clearly, either one will do just fine.
Alex
Threads are probably a weak point of mine. I have little experience with
them, and so I am inclined to paranoia while using them. What
implications does this have for "func"? In my case, it is a function in
an extension module. Does it have to be thread safe? If it isn't, what
limitations does this impose on the use of this wrapper?

All that aside, I read the previous threads but it took until just now
to understand how this works. :-) I'll probably have a go at
implementing this in D, as that will be more convenient for the library...

-Kirk McDonald
Jul 3 '06 #7
Kirk McDonald <ki************ **@gmail.comwro te:
...
def func(callback):
for i in [1, 2, 3, 4, 5]:
callback(i)
...
Threads are probably a weak point of mine. I have little experience with
them, and so I am inclined to paranoia while using them. What
Paranoia is the correct state of mind to start from when threads are
involved.
implications does this have for "func"? In my case, it is a function in
an extension module. Does it have to be thread safe? If it isn't, what
limitations does this impose on the use of this wrapper?
The 'func' you gave as an example IS threadsafe (if callback is): it
does not access any global object that might be used by other threads at
the same time. If the real func *DOES* access (or, even worse, modify)
global objects -- or more generally objects that might be modified by
other threads -- then you must try to make things safe again by adding
locks (with a strong risk of deadlocking, of course).

This problem is basically (to a large extent) tied to your very specs:
you want the whole callstack up to the point where 'func' calls
'callback' to be essentially ``frozen'' while somehow the 'callback'
(the only point in which you allow intervention in the code, since you
forbid alterations of 'func'!) magically hands over the item to a yield
statement which provides the item to calling-code. Now calling code can
do anything it wants (to globals or any other objects that, for all
you've told us, 'func' might be accessing or altering too) -- and this
must not damage func's behavior.

If there are no other threads in your process, beyond the main one (on
which the generator-wrapper is called) and the specialized one which the
generator-wrapper created (to run func in), then the issues are pretty
much the same as if the generator-wrapper was able to avoid using
threads -- it's just slightly less deterministic where control may shift
between the two threads, but I believe (without being able to prove it)
that this is unlikely to matter... the key issue remains ensuring that
the *caller* of the wrapper, and func itself, don't tread on each
other's toes, regarding all that's "suspended" on the stack while the
caller does whatever it wants with the just-yielded item.

If there ARE other threads in your process, and func is not threadsafe
with respect to them, then of course you should not be calling it
(wrapped or not) without precautions in terms of locking &c -- the
wrapper doesn't matter.
All that aside, I read the previous threads but it took until just now
to understand how this works. :-) I'll probably have a go at
implementing this in D, as that will be more convenient for the library...
OK, sounds good. D looks interesting (I found Walter Bright's short
presentation of it at Google quite engaging), and I might like to give
it a try if it was as handy to interface to Python as C++ (or Java or C#
or Objective C) already are.
Alex
Jul 3 '06 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
1811
by: Peter Otten | last post by:
It's easy to write a function that wraps a generator and provides a callback. E. g.: import os, sys def walk(path, visit): """ Emulate os.path.walk() (simplified) using os.walk()""" for dir, folders, files in os.walk(path): visit(dir, folders + files)
4
2341
by: Wai Yip Tung | last post by:
I'm attempting to turn some process than uses callback to return result into a more user friendly generator. I'm hitting some road block so any pointer would be appriciated. Let say there is an existing method producer(x, cb). It calls the supplied cb() every time when there is data available. The definititon of cb should be: def cb(data)
4
2054
by: ma740988 | last post by:
// file sltest.h #ifndef SLTEST_H #define SLTEST_H class CallbackBase // herb shutters gotW source .. { public: virtual void operator()() const { }; virtual ~CallbackBase() = 0; };
15
2006
by: Felix Kater | last post by:
Hi, in a given library I register callback functions with this function: bool set_callback(int index, int (*callback_function)(long)); I need the callback function to also pass the index which is, however, not (re-)passed by the callback function. Normally, I would just register callback functions for each
8
572
by: kurtcobain1978 | last post by:
-------------------------------------------------------------------------------- I need to do the exactly same thing in VB.NET. Load a unmanaged C DLL dynamically and then call a function in which I pass the callback function as an argument. My C function being called callback as type _cdecl. Does anybody have any ideas?
0
1799
by: Robert | last post by:
After failing on a yield/iterator-continuation problem in Python (see below) I tried the Ruby (1.8.2) language first time on that construct: The example tries to convert a block callback interface (Net::FTP.retrbinary) into a read()-like iterator function in order to virtualize the existing FTP class as kind of file system. 4 bytes max per read in this first simple test below. But it fails on the second continuation with ThreadError after...
6
7681
by: smmk25 | last post by:
Before I state the problem, I just want to let the readers know, I am knew to C++\CLI and interop so please forgive any newbie questions. I have a huge C library which I want to be able to use in a .NET application and thus am looking into writing a managed C++ wrapper for in vs2005. Furthermore, this library has many callback hooks which need to be implemented by the C++ wrapper. These callback functions are declared as "extern C...
5
4322
by: Jef Driesen | last post by:
I have a C DLL that I want to use from a C# project. The C header file contains these declarations: typedef void (*callback_t) (const unsigned char *data, unsigned int size, void *userdata); void myfunction (callback_t callback, void *userdata); How do I translate this to C#? I tried with:
0
8818
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9355
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
9307
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9225
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
6069
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4589
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4860
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3296
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
2203
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.