Events and file polling in a C extension

[Dirk Thierbach]

I am writing an Python 2.3 extension in C, and I would like to create
events (Tk-style) from an external source. I use a library that
already parses the events into a C structure. This library can
also supply file ids for poll() or select() calls (all this is on
Linux, BTW).

What's the preferred way to doing this? Can I register my own event
handlers to the Python runtime, including the file ids to poll/to do
select on, as in Tcl/Tk? Or do I have to create a new thread, and
do a blocking select/poll-call inside the thread? If yes, how much
attention do I have to pay to Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS?
E.g. use them as often as possible, or only at cricitical points?
How do I create a new thread with the C API, anyway? Or should I wrap
that into a Python module that calls some auxiliary functions
in the C module?

As I am comparatively new to Python and the C API, I would be glad for
some boilerplate C-code (if there is any available somewhere) I could
use as a starting point, so I don't have to re-invent the wheel myself
another time :-)

Thanks for any help,

- Dirk

[Donn Cave]

Quoth Dirk Thierbach <dt********@gmx.de>:
| I am writing an Python 2.3 extension in C, and I would like to create
| events (Tk-style) from an external source. I use a library that
| already parses the events into a C structure. This library can
| also supply file ids for poll() or select() calls (all this is on
| Linux, BTW).

So what would you do after select() returns with I/O ready? Can
the library then read and parse, and return the result?

| What's the preferred way to doing this? Can I register my own event
| handlers to the Python runtime, including the file ids to poll/to do
| select on, as in Tcl/Tk? Or do I have to create a new thread, and
| do a blocking select/poll-call inside the thread? If yes, how much
| attention do I have to pay to Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS?
| E.g. use them as often as possible, or only at cricitical points?
| How do I create a new thread with the C API, anyway? Or should I wrap
| that into a Python module that calls some auxiliary functions
| in the C module?

I would stay away from threads unless you have a strong need for
concurrency that isn't already satisfied with select/poll. It seems
to me the simplest thing by far would be the arrangement I suggested
above, with the I/O dispatching in Python and the external C functions
responsible only for data transformation, but it depends a lot on
your application. Python doesn't come with an event handler model,
except when used with say a graphic toolkit - so for example Python/Tk
will (I assume) look a lot like Tcl/Tk in this respect.

Donn Cave, do**@drizzle.com

[Dirk Thierbach]

Donn Cave <do**@drizzle.com> wrote:

Quoth Dirk Thierbach <dt********@gmx.de>: So what would you do after select() returns with I/O ready? Can
the library then read and parse, and return the result?
Yes.
I would stay away from threads unless you have a strong need for
concurrency that isn't already satisfied with select/poll.
So would I, but at the moment I don't see how, unless Python supports
some sort of event handling.
It seems to me the simplest thing by far would be the arrangement I
suggested above,
Which one? Handling select in the main program? That doesn't work,
because...
Python doesn't come with an event handler model, except when used
with say a graphic toolkit - so for example Python/Tk will (I
assume) look a lot like Tcl/Tk in this respect.

.... I am exactly after something that looks like Python/Tk with
respect to Tk-bindings. I want to create an object, let the main
program do some bindings on that object, and then the main program
can go on to do whatever it likes. The object should dispatch any
events that arrive on the file descriptor by invoking the bindings,
just like the widget bindings in Python/Tk.

Of course I could create some sort of main loop myself, but that
seems inelegant and not portable. After all, Python/Tk doesn't
require you to call some sort of main loop, too.

- Dirk

[Josiah Carlson]

Dirk Thierbach <dt********@gmx.de> wrote:

Donn Cave <do**@drizzle.com> wrote:

Quoth Dirk Thierbach <dt********@gmx.de>:

So what would you do after select() returns with I/O ready? Can
the library then read and parse, and return the result?

Yes.

I would stay away from threads unless you have a strong need for
concurrency that isn't already satisfied with select/poll.

So would I, but at the moment I don't see how, unless Python supports
some sort of event handling.

I believe that most languages support event handling in one way or
another. Here is one way...

def handle_event(event):
if event.typ == 'new_connection':
#handle a new connection
elif event.typ == ...

Or, if you want to use something that already has much of this designed
and built for sockets, check out asyncore or asynchat. Use
asyncore.loop(timeout, 1) to poll the sockets every 1/100th of a second
or so, and you should be good.

It seems to me the simplest thing by far would be the arrangement I
suggested above,

Which one? Handling select in the main program? That doesn't work,
because...

Python doesn't come with an event handler model, except when used
with say a graphic toolkit - so for example Python/Tk will (I
assume) look a lot like Tcl/Tk in this respect.

... I am exactly after something that looks like Python/Tk with
respect to Tk-bindings. I want to create an object, let the main
program do some bindings on that object, and then the main program
can go on to do whatever it likes. The object should dispatch any
events that arrive on the file descriptor by invoking the bindings,
just like the widget bindings in Python/Tk.

These are not hard to make work reasonably. They can be hard to do well
and in a general fashion. If you want a general event-based framework,
give Twisted a look (though it does seem to be a bit more socket related).
Of course I could create some sort of main loop myself, but that
seems inelegant and not portable. After all, Python/Tk doesn't
require you to call some sort of main loop, too.

While I do not develop with Tk, I believe you are incorrect. Checking
Idle, I notice a call to: root.mainloop(), which seems to be the event
handler loop.

In regards to writing your own main loop; what is so inelegant and not
portable about it? Having done so more than a dozen times, I would say
that writing your own main loop is usually the way to go, unless you
need to hook into some GUI toolkit or a pre-existing event framework, at
which point you can use the toolkit or framework's event processor to
handle dispatch.

- Josiah

[Josiah Carlson]

Josiah Carlson <jc******@uci.edu> wrote:

I believe that most languages support event handling in one way or
another. Here is one way...

def handle_event(event):
if event.typ == 'new_connection':
#handle a new connection
elif event.typ == ...

Or, if you want to use something that already has much of this designed
and built for sockets, check out asyncore or asynchat. Use
asyncore.loop(timeout, 1) to poll the sockets every 1/100th of a second
or so, and you should be good.

That is not to say that asyncore.loop(timeout, 1) will poll the sockets
every 100th a second, just that you should make that call about every
1/100th a second (timeout of 0 would be a true poll).

- Josiah