(Win32 API) callback to Python, threading hiccups

[Francois De Serres]

Hiho,

could somebody please enlighten me about the mechanics of C callbacks to
Python? My domain is more specifically callbacks from the win32 API, but
I'm not sure that's where the problem lies. Here's a description...

I want a callback-based MIDI input/processing, so PortMidi was not an
alternative. I have written a C extension module that links to the mmsys
MIDI API. I separated the win32-dependant code from the Python extension
code, so a) the module interface is system-neutral, b) the
implementation can be tested (re-used) outside of Python. So, that code
is tested OK, but it might be useful to sketch it's design:
- as you may know, the MIDI input on win32 is already managed thru a
callback mechanism; the driver calls back your program with data buffers
- yet, I don't call directly into Python from the callback, since some
restrictions apply on what system calls you can make from there, and I
don't know what Python's interpreter / scripts might call.
- so, on callback, I create a new thread, after checking that the
previous one has returned already (WaitOnSingleObject(mythread)) so we
only have one thread involved.
- this is this thread that calls the user callback, yet this callback
isn't yet a Python callable, we're still in native C code.
- on the side of the extension module now, I merely copied the callback
example from the Python/C API doc, and added GIL management around the call:

static PyObject * my_callback = NULL; //this gets fixed by a
setCallback() func
static void external_callback(const MidiData * const data) {
if (my_callback && (my_callback != Py_None)) {
if (! data) {
PyErr_SetString(PyExc_IndexError, getLastErrorMessage());
} else {
PyObject * arglist = NULL;
PyObject * result = NULL;
arglist = Py_BuildValue("(i,i,s#)", data->deviceIndex,
data->timestamp, data->buffer, data->size);// 0, 0, "test", 4);//

PyGILState_STATE gil = PyGILState_Ensure();
result = PyEval_CallObject(my_callback, arglist);
PyGILState_Release(gil);

Py_DECREF(arglist);
Py_DECREF(result);
}
}
}

- this one above is what is actually passed as callback to the 'native'
C part. So, here's what (I presume) happens:
1. the driver calls into my C code
2. my C code spawns a thread that calls into the extension
3. the extension calls into Python, after acquiring the GIL

Now, here's the hiccup:
inside a Python script, anytime a Python object is accessed by both the
(Python) callback and the main program, I get a GPF :/

You bet I tried to use locks, Queues and whatnot, but nothing will do,
it seems I get protection faults on accessing... thread exclusion objects.

Yet, there is a way where it works seamlessly: it's when the __main__
actually spawns a threading.Thread to access the shared object. Hence I
am lost.

This works:

def input_callback(msg):
midiio.send(msg) ##MIDI thru, a call into my extension that wraps
the implementation call with BEGIN_ALLOW_THREADS and END_...
__main__:
raw_input()

This results in GPF as soon as the callback is fired:

tape = Queue.Queue()
def input_callback(msg):
tape.put(msg)
__main__:
while True:
print tape.get()

This works like a charm:

tape = Queue.Queue()
def input_callback(msg):
tape.put(msg)
def job():
while True:
print tape.get()
__main__:
t = threading.Thread(target = job)
t.start()
raw_input()

Note that I also get a GPF in the later case if I stop the program with
KeyInterrupt, but I guess it's something I'll look into afterwards...

While I can think of hiding away the added threading within a wrapper
module, I won't sleep well untill I figure out what's really happening.
Could someone save my precious and already sparse sleeping time, by
pointing me to what I'm missing here?

WinXP SP1
Python 2.4.1
MinGW GCC 3.4.2

TIA,
Francois De Serres

[Christopher Subich]

Francois De Serres wrote:

- so, on callback, I create a new thread, after checking that the
previous one has returned already (WaitOnSingleObject(mythread)) so we
only have one thread involved.

Uh... to me, this looks like a frighteningly inefficient way of doing
things. How about using a synchronous queue to post the data to a
processing thread? That way, you don't have to create an entierly new
thread each time you receive data in the callback.

[Francois De Serres]

Christopher Subich wrote:

Francois De Serres wrote:

- so, on callback, I create a new thread, after checking that the
previous one has returned already (WaitOnSingleObject(mythread)) so we
only have one thread involved.

Uh... to me, this looks like a frighteningly inefficient way of doing
things. How about using a synchronous queue to post the data to a
processing thread? That way, you don't have to create an entierly new
thread each time you receive data in the callback.

Thanks for the tip, and sorry for frightening you. Maybe you can help as
well with my issue?

Francois

[Scott David Daniels]

Francois De Serres wrote:

PyGILState_STATE gil = PyGILState_Ensure();
result = PyEval_CallObject(my_callback, arglist);
PyGILState_Release(gil);
Py_DECREF(arglist);
Py_DECREF(result);

I think this should be:
PyGILState_STATE gil = PyGILState_Ensure();
result = PyEval_CallObject(my_callback, arglist);
Py_DECREF(arglist);
Py_DECREF(result);
PyGILState_Release(gil);

The DECREFs need to be protected, that is where storage is
recycled and such, and you still need Python's data structures
to do that kind of work.

--Scott David Daniels

[Tim Roberts]

Scott David Daniels <Sc***********@Acm.Org> wrote:

Francois De Serres wrote:

PyGILState_STATE gil = PyGILState_Ensure();
result = PyEval_CallObject(my_callback, arglist);
PyGILState_Release(gil);
Py_DECREF(arglist);
Py_DECREF(result);

I think this should be:
PyGILState_STATE gil = PyGILState_Ensure();
result = PyEval_CallObject(my_callback, arglist);
Py_DECREF(arglist);
Py_DECREF(result);
PyGILState_Release(gil);

The DECREFs need to be protected, that is where storage is
recycled and such, and you still need Python's data structures
to do that kind of work.

I freely admit to being woefully underinformed about the GIL, but I'm
wondering if your statement is really true. If the purpose of the GIL is
simply to make things thread-safe, then I would have guessed that the first
one was correct. If someone else holds a reference to "arglist", then the
DECREF is just a nice, atomic decrement. If no one else holds a reference
to "arglist", then it's quite safe to delete it.

Is there more to the GIL than I'm assuming?
--
- Tim Roberts, ti**@probo.com
Providenza & Boekelheide, Inc.

[Gregory Bond]

Tim Roberts wrote:

PyGILState_STATE gil = PyGILState_Ensure();
result = PyEval_CallObject(my_callback, arglist);
PyGILState_Release(gil);
Py_DECREF(arglist);
Py_DECREF(result);

If someone else holds a reference to "arglist", then the
DECREF is just a nice, atomic decrement. If no one else holds a reference
to "arglist", then it's quite safe to delete it.

What if anothere thread takes the GIL as soon as you release it, and is
attempting to allcate a new opbject, and (at the same time - you might
be on a multiprocessor!) your PyDECREF removes the last reference and
starts freeing objects?