Threading a lengthy C function

[Leo Breebaart]

Hi all,

In a GUI program I'm currently developing (wxWindows, if that
matters), I am trying to get some time-consuming tasks (triggered
by e.g. the user choosing a menu item) out of the interface
thread and into 'worker' threads of their own, in order to stop
them from locking up the GUI.
If I subclass threading.Thread, and I override run() as follows:

def run(self):
retvalue = time.sleep(50)

then everything works perfectly fine when I start up this Thread
in response to the user trigger -- the GUI unlocks and the
application windows will repaint correctly, etc.

If, however, I use the kind of run() method that I *really* need:

def run(self):
retvalue = mypackage.sleep(50)

where mypackage.sleep() is in fact a SWIG-generated wrapper
around the C library sleep() function, I have no such luck: the
entire application will *still* lock up completely until this
task is done.

My guess would be (but I'm new to Python, so I may be entirely
wrong) that this has something to do with that Global Interpreter
Lock thing I keep reading about. I can imagine that if my lengthy
task is a Python task, the interpreter will get the chance to
switch between threads, but that if the task is an external C
function, this will be executed 'atomically' with no chance for
the interpreter to release the GIL to other threads.

I'd like to know if this interpretation of what's happening is
correct (or if it isn't, then what is?), but most importantly I
was wondering if anybody could point me in the direction of a
solution or workaround to my actual problem.

Many thanks in advance,

--
Leo Breebaart <le*@lspace.org>

[Duncan Booth]

Leo Breebaart <le*@lspace.org> wrote in news:bp**********@news.tudelft.nl:

I'd like to know if this interpretation of what's happening is
correct (or if it isn't, then what is?), but most importantly I
was wondering if anybody could point me in the direction of a
solution or workaround to my actual problem.

Your interpretation sounds spot on.

What you need to do is to release the GIL as the last thing that happens in
your C code before you call the lengthy function, and reclaim it as the
first thing you do when the function returns.

It may be easiest simply to wrap the library function in another C function
that does this, and then use SWIG to wrap the new function instead of the
original. e.g.

#include <python.h>
int safe_thefunction(char *arg, int arg2)
{
int res;
Py_BEGIN_ALLOW_THREADS
res = thefunction(arg, arg2);
Py_END_ALLOW_THREADS
return res;
}

and then wrap safe_thefunction instead of thefunction.

If the function you are wrapping can call back into Python then you must
also be sure to reclaim the GIL during each callback. This is easy enough
to do if the callbacks all happen on the same thread, but can be a pain if
there are asynchronous callbacks on other threads.

--
Duncan Booth du****@rcp.co.uk
int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3"
"\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure?

[Leo Breebaart]

My thanks to everyone who participated in this, ahem, thread.
Your responses have been very helpful -- this newsgroup is a
wonderful resource for the beginning Python programmer.
Duncan Booth <du****@NOSPAMrcp.co.uk> writes:

It may be easiest simply to wrap the library function in another C function
that does this, and then use SWIG to wrap the new function instead of the
original. e.g.

#include <python.h>
int safe_thefunction(char *arg, int arg2)
{
int res;
Py_BEGIN_ALLOW_THREADS
res = thefunction(arg, arg2);
Py_END_ALLOW_THREADS
return res;
}

This is more or less exactly what I ended up doing, and
everything works like a charm now.

--
Leo Breebaart <le*@lspace.org>