extension call backs and the GIL

[Robin Becker]

I'm trying to better my understanding of multithreading. I believe a
thread that calls an extension must have the GIL. What happens if the
extension then calls back into Python? Can a function or method somehow
'belong' to a different thread? My understanding was that things should
be OK provided the extension code doesn't mess with the GIL (which it
doesn't).

I'm asking because the ReportLab tool kit contains an accelerator for a
stringWidth function that relies on a callback to fill in some global
data upon demand. A user reports thread problems when using the
accelerator.
--
Robin Becker

[Aahz]

In article <pU************ **@jessikat.fsn et.co.uk>,
Robin Becker <ro***@jessikat .fsnet.co.uk> wrote:

I'm trying to better my understanding of multithreading. I believe a
thread that calls an extension must have the GIL. What happens if the
extension then calls back into Python? Can a function or method somehow
'belong' to a different thread? My understanding was that things should
be OK provided the extension code doesn't mess with the GIL (which it
doesn't).

You need to reacquire the GIL prior to making the callback.
--
Aahz (aa**@pythoncra ft.com) <*> http://www.pythoncraft.com/

"It is easier to optimize correct code than to correct optimized code."
--Bill Harlan

[Robin Becker]

In article <bl**********@p anix3.panix.com >, Aahz <aa**@pythoncra ft.com>
writes

In article <pU************ **@jessikat.fsn et.co.uk>,
Robin Becker <ro***@jessikat .fsnet.co.uk> wrote:

I'm trying to better my understanding of multithreading. I believe a
thread that calls an extension must have the GIL. What happens if the
extension then calls back into Python? Can a function or method somehow
'belong' to a different thread? My understanding was that things should
be OK provided the extension code doesn't mess with the GIL (which it
doesn't).

You need to reacquire the GIL prior to making the callback.

So does calling an extension release the GIL? I believed the opposite
for some reason.
--
Robin Becker

[Bernhard Herzog]

Robin Becker <ro***@jessikat .fsnet.co.uk> writes:

You need to reacquire the GIL prior to making the callback.

So does calling an extension release the GIL?

No. It would be very inconvenient if it did because what most functions
do first is to call one of the PyArg_ParseTupl e functions which could
execute arbitrary python code.

I think it would help if you described a bit more what the actual
problem is or what the user does.

I looked briefly at the sources and assuming tha the accelerator in
question is lib/_rl_accel.c and the callback whatever has been assigned
to _SWRecover, could it be that the problem is that your user tries to
use different callbacks in different threads? _SWRecover is a global
after all and shared by all threads.

Bernhard

--
Intevation GmbH http://intevation.de/
Sketch http://sketch.sourceforge.net/
Thuban http://thuban.intevation.org/

[Robin Becker]

In article <6q************ @salmakis.intev ation.de>, Bernhard Herzog
<bh@intevation. de> writes

Robin Becker <ro***@jessikat .fsnet.co.uk> writes:

You need to reacquire the GIL prior to making the callback. So does calling an extension release the GIL?

No. It would be very inconvenient if it did because what most functions
do first is to call one of the PyArg_ParseTupl e functions which could
execute arbitrary python code.

Thank goodness for that :)

I think it would help if you described a bit more what the actual
problem is or what the user does.

I looked briefly at the sources and assuming tha the accelerator in
question is lib/_rl_accel.c and the callback whatever has been assigned
to _SWRecover, could it be that the problem is that your user tries to
use different callbacks in different threads? _SWRecover is a global
after all and shared by all threads.
I'm not certain, but yes I think that's probably what the problem comes
down to; if so I can claim there are already too many breakages in our
font system to worry about that :(

Bernhard

--
Robin Becker

[Aahz]

In article <OW************ **@jessikat.fsn et.co.uk>,
Robin Becker <ro***@jessikat .fsnet.co.uk> wrote:

In article <bl**********@p anix3.panix.com >, Aahz <aa**@pythoncra ft.com>
writes

In article <pU************ **@jessikat.fsn et.co.uk>,
Robin Becker <ro***@jessikat .fsnet.co.uk> wrote:

I'm trying to better my understanding of multithreading. I believe a
thread that calls an extension must have the GIL. What happens if the
extension then calls back into Python? Can a function or method somehow
'belong' to a different thread? My understanding was that things should
be OK provided the extension code doesn't mess with the GIL (which it
doesn't).

You need to reacquire the GIL prior to making the callback.

So does calling an extension release the GIL? I believed the opposite
for some reason.

Looks like Bernhard has put you on the right track, but just for the
record: calling an extension does not automatically release the GIL; you
need to explicitly call the Py_BEGIN_ALLOW_ THREADS macro. I was just
assuming that because you said "accelerato r" you were already releasing
the GIL.
--
Aahz (aa**@pythoncra ft.com) <*> http://www.pythoncraft.com/

"It is easier to optimize correct code than to correct optimized code."
--Bill Harlan

[Robin Becker]

In article <bl**********@p anix3.panix.com >, Aahz <aa**@pythoncra ft.com>
writes
.......

Looks like Bernhard has put you on the right track, but just for the
record: calling an extension does not automatically release the GIL; you
need to explicitly call the Py_BEGIN_ALLOW_ THREADS macro. I was just
assuming that because you said "accelerato r" you were already releasing
the GIL.

The code is intended to run single thread so we never have done anything
about threading. It's a real pain. In a zope world we ought really to
give every request a fresh context ie each thread would see an entirely
'clean' python and then be forced to load its own fonts/resources. It
just sucks from the overall system viewpoint. How does one ensure that
each thread sees its own version of sys.modules etc etc or is that
considered a dumb question?
--
Robin Becker