Multiple independent Python interpreters in a C/C++ program?

[skip]

This question was posed to me today. Given a C/C++ program we can clearly
embed a Python interpreter in it. Is it possible to fire up multiple
interpreters in multiple threads? For example:

C++ main
thread 1
Py_Initialize()
thread 2
Py_Initialize()

Do I wind up with two completely independent interpreters, one per thread?
I'm thinking this doesn't work (there are bits which aren't thread-safe and
are only protected by the GIL), but wanted to double-check to be sure.

Thanks,

Skip

[Christian Heimes]

Diez B. Roggisch schrieb:

AFAIK there was a thread a few month ago that stated that this is
actually possible - but mostly 3rd-party-c-extension aren't capable of
supporting that. Martin von Loewis had a word in that, maybe googling
with that in mind reveals the discussion.

And of course its a *bad* idea to pass objects between threads...

http://www.python.org/dev/peps/pep-3121/

Christian

[Rhamphoryncus]

On Apr 11, 10:24 am, [e-mail] wrote:

This question was posed to me today. Given a C/C++ program we can clearly
embed a Python interpreter in it. Is it possible to fire up multiple
interpreters in multiple threads? For example:

C++ main
thread 1
Py_Initialize()
thread 2
Py_Initialize()

Do I wind up with two completely independent interpreters, one per thread?
I'm thinking this doesn't work (there are bits which aren't thread-safe and
are only protected by the GIL), but wanted to double-check to be sure.

Since they interpreters would never truly be separate, I think it's
best to bite the bullet and use multiple threads within one
interpreter. The only really difference is pure python modules aren't
duplicated, but why would you need that? Surely not for any kind of
security.

[sturlamolden]

On Apr 11, 6:24 pm, [e-mail] wrote:

Do I wind up with two completely independent interpreters, one per thread?
I'm thinking this doesn't work (there are bits which aren't thread-safe and
are only protected by the GIL), but wanted to double-check to be sure.

You can create a new interpreter with a call to Py_NewInterpreter.

However, the GIL is a global object for the process. If you have more
than two interpreters in the process, they share the same GIL.

In tcl, each thread has its own interpreter instance and no GIL is
shared. This circumvents most of the problems with a global GIL.

In theory, a GIL private to each interpreter would make Python more
scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache's
mod_python, is claimed to depend on this behaviour.

[sturlamolden]

On Apr 11, 6:24 pm, [e-mail] wrote:

Do I wind up with two completely independent interpreters, one per thread?
I'm thinking this doesn't work (there are bits which aren't thread-safe and
are only protected by the GIL), but wanted to double-check to be sure.

You can create a new subinterpreter with a call to Py_NewInterpreter.
You get a nwe interpreter, but not an independent one. The GIL is a
global object for the process. If you have more than one interpreter
in the process, they share the same GIL.

In tcl, each thread has its own interpreter instance and no GIL is
shared. This circumvents most of the problems with a global GIL.

In theory, a GIL private to each (sub)interpreter would make Python
more scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache's
mod_python, is claimed to depend on this behaviour.

[sturlamolden]

On Apr 12, 7:05 pm, sturlamolden <[e-mail]:

In theory, a GIL private to each (sub)interpreter would make Python
more scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache's
mod_python, is claimed to depend on this behaviour.

I just looked into the reason why ctypes, mod_python, etc. depend on a
shared GIL. Well ... PyGILState_Ensure() does not take an argument, so
it does not know which interpreter's GIL to acquire. Duh!

The sad fact is, the functions in Python's C API does not take the
interpreter as an argument, so they default to the one that is
currently active (i.e. the one that PyThreadState_Get()->interp points
to). This is unlike Java's JNI, in which all functions take the
"environment" (a Java VM instance) as the first argument.

IMHO, I consider this a major design flaw in Python's C API. In a more
well thought API, PyGILState_Ensure would take the interpreter
returned by Py_NewInterpreter as an argument, and thus know the
interpreter with which to synchronize.

I complained about this before, but the answer I got was that the
simplified GIL API would not work if interpreters has a separate GIL.
Obviously it would not work as long as the PyGILState_Ensure does not
take any arguments. The lack of a leading VM argument in
PyGILState_Ensure, and almost every function in Python's C API, is the
heart of the problem.

[Graham Dumpleton]

On Apr 13, 3:05*am, sturlamolden <[e-mail]:

On Apr 11, 6:24 pm, [e-mail] wrote:

Do I wind up with two completely independent interpreters, one per thread?
I'm thinking this doesn't work (there are bits which aren't thread-safe and
are only protected by the GIL), but wanted to double-check to be sure.

You can create a new subinterpreter with a call to Py_NewInterpreter.
You get a nwe interpreter, but not an independent one. The GIL is a
global object for the process. If you have more than one interpreter
in the process, they share the same GIL.

In tcl, each thread has its own interpreter instance and no GIL is
shared. This circumvents most of the problems with a global GIL.

In theory, a GIL private to each (sub)interpreter would make Python
more scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache'smod_python, is claimed to depend on this behaviour.

I wouldn't use mod_python as a good guide on how to do this as it
doesn't properly use PyGILState_Ensure() for main interpreter like it
should. If you want an example of how to do this, have a look at code
for mod_wsgi instead. If you want it to work for Python 3.0 as well as
Python 2.X, make sure you look at mod_wsgi source code from subversion
repository trunk as tweak had to be made to source to support Python
3.0. This is because in Python 3.0 it is no longer sufficient to hold
only the GIL when using string/unicode functions, you also need a
proper thread state to be active now.

Do note that although multiple sub interpreters can be made to work,
destroying sub interpreters within the context of a running process,
ie., before the process ends, can be a cause for various problems with
third party C extension modules and thus would advise that once a sub
interpreter has been created, you keep it and use it for the life of
the process.

Graham

[Rhamphoryncus]

On Apr 12, 2:02 pm, sturlamolden <[e-mail]:

On Apr 12, 7:05 pm, sturlamolden <[e-mail]:

In theory, a GIL private to each (sub)interpreter would make Python
more scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache's
mod_python, is claimed to depend on this behaviour.

I just looked into the reason why ctypes, mod_python, etc. depend on a
shared GIL. Well ... PyGILState_Ensure() does not take an argument, so
it does not know which interpreter's GIL to acquire. Duh!

The sad fact is, the functions in Python's C API does not take the
interpreter as an argument, so they default to the one that is
currently active (i.e. the one that PyThreadState_Get()->interp points
to). This is unlike Java's JNI, in which all functions take the
"environment" (a Java VM instance) as the first argument.

IMHO, I consider this a major design flaw in Python's C API. In a more
well thought API, PyGILState_Ensure would take the interpreter
returned by Py_NewInterpreter as an argument, and thus know the
interpreter with which to synchronize.

I complained about this before, but the answer I got was that the
simplified GIL API would not work if interpreters has a separate GIL.
Obviously it would not work as long as the PyGILState_Ensure does not
take any arguments. The lack of a leading VM argument in
PyGILState_Ensure, and almost every function in Python's C API, is the
heart of the problem.

Alternatively, the multiple interpreter API could be ripped out, and
you could just use separate threads. Do you have a use case that
requires it?