473,231 Members | 1,995 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,231 software developers and data experts.

Multiple independent Python interpreters in a C/C++ program?

This question was posed to me today. Given a C/C++ program we can clearly
embed a Python interpreter in it. Is it possible to fire up multiple
interpreters in multiple threads? For example:

C++ main
thread 1
Py_Initialize()
thread 2
Py_Initialize()

Do I wind up with two completely independent interpreters, one per thread?
I'm thinking this doesn't work (there are bits which aren't thread-safe and
are only protected by the GIL), but wanted to double-check to be sure.

Thanks,

Skip
Apr 11 '08 #1
7 12351
Diez B. Roggisch schrieb:
AFAIK there was a thread a few month ago that stated that this is
actually possible - but mostly 3rd-party-c-extension aren't capable of
supporting that. Martin von Loewis had a word in that, maybe googling
with that in mind reveals the discussion.

And of course its a *bad* idea to pass objects between threads...
http://www.python.org/dev/peps/pep-3121/

Christian

Jun 27 '08 #2
On Apr 11, 10:24 am, s...@pobox.com wrote:
This question was posed to me today. Given a C/C++ program we can clearly
embed a Python interpreter in it. Is it possible to fire up multiple
interpreters in multiple threads? For example:

C++ main
thread 1
Py_Initialize()
thread 2
Py_Initialize()

Do I wind up with two completely independent interpreters, one per thread?
I'm thinking this doesn't work (there are bits which aren't thread-safe and
are only protected by the GIL), but wanted to double-check to be sure.
Since they interpreters would never truly be separate, I think it's
best to bite the bullet and use multiple threads within one
interpreter. The only really difference is pure python modules aren't
duplicated, but why would you need that? Surely not for any kind of
security.
Jun 27 '08 #3
On Apr 11, 6:24 pm, s...@pobox.com wrote:
Do I wind up with two completely independent interpreters, one per thread?
I'm thinking this doesn't work (there are bits which aren't thread-safe and
are only protected by the GIL), but wanted to double-check to be sure.
You can create a new interpreter with a call to Py_NewInterpreter.

However, the GIL is a global object for the process. If you have more
than two interpreters in the process, they share the same GIL.

In tcl, each thread has its own interpreter instance and no GIL is
shared. This circumvents most of the problems with a global GIL.

In theory, a GIL private to each interpreter would make Python more
scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache's
mod_python, is claimed to depend on this behaviour.
Jun 27 '08 #4
On Apr 11, 6:24 pm, s...@pobox.com wrote:
Do I wind up with two completely independent interpreters, one per thread?
I'm thinking this doesn't work (there are bits which aren't thread-safe and
are only protected by the GIL), but wanted to double-check to be sure.
You can create a new subinterpreter with a call to Py_NewInterpreter.
You get a nwe interpreter, but not an independent one. The GIL is a
global object for the process. If you have more than one interpreter
in the process, they share the same GIL.

In tcl, each thread has its own interpreter instance and no GIL is
shared. This circumvents most of the problems with a global GIL.

In theory, a GIL private to each (sub)interpreter would make Python
more scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache's
mod_python, is claimed to depend on this behaviour.

Jun 27 '08 #5
On Apr 12, 7:05 pm, sturlamolden <sturlamol...@yahoo.nowrote:
In theory, a GIL private to each (sub)interpreter would make Python
more scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache's
mod_python, is claimed to depend on this behaviour.
I just looked into the reason why ctypes, mod_python, etc. depend on a
shared GIL. Well ... PyGILState_Ensure() does not take an argument, so
it does not know which interpreter's GIL to acquire. Duh!

The sad fact is, the functions in Python's C API does not take the
interpreter as an argument, so they default to the one that is
currently active (i.e. the one that PyThreadState_Get()->interp points
to). This is unlike Java's JNI, in which all functions take the
"environment" (a Java VM instance) as the first argument.

IMHO, I consider this a major design flaw in Python's C API. In a more
well thought API, PyGILState_Ensure would take the interpreter
returned by Py_NewInterpreter as an argument, and thus know the
interpreter with which to synchronize.

I complained about this before, but the answer I got was that the
simplified GIL API would not work if interpreters has a separate GIL.
Obviously it would not work as long as the PyGILState_Ensure does not
take any arguments. The lack of a leading VM argument in
PyGILState_Ensure, and almost every function in Python's C API, is the
heart of the problem.



Jun 27 '08 #6
On Apr 13, 3:05*am, sturlamolden <sturlamol...@yahoo.nowrote:
On Apr 11, 6:24 pm, s...@pobox.com wrote:
Do I wind up with two completely independent interpreters, one per thread?
I'm thinking this doesn't work (there are bits which aren't thread-safe and
are only protected by the GIL), but wanted to double-check to be sure.

You can create a new subinterpreter with a call to Py_NewInterpreter.
You get a nwe interpreter, but not an independent one. The GIL is a
global object for the process. If you have more than one interpreter
in the process, they share the same GIL.

In tcl, each thread has its own interpreter instance and no GIL is
shared. This circumvents most of the problems with a global GIL.

In theory, a GIL private to each (sub)interpreter would make Python
more scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache'smod_python, is claimed to depend on this behaviour.
I wouldn't use mod_python as a good guide on how to do this as it
doesn't properly use PyGILState_Ensure() for main interpreter like it
should. If you want an example of how to do this, have a look at code
for mod_wsgi instead. If you want it to work for Python 3.0 as well as
Python 2.X, make sure you look at mod_wsgi source code from subversion
repository trunk as tweak had to be made to source to support Python
3.0. This is because in Python 3.0 it is no longer sufficient to hold
only the GIL when using string/unicode functions, you also need a
proper thread state to be active now.

Do note that although multiple sub interpreters can be made to work,
destroying sub interpreters within the context of a running process,
ie., before the process ends, can be a cause for various problems with
third party C extension modules and thus would advise that once a sub
interpreter has been created, you keep it and use it for the life of
the process.

Graham
Jun 27 '08 #7
On Apr 12, 2:02 pm, sturlamolden <sturlamol...@yahoo.nowrote:
On Apr 12, 7:05 pm, sturlamolden <sturlamol...@yahoo.nowrote:
In theory, a GIL private to each (sub)interpreter would make Python
more scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache's
mod_python, is claimed to depend on this behaviour.

I just looked into the reason why ctypes, mod_python, etc. depend on a
shared GIL. Well ... PyGILState_Ensure() does not take an argument, so
it does not know which interpreter's GIL to acquire. Duh!

The sad fact is, the functions in Python's C API does not take the
interpreter as an argument, so they default to the one that is
currently active (i.e. the one that PyThreadState_Get()->interp points
to). This is unlike Java's JNI, in which all functions take the
"environment" (a Java VM instance) as the first argument.

IMHO, I consider this a major design flaw in Python's C API. In a more
well thought API, PyGILState_Ensure would take the interpreter
returned by Py_NewInterpreter as an argument, and thus know the
interpreter with which to synchronize.

I complained about this before, but the answer I got was that the
simplified GIL API would not work if interpreters has a separate GIL.
Obviously it would not work as long as the PyGILState_Ensure does not
take any arguments. The lack of a leading VM argument in
PyGILState_Ensure, and almost every function in Python's C API, is the
heart of the problem.
Alternatively, the multiple interpreter API could be ripped out, and
you could just use separate threads. Do you have a use case that
requires it?
Jun 27 '08 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Vbfoo Bar | last post by:
Hello, To implement a Zope intranet on a linux RedHat ES 3, I had to install a Python 2.3.4 that I have compiled myself. I would like to compare the respective performance of this python...
4
by: logicalfeline | last post by:
How do you prevent multiple instances of a program from running? Cat
11
by: Clark Stevens | last post by:
I just finished a WinForms app in VB.NET. I want to allow the user to be able to run multiple instances of the program like you can with Notepad and Wordpad. The way it is now, once I run the...
8
by: =?iso-8859-1?B?QW5kcuk=?= | last post by:
I would like to find out how I can launch an independent Python program from existing one in a cross-platform way. The result I am after is that a new terminal window should open (for io...
3
by: Marcin Kalicinski | last post by:
How do I use multiple Python interpreters within the same process? I know there's a function Py_NewInterpreter. However, how do I use functions like Py_RunString etc. with it? They don't take any...
0
by: Stanley | last post by:
For a multiple independent form application of VC.net 2003, how to program a button in Form 2 to make it have the ability to jump back to the starting form ?
1
by: Davy | last post by:
Hi all, How to make a standalone Python/Tk program(e.g. exe file on Windows)? Any suggestions are welcome! Best regards, Davy
8
by: akineko | last post by:
Hello everyone, This may not be a Python specific challenge. I have a GUI program written in Python + Tkinter. It works very well. Now, I would like to start it from a shell script. As my...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.