473,714 Members | 2,264 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Multithreaded C API Python questions

Hi

A couple of mulithreaded C API python questions:

I) The PyGILState_Ensu re() simply ensures python api call ability, it
doesnt actually lock the GIL, right?

PyGILState_STAT E gstate;
gstate = PyGILState_Ensu re();

II) Am I required to lock the GIL prior to running any python functions
(in a threaded app)?

PyThreadState *pts = PyGILState_GetT hisThreadState( );
PyEval_AcquireT hread(pts);
PyObject_CallOb ject(...);

III) Shouldn't the GIL be released after 100 bytecodes or so to enable
other waiting threads to get the GIL?

I'm unable to get access to python as long as another python call is
executing. The PyEval_AcquireT hread() call blocks until the first call
returns. I was hoping that the python system itself would release the
GIL after some execution, but it itsnt.

I am dependent upon the ability to have to threads executing in python
land at the same time. How can this be done?
Regards,
Svein Seldal
Nov 9 '06 #1
9 5830
Svein Seldal wrote:
Hi

A couple of mulithreaded C API python questions:

I) The PyGILState_Ensu re() simply ensures python api call ability, it
doesnt actually lock the GIL, right?

PyGILState_STAT E gstate;
gstate = PyGILState_Ensu re();

II) Am I required to lock the GIL prior to running any python functions
(in a threaded app)?

PyThreadState *pts = PyGILState_GetT hisThreadState( );
PyEval_AcquireT hread(pts);
PyObject_CallOb ject(...);

III) Shouldn't the GIL be released after 100 bytecodes or so to enable
other waiting threads to get the GIL?

I'm unable to get access to python as long as another python call is
executing. The PyEval_AcquireT hread() call blocks until the first call
returns. I was hoping that the python system itself would release the
GIL after some execution, but it itsnt.

I am dependent upon the ability to have to threads executing in python
land at the same time. How can this be done?
PyGILState_Ensu re/Release guarantees to establish the GIL - even if it was already held (useful if you deal with complex call ordering/dependencies)
PyObject_CallOb ject(...) calls Python code. Thus the interpreter will switch and do as usuall during that.
In your/C/system... code you are responsible to release the GIL or not to enable other Python threads (and prevent from deadlocks)
Usually you'd do this
* if you do time consuming C/system stuff
* or if the code can possibly renter Python through the system (e.g. when you call a Windows function which itself can create Windows messages to be routed back into Python message handlers)
-robert
Nov 9 '06 #2
robert wrote:
PyGILState_Ensu re/Release guarantees to establish the GIL - even if it
was already held (useful if you deal with complex call
ordering/dependencies)
I understand this to mean that I dont need to explicitly lock the GIL
(with PyEval_AcquireL ock() or PyEval_AcquireT hread()).

Well I cant figure out the PyGILState_Ensu re() because if I only use
this function to establish the GIL, when calling python, python will
shortly after crash with "Fatal Python error: ceval: tstate mix-up".
This happens consistently when the main app and the extra thread has
called python and both are busy executing python code.
PyObject_CallOb ject(...) calls Python code. Thus the interpreter
will switch and do as usuall during that.
BTW What do you mean by switch?
In your/C/system... code you are responsible to release the GIL or not
to enable other Python threads (and prevent from deadlocks) Usually
you'd do this * if you do time consuming C/system stuff
* or if the code can possibly renter Python through the system (e.g.
when you call a Windows function which itself can create Windows
messages to be routed back into Python message handlers)
The main problem is that not done this way, it's the other way around.
My main C app will call a python function which will be a lengthy time
consuming process.

The main C app will call python and it will never return from
PyObject_CallOb ject(...), while the extra thread should call a py
function to deliver occational messages into C.
Svein
Nov 9 '06 #3
Svein Seldal wrote:
I'm unable to get access to python as long as another python call is
executing. The PyEval_AcquireT hread() call blocks until the first call
returns. I was hoping that the python system itself would release the
GIL after some execution, but it itsnt.

I am dependent upon the ability to have to threads executing in python
land at the same time. How can this be done?
Are you creating your threads through python or through C code? If you
are only creating it through C code, then make sure you initialize the
GIL with the following call at the beginning of your application:

PyEval_InitThre ads()

From my experience, calling PyGILState_Ensu re() was enough. I didn't
need to call any extra threading functions. But make sure that every
call to PyGILState_Ensu re() is matched with a call to PyGILState_Rele ase()

-Farshid
Nov 9 '06 #4
Svein Seldal wrote:
robert wrote:
>PyGILState_Ens ure/Release guarantees to establish the GIL - even if it
was already held (useful if you deal with complex call
ordering/dependencies)

I understand this to mean that I dont need to explicitly lock the GIL
(with PyEval_AcquireL ock() or PyEval_AcquireT hread()).

Well I cant figure out the PyGILState_Ensu re() because if I only use
this function to establish the GIL, when calling python, python will
shortly after crash with "Fatal Python error: ceval: tstate mix-up".
This happens consistently when the main app and the extra thread has
called python and both are busy executing python code.
Do did't tell enough to see what you want to do on the big picture.

usually you create a thread through Python.
If you boot Python at all from C you have to do
PyEval_InitThre ads
or maybe more simple use the high level layer PyRun_...

If you boot a new Python thread manually or a separated interpreter (with separated module state) from outside
the you have to also do:

(PyInterpreterS tate* PyInterpreterSt ate_New() )
PyThreadState* PyThreadState_N ew( PyInterpreterSt ate *interp)
void PyEval_AcquireT hread( PyThreadState *tstate)
If you are already in a Python thread but don't for some reason not know the current thread state:
PyThreadState* PyThreadState_G et( )
but usually you release with PyThreadState* PyEval_SaveThre ad( ) / Py_BEGIN_ALLOW_ THREADS
>PyObject_CallO bject(...) calls Python code. Thus the interpreter
will switch and do as usuall during that.

BTW What do you mean by switch?
the automatic scheduling every sys.getcheckint erval() etc.. you mentioned
You'd only have to take care for that, if your C-code does things (e.g. calling python basic funcs) for a long time and doesn't release the lock with Py_BEGIN_ALLOW_ THREADS ...
The main problem is that not done this way, it's the other way around.
My main C app will call a python function which will be a lengthy time
consuming process.

The main C app will call python and it will never return from
PyObject_CallOb ject(...), while the extra thread should call a py
function to deliver occational messages into C.
Maybe simply boot Python in the main thread (Py_Initialize( )) and run off Python possibly as simple as PyRun_String
and let Python do thread.start_ne w(...)
You'd not have to worry much about states.

( and later you'd probably even more simple do "python myapp.py" and expose your C app as extension module :-) )
-robert
Nov 9 '06 #5
"Svein Seldal" <"svein at seldal dot com">wrote:
8<---------------------------------------------------
I am dependent upon the ability to have to threads executing in python
land at the same time. How can this be done?
call time.sleep(0.00 1) in each, as well as the main thread, to politely give the
rest a chance....

Fiddle with the sleep times in your setup to get the best performance - and
don't be misled into thinking that faster is better - its not always true -
there comes a point where the overhead of swapping a thread consumes the total
resource of the process, and no work gets done.

- HTH

- Hendrik

Nov 10 '06 #6
Hi!

I think I've found the bug, but I need to confirm this behavior.

My findings is that if you use PyEval_InitThre ads(), it is crucial to
release the GIL with PyEval_ReleaseT hread() afterwards.

The API docs states that you can release the GIL with
PyEval_ReleaseL ock() or PyEval_ReleaseT hread() AFAICS.
http://docs.python.org/api/threads.html under docs of void
PyEval_InitThre ads().

However, if I do use PyEval_ReleaseL ock() it will crash shortly after
with "Fatal Python error: ceval: tstate mix-up" in a multithreaded C
environment.

If I use PyEval_ReleaseT hread() to release the GIL, my app seems stable.
I release the GIL like this:

PyThreadState *pts = PyGILState_GetT hisThreadState( );
PyEval_ReleaseT hread(pts);

Now, is this a feature or expected behavior in python (i.e. unclear API
documentation), or is this a bug in python itself?
Regards,
Svein
PS:

For reference I did something like this in pseudo-code:

Py_Initialize() ;
PyEval_InitThre ads();

other_py_inits( ); // Load py modules, etc.

PyEval_ReleaseL ock(); // <-- MAKES THE APP UNSTABLE

create_c_thread ();

PyGILState_STAT E gstate;
gstate = PyGILState_Ensu re();

call_py_functio n_main(); // Py main() wont return

PyGILState_Rele ase(gstate);
And the "main" of the C thread function looks like this:
while(1)
{
PyGILState_STAT E gstate;
gstate = PyGILState_Ensu re();

call_py_functio n_send();

PyGILState_Rele ase(gstate);
}
Nov 14 '06 #7
Svein Seldal wrote:

You seem to use the functions in very rude manner - wanting to force the GIL around at lowest level.
Better forget the word GIL and think about acquiring and releasing Threads.
For each thread wanting to execute Python stuff there has to be a thread state (ts). And then you have to enter/leave for executing Python by using the thread state.
PyEval_InitThre ads enters Python (incl. the GIL) first time.
To let Python free in main thread do
ts=PyEval_SaveT hread() // or more easy: Py_BEGIN_ALLOW_ THREADS

to reenter Python do simply:

PyEval_RestoreT hread(ts) // or: Py_END_ALLOW_TH READS

Forget all the low level PyGIL... functions.

when you really want another c thread to enter Python first time with no thread state existing through Python itsel (better make the thread in Python with thread.start_ne w ?), then you have to do once

ts_cthread = PyThreadState_N ew(interp)

then enter Python in this thread:

PyEval_AcquireT hread( ts_cthread )

to leave again:

PyEval_ReleaseT hread( ts_cthread )
If you just loop the call_py_functio n_send() mainly in this thread you probably better create the thread at all in Python and make the loop there. You probably stick too much to C-level at any price :-) Probably you just do a PyRun_xxxx in main thread and then everything else in Python, and expose C-parts for the thread-loop to Python as function (in other_py_inits) - where in the c-function you probably have the usual Py_BEGIN_ALLOW_ THREADS/Py_END_ALLOW_TH READS bracket during time consuming/system/reenter-endangered message stuff.
Robert


I think I've found the bug, but I need to confirm this behavior.

My findings is that if you use PyEval_InitThre ads(), it is crucial to
release the GIL with PyEval_ReleaseT hread() afterwards.

The API docs states that you can release the GIL with
PyEval_ReleaseL ock() or PyEval_ReleaseT hread() AFAICS.
http://docs.python.org/api/threads.html under docs of void
PyEval_InitThre ads().

However, if I do use PyEval_ReleaseL ock() it will crash shortly after
with "Fatal Python error: ceval: tstate mix-up" in a multithreaded C
environment.

If I use PyEval_ReleaseT hread() to release the GIL, my app seems stable.
I release the GIL like this:

PyThreadState *pts = PyGILState_GetT hisThreadState( );
PyEval_ReleaseT hread(pts);

Now, is this a feature or expected behavior in python (i.e. unclear API
documentation), or is this a bug in python itself?
Regards,
Svein
PS:

For reference I did something like this in pseudo-code:

Py_Initialize() ;
PyEval_InitThre ads();

other_py_inits( ); // Load py modules, etc.

PyEval_ReleaseL ock(); // <-- MAKES THE APP UNSTABLE

create_c_thread ();

PyGILState_STAT E gstate;
gstate = PyGILState_Ensu re();

call_py_functio n_main(); // Py main() wont return

PyGILState_Rele ase(gstate);
And the "main" of the C thread function looks like this:
while(1)
{
PyGILState_STAT E gstate;
gstate = PyGILState_Ensu re();

call_py_functio n_send();

PyGILState_Rele ase(gstate);
}
Nov 16 '06 #8
robert wrote:
Forget all the low level PyGIL... functions.'
Quite the contrary, I would say!

From the Py C API doc, I interpret the PyGIL... functions to be higher
leveled than eg. PyEval_SaveThre ad(). I've checked into the source of
python to find out what's really going on, and I've learnt a couple of
interesting facts. The PyGIL... are helper functions build over the
PyEval... functions, not the other way around.

A real beauty about the PyGILState_Ensu re() is the fact that if it's
called from a thread that is (yet) unknown to python, it will actually
create the thread state object for you (with PyThreadState_N ew) and
acquire the lock. Hence, I dont have to worry about the process of
swapping thread states with PyEval_xxxThrea d() functions. I ONLY need to
use PyGILState_Ensu re() prior to any py-ops.

From my previous post, I asked about the difference between
PyEval_ReleaseL ock() and PyEval_ReleaseT hread() and I've found one major
difference. Both release the GIL as the docs states, but the
PyEval_ReleaseT hread() saves the current thread state as well. When I
used PyEval_ReleaseL ock() this didnt happen, preventing proper saving of
the current thread, causing py crash when control were handed back to
this thread.

By using:

PyEval_InitThre ads();
py_ops();
PyThreadState *pts = PyGILState_GetT hisThreadState( );
PyEval_ReleaseT hread(pts);

And for each py-op later on (from arbitrary thread):

PyGILState_STAT E gstate;
gstate = PyGILState_Ensu re();
py_ops();
PyGILState_Rele ase(gstate);

Then you're home free. Everything related to threading is handled by
Py/API itself.
Probably you just do a PyRun_xxxx in main thread and then everything
else in Python, and expose C-parts for the thread-loop to Python as
function (in other_py_inits) - where in the c-function you probably have
the usual Py_BEGIN_ALLOW_ THREADS/Py_END_ALLOW_TH READS bracket during
time consuming/system/reenter-endangered message stuff.
The app i'm building is a plugin to a server system, and I cant control
nor remove any threads that the server uses. I am given one main thread
to run python forever, and messages that are to be delivered into python
are called from another.

I could do it like you propose: The data coming from my server will
arrive into a c-function called by a server thread. The c-part of the
thread-loop would then be run as another thread (started from python).
Its fully feasible, yet the challenge is to make proper data sync-ing
between these two threads.

Well, I think I've achieved what I wanted. Python seems apparently
stable, and reading from the python sources, I cant see any immediate
reasons why it shouldn't.

Thanks for letting me closer to a working solution!
Regads,
Svein
Nov 16 '06 #9
Svein Seldal wrote:
A real beauty about the PyGILState_Ensu re() is the fact that if it's
called from a thread that is (yet) unknown to python, it will actually
create the thread state object for you (with PyThreadState_N ew) and
acquire the lock. Hence, I dont have to worry about the process of
swapping thread states with PyEval_xxxThrea d() functions. I ONLY need to
use PyGILState_Ensu re() prior to any py-ops.
Interesting - the PyGILState_Ensu re was new to py2.3. I didn't notice so far, that it will even auto-create a fresh tread state (from default InterpreterStat e only). From the docs its not that clear.

The confusion in usage is that one has to do

Py_Initialize()
PyEval_InitThre ads()
ts=PyEval_SaveT hread()

in order to start clean for further pure "meta-high-level" PyGILState_Ensu re/Release.

They obviously meant PyGILState_Ensu re to make live easier, but the discussion shows, it adds additional worries probably forcing each user into investigating the Python C-code :-)

There should probably a extra init function for this meta-high-level usage like ..

PyGILState_PyIn itThreads() // which would not pre-acquire the lock! - as anyone expects to enter then with PyGILState_Ensu re

.. and these 3 Functions grouped together in the docs for an easy-to-use meta-high-level thread state API.

The current thread state API doc, as you read it from top to bottom now, is in fact totally confusing for anyone who didn't develop Python himself :-)
Robert
Nov 17 '06 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
2283
by: paul h | last post by:
hi there, i'm learning to use python for cgi, and i've hit a few road blocks. basically, i need to know how to achieve the following things using python, on linux. create a new directory, and sub-directory. so if the user inputs "HELLO", i can create a dir called HELLO, and then some other dirs in there. i also need to know how i can write various things into a file. the following doesnt seem to work for me:
3
1299
by: Eli Daniel | last post by:
Hi, I am relative new to Python. Please tell me if the following is possible. I have a command line shell written in C. This shell executes some commands which I would like to import to the python shell. Here is an example command from my shell: openconnection www.yahoo.com 12 2000 So I've created a dll (myshell_d.dll) according to the manual
0
1008
by: Derek Allen | last post by:
I'm using SWIG to generate glue code for my embedded/extended python app. There are a couple of very frustrating issues that I would like to see if anyone else has solutions for: - Once I have created and initialized a module with SWIG I can call it by executing a python file with no problems. However I would like to be able to type in a single-line python function and have it execute from my module. One line cannot contain both "import...
3
5072
by: Leonard J. Reder | last post by:
Hello list, I have been searching on the web for a while now for a specific Python implementation of an FSM. More specifically what I am looking for is a Python implementation of the so called UML Harel State Machine notion of a state machine. These are multi-threaded capable state machines with hierarchical representation capability (nested instances of state machines). They also have a unique conditional trigger symbol scheme for...
0
1427
by: adsheehan | last post by:
Hi, I am embedding Python into a multi-threaded C++ application runnig on Solaris and need urgent clarification on the embedding architecture and its correct usage (as I am experience weird behaviors). Can anyone clarify:
1
1775
by: notanotheridiot | last post by:
Hi- I'm trying to exec some arbitrary code in one thread of an application and read anything it prints to stdout or stderr in another thread. My question is how? I've tried changing sys.stdout, but that changes stdout for the whole application, not just that one thread, which means that any status updates that I DO want printed to stdout don't get printed.
21
1561
by: nateastle | last post by:
I have a simple assignment for school but am unsure where to go. The assignment is to read in a text file, split out the words and say which line each word appears in alphabetical order. I have the basic outline of the program done which is: def Xref(filename): try: fp = open(filename, "r") lines = fp.readlines() fp.close()
3
1678
by: johnny | last post by:
I have python script does ftp download in a multi threaded way. Each thread downloads a file, close the file, calls the comman line to convert the .doc to pdf. Command line should go ahead and convert the file. My question is, when each thread calls the command line, does one command line process all the request, or each thread creates a one command line process for themselves and executes the command? For some reason I am getting "File...
6
1131
by: LessPaul | last post by:
I recently discovered Python and see it as a great language to use for personal projects (and more). I made my living for over a decade as a coder in C, C++, ADA, Fortran, and Assembly before moving to systems engineering. I'm now retired, and would love to code again. I see Python as the perfect language to get a good program working in a short time. My question is in regard to GUI platforms. My primary target would be Windows, but I...
0
8808
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9316
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9177
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9021
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7954
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6637
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5951
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4727
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3159
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.