473,699 Members | 2,912 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Py_NewInterpret er(), is this a bug in the python core?

/*
Is this a bug in Py_NewInterpret er?

The function below "MyThread" is instantiated from a windows worker
thread, but I don't
think that is relevant.
(I can try this on a linux box, but I would have to compile a python
library with debugging
enabled.)

The following code fragment throws an exception in a debug version of
python:
*/

UINT MyThread(LPVOID lpParam)
{
{
cs.Lock(); // this is a CCriticalSectio n lock
if (!Py_IsInitiali zed())
{
Py_Initialize() ;
PyEval_InitThre ads();

// global pointer to the main PyThreadState object
mainThreadState = PyThreadState_G et();
PyEval_ReleaseL ock();
}
cs.Unlock();
}

ASSERT(Py_IsIni tialized());
ASSERT(PyEval_T hreadsInitializ ed());
ASSERT(mainThre adState);
threadnum++;

// get the global lock
PyEval_AcquireL ock();
PyGILState_STAT E gstate;
gstate = PyGILState_Ensu re(); // Is tis necessary?
PyThreadState_S wap(mainThreadS tate);
PyThreadState* nts = Py_NewInterpret er();

/*

The exception is thrown inside the above function call:
This statement tries to swap the new threadstate 'tstate' with the
current one
save_tstate = PyThreadState_S wap(tstate);

Inside PyThreadState_S wap the code uses another way
'PyGILState_Get ThisThreadState ()' to find the current thread state and
compares this with the newly set thread state.
Naturally you would expect the two to be equal but that test fails:
#if defined(Py_DEBU G) && defined(WITH_TH READ)
if (new) {
PyThreadState *check = PyGILState_GetT hisThreadState( );
if (check && check != new)
Py_FatalError(" Invalid thread state for this thread");
}
#endif

The variable 'check' looks as if it is the 'previous' thread state, as
if changing the thread state
is not been done properly. Py_FatalError is called and that's the end.

Is there a mistake in my code, or is there something wrong in how
Py_NewInterpret er is implemented?
Thanks

Martin

PS: Below the rest of my simple test worker thread function.
*/


ASSERT(nts == PyThreadState_G et());

// lock (already locked) - swap in thread state - swap out thread
state - unlock

init_testclass( );
int ret = 0;

ret = PyRun_SimpleStr ing("import sys");
ret = PyRun_SimpleStr ing("class redir:\n def __init__(self, id):\n
self.id = id\n def write(self, s):\n f = open('stdoutput s_%s.txt' %
self.id, 'a')\n f.write('%s: %s' % (self.id, s))\n f.close()\n");
char str[100];
sprintf(str,"r = redir('0x%x')", &nts);
ret = PyRun_SimpleStr ing(str);
ret = PyRun_SimpleStr ing("sys.stderr = r");
sprintf(str,"s = redir('0x%x')", &nts);
ret = PyRun_SimpleStr ing(str);
ret = PyRun_SimpleStr ing("sys.stdout = s");

ret = PyRun_SimpleStr ing("import testclass");
ret = PyRun_SimpleStr ing("t = testclass.testc lass()");
sprintf(str,"pr int 't = ', t ");
ret = PyRun_SimpleStr ing(str);
ret = PyRun_SimpleStr ing("print t.run(10)");
Py_EndInterpret er(nts);
PyGILState_Rele ase(gstate);
PyEval_ReleaseL ock();

return 0;
}

Jul 10 '06 #1
3 4244
freesteel schrieb:
/*
Is this a bug in Py_NewInterpret er?

The function below "MyThread" is instantiated from a windows worker
thread, but I don't
think that is relevant.
(I can try this on a linux box, but I would have to compile a python
library with debugging
enabled.)

The following code fragment throws an exception in a debug version of
python:
*/

UINT MyThread(LPVOID lpParam)
{
{
cs.Lock(); // this is a CCriticalSectio n lock
if (!Py_IsInitiali zed())
{
Py_Initialize() ;
PyEval_InitThre ads();

// global pointer to the main PyThreadState object
mainThreadState = PyThreadState_G et();
PyEval_ReleaseL ock();
}
cs.Unlock();
}

ASSERT(Py_IsIni tialized());
ASSERT(PyEval_T hreadsInitializ ed());
ASSERT(mainThre adState);
threadnum++;

// get the global lock
PyEval_AcquireL ock();
PyGILState_STAT E gstate;
gstate = PyGILState_Ensu re(); // Is tis necessary?
PyThreadState_S wap(mainThreadS tate);
PyThreadState* nts = Py_NewInterpret er();

/*

The exception is thrown inside the above function call:
This statement tries to swap the new threadstate 'tstate' with the
current one
save_tstate = PyThreadState_S wap(tstate);

Inside PyThreadState_S wap the code uses another way
'PyGILState_Get ThisThreadState ()' to find the current thread state and
compares this with the newly set thread state.
Naturally you would expect the two to be equal but that test fails:
#if defined(Py_DEBU G) && defined(WITH_TH READ)
if (new) {
PyThreadState *check = PyGILState_GetT hisThreadState( );
if (check && check != new)
Py_FatalError(" Invalid thread state for this thread");
}
#endif

The variable 'check' looks as if it is the 'previous' thread state, as
if changing the thread state
is not been done properly. Py_FatalError is called and that's the end.

Is there a mistake in my code, or is there something wrong in how
Py_NewInterpret er is implemented?
As far as I know, the PyGILState_... functions are incompatible with multiple
Python interpreters.

Thomas

Jul 10 '06 #2
Yes, I see that now in the documentation, which to me is quite
confusing.
So, how do you use python in a multithreaded environment, where for
example you want to run some embeded python code from a number of
different C threads?

This article: http://www.linuxjournal.com/article/3641 is quite good,
but must have been written before these PyGILState_* functions.

I only used Py_NewInterpret er to have a fresh 'import sys'.

Somebody enlighten me, please.

Martin

Jul 10 '06 #3
freesteel wrote:
Yes, I see that now in the documentation, which to me is quite
confusing.
So, how do you use python in a multithreaded environment, where for
example you want to run some embeded python code from a number of
different C threads?

This article: http://www.linuxjournal.com/article/3641 is quite good,
but must have been written before these PyGILState_* functions.

I only used Py_NewInterpret er to have a fresh 'import sys'.

Somebody enlighten me, please.

Martin
If you try to replicate the code from the linux journal
(http://www.linuxjournal.com/article/3641) and compile with Py_DEBUG
defined, when running it you will find that you get an exception and a
fatal error message from the python core. The exception is thrown from
pystate.c, line 306:
Py_FatalError(" Invalid thread state for this thread");

The exception is thrown in my understanding of the code because there
can only ever be one thread state. If this is true a function to swap
thread states seems rather pointless.
Now, reading up about how to call C API to python from a C thread I
find that with version 2.3 the function pair
PyGILState_Ensu re/PyGILState_Rele ase was introduced. This allows
'grabbing' the global interpreter lock, calling your embedded python
code and release the GIL at the end again.

Am I right in my understanding that the use of this PyGILState_* pair
is meant to 'replace' the prologue of creating a new thread state from
the main thread state, swapping it with the current state, doing your
Python/C API calls and then in an epilogue swap the previous thread
state back in, and deletie the now obsolete previously created thread
state?

At least that how I understand the motivation behind the introduction
of tPyGILState_*, read here:
http://www.python.org/dev/peps/pep-0311/

Now, I tried to use this mechanism, but I am not really successful with
it. Basically, in my C thread, I wrap a number of calls to embedded
python (a few PyRun_SimpleStr ing calls, nothing really fancy) with
PyGILState_Ensu re and PyGILState_Rele ase. The first C thread also
initializes python as well as initializes threading:
Py_Initialize() ;
PyEval_InitThre ads();

(If it helps I can post my whole code here).

When I test this I get deadlocks in most cases after the first thread
has called PyGILState_Rele ase. All other threads at that point 'freeze'
and never return from whatever Python/C API function they are in. I
don't understand why. Moreover, this behavious is not always
reproducable, sometimes the code works as intended. Must be some kind
of race condition? Who can help?

I read that the problem of calling python from a multithreaded
application per thread is very difficult, but a few projects have
managed to solve it, but now we have these PyGILStates and all is much
easier?

Martin

Jul 13 '06 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
2146
by: Christoph Wiedemann | last post by:
Hello, i have some trouble to understand the Py_NewInterpreter API function. The docs state, that Py_NewInterpreter returns a new PyThreadState instance. One can switch between interpreters by swapping thread states. I've used that, and it worked well, until i decided to use the PyGILState_* API functions. After digging a while, i found, that these functions assume, that there is only one PyThreadState instance per thread. However,...
62
4409
by: robert | last post by:
I'd like to use multiple CPU cores for selected time consuming Python computations (incl. numpy/scipy) in a frictionless manner. Interprocess communication is tedious and out of question, so I thought about simply using a more Python interpreter instances (Py_NewInterpreter) with extra GIL in the same process. I expect to be able to directly push around Python Object-Trees between the 2 (or more) interpreters by doing some careful locking. ...
0
2131
Bulldog
by: Bulldog | last post by:
I am working on a C++ win32 DLL which has calls to python, mainly import some modules, call some functions and return some values. Right now I am using the Py_Initialize() and Py_Uninitialize() functions to get to the embedded interpreter. I am having an issue with making repeated calls to the function(s) that my DLL exports, namely I get an error trying to import a python module that I imported the previous time I placed a call to that DLL. ...
0
8623
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9054
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8941
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8896
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7784
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
5879
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4390
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
3071
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2362
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.