473,320 Members | 2,029 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Py_NewInterpreter(), is this a bug in the python core?

/*
Is this a bug in Py_NewInterpreter?

The function below "MyThread" is instantiated from a windows worker
thread, but I don't
think that is relevant.
(I can try this on a linux box, but I would have to compile a python
library with debugging
enabled.)

The following code fragment throws an exception in a debug version of
python:
*/

UINT MyThread(LPVOID lpParam)
{
{
cs.Lock(); // this is a CCriticalSection lock
if (!Py_IsInitialized())
{
Py_Initialize();
PyEval_InitThreads();

// global pointer to the main PyThreadState object
mainThreadState = PyThreadState_Get();
PyEval_ReleaseLock();
}
cs.Unlock();
}

ASSERT(Py_IsInitialized());
ASSERT(PyEval_ThreadsInitialized());
ASSERT(mainThreadState);
threadnum++;

// get the global lock
PyEval_AcquireLock();
PyGILState_STATE gstate;
gstate = PyGILState_Ensure(); // Is tis necessary?
PyThreadState_Swap(mainThreadState);
PyThreadState* nts = Py_NewInterpreter();

/*

The exception is thrown inside the above function call:
This statement tries to swap the new threadstate 'tstate' with the
current one
save_tstate = PyThreadState_Swap(tstate);

Inside PyThreadState_Swap the code uses another way
'PyGILState_GetThisThreadState()' to find the current thread state and
compares this with the newly set thread state.
Naturally you would expect the two to be equal but that test fails:
#if defined(Py_DEBUG) && defined(WITH_THREAD)
if (new) {
PyThreadState *check = PyGILState_GetThisThreadState();
if (check && check != new)
Py_FatalError("Invalid thread state for this thread");
}
#endif

The variable 'check' looks as if it is the 'previous' thread state, as
if changing the thread state
is not been done properly. Py_FatalError is called and that's the end.

Is there a mistake in my code, or is there something wrong in how
Py_NewInterpreter is implemented?
Thanks

Martin

PS: Below the rest of my simple test worker thread function.
*/


ASSERT(nts == PyThreadState_Get());

// lock (already locked) - swap in thread state - swap out thread
state - unlock

init_testclass();
int ret = 0;

ret = PyRun_SimpleString("import sys");
ret = PyRun_SimpleString("class redir:\n def __init__(self, id):\n
self.id = id\n def write(self, s):\n f = open('stdoutputs_%s.txt' %
self.id, 'a')\n f.write('%s: %s' % (self.id, s))\n f.close()\n");
char str[100];
sprintf(str,"r = redir('0x%x')", &nts);
ret = PyRun_SimpleString(str);
ret = PyRun_SimpleString("sys.stderr = r");
sprintf(str,"s = redir('0x%x')", &nts);
ret = PyRun_SimpleString(str);
ret = PyRun_SimpleString("sys.stdout = s");

ret = PyRun_SimpleString("import testclass");
ret = PyRun_SimpleString("t = testclass.testclass()");
sprintf(str,"print 't = ', t ");
ret = PyRun_SimpleString(str);
ret = PyRun_SimpleString("print t.run(10)");
Py_EndInterpreter(nts);
PyGILState_Release(gstate);
PyEval_ReleaseLock();

return 0;
}

Jul 10 '06 #1
3 4223
freesteel schrieb:
/*
Is this a bug in Py_NewInterpreter?

The function below "MyThread" is instantiated from a windows worker
thread, but I don't
think that is relevant.
(I can try this on a linux box, but I would have to compile a python
library with debugging
enabled.)

The following code fragment throws an exception in a debug version of
python:
*/

UINT MyThread(LPVOID lpParam)
{
{
cs.Lock(); // this is a CCriticalSection lock
if (!Py_IsInitialized())
{
Py_Initialize();
PyEval_InitThreads();

// global pointer to the main PyThreadState object
mainThreadState = PyThreadState_Get();
PyEval_ReleaseLock();
}
cs.Unlock();
}

ASSERT(Py_IsInitialized());
ASSERT(PyEval_ThreadsInitialized());
ASSERT(mainThreadState);
threadnum++;

// get the global lock
PyEval_AcquireLock();
PyGILState_STATE gstate;
gstate = PyGILState_Ensure(); // Is tis necessary?
PyThreadState_Swap(mainThreadState);
PyThreadState* nts = Py_NewInterpreter();

/*

The exception is thrown inside the above function call:
This statement tries to swap the new threadstate 'tstate' with the
current one
save_tstate = PyThreadState_Swap(tstate);

Inside PyThreadState_Swap the code uses another way
'PyGILState_GetThisThreadState()' to find the current thread state and
compares this with the newly set thread state.
Naturally you would expect the two to be equal but that test fails:
#if defined(Py_DEBUG) && defined(WITH_THREAD)
if (new) {
PyThreadState *check = PyGILState_GetThisThreadState();
if (check && check != new)
Py_FatalError("Invalid thread state for this thread");
}
#endif

The variable 'check' looks as if it is the 'previous' thread state, as
if changing the thread state
is not been done properly. Py_FatalError is called and that's the end.

Is there a mistake in my code, or is there something wrong in how
Py_NewInterpreter is implemented?
As far as I know, the PyGILState_... functions are incompatible with multiple
Python interpreters.

Thomas

Jul 10 '06 #2
Yes, I see that now in the documentation, which to me is quite
confusing.
So, how do you use python in a multithreaded environment, where for
example you want to run some embeded python code from a number of
different C threads?

This article: http://www.linuxjournal.com/article/3641 is quite good,
but must have been written before these PyGILState_* functions.

I only used Py_NewInterpreter to have a fresh 'import sys'.

Somebody enlighten me, please.

Martin

Jul 10 '06 #3
freesteel wrote:
Yes, I see that now in the documentation, which to me is quite
confusing.
So, how do you use python in a multithreaded environment, where for
example you want to run some embeded python code from a number of
different C threads?

This article: http://www.linuxjournal.com/article/3641 is quite good,
but must have been written before these PyGILState_* functions.

I only used Py_NewInterpreter to have a fresh 'import sys'.

Somebody enlighten me, please.

Martin
If you try to replicate the code from the linux journal
(http://www.linuxjournal.com/article/3641) and compile with Py_DEBUG
defined, when running it you will find that you get an exception and a
fatal error message from the python core. The exception is thrown from
pystate.c, line 306:
Py_FatalError("Invalid thread state for this thread");

The exception is thrown in my understanding of the code because there
can only ever be one thread state. If this is true a function to swap
thread states seems rather pointless.
Now, reading up about how to call C API to python from a C thread I
find that with version 2.3 the function pair
PyGILState_Ensure/PyGILState_Release was introduced. This allows
'grabbing' the global interpreter lock, calling your embedded python
code and release the GIL at the end again.

Am I right in my understanding that the use of this PyGILState_* pair
is meant to 'replace' the prologue of creating a new thread state from
the main thread state, swapping it with the current state, doing your
Python/C API calls and then in an epilogue swap the previous thread
state back in, and deletie the now obsolete previously created thread
state?

At least that how I understand the motivation behind the introduction
of tPyGILState_*, read here:
http://www.python.org/dev/peps/pep-0311/

Now, I tried to use this mechanism, but I am not really successful with
it. Basically, in my C thread, I wrap a number of calls to embedded
python (a few PyRun_SimpleString calls, nothing really fancy) with
PyGILState_Ensure and PyGILState_Release. The first C thread also
initializes python as well as initializes threading:
Py_Initialize();
PyEval_InitThreads();

(If it helps I can post my whole code here).

When I test this I get deadlocks in most cases after the first thread
has called PyGILState_Release. All other threads at that point 'freeze'
and never return from whatever Python/C API function they are in. I
don't understand why. Moreover, this behavious is not always
reproducable, sometimes the code works as intended. Must be some kind
of race condition? Who can help?

I read that the problem of calling python from a multithreaded
application per thread is very difficult, but a few projects have
managed to solve it, but now we have these PyGILStates and all is much
easier?

Martin

Jul 13 '06 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: Christoph Wiedemann | last post by:
Hello, i have some trouble to understand the Py_NewInterpreter API function. The docs state, that Py_NewInterpreter returns a new PyThreadState instance. One can switch between interpreters by...
62
by: robert | last post by:
I'd like to use multiple CPU cores for selected time consuming Python computations (incl. numpy/scipy) in a frictionless manner. Interprocess communication is tedious and out of question, so I...
0
Bulldog
by: Bulldog | last post by:
I am working on a C++ win32 DLL which has calls to python, mainly import some modules, call some functions and return some values. Right now I am using the Py_Initialize() and Py_Uninitialize()...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.