Py_NewInterpret er(), is this a bug in the python core?

[freesteel]

/*
Is this a bug in Py_NewInterpret er?

The function below "MyThread" is instantiated from a windows worker
thread, but I don't
think that is relevant.
(I can try this on a linux box, but I would have to compile a python
library with debugging
enabled.)

The following code fragment throws an exception in a debug version of
python:
*/

UINT MyThread(LPVOID lpParam)
{
{
cs.Lock(); // this is a CCriticalSectio n lock
if (!Py_IsInitiali zed())
{
Py_Initialize() ;
PyEval_InitThre ads();

// global pointer to the main PyThreadState object
mainThreadState = PyThreadState_G et();
PyEval_ReleaseL ock();
}
cs.Unlock();
}

ASSERT(Py_IsIni tialized());
ASSERT(PyEval_T hreadsInitializ ed());
ASSERT(mainThre adState);
threadnum++;

// get the global lock
PyEval_AcquireL ock();
PyGILState_STAT E gstate;
gstate = PyGILState_Ensu re(); // Is tis necessary?
PyThreadState_S wap(mainThreadS tate);
PyThreadState* nts = Py_NewInterpret er();

/*

The exception is thrown inside the above function call:
This statement tries to swap the new threadstate 'tstate' with the
current one
save_tstate = PyThreadState_S wap(tstate);

Inside PyThreadState_S wap the code uses another way
'PyGILState_Get ThisThreadState ()' to find the current thread state and
compares this with the newly set thread state.
Naturally you would expect the two to be equal but that test fails:
#if defined(Py_DEBU G) && defined(WITH_TH READ)
if (new) {
PyThreadState *check = PyGILState_GetT hisThreadState( );
if (check && check != new)
Py_FatalError(" Invalid thread state for this thread");
}
#endif

The variable 'check' looks as if it is the 'previous' thread state, as
if changing the thread state
is not been done properly. Py_FatalError is called and that's the end.

Is there a mistake in my code, or is there something wrong in how
Py_NewInterpret er is implemented?
Thanks

Martin

PS: Below the rest of my simple test worker thread function.
*/


ASSERT(nts == PyThreadState_G et());

// lock (already locked) - swap in thread state - swap out thread
state - unlock

init_testclass( );
int ret = 0;

ret = PyRun_SimpleStr ing("import sys");
ret = PyRun_SimpleStr ing("class redir:\n def __init__(self, id):\n
self.id = id\n def write(self, s):\n f = open('stdoutput s_%s.txt' %
self.id, 'a')\n f.write('%s: %s' % (self.id, s))\n f.close()\n");
char str[100];
sprintf(str,"r = redir('0x%x')", &nts);
ret = PyRun_SimpleStr ing(str);
ret = PyRun_SimpleStr ing("sys.stderr = r");
sprintf(str,"s = redir('0x%x')", &nts);
ret = PyRun_SimpleStr ing(str);
ret = PyRun_SimpleStr ing("sys.stdout = s");

ret = PyRun_SimpleStr ing("import testclass");
ret = PyRun_SimpleStr ing("t = testclass.testc lass()");
sprintf(str,"pr int 't = ', t ");
ret = PyRun_SimpleStr ing(str);
ret = PyRun_SimpleStr ing("print t.run(10)");
Py_EndInterpret er(nts);
PyGILState_Rele ase(gstate);
PyEval_ReleaseL ock();

return 0;
}

[Thomas Heller]

freesteel schrieb:

/*
Is this a bug in Py_NewInterpret er?

The function below "MyThread" is instantiated from a windows worker
thread, but I don't
think that is relevant.
(I can try this on a linux box, but I would have to compile a python
library with debugging
enabled.)

The following code fragment throws an exception in a debug version of
python:
*/

UINT MyThread(LPVOID lpParam)
{
{
cs.Lock(); // this is a CCriticalSectio n lock
if (!Py_IsInitiali zed())
{
Py_Initialize() ;
PyEval_InitThre ads();

// global pointer to the main PyThreadState object
mainThreadState = PyThreadState_G et();
PyEval_ReleaseL ock();
}
cs.Unlock();
}

ASSERT(Py_IsIni tialized());
ASSERT(PyEval_T hreadsInitializ ed());
ASSERT(mainThre adState);
threadnum++;

// get the global lock
PyEval_AcquireL ock();
PyGILState_STAT E gstate;
gstate = PyGILState_Ensu re(); // Is tis necessary?
PyThreadState_S wap(mainThreadS tate);
PyThreadState* nts = Py_NewInterpret er();

/*

The exception is thrown inside the above function call:
This statement tries to swap the new threadstate 'tstate' with the
current one
save_tstate = PyThreadState_S wap(tstate);

Inside PyThreadState_S wap the code uses another way
'PyGILState_Get ThisThreadState ()' to find the current thread state and
compares this with the newly set thread state.
Naturally you would expect the two to be equal but that test fails:
#if defined(Py_DEBU G) && defined(WITH_TH READ)
if (new) {
PyThreadState *check = PyGILState_GetT hisThreadState( );
if (check && check != new)
Py_FatalError(" Invalid thread state for this thread");
}
#endif

The variable 'check' looks as if it is the 'previous' thread state, as
if changing the thread state
is not been done properly. Py_FatalError is called and that's the end.

Is there a mistake in my code, or is there something wrong in how
Py_NewInterpret er is implemented?

As far as I know, the PyGILState_... functions are incompatible with multiple
Python interpreters.

Thomas

[freesteel]

Yes, I see that now in the documentation, which to me is quite
confusing.
So, how do you use python in a multithreaded environment, where for
example you want to run some embeded python code from a number of
different C threads?

This article: http://www.linuxjournal.com/article/3641 is quite good,
but must have been written before these PyGILState_* functions.

I only used Py_NewInterpret er to have a fresh 'import sys'.

Somebody enlighten me, please.

Martin

[freesteel]

freesteel wrote:

Yes, I see that now in the documentation, which to me is quite
confusing.
So, how do you use python in a multithreaded environment, where for
example you want to run some embeded python code from a number of
different C threads?

This article: http://www.linuxjournal.com/article/3641 is quite good,
but must have been written before these PyGILState_* functions.

I only used Py_NewInterpret er to have a fresh 'import sys'.

Somebody enlighten me, please.

Martin

If you try to replicate the code from the linux journal
(http://www.linuxjournal.com/article/3641) and compile with Py_DEBUG
defined, when running it you will find that you get an exception and a
fatal error message from the python core. The exception is thrown from
pystate.c, line 306:
Py_FatalError(" Invalid thread state for this thread");

The exception is thrown in my understanding of the code because there
can only ever be one thread state. If this is true a function to swap
thread states seems rather pointless.
Now, reading up about how to call C API to python from a C thread I
find that with version 2.3 the function pair
PyGILState_Ensu re/PyGILState_Rele ase was introduced. This allows
'grabbing' the global interpreter lock, calling your embedded python
code and release the GIL at the end again.

Am I right in my understanding that the use of this PyGILState_* pair
is meant to 'replace' the prologue of creating a new thread state from
the main thread state, swapping it with the current state, doing your
Python/C API calls and then in an epilogue swap the previous thread
state back in, and deletie the now obsolete previously created thread
state?

At least that how I understand the motivation behind the introduction
of tPyGILState_*, read here:
http://www.python.org/dev/peps/pep-0311/

Now, I tried to use this mechanism, but I am not really successful with
it. Basically, in my C thread, I wrap a number of calls to embedded
python (a few PyRun_SimpleStr ing calls, nothing really fancy) with
PyGILState_Ensu re and PyGILState_Rele ase. The first C thread also
initializes python as well as initializes threading:
Py_Initialize() ;
PyEval_InitThre ads();

(If it helps I can post my whole code here).

When I test this I get deadlocks in most cases after the first thread
has called PyGILState_Rele ase. All other threads at that point 'freeze'
and never return from whatever Python/C API function they are in. I
don't understand why. Moreover, this behavious is not always
reproducable, sometimes the code works as intended. Must be some kind
of race condition? Who can help?

I read that the problem of calling python from a multithreaded
application per thread is very difficult, but a few projects have
managed to solve it, but now we have these PyGILStates and all is much
easier?

Martin