Py_NewInterpreter(), is this a bug in the python core?

[freesteel]

/*
Is this a bug in Py_NewInterpreter?

The function below "MyThread" is instantiated from a windows worker
thread, but I don't
think that is relevant.
(I can try this on a linux box, but I would have to compile a python
library with debugging
enabled.)

The following code fragment throws an exception in a debug version of
python:
*/

UINT MyThread(LPVOID lpParam)
{
{
cs.Lock(); // this is a CCriticalSection lock
if (!Py_IsInitialized())
{
Py_Initialize();
PyEval_InitThreads();

// global pointer to the main PyThreadState object
mainThreadState = PyThreadState_Get();
PyEval_ReleaseLock();
}
cs.Unlock();
}

ASSERT(Py_IsInitialized());
ASSERT(PyEval_ThreadsInitialized());
ASSERT(mainThreadState);
threadnum++;

// get the global lock
PyEval_AcquireLock();
PyGILState_STATE gstate;
gstate = PyGILState_Ensure(); // Is tis necessary?
PyThreadState_Swap(mainThreadState);
PyThreadState* nts = Py_NewInterpreter();

/*

The exception is thrown inside the above function call:
This statement tries to swap the new threadstate 'tstate' with the
current one
save_tstate = PyThreadState_Swap(tstate);

Inside PyThreadState_Swap the code uses another way
'PyGILState_GetThisThreadState()' to find the current thread state and
compares this with the newly set thread state.
Naturally you would expect the two to be equal but that test fails:
#if defined(Py_DEBUG) && defined(WITH_THREAD)
if (new) {
PyThreadState *check = PyGILState_GetThisThreadState();
if (check && check != new)
Py_FatalError("Invalid thread state for this thread");
}
#endif

The variable 'check' looks as if it is the 'previous' thread state, as
if changing the thread state
is not been done properly. Py_FatalError is called and that's the end.

Is there a mistake in my code, or is there something wrong in how
Py_NewInterpreter is implemented?
Thanks

Martin

PS: Below the rest of my simple test worker thread function.
*/


ASSERT(nts == PyThreadState_Get());

// lock (already locked) - swap in thread state - swap out thread
state - unlock

init_testclass();
int ret = 0;

ret = PyRun_SimpleString("import sys");
ret = PyRun_SimpleString("class redir:\n def __init__(self, id):\n
self.id = id\n def write(self, s):\n f = open('stdoutputs_%s.txt' %
self.id, 'a')\n f.write('%s: %s' % (self.id, s))\n f.close()\n");
char str[100];
sprintf(str,"r = redir('0x%x')", &nts);
ret = PyRun_SimpleString(str);
ret = PyRun_SimpleString("sys.stderr = r");
sprintf(str,"s = redir('0x%x')", &nts);
ret = PyRun_SimpleString(str);
ret = PyRun_SimpleString("sys.stdout = s");

ret = PyRun_SimpleString("import testclass");
ret = PyRun_SimpleString("t = testclass.testclass()");
sprintf(str,"print 't = ', t ");
ret = PyRun_SimpleString(str);
ret = PyRun_SimpleString("print t.run(10)");
Py_EndInterpreter(nts);
PyGILState_Release(gstate);
PyEval_ReleaseLock();

return 0;
}

[Thomas Heller]

freesteel schrieb:

/*
Is this a bug in Py_NewInterpreter?

The function below "MyThread" is instantiated from a windows worker
thread, but I don't
think that is relevant.
(I can try this on a linux box, but I would have to compile a python
library with debugging
enabled.)

The following code fragment throws an exception in a debug version of
python:
*/

UINT MyThread(LPVOID lpParam)
{
{
cs.Lock(); // this is a CCriticalSection lock
if (!Py_IsInitialized())
{
Py_Initialize();
PyEval_InitThreads();

// global pointer to the main PyThreadState object
mainThreadState = PyThreadState_Get();
PyEval_ReleaseLock();
}
cs.Unlock();
}

ASSERT(Py_IsInitialized());
ASSERT(PyEval_ThreadsInitialized());
ASSERT(mainThreadState);
threadnum++;

// get the global lock
PyEval_AcquireLock();
PyGILState_STATE gstate;
gstate = PyGILState_Ensure(); // Is tis necessary?
PyThreadState_Swap(mainThreadState);
PyThreadState* nts = Py_NewInterpreter();

/*

The exception is thrown inside the above function call:
This statement tries to swap the new threadstate 'tstate' with the
current one
save_tstate = PyThreadState_Swap(tstate);

Inside PyThreadState_Swap the code uses another way
'PyGILState_GetThisThreadState()' to find the current thread state and
compares this with the newly set thread state.
Naturally you would expect the two to be equal but that test fails:
#if defined(Py_DEBUG) && defined(WITH_THREAD)
if (new) {
PyThreadState *check = PyGILState_GetThisThreadState();
if (check && check != new)
Py_FatalError("Invalid thread state for this thread");
}
#endif

The variable 'check' looks as if it is the 'previous' thread state, as
if changing the thread state
is not been done properly. Py_FatalError is called and that's the end.

Is there a mistake in my code, or is there something wrong in how
Py_NewInterpreter is implemented?

As far as I know, the PyGILState_... functions are incompatible with multiple
Python interpreters.

Thomas

[freesteel]

Yes, I see that now in the documentation, which to me is quite
confusing.
So, how do you use python in a multithreaded environment, where for
example you want to run some embeded python code from a number of
different C threads?

This article: http://www.linuxjournal.com/article/3641 is quite good,
but must have been written before these PyGILState_* functions.

I only used Py_NewInterpreter to have a fresh 'import sys'.

Somebody enlighten me, please.

Martin

[freesteel]

freesteel wrote:

Yes, I see that now in the documentation, which to me is quite
confusing.
So, how do you use python in a multithreaded environment, where for
example you want to run some embeded python code from a number of
different C threads?

This article: http://www.linuxjournal.com/article/3641 is quite good,
but must have been written before these PyGILState_* functions.

I only used Py_NewInterpreter to have a fresh 'import sys'.

Somebody enlighten me, please.

Martin

If you try to replicate the code from the linux journal
(http://www.linuxjournal.com/article/3641) and compile with Py_DEBUG
defined, when running it you will find that you get an exception and a
fatal error message from the python core. The exception is thrown from
pystate.c, line 306:
Py_FatalError("Invalid thread state for this thread");

The exception is thrown in my understanding of the code because there
can only ever be one thread state. If this is true a function to swap
thread states seems rather pointless.
Now, reading up about how to call C API to python from a C thread I
find that with version 2.3 the function pair
PyGILState_Ensure/PyGILState_Release was introduced. This allows
'grabbing' the global interpreter lock, calling your embedded python
code and release the GIL at the end again.

Am I right in my understanding that the use of this PyGILState_* pair
is meant to 'replace' the prologue of creating a new thread state from
the main thread state, swapping it with the current state, doing your
Python/C API calls and then in an epilogue swap the previous thread
state back in, and deletie the now obsolete previously created thread
state?

At least that how I understand the motivation behind the introduction
of tPyGILState_*, read here:
http://www.python.org/dev/peps/pep-0311/

Now, I tried to use this mechanism, but I am not really successful with
it. Basically, in my C thread, I wrap a number of calls to embedded
python (a few PyRun_SimpleString calls, nothing really fancy) with
PyGILState_Ensure and PyGILState_Release. The first C thread also
initializes python as well as initializes threading:
Py_Initialize();
PyEval_InitThreads();

(If it helps I can post my whole code here).

When I test this I get deadlocks in most cases after the first thread
has called PyGILState_Release. All other threads at that point 'freeze'
and never return from whatever Python/C API function they are in. I
don't understand why. Moreover, this behavious is not always
reproducable, sometimes the code works as intended. Must be some kind
of race condition? Who can help?

I read that the problem of calling python from a multithreaded
application per thread is very difficult, but a few projects have
managed to solve it, but now we have these PyGILStates and all is much
easier?

Martin