Problem with embedded python

[Ugo Di Girolamo]

I have the following code, that seems to make sense to me.
However, it crashes about 1/3 of the times.
My platform is Python 2.4.1 on WXP (I tried the release version from
the msi and the debug version built by me, both downloaded today to
have the latest version).
The crash happens while the main thread is in Py_Finalize.
I traced the crash to _Py_ForgetReference(op) in object.c at line 1847,
where I have op->_ob_prev == NULL.
What am I doing wrong? I'm definitely not too sure about the way I'm
handling the GIL.
Thanks in adv for any suggestion/ comment
Cheers and ciao
Ugo
////////////////////////// TestPyThreads.py //////////////////////////
#include <windows.h>
#include "Python.h"
int main()
{
PyEval_InitThreads();
Py_Initialize();
PyGILState_STATE main_restore_state = PyGILState_UNLOCKED;
PyGILState_Release(main_restore_state);
// start the thread
{
PyGILState_STATE state = PyGILState_Ensure();
int trash = PyRun_SimpleString(
"import thread\n"
"import time\n"
"def foo():\n"
" f = open('pippo.out', 'w', 0)\n"
" i = 0;\n"
" while 1:\n"
" f.write('%d\\n'%i)\n"
" time.sleep(0.01)\n"
" i += 1\n"
"t = thread.start_new_thread(foo, ())\n"
);
PyGILState_Release(state);
}
// wait 300 ms
Sleep(300);
PyGILState_Ensure();
Py_Finalize();
return 0;

}

[Tom Cocagne]

From looking at your example, it looks like you're making the problem FAR
more difficult than it needs to be. The main thing to keep in mind is that
Python threads do not correspond to operating system threads. In an
application using a single OS-level thread, you can use as many Python
threads as you like. The thread-switching mechanism in Python is to
transfer control between threads after the execution of 100 bytecodes. The
GIL is used to insure that only 1 operating system thread is executing
bytecodes at any given time. This allows os threads to sleep on blocking
calls (usually I/O) without necessarily stalling the interpreter.

Your example appears to be written under the assumption that the spawned
Python thread will operate concurrently with your main thread. That will
not be the case. The PyRun_SimpleString() call will operate entirely within
the main thread and will not return until the Python thread has completed
operation (so no GIL manipulation is needed). The subsequent Sleep() call
will simply halt the process for 300ms.

Regular threading is difficult enough to get right without throwing the
Python-thread / OS-thread interaction into the mix. It's a tough nut to
crack so don't feel bad if it takes a while to get everything straight ;-)
The quickest and most practical approach would probably be to read through
some code that's similar to what you're shooting for. I'd suggest taking a
look at some of the modules in the Python source code (particularly the
network related ones) to get a better handle on how threads are managed.

Cheers,

Cocagne
Ugo Di Girolamo wrote:

I have the following code, that seems to make sense to me.
However, it crashes about 1/3 of the times.
My platform is Python 2.4.1 on WXP (I tried the release version from
the msi and the debug version built by me, both downloaded today to
have the latest version).
The crash happens while the main thread is in Py_Finalize.
I traced the crash to _Py_ForgetReference(op) in object.c at line 1847,
where I have op->_ob_prev == NULL.
What am I doing wrong? I'm definitely not too sure about the way I'm
handling the GIL.
Thanks in adv for any suggestion/ comment
Cheers and ciao
Ugo
////////////////////////// TestPyThreads.py //////////////////////////
#include <windows.h>
#include "Python.h"
int main()
{
PyEval_InitThreads();
Py_Initialize();
PyGILState_STATE main_restore_state = PyGILState_UNLOCKED;
PyGILState_Release(main_restore_state);
// start the thread
{
PyGILState_STATE state = PyGILState_Ensure();
int trash = PyRun_SimpleString(
"import thread\n"
"import time\n"
"def foo():\n"
" f = open('pippo.out', 'w', 0)\n"
" i = 0;\n"
" while 1:\n"
" f.write('%d\\n'%i)\n"
" time.sleep(0.01)\n"
" i += 1\n"
"t = thread.start_new_thread(foo, ())\n"
);
PyGILState_Release(state);
}
// wait 300 ms
Sleep(300);
PyGILState_Ensure();
Py_Finalize();
return 0;

}

[ugodiggi]

Tom Cocagne wrote:

From looking at your example, it looks like you're making the problem FAR more difficult than it needs to be. The main thing to keep in mind is that Python threads do not correspond to operating system threads. In an
application using a single OS-level thread, you can use as many Python threads as you like. The thread-switching mechanism in Python is to
transfer control between threads after the execution of 100 bytecodes. The GIL is used to insure that only 1 operating system thread is executing bytecodes at any given time. This allows os threads to sleep on blocking calls (usually I/O) without necessarily stalling the interpreter.

Your example appears to be written under the assumption that the spawned Python thread will operate concurrently with your main thread. That will not be the case. The PyRun_SimpleString() call will operate entirely within the main thread and will not return until the Python thread has completed operation (so no GIL manipulation is needed). The subsequent Sleep() call will simply halt the process for 300ms.

Regular threading is difficult enough to get right without throwing the Python-thread / OS-thread interaction into the mix. It's a tough nut to crack so don't feel bad if it takes a while to get everything straight ;-) The quickest and most practical approach would probably be to read through some code that's similar to what you're shooting for. I'd suggest taking a look at some of the modules in the Python source code (particularly the network related ones) to get a better handle on how threads are managed.
Cheers,

Cocagne

<removed code>

Tom,

Thank you for your answer.

This code is not the real "problem child", but a slimmed down version
that seems to reproduce a problem that I see in a much larger system.

My understanding of the python GIL works as follows:
1) when I start the embedded python interpreter (Py_Initialize,
PyEval_InitThreads), I own the GIL.
So I need to release it for other threads to be able to do anything.
I don't have other threads, so nothing is happening.

2) when I want to use PyRun_SimpleString, I need to acquire the GIL
before, and release it after. By doing this, no python thread will run
while I'm running my simple string, regardless of anything because the
C is owning the GIL.

3) after I release the GIL, all the python threads are actually free to
run, and they actually will. Since I'm on WXP, python threads ARE
actually os threads, as I can see from my debugger.
I do get my nice pippo.out file with the numbers from 0 to 29 or 30,
which is coherent with the sleeps.

4) when I want to shut down the whole thing, I acquire the GIL and then
I call Py_Finalize, which should pretty much kill all the threads
cleanly.

The crash that I'm getting is during step 4, in the python thread,
while in the main thread I'm calling Py_Finalize.
Maybe I'm doing something wrong in one of the steps, but I cannot see
what.

My best guess is that the words from the ref manual:
"The lock is also released and reacquired around potentially blocking
I/O operations like reading or writing a file, so that other threads
can run while the thread that requests the I/O is waiting for the I/O
operation to complete. "
are haunting me, but I'm not sure how.

Thanks again

Cheers & ciao

Ugo