473,722 Members | 2,218 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Problem with embedded python

I have the following code, that seems to make sense to me.
However, it crashes about 1/3 of the times.
My platform is Python 2.4.1 on WXP (I tried the release version from
the msi and the debug version built by me, both downloaded today to
have the latest version).
The crash happens while the main thread is in Py_Finalize.
I traced the crash to _Py_ForgetRefer ence(op) in object.c at line 1847,
where I have op->_ob_prev == NULL.
What am I doing wrong? I'm definitely not too sure about the way I'm
handling the GIL.
Thanks in adv for any suggestion/ comment
Cheers and ciao
Ugo
////////////////////////// TestPyThreads.p y //////////////////////////
#include <windows.h>
#include "Python.h"
int main()
{
PyEval_InitThre ads();
Py_Initialize() ;
PyGILState_STAT E main_restore_st ate = PyGILState_UNLO CKED;
PyGILState_Rele ase(main_restor e_state);
// start the thread
{
PyGILState_STAT E state = PyGILState_Ensu re();
int trash = PyRun_SimpleStr ing(
"import thread\n"
"import time\n"
"def foo():\n"
" f = open('pippo.out ', 'w', 0)\n"
" i = 0;\n"
" while 1:\n"
" f.write('%d\\n' %i)\n"
" time.sleep(0.01 )\n"
" i += 1\n"
"t = thread.start_ne w_thread(foo, ())\n"
);
PyGILState_Rele ase(state);
}
// wait 300 ms
Sleep(300);
PyGILState_Ensu re();
Py_Finalize();
return 0;

}
Jul 19 '05 #1
2 2769

From looking at your example, it looks like you're making the problem FAR
more difficult than it needs to be. The main thing to keep in mind is that
Python threads do not correspond to operating system threads. In an
application using a single OS-level thread, you can use as many Python
threads as you like. The thread-switching mechanism in Python is to
transfer control between threads after the execution of 100 bytecodes. The
GIL is used to insure that only 1 operating system thread is executing
bytecodes at any given time. This allows os threads to sleep on blocking
calls (usually I/O) without necessarily stalling the interpreter.

Your example appears to be written under the assumption that the spawned
Python thread will operate concurrently with your main thread. That will
not be the case. The PyRun_SimpleStr ing() call will operate entirely within
the main thread and will not return until the Python thread has completed
operation (so no GIL manipulation is needed). The subsequent Sleep() call
will simply halt the process for 300ms.

Regular threading is difficult enough to get right without throwing the
Python-thread / OS-thread interaction into the mix. It's a tough nut to
crack so don't feel bad if it takes a while to get everything straight ;-)
The quickest and most practical approach would probably be to read through
some code that's similar to what you're shooting for. I'd suggest taking a
look at some of the modules in the Python source code (particularly the
network related ones) to get a better handle on how threads are managed.

Cheers,

Cocagne
Ugo Di Girolamo wrote:
I have the following code, that seems to make sense to me.
However, it crashes about 1/3 of the times.
My platform is Python 2.4.1 on WXP (I tried the release version from
the msi and the debug version built by me, both downloaded today to
have the latest version).
The crash happens while the main thread is in Py_Finalize.
I traced the crash to _Py_ForgetRefer ence(op) in object.c at line 1847,
where I have op->_ob_prev == NULL.
What am I doing wrong? I'm definitely not too sure about the way I'm
handling the GIL.
Thanks in adv for any suggestion/ comment
Cheers and ciao
Ugo
////////////////////////// TestPyThreads.p y //////////////////////////
#include <windows.h>
#include "Python.h"
int main()
{
PyEval_InitThre ads();
Py_Initialize() ;
PyGILState_STAT E main_restore_st ate = PyGILState_UNLO CKED;
PyGILState_Rele ase(main_restor e_state);
// start the thread
{
PyGILState_STAT E state = PyGILState_Ensu re();
int trash = PyRun_SimpleStr ing(
"import thread\n"
"import time\n"
"def foo():\n"
" f = open('pippo.out ', 'w', 0)\n"
" i = 0;\n"
" while 1:\n"
" f.write('%d\\n' %i)\n"
" time.sleep(0.01 )\n"
" i += 1\n"
"t = thread.start_ne w_thread(foo, ())\n"
);
PyGILState_Rele ase(state);
}
// wait 300 ms
Sleep(300);
PyGILState_Ensu re();
Py_Finalize();
return 0;

}


Jul 19 '05 #2
Tom Cocagne wrote:
From looking at your example, it looks like you're making the problem FAR more difficult than it needs to be. The main thing to keep in mind is that Python threads do not correspond to operating system threads. In an
application using a single OS-level thread, you can use as many Python threads as you like. The thread-switching mechanism in Python is to
transfer control between threads after the execution of 100 bytecodes. The GIL is used to insure that only 1 operating system thread is executing bytecodes at any given time. This allows os threads to sleep on blocking calls (usually I/O) without necessarily stalling the interpreter.

Your example appears to be written under the assumption that the spawned Python thread will operate concurrently with your main thread. That will not be the case. The PyRun_SimpleStr ing() call will operate entirely within the main thread and will not return until the Python thread has completed operation (so no GIL manipulation is needed). The subsequent Sleep() call will simply halt the process for 300ms.

Regular threading is difficult enough to get right without throwing the Python-thread / OS-thread interaction into the mix. It's a tough nut to crack so don't feel bad if it takes a while to get everything straight ;-) The quickest and most practical approach would probably be to read through some code that's similar to what you're shooting for. I'd suggest taking a look at some of the modules in the Python source code (particularly the network related ones) to get a better handle on how threads are managed.
Cheers,

Cocagne

<removed code>

Tom,

Thank you for your answer.

This code is not the real "problem child", but a slimmed down version
that seems to reproduce a problem that I see in a much larger system.

My understanding of the python GIL works as follows:
1) when I start the embedded python interpreter (Py_Initialize,
PyEval_InitThre ads), I own the GIL.
So I need to release it for other threads to be able to do anything.
I don't have other threads, so nothing is happening.

2) when I want to use PyRun_SimpleStr ing, I need to acquire the GIL
before, and release it after. By doing this, no python thread will run
while I'm running my simple string, regardless of anything because the
C is owning the GIL.

3) after I release the GIL, all the python threads are actually free to
run, and they actually will. Since I'm on WXP, python threads ARE
actually os threads, as I can see from my debugger.
I do get my nice pippo.out file with the numbers from 0 to 29 or 30,
which is coherent with the sleeps.

4) when I want to shut down the whole thing, I acquire the GIL and then
I call Py_Finalize, which should pretty much kill all the threads
cleanly.

The crash that I'm getting is during step 4, in the python thread,
while in the main thread I'm calling Py_Finalize.
Maybe I'm doing something wrong in one of the steps, but I cannot see
what.

My best guess is that the words from the ref manual:
"The lock is also released and reacquired around potentially blocking
I/O operations like reading or writing a file, so that other threads
can run while the thread that requests the I/O is waiting for the I/O
operation to complete. "
are haunting me, but I'm not sure how.

Thanks again

Cheers & ciao

Ugo

Jul 19 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

12
3094
by: Brandon | last post by:
Java seems to have taken off as the platform and language of choice for many embedded devices. Would it be feasible for Python(perhaps running on an embedded version of Linux) to act in such a capacity. Most of my experience with Python has been with Unix-type scripting tasks and using it when it is an applications built in scripting, but I know some people try to use to build larger complex applications. Is the Python interpreter portable...
13
2813
by: Alexander May | last post by:
Hi, I love Python! I've been using it for a couple of years now and have found it to be a highly productive language. I evangelize it to my developer friends and am probably responsible for the sale of at least 10 Alex Martelli books. I am now in the fortunate position of being able to use Python for a large project, and as such I have a question. We are developing a distributed application running on approximately six thousand...
6
4439
by: Farshid Lashkari | last post by:
Hi, My application has python embedded into it. I noticed that when I run any python code the output is buffered and doesn't get flushed until my application exits. To fix this I simply flush sys.stdout and sys.stderr every once in while by using the following code: //Get handle to python stdout file and flush it PyObject *pyStdout = PySys_GetObject("stdout"); if(pyStdout && PyFile_Check(pyStdout)) {
3
2001
by: stefan | last post by:
Hi Folks, I currenty extended some of my C++ functionality to python and also embedded python to use python functionality in my C++ system (and use as well these extended functions). While this works fine with the core python functionality, as soon as I run a script (on the embedded system) which tries to import modules which are not in the core system, like "xml" or "re", it fails and says it cannot find the related dll (for example...
4
3043
by: Dennis Clark | last post by:
Hi all, I've looked through the threads about embedded Python that are a year and a half old, and I thought that I'd ask this question now to see if anything has changed. Has anyone, or is anyone working with Python in an embedded Linux environment? Mine is NO where near as constrained as a cell phone since I've got plenty of memory to work with, I'm just running a Linux 2.4 kernel on an ARM9 platform.
3
3069
by: Godzilla | last post by:
Has anyone install Python on Windows XP Embedded? We wish to evaluate the possible solution of installing Python with WinXPE on a PC/104 plus module. Thank you.
1
1210
by: Paul Miller | last post by:
I had some code that used to work that now doesn't. It's an embedded Python interpreter that uses numpy internally. The code calls "import_array()", which now fails (and generates a "ImportError: No module named _numpy" error). This is on the latest OS X 10.4 release. I have Numeric installed in the Python site-packages directory, and it loads and works properly from a normal Python command prompt. I also just noticed that other...
20
22527
by: Jack | last post by:
Is there a Python packaging that is specifically for embedded systems? ie, very small and configurable so the user gets to select what modules to install? For Linux-based embedded systems in particular? I'm thinking of running it on the Linksys's Linux-based open source router WRT54G. It has 4MB flash and 16MB RAM. I think another model has 16MB flash. Any possibilities of running Python on these systems?
2
1613
by: Carl J. Van Arsdall | last post by:
I'm aware of a couple python projects for embedded systems. I am currently considering using Python on an embedded platform to develop a simple application as a personal project, mostly to see if it will work. I was wondering if anyone here was using python for anything of that nature? For those that are involved in these types of projects, how does development in python differ for embedded projects versus a non-embedded project? Is...
0
8738
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9381
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
9151
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9086
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
6680
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
4501
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4762
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
2600
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2146
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.