473,883 Members | 1,611 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

multithreading windows and embedding python

I have posted about this problem before. SInce then I found a much
better article to help with embedding python in a multithreaded
application:

http://www.linuxjournal.com/article/3641

I found this article very good and it clarified for me what needs
doing. Now I have an example application that almost works, but it is
far from reliable. If I run this several times I get crashes telling me
that the heap is modified after deallocation. Can anybody else
reproduce this?

At the bottom of this file I left a debug dump, and a stack dump.

Here is my application, compile in windows using a standard windows
application project.


#include <afxmt.h>
#include <afxwin.h>
#include <stdio.h>
#include <Python.h>
#include <Windows.h>
#include <process.h>

static int threadnum = 0;


UINT MyThread(LPVOID lpParam)
{
ASSERT(Py_IsIni tialized());
threadnum++;
PyThreadState* mainThreadState = (PyThreadState *)lpParam;

// get the global lock
PyEval_AcquireL ock();
// get a reference to the PyInterpreterSt ate
PyInterpreterSt ate * mainInterpreter State = mainThreadState->interp;
PyThreadState_S wap(mainThreadS tate);

// create a thread state object for this thread
PyThreadState * myThreadState =
PyThreadState_N ew(mainInterpre terState);
// free the lock
PyEval_ReleaseL ock();
// lock - swap in thread state - swap out thread state - unlock
PyEval_AcquireL ock();
PyThreadState_S wap(myThreadSta te);

int num = 0;
int ret = 0;
ret = PyRun_SimpleStr ing("x = []");
ret = PyRun_SimpleStr ing("for i in range(10):\n x.append(i)");
char cmd[100];
sprintf(cmd, "f = open('c:/windows/temp/test%d.txt', 'w')",
threadnum);
ret = PyRun_SimpleStr ing(cmd);
ret = PyRun_SimpleStr ing("f.write('% s\\n' % x.__str__())");
sprintf(cmd, "f.write('0x%d\ \n')", &myThreadState) ;
ret = PyRun_SimpleStr ing(cmd);
ret = PyRun_SimpleStr ing("f.close()" );

PyThreadState_S wap(NULL);
PyEval_ReleaseL ock();
// clean up
// grab the lock
PyEval_AcquireL ock();
// swap my thread state out of the interpreter
PyThreadState_S wap(NULL);
// clear out any cruft from thread state object
PyThreadState_C lear(myThreadSt ate);
// delete my thread state object
PyThreadState_D elete(myThreadS tate);
// release the lock
PyEval_ReleaseL ock();

return 0;
}

class CMyWinApp : public CWinApp
{
public:
CMyWinApp() { }
BOOL InitInstance()
{
Py_Initialize() ;
PyEval_InitThre ads();

// save a pointer to the main PyThreadState object
PyThreadState * mainThreadState = PyThreadState_G et();
// release the lock
PyEval_ReleaseL ock();

const int nhandles = 100;
HANDLE hnd[nhandles];
CWinThread* pThread[nhandles];
for (int ih = 0; ih < nhandles; ih++)
{
pThread[ih] = AfxBeginThread( MyThread, mainThreadState ,
THREAD_PRIORITY _NORMAL, CREATE_SUSPENDE D);
pThread[ih]->m_bAutoDelet e = false;
pThread[ih]->ResumeThread() ;

hnd[ih] = pThread[ih]->m_hThread;
}

int nwaits, nfails;
do
{
nwaits = 0;
nfails = 0;
for (int ih = 0; ih < nhandles; ih++)
{
DWORD ret = WaitForSingleOb ject(hnd[ih], INFINITE);
switch (ret)
{
case WAIT_OBJECT_0:
printf("WAIT_OB JECT_0\n");
break;

case WAIT_TIMEOUT:
++nwaits;
printf("WAIT_TI MEOUT\n");
break;

case WAIT_FAILED:
++nfails;
printf("WAIT_FA ILED\n");
break;
}
}
}
while (nwaits > 0);
ASSERT(nfails == 0);

// delete all windows threads
for (int ih = 0; ih < nhandles; ++ih)
delete pThread[ih];

PyEval_AcquireL ock();
PyThreadState_S wap(mainThreadS tate);
Py_Finalize();

return TRUE;
};
};

CMyWinApp app;


Debug dump:

'pyembed_test.e xe': Loaded
'C:\mdunschen\p yembed_test\Deb ug\pyembed_test .exe', Symbols loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\ntdll.dll ', No symbols
loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\mscoree.d ll', No
symbols loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\kernel32. dll', No
symbols loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\advapi32. dll', No
symbols loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\rpcrt4.dl l', No symbols
loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\user32.dl l', No symbols
loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\gdi32.dll ', No symbols
loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\comctl32. dll', No
symbols loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\shlwapi.d ll', No
symbols loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\msvcrt.dl l', No symbols
loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\oleacc.dl l', No symbols
loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\msvcp60.d ll', No
symbols loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\ole32.dll ', No symbols
loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\oleaut32. dll', No
symbols loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\winspool. drv', No
symbols loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\comdlg32. dll', No
symbols loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\shell32.d ll', No
symbols loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\python24. dll', No
symbols loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\msvcr71.d ll', Symbols
loaded.
'pyembed_test.e xe': Loaded
'C:\WINDOWS\Win SxS\x86_Microso ft.Windows.Comm on-Controls_6595b6 4144ccf1df_6.0. 2600.2180_x-ww_a84f1ff9\com ctl32.dll',
No symbols loaded.
'pyembed_test.e xe': Loaded
'C:\WINDOWS\Mic rosoft.NET\Fram ework\v1.1.4322 \mscorwks.dll', No symbols
loaded.
'pyembed_test.e xe': Loaded
'C:\WINDOWS\Mic rosoft.NET\Fram ework\v1.1.4322 \fusion.dll', No symbols
loaded.
'pyembed_test.e xe': Loaded
'C:\WINDOWS\Mic rosoft.NET\Fram ework\v1.1.4322 \mscorlib.dll', No symbols
loaded.
'pyembed_test.e xe': Loaded
'C:\WINDOWS\ass embly\NativeIma ges1_v1.1.4322\ mscorlib\1.0.50 00.0__b77a5c561 934e089_ca105f6 f\mscorlib.dll' ,
No symbols loaded.
'pyembed_test.e xe': Loaded
'C:\WINDOWS\Mic rosoft.NET\Fram ework\v1.1.4322 \diasymreader.d ll', No
symbols loaded.
'DefaultDomain' : Loaded
'c:\windows\mic rosoft.net\fram ework\v1.1.4322 \mscorlib.dll', No symbols
loaded.
'pyembed_test.e xe': Loaded
'C:\WINDOWS\Mic rosoft.NET\Fram ework\v1.1.4322 \mscorsn.dll', No symbols
loaded.
'pyembed_test': Loaded
'c:\mdunschen\p yembed_test\Deb ug\pyembed_test .exe', Symbols loaded.
'pyembed_test.e xe': Loaded
'C:\WINDOWS\Mic rosoft.NET\Fram ework\v1.1.4322 \mscorjit.dll', No symbols
loaded.
'pyembed_test.e xe': Loaded 'C:\WINDOWS\sys tem32\uxtheme.d ll', No
symbols loaded.
The thread '_threadstartex ' (0x500) has exited with code 0 (0x0).
The thread '_threadstartex ' (0xadc) has exited with code 0 (0x0).
The thread '<No Name>' (0xadc) has exited with code 0 (0x0).
The thread '<No Name>' (0x500) has exited with code 0 (0x0).
The thread '_threadstartex ' (0x928) has exited with code 0 (0x0).
The thread '<No Name>' (0x928) has exited with code 0 (0x0).
The thread '_threadstartex ' (0xe48) has exited with code 0 (0x0).
The thread '<No Name>' (0xe48) has exited with code 0 (0x0).
The thread '_threadstartex ' (0x984) has exited with code 0 (0x0).
The thread '<No Name>' (0x984) has exited with code 0 (0x0).
The thread '_threadstartex ' (0x108) has exited with code 0 (0x0).
HEAP[pyembed_test.ex e]: HEAP: Free Heap block eb5bd8 modified at eb5c24
after it was freed
Unhandled exception at 0x7c901230 in pyembed_test.ex e: User breakpoint.


Stack dump:
pyembed_test.ex e!_heap_alloc_d bg(unsigned int nSize=15400960, int nBlockUse=10737 41920, const char * szFileName=0x00 000064, int nLine=1242368) Line 359 + 0x1e C

pyembed_test.ex e!_heap_alloc_b ase(unsigned int size=100) Line 212 C
pyembed_test.ex e!_heap_alloc_d bg(unsigned int nSize=64, int
nBlockUse=12582 916, const char * szFileName=0x00 4c4dd4, int nLine=311)
Line 397 + 0x9 C
pyembed_test.ex e!_nh_malloc_db g(unsigned int nSize=64, int nhFlag=0,
int nBlockUse=12582 916, const char * szFileName=0x00 4c4dd4, int
nLine=311) Line 260 + 0x15 C
pyembed_test.ex e!_malloc_dbg(u nsigned int nSize=64, int
nBlockUse=12582 916, const char * szFileName=0x00 4c4dd4, int nLine=311)
Line 176 + 0x1b C
pyembed_test.ex e!operator new(unsigned int nSize=64, int
nType=12582916, const char * lpszFileName=0x 004c4dd4, int nLine=311)
Line 403 + 0x15 C++
pyembed_test.ex e!CObject::oper ator new(unsigned int nSize=64, const
char * lpszFileName=0x 004c4dd4, int nLine=311) Line 93 + 0x16 C++
pyembed_test.ex e!AfxBeginThrea d(unsigned int (void *)*
pfnThreadProc=0 x00401b80, void * pParam=0x00df7b 10, int nPriority=0,
unsigned int nStackSize=4, unsigned long dwCreateFlags=0 ,
_SECURITY_ATTRI BUTES * lpSecurityAttrs =0x00000000) Line 311 + 0x11 C++
00aaa43d()
pyembed_test.ex e!CMyWinApp::In itInstance() Line 87 + 0x26 bytes C++
pyembed_test.ex e!AfxWinMain(HI NSTANCE__ * hInstance=0x004 00000,
HINSTANCE__ * hPrevInstance=0 x00000000, char * lpCmdLine=0x001 41f12,
int nCmdShow=5) Line 39 + 0xb C++
pyembed_test.ex e!WinMain(HINST ANCE__ * hInstance=0x004 00000,
HINSTANCE__ * hPrevInstance=0 x00000000, char * lpCmdLine=0x001 41f12,
int nCmdShow=5) Line 25 C++
pyembed_test.ex e!WinMainCRTSta rtup() Line 251 + 0x30 C

Jun 19 '06 #1
1 3086

freesteel wrote:
....
pThread[ih] = AfxBeginThread( MyThread, mainThreadState ,
THREAD_PRIORITY _NORMAL, CREATE_SUSPENDE D);
....

Here the call to AfxBeginThread is wrong, there is one argument
missing, it should be:

pThread[ih] = AfxBeginThread( MyThread, mainThreadState ,
THREAD_PRIORITY _NORMAL, 0, CREATE_SUSPENDE D);

Because there are so many default arguments of similar types the
compiler did not notice that I passed 'CREATE_SUSPEND ED' as a stack
size, and use the default 'creation' state of 'start right away' for
the thread.

Don't you love default args and types like 'void*' ?

Martin

Jul 10 '06 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
2518
by: jordi | last post by:
Hi, I'm starting to use Python embedded in a C program. I'm using Python to execute several scripts using as a variables information retrieved for several multithread "agents" written in C. The method is: 1- Create a thread for each agent (I use a pool thread, every agent run in a different thread) 2- Precompile a different script for each agent. 3- Make the agent retrieve its data
0
1860
by: Atul Kshirsagar | last post by:
I am embedding python in my C++ application. I am using Python *2.3.2* with a C++ extention DLL in multi-threaded environment. I am using SWIG-1.3.19 to generate C++ to Python interface. Now to explain it in details, 1. Python initialization and finalization is done in the *main* thread. 2. For each new thread I create a separate sub-interpreter . 3. Using PyRun_String("import myModule"...) before execution of python
2
2882
by: Roose | last post by:
With some googling I have found these resources: http://docs.python.org/ext/win-dlls.html http://www.python.org/doc/faq/windows.html I have a large Win32/MFC/C/C++ application that has an embedded scripting language (a very limited one). I would like to rip it out and replace it with Python. I am thinking that this would be relatively simple since the scripting language is a very small interface between the UI and the engine --...
1
1928
by: amit | last post by:
Hello, I am currently studying how to embedd python. I am developing a graphical C++ application. My goal is to embedd python script that will control some kind of animation. I have some questions about python embedding: 1) Is there a good text book or other resource on embedding/extending? (I find it hard to learn only by the tutorial and C/Python API from the python.org site)
4
1933
by: davidstummer | last post by:
I was wondering if anyone could point me to an example. Currently i have a c++ program which calls and c++ dll (i created both). The dll uses SendMessage to pass messages back to the calling .exe, and the ..exe process the messages in it's Windows Procedure function WindProc(). What i want is to receive these messages ( the contents of each message will be plain text), in python using a callback (i think this is what i need).
1
2353
by: abhinav | last post by:
Hi guys.I have read that one cannot perform true multithreading in python due to global interpreter lock mechanism.Suppose i have to implement a crawler on a say cluster system like clusterknoppix so that i can use parallel virtual machine (PVM)for programming in multiprocessor environment or say open MPI.Can i integrate python with PVM or MPI.Can i embed python into C for programming in multiprocessor environment.Is there any way of...
3
2931
by: John Pye | last post by:
Hi all I have been working on some new code that embeds python in an C application. The embedding is working fine under Linux but crashing under Windows (XP) when I reach the following step. PyRun_AnyFile(f,name); If there's some python exception being thrown by the PyRun_AnyFile call, how can I retrieve it from C?
0
1191
by: Thomas Schreiner | last post by:
Hi, I'm extending a windows application (C++) by embedding Python calls. It seems to be a known problem that windows applications detach immediately from the calling console, so that all output to stdout (from both C++ and Python) doesn't get shown anywhere. A workaround seems to be the allocation of a windows console and redirecting stdout to it:
0
2123
by: Tim Spens | last post by:
--- On Fri, 6/27/08, Tim Spens <t_spens@yahoo.comwrote: I think I know where the problem is but I'm unsure how to fix it. When I call Register_Handler(...) from python via callback.setHandler1(callback1) this only seems to affect pythons ability to trigger an "event" in c. PyObject *Handler is always NULL even after I call Register_Handler(...). I thought there was some magic here that was assigning the pointer *Handler to my python...
0
9943
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
1
10858
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10419
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9577
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7974
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5804
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5996
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4619
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
3237
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.