473,389 Members | 1,067 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,389 software developers and data experts.

Python C/API based multithread python program locks

Hi ALL,

I have written a multithreaded python program where each thread calls
a C function
(via Python/C extension module) to execute some tasks on a remote
node. The number
of threads == the number of nodes specified by the user.
The issue is it works most of the time, but occassionally (I mean this
is quite random ) it hangs and it does not generate any errors as
such. While trying to debug, sometimes even the gdb hangs, but i
managed to get a backtrace of a hung thread:

#0 0xb75ebc32 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0xb75d11ee in pthread_cond_wait@@GLIBC_2.3.2 ()
from/lib/tls/libpthread.so.0
#2 0x0809bb3f in PyThread_acquire_lock ()
#3 0x0809e45c in _PyObject_GC_Del ()
#4 0x0807cad6 in PyEval_GetFuncDesc ()
#5 0x0807abc4 in PyEval_EvalCode ()
#6 0x0807b65e in PyEval_EvalCodeEx ()
#7 0x0807cbbb in PyEval_GetFuncDesc ()
#8 0x0807ab33 in PyEval_EvalCode ()
#9 0x0807b65e in PyEval_EvalCodeEx ()
#10 0x0807cbbb in PyEval_GetFuncDesc ()
#11 0x0807ab33 in PyEval_EvalCode ()
#12 0x0807b65e in PyEval_EvalCodeEx ()
#13 0x0807cbbb in PyEval_GetFuncDesc ()
#14 0x0807ab33 in PyEval_EvalCode ()
#15 0x0807b65e in PyEval_EvalCodeEx ()
#16 0x08078555 in PyEval_EvalCode ()
#17 0x08098569 in PyRun_FileExFlags ()
#18 0x080974d0 in PyRun_SimpleFileExFlags ()
#19 0x08096e1a in PyRun_AnyFileExFlags ()
#20 0x08053ac9 in Py_Main ()
#21 0x08053519 in main ()
So just to weed out the possibility that it is not because of some
error in the code, I iteratively called the same function (which
creates say 100 threads) in a for loop - for 500 times. I found that
it tends to hang at different iterations -- say may be at iteration
#480 or #12 or sometimes it sails smoothly.
in the python program -- the outputs from all threads are synchronized
via thread.join()

In the extension C srcs, i have used Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS brackets to take care of GIL. I have separately
tested the C functions and it seemed to work fine.

Any ideas what could be the possible problem ? The test system is RHEL
3 and Python version 2.2.2

Please let me know if there any useful pointers to solve this issue.

Thanks
kanji
Jul 18 '05 #1
1 2069
"kanji" <ka********@yahoo.com> wrote:

The issue is it works most of the time, but occassionally (I mean this
is quite random ) it hangs and it does not generate any errors as
such. While trying to debug, sometimes even the gdb hangs, but i
managed to get a backtrace of a hung thread:

#0 0xb75ebc32 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0xb75d11ee in pthread_cond_wait@@GLIBC_2.3.2 ()
from/lib/tls/libpthread.so.0
#2 0x0809bb3f in PyThread_acquire_lock ()
#3 0x0809e45c in _PyObject_GC_Del ()
My guess is that the problem is not in this thread: Some other thread is
hung or crashed, while holding the GIL, and this thread here is just waiting
on the GIL.
So just to weed out the possibility that it is not because of some
error in the code, I iteratively called the same function (which
creates say 100 threads) in a for loop - for 500 times. I found that
it tends to hang at different iterations -- say may be at iteration
#480 or #12 or sometimes it sails smoothly.


You have a race condition. There may be some shared resource that is not
accessed in a thread-safe manner. Perhaps some C global variable used by
your code or a library that your code calls?

Or perhaps you just have a blocking I/O-call waiting for something that
never happens.

Good luck - you're gonna need it :/

- Anders

Jul 18 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: JD | last post by:
This is more details about my problem, which I already managed to post as a Python bug to Sourceforge This problem is not related to the bug I encountered earlier, and this is related to a...
0
by: Mark Hahn | last post by:
I would like to announce a new interpreted object-oriented language very closely based on Python, that is Prototype-based, like Self (http://research.sun.com/research/self/language.html) instead of...
0
by: Mark Hahn | last post by:
Ben Collins and I have developed a new interpreted object-oriented language very closely based on Python, that is Prototype-based, like Self (http://research.sun.com/research/self/language.html)...
5
by: Francois De Serres | last post by:
Hiho, could somebody please enlighten me about the mechanics of C callbacks to Python? My domain is more specifically callbacks from the win32 API, but I'm not sure that's where the problem...
4
by: Jeronimo Bertran | last post by:
I have a multithreaded application that manipulates a Queue: protected Queue outputQueue; when I add elements to the queue I use a critical section private void Add(OutputRecord record) {...
0
by: r_obert | last post by:
Hello, I'm trying to create a worker thread for my VC++ program, and was wondering whether I should be linking with the Multithread /MT or Multithread DLL /MD option? I'm not quite sure, in...
3
by: James Stroud | last post by:
Hello All, I am helping someone write a python script to run their DOS application through an SSH terminal. It seems that this program wants to access a DOS shell and send output there. If...
1
by: Jerry Fleming | last post by:
Hi, I have wrote a game with python curses. The problem is that I want to confirm before quitting, while my implementation doesn't seem to work. Anyone can help me? #!/usr/bin/python # #...
6
by: jmartin | last post by:
Hi, I have made a multithread version of a program (load a file into database), and with two processors I get the double of time in the multithread than in the process (unithread) version. I...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.