473,287 Members | 1,515 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,287 software developers and data experts.

bug: subprocess.Popen() hangs

This is a bug in python 2.4 under Linux 2.6.

I occasionally see subprocess.Popen() fail to return, and I have
finally figured out roughly what's going on. It involves the GC and
stderr.

1. os.fork()

2. Parent blocks reading from errpipe_read (subprocess.py:982)

3. In child, a GC occurs before the exec.

4. The GC attempts to free a file descriptor, calling file_dealloc.

5. That function gets an error closing the descriptor ("close failed:
[Errno 9] Bad file descriptor\n," is the string I extracted via gdb).

6. It attempts to write the error to stderr and blocks. Since it never
execs or writes to errpipe_write, both child and parent are hung.

Here is the pstack output on the child:

#0 0x006587a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x0072f11b in __write_nocancel () from /lib/tls/libc.so.6
#2 0x006d409f in _IO_new_file_write () from /lib/tls/libc.so.6
#3 0x006d42ec in _IO_new_file_xsputn () from /lib/tls/libc.so.6
#4 0x006afce9 in buffered_vfprintf () from /lib/tls/libc.so.6
#5 0x006afe8b in vfprintf () from /lib/tls/libc.so.6
#6 0x080dd4af in mywrite ()
#7 0x080dd505 in PySys_WriteStderr ()
#8 0x08064cd0 in file_dealloc ()
#9 0x08079c88 in dict_dealloc ()
#10 0x0808931d in subtype_dealloc ()
#11 0x08079af3 in PyDict_Clear ()
#12 0x0807bb6a in dict_tp_clear ()
#13 0x080e0ead in collect ()
#14 0x080e1912 in _PyObject_GC_New ()
#15 0x0805e50b in PyInstance_NewRaw ()
#16 0x0805e5d7 in PyInstance_New ()
#17 0x0805bdc0 in PyObject_Call ()
#18 0x080aecef in PyEval_CallObjectWithKeywords ()
#19 0x080ca975 in PyErr_NormalizeException ()
#20 0x080b492c in PyEval_EvalFrame ()
#21 0x080b5eb2 in PyEval_EvalCodeEx ()
#22 0x080b3c83 in PyEval_EvalFrame ()
#23 0x080b5734 in PyEval_EvalFrame ()
#24 0x080b5eb2 in PyEval_EvalCodeEx ()
#25 0x080fce92 in function_call ()
#26 0x0805bdc0 in PyObject_Call ()
#27 0x080641e5 in instancemethod_call ()
#28 0x0805bdc0 in PyObject_Call ()
#29 0x0808ffce in slot_tp_init ()
#30 0x08088b3a in type_call ()
#31 0x0805bdc0 in PyObject_Call ()
#32 0x080b0f05 in PyEval_EvalFrame ()
#33 0x080b5eb2 in PyEval_EvalCodeEx ()
#34 0x080fce92 in function_call ()
#35 0x0805bdc0 in PyObject_Call ()
#36 0x080641e5 in instancemethod_call ()
#37 0x0805bdc0 in PyObject_Call ()
#38 0x0808ffce in slot_tp_init ()
#39 0x08088b3a in type_call ()
#40 0x0805bdc0 in PyObject_Call ()
#41 0x080b0f05 in PyEval_EvalFrame ()
#42 0x080b5734 in PyEval_EvalFrame ()
#43 0x080b5eb2 in PyEval_EvalCodeEx ()
#44 0x080fce92 in function_call ()
#45 0x0805bdc0 in PyObject_Call ()
#46 0x080641e5 in instancemethod_call ()
#47 0x0805bdc0 in PyObject_Call ()
#48 0x0808ffce in slot_tp_init ()
#49 0x08088b3a in type_call ()
#50 0x0805bdc0 in PyObject_Call ()
#51 0x080b0f05 in PyEval_EvalFrame ()
#52 0x080b5eb2 in PyEval_EvalCodeEx ()
#53 0x080fce92 in function_call ()
#54 0x0805bdc0 in PyObject_Call ()
#55 0x080b075f in PyEval_EvalFrame ()
#56 0x080b5734 in PyEval_EvalFrame ()
#57 0x080b5734 in PyEval_EvalFrame ()
#58 0x080b5734 in PyEval_EvalFrame ()
#59 0x080b5eb2 in PyEval_EvalCodeEx ()
#60 0x080b3c83 in PyEval_EvalFrame ()
#61 0x080b5734 in PyEval_EvalFrame ()
#62 0x080b5734 in PyEval_EvalFrame ()
#63 0x080b5eb2 in PyEval_EvalCodeEx ()
#64 0x080b3c83 in PyEval_EvalFrame ()
#65 0x080b5eb2 in PyEval_EvalCodeEx ()
#66 0x080b3c83 in PyEval_EvalFrame ()
#67 0x080b5eb2 in PyEval_EvalCodeEx ()
#68 0x080b3c83 in PyEval_EvalFrame ()
#69 0x080b5734 in PyEval_EvalFrame ()
#70 0x080b5734 in PyEval_EvalFrame ()
#71 0x080b5734 in PyEval_EvalFrame ()
#72 0x080b5734 in PyEval_EvalFrame ()
#73 0x080b5734 in PyEval_EvalFrame ()
#74 0x080b5eb2 in PyEval_EvalCodeEx ()
#75 0x080fce92 in function_call ()
#76 0x0805bdc0 in PyObject_Call ()
#77 0x080b075f in PyEval_EvalFrame ()
#78 0x080b5eb2 in PyEval_EvalCodeEx ()
#79 0x080b3c83 in PyEval_EvalFrame ()
#80 0x080b5eb2 in PyEval_EvalCodeEx ()
#81 0x080b3c83 in PyEval_EvalFrame ()
#82 0x080b5eb2 in PyEval_EvalCodeEx ()
#83 0x080b3c83 in PyEval_EvalFrame ()
#84 0x080b5734 in PyEval_EvalFrame ()
#85 0x080b5734 in PyEval_EvalFrame ()
#86 0x080b5eb2 in PyEval_EvalCodeEx ()
#87 0x080b601a in PyEval_EvalCode ()
#88 0x080d9ff4 in PyRun_FileExFlags ()
#89 0x080da8d6 in PyRun_SimpleFileExFlags ()
#90 0x08055617 in Py_Main ()
#91 0x08054e3f in main ()

Oct 25 '07 #1
1 1914
Jonathan Amsterdam <jb*********@gmail.comwrote:
This is a bug in python 2.4 under Linux 2.6.

I occasionally see subprocess.Popen() fail to return, and I have
finally figured out roughly what's going on. It involves the GC and
stderr.
Interesting

Do you have a program to demonstrate the problem?

You are best off reporting bugs here - then they won't get lost!

http://bugs.python.org/

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick
Oct 26 '07 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: Kurt B. Kaiser | last post by:
Patch / Bug Summary ___________________ Patches : 385 open (+21) / 3790 closed (+21) / 4175 total (+42) Bugs : 1029 open (+43) / 6744 closed (+43) / 7773 total (+86) RFE : 262 open...
6
by: gregpinero | last post by:
Let's say I have this Python file called loop.py: import sys print 'hi' sys.stdout.flush() while 1: pass And I want to call it from another Python process and read the value 'hi'. How...
12
by: bhunter | last post by:
Hi, I've used subprocess with 2.4 several times to execute a process, wait for it to finish, and then look at its output. Now I want to spawn the process separately, later check to see if it's...
1
by: Marko Rauhamaa | last post by:
This tiny program hangs: ======================================================================== #!/usr/bin/env python import subprocess a = subprocess.Popen('cat',shell = True,stdin =...
7
by: skunkwerk | last post by:
Hi, i'm trying to call subprocess.popen on the 'rename' function in linux. When I run the command from the shell, like so: rename -vn 's/\.htm$/\.html/' *.htm it works fine... however when I...
4
by: grayaii | last post by:
There are so many threads on this subject, but I ran across a situation on Windows that I can't figure out. I'm trying to run this little command-line exe and when I launch like this, it hangs:...
2
by: dudeja.rajat | last post by:
On Mon, Sep 8, 2008 at 11:50 AM, <dudeja.rajat@gmail.comwrote: Ok, I re-phrase my question: there is a batch file that executes a exe file. The batch just works if run from command prompt and...
0
by: Kenneth McDonald | last post by:
When making calls of the form Popen(cmd, shell=True, stdout=subprocess.PIPE), we've been getting occasional, predictable hangs. Will Popen accumulate a certain amount of stdout and then block...
0
by: Christian Heimes | last post by:
Kenneth McDonald wrote: The subprocess module has already an API method for your use case. The communicate() method of a subprocess.Popen instance takes an optional stdin string and returns the...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.