473,287 Members | 1,827 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,287 software developers and data experts.

Subprocess with and without shell

I'm trying to figure out why Popen captures the stderr of a specific
command when it runs through the shell but not without it. IOW:

cmd = [my_exe, arg1, arg2, ..., argN]
if 1: # this captures both stdout and stderr as expected
pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE)
else: # this captures only stdout
pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE)

# this prints the empty string if not run through the shell
print "stderr:", pipe.stderr.read()
# this prints correctly in both cases
print "stdout:", pipe.stdout.read()

Any hints ?

George

May 15 '07 #1
3 3782
George Sakkis <ge***********@gmail.comwrote:
I'm trying to figure out why Popen captures the stderr of a specific
command when it runs through the shell but not without it. IOW:

cmd = [my_exe, arg1, arg2, ..., argN]
if 1: # this captures both stdout and stderr as expected
pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE)
else: # this captures only stdout
pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE)

# this prints the empty string if not run through the shell
print "stderr:", pipe.stderr.read()
# this prints correctly in both cases
print "stdout:", pipe.stdout.read()

Any hints ?
Post an example which replicates the problem!

My effort works as expected

-- z.py ----------------------------------------------------
#!/usr/bin/python
from subprocess import Popen, PIPE
cmd = ["./zz.py"]
for i in range(2):
if i: # this captures both stdout and stderr as expected
print "With shell"
pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE)
else: # this captures only stdout
print "Without shell"
pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE)

# this prints the empty string if not run through the shell
print "stderr:", pipe.stderr.read()
# this prints correctly in both cases
print "stdout:", pipe.stdout.read()
---zz.py----------------------------------------------------
#!/usr/bin/python
import sys
print >>sys.stdout, "Stdout"
print >>sys.stderr, "Stderr"
------------------------------------------------------------

Produces

$ ./z.py
Without shell
stderr: Stderr

stdout: Stdout

With shell
stderr: Stderr

stdout: Stdout

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick
May 15 '07 #2
On May 15, 5:30 am, Nick Craig-Wood <n...@craig-wood.comwrote:
George Sakkis <george.sak...@gmail.comwrote:
I'm trying to figure out why Popen captures the stderr of a specific
command when it runs through the shell but not without it. IOW:
cmd = [my_exe, arg1, arg2, ..., argN]
if 1: # this captures both stdout and stderr as expected
pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE)
else: # this captures only stdout
pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE)
# this prints the empty string if not run through the shell
print "stderr:", pipe.stderr.read()
# this prints correctly in both cases
print "stdout:", pipe.stdout.read()
Any hints ?

Post an example which replicates the problem!
I would, but the specific executable being spawned is not a python
script, it's a compiled binary (it's not an extension module either;
it's totally unrelated to python). I don't claim there is a bug or
anything suspicious about Popen, but rather I'd like an explanation of
how can a program display different behavior depending on whether it
runs through the shell or not.

George
May 15 '07 #3
George Sakkis <ge***********@gmail.comwrote:
On May 15, 5:30 am, Nick Craig-Wood <n...@craig-wood.comwrote:
>George Sakkis <george.sak...@gmail.comwrote:
>> I'm trying to figure out why Popen captures the stderr of a specific
command when it runs through the shell but not without it. IOW:
>> cmd = [my_exe, arg1, arg2, ..., argN]
if 1: # this captures both stdout and stderr as expected
pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE)
else: # this captures only stdout
pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE)
>> # this prints the empty string if not run through the shell
print "stderr:", pipe.stderr.read()
# this prints correctly in both cases
print "stdout:", pipe.stdout.read()
>> Any hints ?
>Post an example which replicates the problem!
I would, but the specific executable being spawned is not a python
script, it's a compiled binary (it's not an extension module either;
it's totally unrelated to python). I don't claim there is a bug or
anything suspicious about Popen, but rather I'd like an explanation of
how can a program display different behavior depending on whether it
runs through the shell or not.
George
Well, I would try inspecting your environment ... in the shell
and from within your Python process. See if there's anything
there.

If run a command via an interactive shell and it behaves differently
when run via Popen then see if perhaps it's doing something like
checking to see if it's stdin, or stdout are TTYs (using the C
library functions like isatty() for example). You might try
running the program under a Pexpect rather than SubProcess (since
Pexpect will run the process with it's std* descriptors connected
to pty devices). Alternatively try running the program in a shell
pipeline to see if it behaves more like you're seeing when you run
it under Python. (Since running it in the middle of a pipeline,
perhaps with 2>&1 as well, is ensuring that all of the std* descriptors
are connected to pipes. (You could also run with 2>/tmp/some.FIFO
after doing a mknod p /tmp/some.FIFO (Linux) or mkfifo /tmp/some.FIFO
(BSD) to create the named pipe, of course).

If none of that worked ... try running the program under stace,
truss, ktrace or whatever system call tracing facility your OS
provides ... or under gdb.


--
Jim Dennis,
Starshine: Signed, Sealed, Delivered

Jun 29 '07 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

12
by: km | last post by:
hi all, can any linux command be invoked/ executed without using shell (bash) ? what abt security concerns ? regards, KM
1
by: Neil McNaughton | last post by:
Using xmllint --shell ... there is a great xpath parser. For instance you can do "//body" to extract only the <BODY of an html document. But how can you do this with just the commande line?...
8
by: Rik | last post by:
Hi all, is there a in PHP to get the available locales without shellacces (i.e. locale -a)? Grtz, -- Rik Wasmus
7
by: skunkwerk | last post by:
Hi, i'm trying to call subprocess.popen on the 'rename' function in linux. When I run the command from the shell, like so: rename -vn 's/\.htm$/\.html/' *.htm it works fine... however when I...
9
by: erikcw | last post by:
Hi, I have a cgi script where users are uploading large files for processing. I want to launch a subprocess to process the file so the user doesn't have to wait for the page to load. What is...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.