473,700 Members | 2,500 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Subprocess with and without shell

I'm trying to figure out why Popen captures the stderr of a specific
command when it runs through the shell but not without it. IOW:

cmd = [my_exe, arg1, arg2, ..., argN]
if 1: # this captures both stdout and stderr as expected
pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE)
else: # this captures only stdout
pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE)

# this prints the empty string if not run through the shell
print "stderr:", pipe.stderr.rea d()
# this prints correctly in both cases
print "stdout:", pipe.stdout.rea d()

Any hints ?

George

May 15 '07 #1
3 3825
George Sakkis <ge***********@ gmail.comwrote:
I'm trying to figure out why Popen captures the stderr of a specific
command when it runs through the shell but not without it. IOW:

cmd = [my_exe, arg1, arg2, ..., argN]
if 1: # this captures both stdout and stderr as expected
pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE)
else: # this captures only stdout
pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE)

# this prints the empty string if not run through the shell
print "stderr:", pipe.stderr.rea d()
# this prints correctly in both cases
print "stdout:", pipe.stdout.rea d()

Any hints ?
Post an example which replicates the problem!

My effort works as expected

-- z.py ----------------------------------------------------
#!/usr/bin/python
from subprocess import Popen, PIPE
cmd = ["./zz.py"]
for i in range(2):
if i: # this captures both stdout and stderr as expected
print "With shell"
pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE)
else: # this captures only stdout
print "Without shell"
pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE)

# this prints the empty string if not run through the shell
print "stderr:", pipe.stderr.rea d()
# this prints correctly in both cases
print "stdout:", pipe.stdout.rea d()
---zz.py----------------------------------------------------
#!/usr/bin/python
import sys
print >>sys.stdout, "Stdout"
print >>sys.stderr, "Stderr"
------------------------------------------------------------

Produces

$ ./z.py
Without shell
stderr: Stderr

stdout: Stdout

With shell
stderr: Stderr

stdout: Stdout

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick
May 15 '07 #2
On May 15, 5:30 am, Nick Craig-Wood <n...@craig-wood.comwrote:
George Sakkis <george.sak...@ gmail.comwrote:
I'm trying to figure out why Popen captures the stderr of a specific
command when it runs through the shell but not without it. IOW:
cmd = [my_exe, arg1, arg2, ..., argN]
if 1: # this captures both stdout and stderr as expected
pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE)
else: # this captures only stdout
pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE)
# this prints the empty string if not run through the shell
print "stderr:", pipe.stderr.rea d()
# this prints correctly in both cases
print "stdout:", pipe.stdout.rea d()
Any hints ?

Post an example which replicates the problem!
I would, but the specific executable being spawned is not a python
script, it's a compiled binary (it's not an extension module either;
it's totally unrelated to python). I don't claim there is a bug or
anything suspicious about Popen, but rather I'd like an explanation of
how can a program display different behavior depending on whether it
runs through the shell or not.

George
May 15 '07 #3
George Sakkis <ge***********@ gmail.comwrote:
On May 15, 5:30 am, Nick Craig-Wood <n...@craig-wood.comwrote:
>George Sakkis <george.sak...@ gmail.comwrote:
>> I'm trying to figure out why Popen captures the stderr of a specific
command when it runs through the shell but not without it. IOW:
>> cmd = [my_exe, arg1, arg2, ..., argN]
if 1: # this captures both stdout and stderr as expected
pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE)
else: # this captures only stdout
pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE)
>> # this prints the empty string if not run through the shell
print "stderr:", pipe.stderr.rea d()
# this prints correctly in both cases
print "stdout:", pipe.stdout.rea d()
>> Any hints ?
>Post an example which replicates the problem!
I would, but the specific executable being spawned is not a python
script, it's a compiled binary (it's not an extension module either;
it's totally unrelated to python). I don't claim there is a bug or
anything suspicious about Popen, but rather I'd like an explanation of
how can a program display different behavior depending on whether it
runs through the shell or not.
George
Well, I would try inspecting your environment ... in the shell
and from within your Python process. See if there's anything
there.

If run a command via an interactive shell and it behaves differently
when run via Popen then see if perhaps it's doing something like
checking to see if it's stdin, or stdout are TTYs (using the C
library functions like isatty() for example). You might try
running the program under a Pexpect rather than SubProcess (since
Pexpect will run the process with it's std* descriptors connected
to pty devices). Alternatively try running the program in a shell
pipeline to see if it behaves more like you're seeing when you run
it under Python. (Since running it in the middle of a pipeline,
perhaps with 2>&1 as well, is ensuring that all of the std* descriptors
are connected to pipes. (You could also run with 2>/tmp/some.FIFO
after doing a mknod p /tmp/some.FIFO (Linux) or mkfifo /tmp/some.FIFO
(BSD) to create the named pipe, of course).

If none of that worked ... try running the program under stace,
truss, ktrace or whatever system call tracing facility your OS
provides ... or under gdb.


--
Jim Dennis,
Starshine: Signed, Sealed, Delivered

Jun 29 '07 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

12
2120
by: km | last post by:
hi all, can any linux command be invoked/ executed without using shell (bash) ? what abt security concerns ? regards, KM
1
4048
by: Neil McNaughton | last post by:
Using xmllint --shell ... there is a great xpath parser. For instance you can do "//body" to extract only the <BODY of an html document. But how can you do this with just the commande line? I need a shell one liner to grab the contents of a <body> XXXX </body>
8
4706
by: Rik | last post by:
Hi all, is there a in PHP to get the available locales without shellacces (i.e. locale -a)? Grtz, -- Rik Wasmus
7
4927
by: skunkwerk | last post by:
Hi, i'm trying to call subprocess.popen on the 'rename' function in linux. When I run the command from the shell, like so: rename -vn 's/\.htm$/\.html/' *.htm it works fine... however when I try to do it in python like so: p = subprocess.Popen(,stdout=subprocess.PIPE,stderr=subprocess.PIPE) print p.communicate()
9
6447
by: erikcw | last post by:
Hi, I have a cgi script where users are uploading large files for processing. I want to launch a subprocess to process the file so the user doesn't have to wait for the page to load. What is the correct way to launch subprocess without waiting for the result to return? Thanks!
0
8726
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8645
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
1
8973
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8925
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7810
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6561
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5903
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
2
2392
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2028
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.