473,848 Members | 1,665 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Dialog with a process via subprocess.Pope n blocks forever

Hi,

I am trying to communicate with a subprocess via the subprocess
module. Consider the following example:
>>from subprocess import Popen, PIPE
Popen("""pyth on -c 'input("hey")'" "", shell=True)
<subprocess.Pop en object at 0x729f0>
>>hey
Here hey is immediately print to stdout of my interpreter, I did not
type in the "hey". But I want to read from the output into a string,
so I do
>>x = Popen("""python -c 'input("hey\n") '""", shell=True, stdout=PIPE, bufsize=2**10)
x.stdout.read (1)
# blocks forever

Is it possible to read to and write to the std streams of a
subprocess? What am I doing wrong?

Regards,
-Justin

Feb 28 '07 #1
13 5140
<ba**********@g ooglemail.comwr ote:

Is it possible to read to and write to the std streams of a
subprocess? What am I doing wrong?
I think this problem lies deeper - there has been a lot of
complaints about blocking and data getting stuck in pipes
and sockets...

I have noticed that the Python file objects seem to be
inherently half duplex, but I am not sure if it is python
or the underlying OS. (Suse 10 in my case)

You can fix it by unblocking using the fcntl module,
but then all your accesses have to be in try - except
clauses.

It may be worth making some sort of FAQ on this
subject, as it appears from time to time.

The standard advice has been to use file.flush()
after file.write(), but if you are threading and
have called file.read(n), then the flushing does
not help - this is why I say that the file object
seems to be inherently half duplex.

It makes perfect sense, of course, if the file is a
real disk file, as you have to finish the read before
you can move the heads to do the write- but for
pipes, sockets and RS-232 serial lines it does not
make so much sense.

Does anybody know where it comes from -
Python, the various OSses, or C?

- Hendrik

Mar 1 '07 #2
Hi,

Thanks for your answer. I had a look into the fcntl module and tried
to unlock the output-file, but
>>fcntl.lockf(x .stdout, fcntl.LOCK_UN)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 9] Bad file descriptor

I wonder why it does work with the sys.stdin It's really a pity, it's
the first time python does not work as expected. =/

Flushing the stdin did not help, too.

Regards,
-Justin

Mar 1 '07 #3
En Wed, 28 Feb 2007 18:27:43 -0300, <ba**********@g ooglemail.comes cribió:
Hi,

I am trying to communicate with a subprocess via the subprocess
module. Consider the following example:
>>>from subprocess import Popen, PIPE
Popen("""pyt hon -c 'input("hey")'" "", shell=True)
<subprocess.Pop en object at 0x729f0>
>>>hey

Here hey is immediately print to stdout of my interpreter, I did not
type in the "hey". But I want to read from the output into a string,
so I do
>>>x = Popen("""python -c 'input("hey\n") '""", shell=True, stdout=PIPE,
bufsize=2**1 0)
x.stdout.rea d(1)
# blocks forever
Blocks, or is the child process waiting for you to input something in
response?
Is it possible to read to and write to the std streams of a
subprocess? What am I doing wrong?
This works for me on Windows XP. Note that I'm using a tuple with
arguments, and raw_input instead of input (just to avoid a traceback on
stderr)

pyx=Popen(("pyt hon", "-c", "raw_input('hey ')"), shell=True, stdout=PIPE)
pyx.stdout.read (1)
1234
'h'
pyx.stdout.read ()
'ey'

I typed that 1234 (response to raw_input).

You may need to use python -u, or redirect stderr too, but what your real
problem is?

--
Gabriel Genellina

Mar 1 '07 #4
Okay, here is what I want to do:

I have a C Program that I have the source for and want to hook with
python into that. What I want to do is: run the C program as a
subprocess.
The C programm gets its "commands" from its stdin and sends its state
to stdout. Thus I have some kind of dialog over stdin.

So, once I start the C Program from the shell, I immediately get its
output in my terminal. If I start it from a subprocess in python and
use python's sys.stdin/sys.stdout as the subprocess' stdout/stdin I
also get it immediately.

BUT If I use PIPE for both (so I can .write() on the stdin and .read()
from the subprocess' stdout stream (better: file descriptor)) reading
from the subprocess stdout blocks forever. If I write something onto
the subprocess' stdin that causes it to somehow proceed, I can read
from its stdout.

Thus a useful dialogue is not possible.

Regards,
-Justin

Mar 1 '07 #5
ba**********@go oglemail.com wrote:
Okay, here is what I want to do:

I have a C Program that I have the source for and want to hook with
python into that. What I want to do is: run the C program as a
subprocess.
The C programm gets its "commands" from its stdin and sends its state
to stdout. Thus I have some kind of dialog over stdin.

So, once I start the C Program from the shell, I immediately get its
output in my terminal. If I start it from a subprocess in python and
use python's sys.stdin/sys.stdout as the subprocess' stdout/stdin I
also get it immediately.

BUT If I use PIPE for both (so I can .write() on the stdin and .read()
from the subprocess' stdout stream (better: file descriptor)) reading
from the subprocess stdout blocks forever. If I write something onto
the subprocess' stdin that causes it to somehow proceed, I can read
from its stdout.

Thus a useful dialogue is not possible.

Regards,
-Justin
Have you considered using pexpect: http://pexpect.sourceforge.net/ ?

George
Mar 1 '07 #6
En Thu, 01 Mar 2007 14:42:00 -0300, <ba**********@g ooglemail.comes cribió:
BUT If I use PIPE for both (so I can .write() on the stdin and .read()
from the subprocess' stdout stream (better: file descriptor)) reading
from the subprocess stdout blocks forever. If I write something onto
the subprocess' stdin that causes it to somehow proceed, I can read
from its stdout.
On http://docs.python.org/lib/popen2-flow-control.html there are some
notes on possible flow control problems you may encounter.
If you have no control over the child process, it may be safer to use a
different thread for reading its output.

--
Gabriel Genellina

Mar 1 '07 #7
<ba**********@g ooglemail.comwr ote:
Hi,

Thanks for your answer. I had a look into the fcntl module and tried
to unlock the output-file, but
>fcntl.lockf(x. stdout, fcntl.LOCK_UN)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 9] Bad file descriptor

I wonder why it does work with the sys.stdin It's really a pity, it's
the first time python does not work as expected. =/

Flushing the stdin did not help, too.
its block, not lock, and one uses file.flush() after using file.write(),
so the stdin is the wrong side - you have to push, you can't pull..

Here is the unblock function I use - it comes from the internet,
possibly from this group, but I have forgotten who wrote it.

# Some magic to make a file non blocking - from the internet

def unblock(f):
"""Given file 'f', sets its unblock flag to true."""

fcntl.fcntl(f.f ileno(), fcntl.F_SETFL, os.O_NONBLOCK)

hope this helps - note that the f is not the file's name but the
thing you get when you write :

f = open(...

- Hendrik

Mar 2 '07 #8
<ba**********@g ooglemail.comwr ote:

8<------------------
The C programm gets its "commands" from its stdin and sends its state
to stdout. Thus I have some kind of dialog over stdin.

So, once I start the C Program from the shell, I immediately get its
output in my terminal. If I start it from a subprocess in python and
use python's sys.stdin/sys.stdout as the subprocess' stdout/stdin I
also get it immediately.
so why don't you just write to your stdout and read from your stdin?
>
BUT If I use PIPE for both (so I can .write() on the stdin and .read()
This confuses me - I assume you mean write to the c program's stdin?
from the subprocess' stdout stream (better: file descriptor)) reading
from the subprocess stdout blocks forever. If I write something onto
the subprocess' stdin that causes it to somehow proceed, I can read
from its stdout.
This sounds like the c program is getting stuck waiting for input...
>
Thus a useful dialogue is not possible.
If you are both waiting for input, you have a Mexican standoff...

And if you are using threads, and you have issued a .read() on
a file, then a .write() to the same file, even followed by a .flush()
will not complete until after the completion of the .read().

So in such a case you have to unblock the file, and do the .read() in
a try - except clause, to "free up" the "file driver" so that the .write()
can complete.

But I am not sure if this is in fact your problem, or if it is just normal
synchronisation hassles...

- Hendrik

Mar 2 '07 #9
If you are both waiting for input, you have a Mexican standoff...

That is not the problem. The problem is, that the buffers are not
flushed correctly. It's a dialogue, so nothing complicated. But python
does not get what the subprocess sends onto the subprocess' standard
out - not every time, anyway.

I'm quite confused, but hopefully will understand what's going on and
come back here.


Mar 2 '07 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

18
4900
by: jas | last post by:
Hi, I would like to start a new process and be able to read/write from/to it. I have tried things like... import subprocess as sp p = sp.Popen("cmd.exe", stdout=sp.PIPE) p.stdin.write("hostname\n") however, it doesn't seem to work. I think the cmd.exe is catching it.
8
6528
by: cypher543 | last post by:
This has been driving me insane for the last hour or so. I have search everywhere, and nothing works. I am trying to use the subprocess module to run a program and get its output line by line. But, it always waits for the process to terminate and then return the output all at once. Can someone please show me some code that actually works for this sort of thing? It doesn't even have to use the subprocess module. Don't worry if the code...
3
2203
by: Pappy | last post by:
SHORT VERSION: Python File B changes sys.stdout to a file so all 'prints' are written to the file. Python file A launches python file B with os.popen("./B 2>&^1 >dev/null &"). Python B's output disappears into never-never land. LONG VERSION: I am working on a site that can kick off large-scale simulations. It will write the output to an html file and a link will be emailed to the user. Also, the site will continue to display...
2
6281
by: Greg Ercolano | last post by:
When I use os.popen(cmd,'w'), I find that under windows, the stdout of the child process disappears, instead of appearing in the DOS window the script is invoked from. eg: C:\type foo.py import os import sys file = os.popen("nslookup", 'w') file.write("google.com\n") file.close()
12
4547
by: bhunter | last post by:
Hi, I've used subprocess with 2.4 several times to execute a process, wait for it to finish, and then look at its output. Now I want to spawn the process separately, later check to see if it's finished, and if it is look at its output. I may want to send a signal at some point to kill the process. This seems straightforward, but it doesn't seem to be working. Here's my test case:
4
5874
by: grayaii | last post by:
There are so many threads on this subject, but I ran across a situation on Windows that I can't figure out. I'm trying to run this little command-line exe and when I launch like this, it hangs: I can run this exe manually via the command prompt and it returns after a few seconds, but more importantly when I run it as follows it works fine:
2
2750
by: dudeja.rajat | last post by:
On Mon, Sep 8, 2008 at 11:50 AM, <dudeja.rajat@gmail.comwrote: Ok, I re-phrase my question: there is a batch file that executes a exe file. The batch just works if run from command prompt and produces output to standard output and the file. Now, I try to call the same batch file from subprocess.Pope() call. The batch file gets called and that internally also calls the exe
7
6241
by: Samuel A. Falvo II | last post by:
I have a shell script script.sh that launches a Java process in the background using the &-operator, like so: #!/bin/bash java ... arguments here ... & In my Python code, I want to invoke this shell script using the Subprocess module. Here is my code: def resultFromRunning_(command):
9
15538
by: Catherine Moroney | last post by:
I have one script (Match1) that calls a Fortran executable as a sub-process, and I want to write another script (Match4) that spawns off several instances of Match1 in parallel and then waits until they all finish running. The only way I can think of doing this is to call it as a sub-process, rather than directly. I'm able to get Match1 working correctly in isolation, using the subprocess.Popen command, but calling an instance of Match1...
0
9893
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9738
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10664
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10722
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10348
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7069
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5735
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5919
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
4137
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.