473,246 Members | 1,491 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,246 software developers and data experts.

popen2 with large input

from popen2 import popen2

r, w = popen2 ( 'tr "[A-Z]" "[a-z]"' )
w.write ( t ) # t is a text file of around 30k bytes
w.close ()
text = r.readlines ()
print text
r.close ()

This simple script halted on

w.write ( t )

Anyone knows what the problem is?
Jul 18 '05 #1
2 2422
cherico wrote:
from popen2 import popen2

r, w = popen2 ( 'tr "[A-Z]" "[a-z]"' )
w.write ( t ) # t is a text file of around 30k bytes
w.close ()
text = r.readlines ()
print text
r.close ()

This simple script halted on

w.write ( t )

Anyone knows what the problem is?


Yep: deadlock... Pipes are synchronized: you can't read from (resp. write to) a
pipe if the process at the other end does not write to (resp. read from) it. If
you try the command "tr '[A-Z]' '[a-z]'" interactively, you'll see that
everytime tr receives a line, it outputs *immediately* the converted line. So if
you write a file having several lines to the pipe, on the first \n, tr will try
to write to its output, and will be stuck since your program is not reading from
it. So it won't read on its input anymore, so your program will be stuck because
it can't write to the pipe. And they'll wait for each other until the end of
times...

If you really want to use the "tr" command for this stuff, you'd better send
your text lines by lines and read the result immediatly, like in:

text = ''
for line in text.splitlines(1):
w.write(line)
w.flush() # Mandatory because of output bufferization - see below
text += r.readline()
w.close()
r.close()

It *may* work better, but you cannot be sure: in fact, you just can't know
exactly when tr will actually output the converted text. Even worse: since
output is usually buffered, you'll only see the output from tr when its standard
output is flushed, and you can't know when that will be...

(BTW, the script above does not work on my Linux box: the first r.readline()
never returns...)

So the conclusion is: don't use pipes unless you're really forced to. They're a
hell to use, since you never know how to synchronize them.

BTW, if the problem you posted is your real problem, why on earth don't you do:
text = t.lower()
???

HTH
--
- Eric Brunel <eric dot brunel at pragmadev dot com> -
PragmaDev : Real Time Software Development Tools - http://www.pragmadev.com

Jul 18 '05 #2
The connection to the child process created by the popen family have
some inherent maximum size for data "in flight". I'm not sure how to
find out what that value is, but it might be anywhere from a few bytes
to a few K.

So tr starts to write its output as it gets input, but you won't read
its output before you've written all your output. If the size of tr's
output is bigger than the size of the buffer for tr's unread output,
you'll deadlock.

As an aside, the particular problem you pose can be solved with Python's
str.translate method. If the actual goal is to "work like tr", then use
that instead and forget about popen.

Anyway, to solve the popen2 problem, you'll need to write something like this:
[untested, and as you can see there's lots of pseudocode]
def getoutput( command, input ):
r, w = popen2(command)
rr = [r]; ww = [w]
output = []
set r and w nonblocking
while 1:
_r, _w, _ = select.select(rr, ww, [], 0)

if _w:
write some stuff from input to w
if nothing left:
w.close(); ww = []
if _r:
read some stuff into output
if nothing to read:
handle the fact that r was closed
if w was closed: break
else: probably an error condition
return "".join(output)

You could also write 'input' into a temporary file and use
commands.getoutput() or os.popen(.., "r").

Jeff

Jul 18 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Guy | last post by:
Hi I was given an exstremly useful answer to a problem that I had in windows. I'm relativly new with python, but find it exstremly useful. I'm creating a script to run test files and record...
1
by: | last post by:
This could possibly be a bug, but I don't understand it fully so I'm posting here first. Searching the list told me other people are having this problem too. I have created a class which...
1
by: lists04 | last post by:
Hi, I have a problem with a curl request and running it under popen2. If I run this request from the command line: curl -i http://www.yahoo.com/test --stderr errfile (also tried redirecting...
3
by: alexrait1 | last post by:
I launch my python. Then I write this: import popen2 popen2.Popen3("mplayer *.mpg") it starts playing for 2 seconds.. and then stops... if I quit python (ctrl - D) mplayer continues to run and...
3
by: jb | last post by:
Hi there: I need help with popen2 usage. I am coding on Windows 2000 environment and I am basically trying to run command line executable program that accepts command line arguments from user....
9
by: Martin P. Hellwig | last post by:
Hi all, I was doing some popen2 tests so that I'm more comfortable using it. I wrote a little python script to help me test that (testia.py): --------------------------------- someline =...
3
by: I. Myself | last post by:
I can't get this to work: # commer.py - to test communication with other process from popen2 import popen2 (child_stdout, child_stdin) = popen2("commer.exe") print "Got here 1" line =...
3
by: mikem76 | last post by:
How do I automatically redirect stdout and stderr when using os.popen2 to start a long running process. If the process prints a lot of stuff to stdout it will eventually stop because it runs out...
1
by: David Bear | last post by:
I'm using popen2 and getting an extra 1 at the end of my output. I didn't see where this was explained in the docs so I clearly don't understand the behavior. My code is simple. (input, output)...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.