473,320 Members | 2,177 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

os.pipe and subprocess under Windows

Hi - I have some code which works under linux. It starts a remote python
process using subprocess and communicates to it via a pipe created by
os.pipe. As far as I understand, child processes should inherit file
descriptors from the parent if close_fds=False on the suprocess.Popen
command line.

This code doesn't work under Window, but gives "bad file descriptor" when
trying to read from the pipe in the child process. I have some example code
which fails. It consists of two files, master.py and slave.py. The file
descriptor for reading from the pipe is passed as a argument to slave.py.

----------------------------------------------------------------
#!/usr/bin/env python
# This is master.py

import os
import os.path
import sys
import subprocess

def runMaster():
# create pipe to communicate with remote process
rpipe, wpipe = os.pipe()

# start remote process
cmdline = [sys.executable,
os.path.join( os.path.dirname(
os.path.abspath(__file__)), 'slave.py' ),
str(rpipe) ]

remote = subprocess.Popen(cmdline, shell=False, bufsize=0,
close_fds=False)

# send text to remote process via pipe
os.write(wpipe, 'hi there$')

# wait until remote exit
remote.wait()

if __name__ == '__main__':
runMaster()

------------------------------------------------------------------

# This is slave.py

import sys
import os

def runSlave(fd):
"""Copy text to stderr from file descriptor until a $ symbol."""
while True:
intext = os.read(fd, 1)
if intext == '$':
break
elif intext:
# write text from pipe to stderr
sys.stderr.write('* %s\n' % intext)

if __name__ == '__main__':
fd = int(sys.argv[1])
runSlave(fd)

-------------------------------------------------------------------

Does anyone have any ideas how to get this to work under Windows? Is it
correct code under unix?

Thanks

Jeremy

--
Jeremy Sanders
http://www.jeremysanders.net/
Nov 17 '08 #1
3 8637
Jeremy Sanders wrote:
As far as I understand, child processes should inherit file
descriptors from the parent if close_fds=False on the suprocess.Popen
command line.

This code doesn't work under Window, but gives "bad file descriptor" when
trying to read from the pipe in the child process.
<http://docs.python.org/library/subprocess.html>:

If close_fds is true, all file descriptors except 0, 1 and 2 will be
closed before the child process is executed. (Unix only). Or, on
Windows, if close_fds is true then no handles will be inherited by the
child process.

Windows has no fork(2).
Nov 17 '08 #2
Lawrence D'Oliveiro wrote:
<http://docs.python.org/library/subprocess.html>:

If close_fds is true, all file descriptors except 0, 1 and 2 will be
closed before the child process is executed. (Unix only). Or, on
Windows, if close_fds is true then no handles will be inherited by the
child process.

Windows has no fork(2).
Yes - I saw that - thanks. This suggests that as I have used
closed_fds=False, then the child process will inherit the handles under
Windows, which it doesn't seem to. This documentation looks wrong to me.

I know Windows has no fork - that's why I used subprocess. This MSDN page
suggests you you need to do something with SetHandleInformation to get them
to be inherited. Doesn't subprocess do that?

http://msdn.microsoft.com/en-us/libr...99(VS.85).aspx

--
Jeremy Sanders
http://www.jeremysanders.net/
Nov 17 '08 #3
Jeremy Sanders wrote:
Hi - I have some code which works under linux. It starts a remote python
process using subprocess and communicates to it via a pipe created by
os.pipe. As far as I understand, child processes should inherit file
descriptors from the parent if close_fds=False on the suprocess.Popen
command line.
Hmm... examining the code for os.pipe in posixmodule.c, it looks like pipes
are create specifically to be non-inheritable in Windows. I can't see why
you would want a non-inheritable pipe, so I would call this a bug.

I suppose I could try this trick from subprocess.py to make the pipes
inheritable:

def _make_inheritable(self, handle):
"""Return a duplicate of handle, which is inheritable"""
return DuplicateHandle(GetCurrentProcess(), handle,
GetCurrentProcess(), 0, 1,
DUPLICATE_SAME_ACCESS)

Pretty nasty to have to do this though, and I would have to add a win32api
dependency, or hack around with the _subprocess module.

Jeremy

--
Jeremy Sanders
http://www.jeremysanders.net/
Nov 17 '08 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Hoegje | last post by:
I am writing a C++ program, which should create a sub- process to start a telnet session to another server. Then it should login to that server (on the telnet login) and execute one or more...
0
by: Tom Brown | last post by:
I need to chain together three linux commands and get the final output. I read the documentation for Popen in the subprocess module for replacing the shell pipe line. I followed the example and...
6
by: placid | last post by:
Hi all, I have been looking into non-blocking read (readline) operations on PIPES on windows XP and there seems to be no way of doing this. Ive read that you could use a Thread to read from the...
4
by: Bart | last post by:
Hi everyone! Im using module that gives errors to stderr/stdout (generated by SWIG) Problem is that I need to parse this errors/information from module. os.popen3 looks nice but this executes...
11
by: 7stud | last post by:
Hi, Can someone explain what a broken pipe is? The following produces a broken pipe error: ---------- import subprocess as sub p = sub.Popen(, stdin=sub.PIPE, stdout=sub.PIPE)
2
by: Rafael Giannetti Viotti | last post by:
Hi, I am working with the subprocess.py module in Python 2.4.4 and I am confused about it's functionality. It uses the standard pipe-fork-exec method to start a subprocess: # create pipes ...
0
by: schickb | last post by:
I'm trying to pipe data that starts life in an array('B') object through several processes. The code below is a simplified example. The data makes it through, but the wait() always hangs. Is there...
1
by: Jean-Paul Calderone | last post by:
On Mon, 12 May 2008 17:35:44 -0700 (PDT), schickb <schickb@gmail.comwrote: Alas, you haven't actually closed src's standard input. Though you did close src.stdin, you didn't close the copy of it...
3
by: Mathieu Prevot | last post by:
Hi I import subprocess and use Popen, but PIPE is not defined. I used 2.5.1, 2.5.2, Python 2.6a3+ (trunk:63576, May 24 2008, 12:13:40), it's always the same. What am I missing ? Thanks...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.