473,320 Members | 1,612 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

managing multiple subprocesses

Hi guys,
I realise this question has been answered in one form or another many
times before but I can't quite find the solution I need. I am trying to
run multiple subprocesses from a python script and then wait until all
subprocesses have completed before continuing. The subprocesses run on
other machines ie this is a coarse grained parallel computation task.

So the commands I am using are:
for x in range(0, limit):
os.system("xgrid .... ./someprogram %d &" % x)

and then to check whether they are all finished I do a very crude:

joblist = commands.getoutput("xgrid .... -job list")
while joblist != "{jobArray = (); }":
time.sleep(1)
joblist = commands.getoutput("xgrid .... -job list")

Xgrid is a system for parallel computation on Mac OS X. It is quite
good, check it out at www.apple.com/acg/xgrid. But anyway the method I
am using above, while it works, it quite often hangs. I'd like to just
run all the subprocesses at once and then check within the script,
rather than an auxilliary, to see if everything has returned. The
output of the commands is not important as that gets redirected to
files outside the script. I have tried all sorts of popens / excevs /
os.systems / commands etc etc. I realise the subprocess module may have
what I need but I can't get python 2.4 on the Mac so I need a 2.3 based
solution. Any help is much appreciated. Cheers.

Jul 18 '05 #1
2 2410
"Marcos" <se******@gmail.com> wrote in message news:<11**********************@l41g2000cwc.googleg roups.com>...
...
os.systems / commands etc etc. I realise the subprocess module may have
what I need but I can't get python 2.4 on the Mac so I need a 2.3 based
solution. Any help is much appreciated. Cheers.


The Python 2.4 subprocess module by Peter Astrand is pure python for
posix systems (a small C extension module is required only for win32
systems). You can bundle it with your code and use it with older
Python versions. It even contains backward compatibility code to
replace True and False if not found so it can run on Python 2.2.

Oren
Jul 18 '05 #2
Oren Tirosh wrote:
os.systems / commands etc etc. I realise the subprocess module may have
what I need but I can't get python 2.4 on the Mac so I need a 2.3 based
solution. Any help is much appreciated. Cheers.


The Python 2.4 subprocess module by Peter Astrand is pure python for
posix systems (a small C extension module is required only for win32
systems). You can bundle it with your code and use it with older
Python versions. It even contains backward compatibility code to
replace True and False if not found so it can run on Python 2.2.


footnote: a stand-alone distribution is available here:

http://www.lysator.liu.se/~astrand/popen5/

direct link to the latest subprocess.py source:

http://tinyurl.com/3mmka

</F>

Jul 18 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Andrei D. | last post by:
Hello Python newsgroup, In the process of developing a big ssh wrapper for sending commands to multiple hosts over the last few months, I (almost accidentally, considering I'm really just an...
4
by: Jane Austine | last post by:
Running Python 2.3 on Win XP It seems like socket is working interdependently with subprocesses of the process which created socket. ------------------------------------ #the server side >>>...
0
by: Eliezer Figueroa | last post by:
Managing Multiple Excel incoming files? I have this situation. I have a client which have several locations they work primary with excel forms and they are thinking in doing reports with them....
1
by: Cliff Williams | last post by:
How are people managing multiple, interdependent projects in the same solution? If I have to rebuild my solution file one more time, I think my head is going to spin around and pop off. I've...
1
by: Bill Godfrey | last post by:
Hiya. I'm sure this question has come up on this group many times before, but I can't find any prior discussion. (Appologies if I missed it.) I have a PHP program looking a bit like this. Read...
4
by: Simon Matthews | last post by:
Hi, I have created multiple .dll's mostly representing libraries used by multiple projects. The dll's are often dependant on each other , for example:- ..dll1 dependant on .dll2 dependant on...
3
by: Dara Durum | last post by:
Hi ! Py2.4, Win32. I need to optimize a program that have a speciality: hash (MD5/SHA1) the file contents (many files). Now I do this in a simple python program, because (what a pity) the...
1
by: Gal Diskin | last post by:
Hi all, I'm writing a python program using threads to open several subprocesses concurrently (using module subprocess) and wait on them. I was wondering if there is a possibilty that a thread will...
2
by: fariba123 | last post by:
i have designed an employee information site. there is an option to generate pay slip. how can i show the employee related data based on the drop down list. i have found code example for...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.