473,231 Members | 1,683 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,231 software developers and data experts.

subprocess.Popen(cmd) question

Hello,

I'm starting some subprocesses inside a loop. The processes run
independent and dont need any communication between each other. Due to
memory issues I need to limit the number of running processes to around
10. How can I insert a break into my loop to wait until some processes
are finished?

Some minimal examplecode:

import subprocess
for i in range(0,100):
cmd='ping localhost'
p=subprocess.Popen(cmd)
p.wait()

Thanks for any ideas.

Wolfgang

Aug 10 '07 #1
1 2051
WolfgangZ wrote:
I'm starting some subprocesses inside a loop. The processes run
independent and dont need any communication between each other. Due to
memory issues I need to limit the number of running processes to around
10. How can I insert a break into my loop to wait until some processes
are finished?

Some minimal examplecode:

import subprocess
for i in range(0,100):
cmd='ping localhost'
p=subprocess.Popen(cmd)
p.wait()
Just polling the processes may be good enough:

import random
import subprocess
import sys
import time
from itertools import islice
from functools import partial

PROCESSES = 100
SIMULTANEOUS = 10

def info(*args):
print >sys.stderr, " ".join(str(a) for a in args)

class Process(object):
def __init__(self, index):
self.index = index
info("starting process #%d" % index)
self.process = subprocess.Popen(["ping", "localhost", "-w", "%d" %
random.randrange(1, 6)])
def __nonzero__(self):
running = self.process.poll() is None
if not running:
# XXX ugly side effect
info("process #%d terminated" % self.index)
return running

def processes():
for i in range(PROCESSES):
yield partial(Process, i)

def main():
starters = processes()
running = [sp() for sp in islice(starters, SIMULTANEOUS)]
while running:
before = len(running)
running = [p for p in running if p]
after = len(running)
if before == after:
info("waiting")
time.sleep(1)
else:
running.extend(sp() for sp in islice(starters, SIMULTANEOUS -
len(running)))
info("that's all, folks")

if __name__ == "__main__":
main()

Peter
Aug 10 '07 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Jim Benson | last post by:
Hi, I use popen to run a binary executable. A code snippit is below. This part runs fine. I use readline() to catch and print output from the binary to the screen. The system obviously buffers...
6
by: Ernesto | last post by:
I'm opening a telnet session with the subprocess call below. I then wait ten seconds and attempt to terminate the telnet window I created. Unfortuantely, the telnet window remains after the...
3
by: George Sakkis | last post by:
I'm trying to figure out why Popen captures the stderr of a specific command when it runs through the shell but not without it. IOW: cmd = if 1: # this captures both stdout and stderr as...
12
by: Eric_Dexter | last post by:
I am trying to modify a programming example and I am coming up with two problems... first is that I can't seem to pass along the arguments to the external command (I have been able to do that with...
6
by: Eric_Dexter | last post by:
I am having trouble contolling vim with subprocess on a windows machine. It appears that vim comes up on the machine all right and it sometimes looks like it is doing the searchs what I am asking...
2
by: dude | last post by:
Working on Windows XP Say I have a Windows executable, foo.exe. foo.exe is a command line tool that can take a number of different arguments and perform corresponding actions. I want to invoke...
5
by: Robert Latest | last post by:
Hello, look at this function: -------------- def test(): child = os.popen('./slow') for line in child: print line -------------
2
by: rparimi | last post by:
I am trying to redirect stderr of a process to a temporary file and then read back the contents of the file, all in the same python script. As a simple exercise, I launched /bin/ls but this doesn't...
1
by: Mark Shewfelt | last post by:
Hello, I am attempting to use Popen() in a Windows service. I have a small Win32 .exe that I normally run through the os.popen2() function. I've written a class to work with the input and output...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.