Main-process doesn't wait for child-processes

[Markus Franz]

Hi.

I created a little script:

for currenturl in sys.argv[1:]:
pid = os.fork()
if pid == 0:
signal.alarm(10)
do_something() # placeholder for the download and print
routine
break

This script is started by ./script.py http://www.website1.com
http://www.website2.com
As you see it creates child-processes for loading all websites specified on
the command line. (So all websites are loaded in parallel.) Each
child-process simply load a websites and print the contents to STDOUT (above
replaced by do_something).
(With signal.alarm(10) I make sure that each child-process will not run
longer than 10 seconds.)
But now I have a difficult problem: This script is executed by a
comand-line-function (shell_exec) in PHP, the response of the Python-script
is used inside the PHP application. In the python script the main process
seem to stop immediately after starting the script, only the child-processes
seem to run. Because of this my PHP application will not wait untill the
child-processes have finised and so I can't use the contents of the
websites. (I think the solution should be that the main-process waits untill
the child-processes terminate.)

Now my question is: How can I force the main-process to wait untill every
child-process is finished?
I tried:

for currenturl in sys.argv[1:]:
pid = os.fork()
if pid == 0:
signal.alarm(10)
do_something() # placeholder for the download and print
routine
break
else:
os.wait()

The version above behaves like a script that loads all given websites in
sequence...
I also tried:

for currenturl in sys.argv[1:]:
pid = os.fork()
if pid == 0:
signal.alarm(10)
do_something() # placeholder for the download and print
routine
break
time.sleep(4)

That didn't help me, too.

The following is a PHP application that may help you to understand the bad
behavoir of my application (you will guess my problem if you execute both
the PHP and the Python script):

<?php
echo shell_exec('./script.py http://www.website1.com
http://www.website2.com');
?>
Thank you.
Best regards

Markus Franz

[Jeff Epler]

Keep track of the processes you create, then wait for them each to exit.
Untested code:

def do(task):
pass

processes = []
try:
for task in tasks:
pid = os.fork()
if pid == 0:
do(task)
os._exit(0)
else:
processes.append(pid)
finally:
while processes:
pid = waitpid(0,0) # Wait for any child, block
# print "reaped", pid
processes.remove(pid)

Instead of looping until 'processes' is empty, you could wait until
waitpid raises os.error with errno==ECHILD ("No child processes").

Jeff