471,119 Members | 1,429 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,119 software developers and data experts.

Problem with running external process

There is a script running continuously which executes an external
script in a loop using os.spawnlp(os.P_WAIT, 'extscript.py') function.
After several hundred executions (cycles of the loop), the main script
gets an exception while trying to start the process. This situation is
repeated until the main script is killed. Before it is killed, no other
command can be executed - I'm getting the message:

-bash: fork: Resource temporarily unavailable.

How to properly execute another process from my script?

Jan 20 '06 #1
8 1938
see the help for ulimit (builtils) :

-$ help limit

Jan 21 '06 #2
Quoth "ToMasz" <t_*******@o2.pl>:
| There is a script running continuously which executes an external
| script in a loop using os.spawnlp(os.P_WAIT, 'extscript.py') function.
| After several hundred executions (cycles of the loop), the main script
| gets an exception while trying to start the process. This situation is
| repeated until the main script is killed. Before it is killed, no other
| command can be executed - I'm getting the message:
|
| -bash: fork: Resource temporarily unavailable.
|
| How to properly execute another process from my script?

Not sure what's causing this. I suggest you invoke "ps" after a
hundred iterations or so - maybe "ps -ef", or "ps wwaux" on BSD
platforms like MacOS. I think the output may show a large number
of processes, and you will know where they came from.

Donn Cave, do**@drizzle.com
Jan 21 '06 #3
This is the result of ulimit command on my machine:

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1014
virtual memory (kbytes, -v) unlimited

It looks like either pipe size or stack size may be too low. But my
main script must be running all the time to collect data so how much
should I increase these values to allow infinite executions of external
process?

Jan 23 '06 #4
On 23 Jan 2006 04:33:17 -0800,
"ToMasz" <t_*******@o2.pl> wrote:
... how much should I increase these values to allow infinite
executions of external process?


That sounds like a bad idea. Long before you've run an infinite number
of processes, a real resource will be exhausted.

You are probably creating zombie processes (you can check with /bin/ps).
To get rid of them, execute os.waitpid(-1, os.WNOHANG) (or something
similar to that) periodically from within your python script. Possibly
execute that statement repeatedly until it indicates that there are no
more zombies. A good time/place to do this is right before you start
another process.

Regards,
Dan

--
Dan Sommers
<http://www.tombstonezero.net/dan/>
Jan 23 '06 #5
Yes, each time the process is created, it remains.

os.waitpid(-1, os.WNOHANG) doesn't work before starting the process
for the first time.

I tried this:

pid = os.fork()
if pid == 0:
os.execl('ext_script.py','ext_script.py')
else:
(pid,status) = os.waitpid(pid, 0)

It seems to work without leaving zombies.

Jan 23 '06 #6
On 23 Jan 2006 10:03:28 -0800,
"ToMasz" <t_*******@o2.pl> wrote:
Yes, each time the process is created, it remains. os.waitpid(-1,
os.WNOHANG) doesn't work before starting the process for the first
time.
Argh. That's what I get for looking at my example too quickly. ;-)

Of course: the best place for os.waitpid is immediately *after*
starting a new process. :-/
I tried this: pid = os.fork()
if pid == 0:
os.execl('ext_script.py','ext_script.py')
else:
(pid,status) = os.waitpid(pid, 0) It seems to work without leaving zombies.


If your parent process just sits around and waits for each child process
to finish before moving on, why not use

os.spawnl(os.P_WAIT, 'ext_script.py')

instead of forking and execing "manually"?

Regards,
Dan

--
Dan Sommers
<http://www.tombstonezero.net/dan/>
Jan 24 '06 #7
No, no, that wasn't my intention (I'm just not conscious enough what's
going on with these fork, exec, spawn.. functions).
My parent process should start the child process and go back to it's
tasks. Before executing it for the next time the parent should check if
the previous child process is done and start it again.

Is it what these lines do?

os.spawnlp(os.P_NOWAIT,'ext_script.py','')
os.waitpid(-1, os.WNOHANG)

Jan 24 '06 #8
In article <11**********************@o13g2000cwo.googlegroups .com>,
"ToMasz" <t_*******@o2.pl> wrote:
No, no, that wasn't my intention (I'm just not conscious enough what's
going on with these fork, exec, spawn.. functions).
My parent process should start the child process and go back to it's
tasks. Before executing it for the next time the parent should check if
the previous child process is done and start it again.

Is it what these lines do?

os.spawnlp(os.P_NOWAIT,'ext_script.py','')
os.waitpid(-1, os.WNOHANG)


No, do you see them doing it? "Check if the previous child
process is done?"

In your original post you said you were using os.P_WAIT,
but I suppose you really were using os.P_NOWAIT all along,
right?

This case may call for a working sample program that makes a
genuine attempt to do what you want to do.

Donn Cave, do**@u.washington.edu
Jan 24 '06 #9

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

3 posts views Thread by Wolfie | last post: by
3 posts views Thread by Myron Turner | last post: by
3 posts views Thread by Oenone | last post: by
11 posts views Thread by Tyron | last post: by
4 posts views Thread by Dylan Parry | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.