process wrapper?

[Steve]

I need help. I'm trying to write a process wrapper class in Python (on
Linux) that let's one:
- read service definitions from a config file (where a service definition
includes a bash command to start the service, and the service is a daemon)
- call a method that will start up the service
- call a method that will shut down the service.
- other stuff not relevant here

Where I'm stumped is in starting up the service in a way that:
- doesn't block the running of the main process (i.e. returns control back
to the wrapper, so it will listen for more commands)
- let's me get some kind of handle on the started service that I can later
use to shut down the service.

I've tried every implementation of os.system, os.popen*, os.spawn* , and
(os.fork + os.exec*) for which I can find an example or that I can imagine,
but I can't come up with anything that works for me. os.spawnv() looked
very promising, but every process I start with it goes defunct.

I won't bore you with examples of stuff that doesn't work. I've tried so
many things, my head is spinning. I don't care if I wrap the process in a
handler that my main process controls/kills or if I spin off a process but
track it's PID, to kill later. I don't care if the PID is in memory or
written to a file I can read later.

Can anyone point me in the right direction?

[Donn Cave]

Quoth "Steve @ Waypath" <st***@waypath.com>:
....
| I've tried every implementation of os.system, os.popen*, os.spawn* , and
| (os.fork + os.exec*) for which I can find an example or that I can imagine,
| but I can't come up with anything that works for me. os.spawnv() looked
| very promising, but every process I start with it goes defunct.

Well, you need to fix that, no? spawnv works for people, and very
likely it's the perfect thing for your application. So start with
a simple Python program that uses spawnv on some ordinary utility,
like "date" for example, and if you can't get it to work you will
have a more specific question to pose.

Donn Cave, do**@drizzle.com

[Steve]

Donn,

Thanks for the reply. I haven't yet implemented a working spawnv call, so
I'm not confident my tests are valid. Here's a sample:

##########
File: test1.py
------------
import os, time
if __name__=='__main__':
os.spawnv(os.P_NOWAIT,'python',['test2.py'])
for step in range(10):
print 'test1 checking in'
time.sleep(1)

##########
File: test2.py
------------
import time
if __name__=='__main__':
for step in range(10):
print 'test2 checking in'
time.sleep(1)

##########
stdout:
# python test.py
test1 checking in
....
test1 checking in
[ 10 iterations of 'test1 checkin in'; none of 'test2 checking in']

###########
While test 1 is running, a ps (in another shell):
# ps x | grep python
11726 pts/1 S 0:00 python test1.py
11727 pts/1 Z 0:00 [python <defunct>]
11729 pts/2 S 0:00 grep python

###########
I see this defunct thing with every spawnv test I try. The defunct process
goes away when the calling process (test1.py, in this case) finishes. Where
am I going wrong here?

Thanks
Steve @ Waypath.com

"Donn Cave" <do**@drizzle.com> wrote in message
news:1080539401.173853@yasure...

Quoth "Steve @ Waypath" <st***@waypath.com>:
...
| I've tried every implementation of os.system, os.popen*, os.spawn* , and
| (os.fork + os.exec*) for which I can find an example or that I can imagine, | but I can't come up with anything that works for me. os.spawnv() looked
| very promising, but every process I start with it goes defunct.

Well, you need to fix that, no? spawnv works for people, and very
likely it's the perfect thing for your application. So start with
a simple Python program that uses spawnv on some ordinary utility,
like "date" for example, and if you can't get it to work you will
have a more specific question to pose.

Donn Cave, do**@drizzle.com

[Donn Cave]

In article <QZ********************@speakeasy.net>,
"Steve @ Waypath" <st***@waypath.com> wrote:

Thanks for the reply. I haven't yet implemented a working spawnv call, so
I'm not confident my tests are valid. Here's a sample:

##########
File: test1.py
------------
import os, time
if __name__=='__main__':
os.spawnv(os.P_NOWAIT,'python',['test2.py'])
OK, that can be fixed.

When the documentation says "path" for some functions, and
"file" for others (e.g., spawnvp), the distinction is that
"path" includes the directory specification, either absolute
or relative to the current working directory. spawnv wants
a path, so the above doesn't work unless "python" is a file
in your current working directory. To resolve this problem,
I would just write in the absolute path, or you may use spawnvp
instead.

But don't give it python's path. Instead, at least consider
invoking the Python program file directly - put its path there
as the 2nd parameter, not python's. Then your argument list
(the 3rd parameter) will be correct as written. If you invoke
python itself, then you'll need to make test2.py the second
argument, directing python to interpret that file. If you
decide to do as I suggest, test2.py will have to be executable
(chmod 755) and start with a line like "#!/usr/bin/python", as
appropriate for your system. From there on, your test1.py
program no longer needs to know how test2.py is actually
implemented.

###########
While test 1 is running, a ps (in another shell):
# ps x | grep python
11726 pts/1 S 0:00 python test1.py
11727 pts/1 Z 0:00 [python <defunct>]
11729 pts/2 S 0:00 grep python

###########
I see this defunct thing with every spawnv test I try. The defunct process
goes away when the calling process (test1.py, in this case) finishes. Where
am I going wrong here?

It's a zombie! UNIX leaves a dead process in the table as
long as its parent might come back and ask for the status.
It goes either when the parent gets its status with a wait
function (like waitpid()), or when the parent exits.

Donn Cave, do**@drizzle.com

[Steve]

"Donn Cave" <do**@u.washington.edu> wrote in message
news:do************************@nntp4.u.washington .edu...

In article <QZ********************@speakeasy.net>,
"Steve @ Waypath" <st***@waypath.com> wrote:

##########
File: test1.py
------------
import os, time
if __name__=='__main__':
os.spawnv(os.P_NOWAIT,'python',['test2.py'])

But don't give it python's path. Instead, at least consider
invoking the Python program file directly - put its path there
as the 2nd parameter, not python's. Then your argument list
(the 3rd parameter) will be correct as written. If you invoke
python itself, then you'll need to make test2.py the second
argument, directing python to interpret that file. If you
decide to do as I suggest, test2.py will have to be executable
(chmod 755) and start with a line like "#!/usr/bin/python", as
appropriate for your system. From there on, your test1.py
program no longer needs to know how test2.py is actually
implemented.

This makes sense to me. I tried this originally, and tried again just now.
For reasons not clear to me, I get the error:
bad interpreter: No such file or directory
when I include the path to the python interperter. I've tried a number of
approaches:
#!python
#!/usr/bin/python
#!/usr/bin/env python

(I've got the x bit set on the test files, so it's not a permissions
problem. ) This, I'm guessing, isn't a python problem, and would seem to be
my roadblock. I've done some preliminary searches for an answer, but nothing
has panned out so far; if anyone has any ideas...

###########
While test 1 is running, a ps (in another shell):
# ps x | grep python
11726 pts/1 S 0:00 python test1.py
11727 pts/1 Z 0:00 [python <defunct>]
11729 pts/2 S 0:00 grep python

###########
I see this defunct thing with every spawnv test I try. The defunct process goes away when the calling process (test1.py, in this case) finishes. Where am I going wrong here?
It's a zombie! UNIX leaves a dead process in the table as
long as its parent might come back and ask for the status.
It goes either when the parent gets its status with a wait
function (like waitpid()), or when the parent exits.

I've read other posts (many yours, I think :) indicating something like
this, but I confess I don't understand the relationship between waitpid()
and spawnv(), how I would use the former with the latter. However, as I
recognize that isn't the solution to my problem, I won't ask you to explain
it--unless you really want to. :)

Donn Cave, do**@drizzle.com

[Donn Cave]

In article <2p********************@speakeasy.net>,
"Steve @ Waypath" <st***@waypath.com> wrote:
....

For reasons not clear to me, I get the error:
bad interpreter: No such file or directory
when I include the path to the python interperter. I've tried a number of
approaches:
#!python
#!/usr/bin/python
#!/usr/bin/env python

(I've got the x bit set on the test files, so it's not a permissions
problem. ) This, I'm guessing, isn't a python problem, and would seem to be
my roadblock. I've done some preliminary searches for an answer, but nothing
has panned out so far; if anyone has any ideas...
The first one wouldn't work for sure. Second one works if python
is in /usr/bin, third one works if python is in some directory in
PATH and env is in /usr/bin. Assuming your own shell is bash,
try "type python" and "type env" to see where those programs are.

[re zombie processes] I've read other posts (many yours, I think :) indicating something like
this, but I confess I don't understand the relationship between waitpid()
and spawnv(), how I would use the former with the latter.

spawnv returns a process ID, if you use os.P_NOWAIT. It returns
the process exit status if you use os.P_WAIT. If you look at the
code, in os.py, you will see that this second variation is essentially
implemented by applying waitpid() to the first.

Donn Cave, do**@drizzle.com

[Steve]

"Donn Cave" <do**@u.washington.edu> wrote in message
news:do************************@nntp6.u.washington .edu...

The first one wouldn't work for sure. Second one works if python
is in /usr/bin, third one works if python is in some directory in
PATH and env is in /usr/bin. Assuming your own shell is bash,
try "type python" and "type env" to see where those programs are.
I think I found the problem here. I was editing in Windows, saving to a
mounted Linux drive, running in Linux. It looks like some end-of-line chars
I didn't account for. I'm still having defunct trouble, but I'm going to
split this one off into a separate thread, going back a few steps.

spawnv returns a process ID, if you use os.P_NOWAIT. It returns
the process exit status if you use os.P_WAIT. If you look at the
code, in os.py, you will see that this second variation is essentially
implemented by applying waitpid() to the first.
So, if I'm following this correctly, if I use os.P_NOWAIT, I shouldn't have
to apply waitpid(), no? And, then, I shouldn't have a zombie process if I
use P_NOWAIT, either. Am I getting this?

Donn Cave, do**@drizzle.com

[Steve]

> I think I found the problem here. I was editing in Windows, saving to a

mounted Linux drive, running in Linux. It looks like some end-of-line chars I didn't account for. I'm still having defunct trouble, but I'm going to
split this one off into a separate thread, going back a few steps.

Donn,

Forget that separate thread. Writing in Linux, instead of Windows, and all
my spawnv problems go away. I've got enough progress here to keep me busy
for a while. I appreciate your help.

If I run into trouble again, you'll hear from me. :)

Best,
Steve

[Donn Cave]

In article <5q********************@speakeasy.net>,
"Steve @ Waypath" <st***@waypath.com> wrote:

"Donn Cave" <do**@u.washington.edu> wrote in message
news:do************************@nntp6.u.washington .edu...

spawnv returns a process ID, if you use os.P_NOWAIT. It returns
the process exit status if you use os.P_WAIT. If you look at the
code, in os.py, you will see that this second variation is essentially
implemented by applying waitpid() to the first.

So, if I'm following this correctly, if I use os.P_NOWAIT, I shouldn't have
to apply waitpid(), no? And, then, I shouldn't have a zombie process if I
use P_NOWAIT, either. Am I getting this?

Well, no, I believe you have it backwards.

Donn Cave, do**@drizzle.com

[Steve]

There should be a waiting period on posting. Yes, of course it's the other
way.

In practice, when does one use P_NOWAIT and then waitpid(), instead of
P_WAIT? I'm guessing the answer is "if you want to do something between the
spawing and the waiting," but is there a standard use I haven't learned yet?

I'm guessing your reponse will finish this thread. So, thanks again for all
the direction.

"Donn Cave" <do**@u.washington.edu> wrote in message
news:do************************@nntp4.u.washington .edu...

In article <5q********************@speakeasy.net>,
"Steve @ Waypath" <st***@waypath.com> wrote:

"Donn Cave" <do**@u.washington.edu> wrote in message
news:do************************@nntp6.u.washington .edu...

spawnv returns a process ID, if you use os.P_NOWAIT. It returns
the process exit status if you use os.P_WAIT. If you look at the
code, in os.py, you will see that this second variation is essentially
implemented by applying waitpid() to the first.

So, if I'm following this correctly, if I use os.P_NOWAIT, I shouldn't have to apply waitpid(), no? And, then, I shouldn't have a zombie process if I use P_NOWAIT, either. Am I getting this?

Well, no, I believe you have it backwards.

Donn Cave, do**@drizzle.com

[Donn Cave]

In article <Ix********************@speakeasy.net>,
"Steve @ Waypath" <st***@waypath.com> wrote:
....

In practice, when does one use P_NOWAIT and then waitpid(), instead of
P_WAIT? I'm guessing the answer is "if you want to do something between the
spawing and the waiting," but is there a standard use I haven't learned yet?

Yes, that's the idea.

Donn Cave, do**@drizzle.com