pythonic exec* spawn*

[Daniel Nogradi]

Is it possible to pass a python object to a python program as argument?

In my program I would like to start executing an other python program
and don't wait until that finishes, only launch it and keep running
the original program. The obvious way I can think of it is using the
exec* or spawn* function families. However, these are generic
functions to start any external program and I was wondering if there
was a special way to start a python program (as opposed to an
arbitrary program), especially because I would like to pass data to
this other program in the form of python objects.

While the original program is running it creates all sorts of data in
a nice pythonic way, for example as a class instance with lot of
member data, and then I wouldn't want to convert all of this into
command line arguments because that would create a huge list. I would
like to pass the whole object at once to the second python program
which should start doing its thing with it while the original program
should keep running.

I was looking around for threading and forking but couldn't come up
with a clear solution.

[kmkz89]

You could possibly pickle an object and send it, but that could be a
bit messy...

[Jeremiah]

you can use the threads commands

[dwelch]

km****@gmail.com wrote:

You could possibly pickle an object and send it, but that could be a
bit messy...

pickle it (or shelve?) and then pass the pickle/shleve filename to the
other process as a command line parameter?

Not sure if this would work or not, but that's where I would start.

-Don

[Rene Pijlman]

Daniel Nogradi:

I would like to pass the whole object at once to the second python
program which should start doing its thing with it while the original
program should keep running.

os.fork() does that (on Mac and Unix).

--
René Pijlman

[Daniel Nogradi]

> >I would like to pass the whole object at once to the second python

program which should start doing its thing with it while the original
program should keep running.

os.fork() does that (on Mac and Unix).

Okay, but how? It seems to me that if the process which issued
os.fork() ends, then the forked process also ends. And what I would
need is that regardless of the execution time of the original program,
the second one which the original starts (or forks) should finish what
it supposed to do. Roughly I have the following in mind:

-------------------------------------------------------------------------------
def longer( data ):
# do some stuff, that takes long
# do something to collect data

data = something

# here call the function longer( data ) in a way
# that execution of the code doesn't stop, so
# we get to the following print and then to the end

print "okay, the first program finished"
------------------------------------------------------------------------------

But the execute of longer( data ) should keep going even though the
original program ended. I'm pretty sure it's something basic and
probably I'm not aware of the right concepts and that's why I can't
find the right place to look in the docs.

[Rene Pijlman]

Daniel Nogradi:

os.fork() does that (on Mac and Unix).
Okay, but how?

Sorry, fork() is implemented strictly on a 'need to know' basis :-)
It seems to me that if the process which issued os.fork() ends, then
the forked process also ends.
No, no, they're not a quantum mechanic photon pair. Every process decides
for itself if and when to end. As long as it's not signalled/killed, that
is.
But the execute of longer( data ) should keep going even though the
original program ended. I'm pretty sure it's something basic and
probably I'm not aware of the right concepts and that's why I can't
find the right place to look in the docs.

You can search for "daemonize".

Here's a recipe:
http://aspn.activestate.com/ASPN/Coo...n/Recipe/66012

The concept is best explained in this book:
http://www.kohala.com/start/apue.html

--
René Pijlman

[Daniel Nogradi]

> >> os.fork() does that (on Mac and Unix).

Okay, but how?

Sorry, fork() is implemented strictly on a 'need to know' basis :-)

It seems to me that if the process which issued os.fork() ends, then
the forked process also ends.

No, no, they're not a quantum mechanic photon pair. Every process decides
for itself if and when to end. As long as it's not signalled/killed, that
is.

But the execute of longer( data ) should keep going even though the
original program ended. I'm pretty sure it's something basic and
probably I'm not aware of the right concepts and that's why I can't
find the right place to look in the docs.

You can search for "daemonize".

Here's a recipe:
http://aspn.activestate.com/ASPN/Coo...n/Recipe/66012

The concept is best explained in this book:
http://www.kohala.com/start/apue.html

Okay, thanks a lot, I'll look into that.

[Daniel Nogradi]

> > >> os.fork() does that (on Mac and Unix).

Okay, but how?

Sorry, fork() is implemented strictly on a 'need to know' basis :-)

It seems to me that if the process which issued os.fork() ends, then
the forked process also ends.

No, no, they're not a quantum mechanic photon pair. Every process decides
for itself if and when to end. As long as it's not signalled/killed, that
is.

But the execute of longer( data ) should keep going even though the
original program ended. I'm pretty sure it's something basic and
probably I'm not aware of the right concepts and that's why I can't
find the right place to look in the docs.

You can search for "daemonize".

Here's a recipe:
http://aspn.activestate.com/ASPN/Coo...n/Recipe/66012

The concept is best explained in this book:
http://www.kohala.com/start/apue.html

Thanks again, the recipe was very useful, and in the meantime I also
found this article,
http://www.informit.com/content/imag...rk_article.pdf
which explains os.fork and os.wait in a python context. Maybe some
others will also make use of it, it clearified the basics for me.

[Nick Craig-Wood]

Daniel Nogradi <no*****@gmail.com> wrote:

Is it possible to pass a python object to a python program as
argument?
Yes - you can pickle it.

http://www.python.org/doc/2.4.2/lib/module-cPickle.html

You can pickle just about anything into a string and unpickle it back
into python objects. You shouldn't accept pickles from untrusted
sources as this may reperesent a security problem.
In my program I would like to start executing an other python program
and don't wait until that finishes, only launch it and keep running
the original program. The obvious way I can think of it is using the
exec* or spawn* function families. However, these are generic
functions to start any external program and I was wondering if there
was a special way to start a python program (as opposed to an
arbitrary program), especially because I would like to pass data to
this other program in the form of python objects.

While the original program is running it creates all sorts of data in
a nice pythonic way, for example as a class instance with lot of
member data, and then I wouldn't want to convert all of this into
command line arguments because that would create a huge list. I would
like to pass the whole object at once to the second python program
which should start doing its thing with it while the original program
should keep running.

I was looking around for threading and forking but couldn't come up
with a clear solution.

If you want to fork(), then I would use the subprocess module to
create a child process with a pipe to it. I would then pass python
objects as pickles back and forth across that pipe.

http://www.python.org/doc/2.4.2/lib/...ubprocess.html

You'll probably find threading easier though.

http://www.python.org/doc/2.4.2/lib/...threading.html

But it will use your multiple CPUs less efficiently than fork()-ing.

--
Nick Craig-Wood <ni**@craig-wood.com> -- http://www.craig-wood.com/nick