470,604 Members | 2,173 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 470,604 developers. It's quick & easy.

build flow? SCons? AAP? process creation?

Hey guys, here's a little question. I'm looking for something like a
build system, where I can set up a bunch of jobs with dependencies, and
have them run in parallel. Ideally, I'd like a system where jobs can
be run in parallel, all the stdout and stderr for each job is kept in a
database of some kind, jobs can be anything that can be run at a
command line, and a failing job won't cause the whole run to fail.
I've been looking into some of the standard stuff (GNU make) and some
of the more pythonic stuff like SCons and AAP. SCons seems to do
parallel execution, which is nice, but I can't figure out from the docs
if it's possible to keep stdout and stderr somewhere. AAP seems to
have a nice interface, but it doesn't do parallel.

So if I do wind up having to write this thing myself, I've been
checking the docs on process creation, and have a couple questions if
anyone can fill me in. It looks like the os.spawn* commands can start
nonblocking sub-processes, but there doesn't seem to be a way to get
stdout and stderr. On the other hand, the popen commands make it easy
to trap stdout and stderr, but I guess I'd have to do the thread setup
and spawning myself. Is there another alternative that I'm missing
here?

Thanks!

Jul 18 '05 #1
3 1503
co************@comcast.net schrieb:
[...]
So if I do wind up having to write this thing myself, I've been
checking the docs on process creation, and have a couple questions if
anyone can fill me in. It looks like the os.spawn* commands can start
nonblocking sub-processes, but there doesn't seem to be a way to get
stdout and stderr. On the other hand, the popen commands make it easy
to trap stdout and stderr, but I guess I'd have to do the thread setup
and spawning myself. Is there another alternative that I'm missing
here?
[...]


Yes, you are missing the subprocess module which was introduced in
Python 2.4: http://docs.python.org/lib/module-subprocess.html

Bye,
Dennis
Jul 18 '05 #2
good point, I hadn't checked the docs too closely. Shame it's only in
2.4 though, we're still running 2.2 around here, but it would be nice
to have a reason to upgrade, anyway. Thanks for the pointer!

Jul 18 '05 #3
In article <11**********************@g14g2000cwa.googlegroups .com>, co************@comcast.net wrote:
Hey guys, here's a little question. I'm looking for something like a
build system, where I can set up a bunch of jobs with dependencies, and
have them run in parallel. Ideally, I'd like a system where jobs can
be run in parallel, all the stdout and stderr for each job is kept in a
database of some kind, jobs can be anything that can be run at a
command line, and a failing job won't cause the whole run to fail.
I've been looking into some of the standard stuff (GNU make) and some
of the more pythonic stuff like SCons and AAP. SCons seems to do
parallel execution, which is nice, but I can't figure out from the docs
if it's possible to keep stdout and stderr somewhere. AAP seems to
have a nice interface, but it doesn't do parallel.


Here's something from the SCons Wiki:
http://www.scons.org/cgi-bin/wiki/BuildLog

-Steve

--
So if you people can't figure out what my birthday is based on subtle
hints and vague statements made at random intervals then I'm beginning
to think that you really don't care. -- Dean Lenort, a.r.k.
Jul 19 '05 #4

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

10 posts views Thread by Hung Jung Lu | last post: by
8 posts views Thread by ted | last post: by
14 posts views Thread by Giovanni Bajo | last post: by
6 posts views Thread by Marco | last post: by
2 posts views Thread by seberino | last post: by
reply views Thread by gamename | last post: by
1 post views Thread by anush | last post: by
6 posts views Thread by Hussein B | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.