Hey guys, here's a little question. I'm looking for something like a
build system, where I can set up a bunch of jobs with dependencies, and
have them run in parallel. Ideally, I'd like a system where jobs can
be run in parallel, all the stdout and stderr for each job is kept in a
database of some kind, jobs can be anything that can be run at a
command line, and a failing job won't cause the whole run to fail.
I've been looking into some of the standard stuff (GNU make) and some
of the more pythonic stuff like SCons and AAP. SCons seems to do
parallel execution, which is nice, but I can't figure out from the docs
if it's possible to keep stdout and stderr somewhere. AAP seems to
have a nice interface, but it doesn't do parallel.
So if I do wind up having to write this thing myself, I've been
checking the docs on process creation, and have a couple questions if
anyone can fill me in. It looks like the os.spawn* commands can start
nonblocking sub-processes, but there doesn't seem to be a way to get
stdout and stderr. On the other hand, the popen commands make it easy
to trap stdout and stderr, but I guess I'd have to do the thread setup
and spawning myself. Is there another alternative that I'm missing
here?
Thanks! 3 1632 co************@ comcast.net schrieb: [...] So if I do wind up having to write this thing myself, I've been checking the docs on process creation, and have a couple questions if anyone can fill me in. It looks like the os.spawn* commands can start nonblocking sub-processes, but there doesn't seem to be a way to get stdout and stderr. On the other hand, the popen commands make it easy to trap stdout and stderr, but I guess I'd have to do the thread setup and spawning myself. Is there another alternative that I'm missing here? [...]
Yes, you are missing the subprocess module which was introduced in
Python 2.4: http://docs.python.org/lib/module-subprocess.html
Bye,
Dennis
good point, I hadn't checked the docs too closely. Shame it's only in
2.4 though, we're still running 2.2 around here, but it would be nice
to have a reason to upgrade, anyway. Thanks for the pointer!
In article <11************ **********@g14g 2000cwa.googleg roups.com>, co************@ comcast.net wrote: Hey guys, here's a little question. I'm looking for something like a build system, where I can set up a bunch of jobs with dependencies, and have them run in parallel. Ideally, I'd like a system where jobs can be run in parallel, all the stdout and stderr for each job is kept in a database of some kind, jobs can be anything that can be run at a command line, and a failing job won't cause the whole run to fail. I've been looking into some of the standard stuff (GNU make) and some of the more pythonic stuff like SCons and AAP. SCons seems to do parallel execution, which is nice, but I can't figure out from the docs if it's possible to keep stdout and stderr somewhere. AAP seems to have a nice interface, but it doesn't do parallel.
Here's something from the SCons Wiki: http://www.scons.org/cgi-bin/wiki/BuildLog
-Steve
--
So if you people can't figure out what my birthday is based on subtle
hints and vague statements made at random intervals then I'm beginning
to think that you really don't care. -- Dean Lenort, a.r.k. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Hung Jung Lu |
last post by:
Hi,
Does anyone know whether there is any generic GUI-based build system
around? (Python-based would be great. Also, for now I am only looking
for Windows OS.) By "build system" I mean something similar the
"make/makefile" system, but somewhat more
generic/powerful/flexible/higher-level/etc, so that if you have a
sequence (maybe even some simple conditional branches and loops) of
system tasks to perform, you can more easily visualize...
|
by: ted |
last post by:
How does the speed of the Scons build tool compare with Ant? Right now with
out Ant builds take around an hour. Hoping to speed that up.
TIA,
Ted
|
by: Giovanni Bajo |
last post by:
Hello,
python24.dll is much bigger than python23.dll. This was discussed already on
the newsgroup, see the thread starting here:
http://mail.python.org/pipermail/python-list/2004-July/229096.html
I don't think I fully understand the reason why additional .pyd modules were
built into the .dll. OTOH, this does not help anyone, since:
- Normal users don't care about the size of the pythonXX.dll, or the number of
|
by: Marco |
last post by:
Is there anything better than using make for fairly large C++ projects?
Could be an open-source, free or inexpensive commercial tool. It needs
to easily integrate with a SCM tool.
I want to hear from experienced users.
thanks in advance
Marco
|
by: Xela |
last post by:
Hi
I am facing the following problem. I load a fact table with around 25
millons lines, and 7 indexes. I load it with 3 million line subsets. I am on
a quadriprocessor Solaris machine. The loading phase seems quite efficient
and seems to be well parallelized. But the build phase is long and seems to
use only one processor. Very low IO wait occurs during this phase. Is there
something can I do to fasten this process? I guess that if the 4...
| |
by: seberino |
last post by:
How does Scons compare to distutils? Should I ignore
it or move to it?
Chris
|
by: gamename |
last post by:
Hi,
I just installed scons on win32 that has cygwin but uses ActiveState
python. If I do "import SCons", the lib isn't found.
The win32 installer seemed very complete, but the scons lib doesn't
seem to be in any dir python knows about.
Is there another install step I need to do?
TIA,
-T
|
by: anush |
last post by:
Can anybody tell how I could go about running python scripts with
scons.
|
by: Hussein B |
last post by:
Hi.
Apache Ant is the de facto building tool for Java (whether JSE, JEE
and JME) application.
With Ant you can do what ever you want: compile, generate docs,
generate code, packing, deploy, connecting to remote servers and every
thing.
Do we have such a tool for Python projects?
Thank you.
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| |
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |