Hi,
I want send my jobs over a whole bunch of machines (using ssh). The
jobs will need to be run in the following pattern:
(Machine A) (Machine B) (Machine C)
Job A1 Job B1 Job C1
Job A2 Job B2 etc
Job A3 etc
etc
Jobs runing on machine A, B, C should be in parallel, however, for
each machine, jobs should run one after another.
How can I do it with the subprocess?
Thanks,
JD 10 2559
JD <Ji*********@gm ail.comwrote:
How can I do it with the subprocess?
You can't. Subprocess is a library to spawn new processes on the local
machine. If you want to handle external machines you need something like
parallel python: <http://www.parallelpyt hon.com/>
--
Lawrence, oluyede.org - neropercaso.it
"It is difficult to get a man to understand
something when his salary depends on not
understanding it" - Upton Sinclair
On Oct 3, 9:46 am, JD <Jiandong...@gm ail.comwrote:
Hi,
I want send my jobs over a whole bunch of machines (using ssh). The
jobs will need to be run in the following pattern:
(Machine A) (Machine B) (Machine C)
Job A1 Job B1 Job C1
Job A2 Job B2 etc
Job A3 etc
etc
Jobs runing on machine A, B, C should be in parallel, however, for
each machine, jobs should run one after another.
How can I do it with the subprocess?
subprocess is not network aware. What you can do is write a simple
python script say run_jobs.py which can take in a command-line
argument (say A or B or C) and will fire a sequence of subprocesses to
execute a series of jobs. This will ensure the serialization condition
like A2 starting after A1's completion.
Now you can write a load distributer kind of script which uses ssh to
login to the various machines and run run_jobs.py with appropriate
argument (Here I assume all machines have access to run_jobs.py -- say
it may reside on a shared mounted file-system).
e.g. in outer script:
ssh machine-A run_jobs.py A
ssh machine-B run_jobs.py B
ssh machine-B run_jobs.py C
....
You may want to fire all these at once so that they all execute in
parallel.
Karthik
>
Thanks,
JD
In message <11************ *********@22g20 00hsm.googlegro ups.com>, JD wrote:
I want send my jobs over a whole bunch of machines (using ssh). The
jobs will need to be run in the following pattern:
(Machine A) (Machine B) (Machine C)
Job A1 Job B1 Job C1
Job A2 Job B2 etc
Job A3 etc
etc
Jobs runing on machine A, B, C should be in parallel, however, for
each machine, jobs should run one after another.
How can I do it with the subprocess?
You could do it with SSH. A command like
ssh machine_a run_job_a1.py
will not terminate until the execution of run_job_a1.py on the remote
machine has terminated. So you end up with a lot of "proxy" subprocesses,
if you like, on the master machine, each one waiting for a remote process
on some master machine to terminate. As the controlling process notices the
termination of each proxy process, it looks to see which slave machine that
maps to, and sends another command to start the next job on that machine.
In message <fe**********@l ust.ihug.co.nz> , Lawrence D'Oliveiro wrote:
... each one waiting for a remote process
on some master machine to terminate.
Of course, "master" there should be "slave". :)
On Wed, 03 Oct 2007 22:19:30 +0200, Lawrence Oluyede wrote:
JD <Ji*********@gm ail.comwrote:
>How can I do it with the subprocess?
You can't. Subprocess is a library to spawn new processes on the local
machine. If you want to handle external machines you need something like
parallel python: <http://www.parallelpyt hon.com/>
Sure he can--you didn't read his post carefully. He wants to run jobs on
other machines using ssh, which is installed on the local machine.
subprocess can call ssh.
Carl Banks
On Wed, 03 Oct 2007 16:46:20 +0000, JD wrote:
Hi,
I want send my jobs over a whole bunch of machines (using ssh). The jobs
will need to be run in the following pattern:
(Machine A) (Machine B) (Machine C)
Job A1 Job B1 Job C1
Job A2 Job B2 etc
Job A3 etc
etc
Jobs runing on machine A, B, C should be in parallel, however, for each
machine, jobs should run one after another.
How can I do it with the subprocess?
It's not too hard if the remote jobs exit gracefully, so that ssh knows
when the remote job is done.
If that's the case, the easiest thing to do might be to simply run a
different script for each machine, and use subprocess.call ().
For example, in one script, do this:
subprocess.call ("ssh A jobA1")
subprocess.call ("ssh A jobA2")
subprocess.call ("ssh A jobA3")
subprocess.call ("ssh A jobA4")
In another, do this:
subprocess.call ("ssh B jobB1")
subprocess.call ("ssh B jobB2")
subprocess.call ("ssh B jobB3")
subprocess.call ("ssh B jobB4")
And so on. Then simply run them at the same time.
If you can't do that--and I recommend you do it that way if you can--then
you can either use threads, or detect when the processes complete
asynchronously using os.wait. I'm not sure if threads and subprocesses
work well together on all machines, to be honest; there could be some
signal issues.
Briefly, here's how os.wait would work. You run parallel subprocesses
like this:
pid_A = subprocess.Pope n("ssh A jobA1").pid
pid_B = subprocess.Pope n("ssh B jobB1").pid
pid_C = subprocess.Pope n("ssh C jobC1").pid
Then call os.wait:
pid,status = os.wait()
os.wait should return the process id of the first subprocess to exit.
You can then tell which machine it was by comparing pid to pid_A, pid_B,
or pid_C, and start the next job accordingly. What you would do is to
call os.wait() in a loop, waiting and spawning the next job for a
particular machine, until all jobs are done. The bookkeeping necessary
to do all this is left an exercise.
Carl Banks
In message <47************ ***********@roa drunner.com>, Carl Banks wrote:
On Wed, 03 Oct 2007 16:46:20 +0000, JD wrote:
>> I want send my jobs over a whole bunch of machines (using ssh). The jobs will need to be run in the following pattern:
(Machine A) (Machine B) (Machine C)
Job A1 Job B1 Job C1
Job A2 Job B2 etc
Job A3 etc
etc
Jobs runing on machine A, B, C should be in parallel, however, for each machine, jobs should run one after another.
How can I do it with the subprocess?
It's not too hard if the remote jobs exit gracefully, so that ssh knows
when the remote job is done.
It's hard to see how an _ungraceful_ exit could stop SSH from knowing when
the remote job is done.
On Oct 4, 4:55 am, Lawrence D'Oliveiro <l...@geek-
central.gen.new _zealandwrote:
In message <4704a187$1$189 27$4c368...@roa drunner.com>, Carl Banks wrote:
On Wed, 03 Oct 2007 16:46:20 +0000, JD wrote:
I want send my jobs over a whole bunch of machines (using ssh). The jobs
will need to be run in the following pattern:
(Machine A) (Machine B) (Machine C)
Job A1 Job B1 Job C1
Job A2 Job B2 etc
Job A3 etc
etc
Jobs runing on machine A, B, C should be in parallel, however, for each
machine, jobs should run one after another.
How can I do it with the subprocess?
It's not too hard if the remote jobs exit gracefully, so that ssh knows
when the remote job is done.
It's hard to see how an _ungraceful_ exit could stop SSH from knowing when
the remote job is done.
Clean exit might have been a better word. Various things can cause
ssh to exit before the job is done; other things can make the process
hang around after the job is finished. The OP needs to make sure the
job, and ssh, exit reliably enough for the given use before depending
on it. Otherwise, resorting to things like lockfiles and timeouts may
be necessary to keep things sequential.
Carl Banks
You don't necessarily need the subprocess module to do this, though you
could use it.
I've done this sort of thing in the past with fork and exec.
To serialize the jobs on the machines, the easiest thing is to just send
the commands all at once to a given machine, like "command1; command2;
command3".
You can use waitpid or similar to check if a series of jobs has finished
on a particular machine.
An example of something similar can be found at http://stromberg.dnsalias.org/~strombrg/loop.html
(If you look at the code, be kind. I wrote it long ago :)
There's a benefit to saving the output from each machine into a single
file for that machine. If you think some machines will produce the same
output, and you don't want to see it over and over, you can analyze the
files with something like http://stromberg.dnsalias.org/~strom...e-classes.html .
On Wed, 03 Oct 2007 16:46:20 +0000, JD wrote:
Hi,
I want send my jobs over a whole bunch of machines (using ssh). The
jobs will need to be run in the following pattern:
(Machine A) (Machine B) (Machine C)
Job A1 Job B1 Job C1
Job A2 Job B2 etc
Job A3 etc
etc
Jobs runing on machine A, B, C should be in parallel, however, for
each machine, jobs should run one after another.
How can I do it with the subprocess?
Thanks,
JD
This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Marc Carter |
last post by:
I am trying to rewrite a PERL automation which started a "monitoring"
application on many machines, via RSH, and then multiplexed their
collective outputs to stdout.
In production there are lots of these subprocesses but here is a
simplified example what I have so far (python n00b alert!)
- SNIP ---------
import subprocess,select,sys
...
|
by: Stewart Midwinter |
last post by:
this has me puzzled; I've created a small test app to show the problem
I'm having.
I want to use subprocess to execute system commands from inside a
Tkinter app running under Cygwin.
When I open a python interpreter and run my subprocess command, all is
well. But when I run the same command from inside a Tkinter app, I'm
getting errors.
|
by: Darren Dale |
last post by:
I'm a developer on the matplotlib project, and I am having trouble with the
subprocess module on windows (Python 2.4.2 on winXP). No trouble to report
with linux. I need to use _subprocess instead of pywin32, but my trouble
exists with either option:
import subprocess
process = subprocess.Popen(, stderr=subprocess.STDOUT,...
|
by: Cameron Laird |
last post by:
Question:
import subprocess, StringIO
input = StringIO.StringIO("abcdefgh\nabc\n")
# I don't know of a compact, evocative, and
# cross-platform way to exhibit this behavior.
# For now, depend on cat(1).
p = subprocess.Popen(, stdout = subprocess.PIPE,
stdin = response)
|
by: Grant Edwards |
last post by:
I'm trying to use the py-gnuplot module on windows, and have
been unable to get it to work reliably under Win2K and WinXP.
By default, it uses popen(gnuplotcmd,'w'), but in some
situations that consistently gets an "invalid operand" IOError
when write() is called on the pipe.
So I switched to subprocess. It works fine when executed...
| |
by: Phoe6 |
last post by:
Hi all,
Consider this scenario, where in I need to use subprocess to execute a
command like 'ping 127.0.0.1' which will have a continuous non-
terminating output in Linux.
# code
# This hangs at this point.
How should I handle these kind of commands (ping 127.0.0.1) with
|
by: bhunter |
last post by:
Hi,
I've used subprocess with 2.4 several times to execute a process, wait
for it to finish, and then look at its output. Now I want to spawn
the process separately, later check to see if it's finished, and if it
is look at its output. I may want to send a signal at some point to
kill the process. This seems straightforward, but it...
|
by: skunkwerk |
last post by:
Hi,
i'm trying to call subprocess.popen on the 'rename' function in
linux. When I run the command from the shell, like so:
rename -vn 's/\.htm$/\.html/' *.htm
it works fine... however when I try to do it in python like so:
p = subprocess.Popen(,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
print p.communicate()
|
by: Gabriel Genellina |
last post by:
En Tue, 19 Aug 2008 16:48:26 -0300, aditya shukla
<adityashukla1983@gmail.comescribi�:
I don't have a Vista machine to test right now, but I'd expect the above
sequence to hang indefinitely. If you want to execute something, don't
forget the final \n
On XP this works fine:
import subprocess
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it. ...
| |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...
| |