473,228 Members | 1,814 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,228 software developers and data experts.

A question about subprocess

JD
Hi,

I want send my jobs over a whole bunch of machines (using ssh). The
jobs will need to be run in the following pattern:

(Machine A) (Machine B) (Machine C)

Job A1 Job B1 Job C1

Job A2 Job B2 etc

Job A3 etc

etc

Jobs runing on machine A, B, C should be in parallel, however, for
each machine, jobs should run one after another.

How can I do it with the subprocess?
Thanks,

JD

Oct 3 '07 #1
10 2543
JD <Ji*********@gmail.comwrote:
How can I do it with the subprocess?
You can't. Subprocess is a library to spawn new processes on the local
machine. If you want to handle external machines you need something like
parallel python: <http://www.parallelpython.com/>

--
Lawrence, oluyede.org - neropercaso.it
"It is difficult to get a man to understand
something when his salary depends on not
understanding it" - Upton Sinclair
Oct 3 '07 #2
On Oct 3, 9:46 am, JD <Jiandong...@gmail.comwrote:
Hi,

I want send my jobs over a whole bunch of machines (using ssh). The
jobs will need to be run in the following pattern:

(Machine A) (Machine B) (Machine C)

Job A1 Job B1 Job C1

Job A2 Job B2 etc

Job A3 etc

etc

Jobs runing on machine A, B, C should be in parallel, however, for
each machine, jobs should run one after another.

How can I do it with the subprocess?
subprocess is not network aware. What you can do is write a simple
python script say run_jobs.py which can take in a command-line
argument (say A or B or C) and will fire a sequence of subprocesses to
execute a series of jobs. This will ensure the serialization condition
like A2 starting after A1's completion.

Now you can write a load distributer kind of script which uses ssh to
login to the various machines and run run_jobs.py with appropriate
argument (Here I assume all machines have access to run_jobs.py -- say
it may reside on a shared mounted file-system).

e.g. in outer script:

ssh machine-A run_jobs.py A
ssh machine-B run_jobs.py B
ssh machine-B run_jobs.py C
....

You may want to fire all these at once so that they all execute in
parallel.

Karthik
>
Thanks,

JD

Oct 4 '07 #3
In message <11*********************@22g2000hsm.googlegroups.c om>, JD wrote:
I want send my jobs over a whole bunch of machines (using ssh). The
jobs will need to be run in the following pattern:

(Machine A) (Machine B) (Machine C)

Job A1 Job B1 Job C1

Job A2 Job B2 etc

Job A3 etc

etc

Jobs runing on machine A, B, C should be in parallel, however, for
each machine, jobs should run one after another.

How can I do it with the subprocess?
You could do it with SSH. A command like

ssh machine_a run_job_a1.py

will not terminate until the execution of run_job_a1.py on the remote
machine has terminated. So you end up with a lot of "proxy" subprocesses,
if you like, on the master machine, each one waiting for a remote process
on some master machine to terminate. As the controlling process notices the
termination of each proxy process, it looks to see which slave machine that
maps to, and sends another command to start the next job on that machine.
Oct 4 '07 #4
In message <fe**********@lust.ihug.co.nz>, Lawrence D'Oliveiro wrote:
... each one waiting for a remote process
on some master machine to terminate.
Of course, "master" there should be "slave". :)
Oct 4 '07 #5
On Wed, 03 Oct 2007 22:19:30 +0200, Lawrence Oluyede wrote:
JD <Ji*********@gmail.comwrote:
>How can I do it with the subprocess?

You can't. Subprocess is a library to spawn new processes on the local
machine. If you want to handle external machines you need something like
parallel python: <http://www.parallelpython.com/>
Sure he can--you didn't read his post carefully. He wants to run jobs on
other machines using ssh, which is installed on the local machine.
subprocess can call ssh.
Carl Banks
Oct 4 '07 #6
On Wed, 03 Oct 2007 16:46:20 +0000, JD wrote:
Hi,

I want send my jobs over a whole bunch of machines (using ssh). The jobs
will need to be run in the following pattern:

(Machine A) (Machine B) (Machine C)

Job A1 Job B1 Job C1

Job A2 Job B2 etc

Job A3 etc

etc

Jobs runing on machine A, B, C should be in parallel, however, for each
machine, jobs should run one after another.

How can I do it with the subprocess?

It's not too hard if the remote jobs exit gracefully, so that ssh knows
when the remote job is done.

If that's the case, the easiest thing to do might be to simply run a
different script for each machine, and use subprocess.call().

For example, in one script, do this:
subprocess.call("ssh A jobA1")
subprocess.call("ssh A jobA2")
subprocess.call("ssh A jobA3")
subprocess.call("ssh A jobA4")

In another, do this:
subprocess.call("ssh B jobB1")
subprocess.call("ssh B jobB2")
subprocess.call("ssh B jobB3")
subprocess.call("ssh B jobB4")

And so on. Then simply run them at the same time.
If you can't do that--and I recommend you do it that way if you can--then
you can either use threads, or detect when the processes complete
asynchronously using os.wait. I'm not sure if threads and subprocesses
work well together on all machines, to be honest; there could be some
signal issues.

Briefly, here's how os.wait would work. You run parallel subprocesses
like this:

pid_A = subprocess.Popen("ssh A jobA1").pid
pid_B = subprocess.Popen("ssh B jobB1").pid
pid_C = subprocess.Popen("ssh C jobC1").pid
Then call os.wait:

pid,status = os.wait()
os.wait should return the process id of the first subprocess to exit.
You can then tell which machine it was by comparing pid to pid_A, pid_B,
or pid_C, and start the next job accordingly. What you would do is to
call os.wait() in a loop, waiting and spawning the next job for a
particular machine, until all jobs are done. The bookkeeping necessary
to do all this is left an exercise.
Carl Banks
Oct 4 '07 #7
In message <47***********************@roadrunner.com>, Carl Banks wrote:
On Wed, 03 Oct 2007 16:46:20 +0000, JD wrote:
>>
I want send my jobs over a whole bunch of machines (using ssh). The jobs
will need to be run in the following pattern:

(Machine A) (Machine B) (Machine C)

Job A1 Job B1 Job C1

Job A2 Job B2 etc

Job A3 etc

etc

Jobs runing on machine A, B, C should be in parallel, however, for each
machine, jobs should run one after another.

How can I do it with the subprocess?


It's not too hard if the remote jobs exit gracefully, so that ssh knows
when the remote job is done.
It's hard to see how an _ungraceful_ exit could stop SSH from knowing when
the remote job is done.
Oct 4 '07 #8
On Oct 4, 4:55 am, Lawrence D'Oliveiro <l...@geek-
central.gen.new_zealandwrote:
In message <4704a187$1$18927$4c368...@roadrunner.com>, Carl Banks wrote:
On Wed, 03 Oct 2007 16:46:20 +0000, JD wrote:
I want send my jobs over a whole bunch of machines (using ssh). The jobs
will need to be run in the following pattern:
(Machine A) (Machine B) (Machine C)
Job A1 Job B1 Job C1
Job A2 Job B2 etc
Job A3 etc
etc
Jobs runing on machine A, B, C should be in parallel, however, for each
machine, jobs should run one after another.
How can I do it with the subprocess?
It's not too hard if the remote jobs exit gracefully, so that ssh knows
when the remote job is done.

It's hard to see how an _ungraceful_ exit could stop SSH from knowing when
the remote job is done.
Clean exit might have been a better word. Various things can cause
ssh to exit before the job is done; other things can make the process
hang around after the job is finished. The OP needs to make sure the
job, and ssh, exit reliably enough for the given use before depending
on it. Otherwise, resorting to things like lockfiles and timeouts may
be necessary to keep things sequential.
Carl Banks

Oct 4 '07 #9

You don't necessarily need the subprocess module to do this, though you
could use it.

I've done this sort of thing in the past with fork and exec.

To serialize the jobs on the machines, the easiest thing is to just send
the commands all at once to a given machine, like "command1; command2;
command3".

You can use waitpid or similar to check if a series of jobs has finished
on a particular machine.

An example of something similar can be found at
http://stromberg.dnsalias.org/~strombrg/loop.html

(If you look at the code, be kind. I wrote it long ago :)

There's a benefit to saving the output from each machine into a single
file for that machine. If you think some machines will produce the same
output, and you don't want to see it over and over, you can analyze the
files with something like
http://stromberg.dnsalias.org/~strom...e-classes.html .
On Wed, 03 Oct 2007 16:46:20 +0000, JD wrote:
Hi,

I want send my jobs over a whole bunch of machines (using ssh). The
jobs will need to be run in the following pattern:

(Machine A) (Machine B) (Machine C)

Job A1 Job B1 Job C1

Job A2 Job B2 etc

Job A3 etc

etc

Jobs runing on machine A, B, C should be in parallel, however, for
each machine, jobs should run one after another.

How can I do it with the subprocess?
Thanks,

JD

Oct 4 '07 #10
JD
Thanks very much for all the answers.

JD

On Oct 3, 6:24 pm, Dan Stromberg <dstrombergli...@gmail.comwrote:
You don't necessarily need thesubprocessmodule to do this, though you
could use it.

I've done this sort of thing in the past with fork and exec.

To serialize the jobs on the machines, the easiest thing is to just send
the commands all at once to a given machine, like "command1; command2;
command3".

You can use waitpid or similar to check if a series of jobs has finished
on a particular machine.

An example of something similar can be found athttp://stromberg.dnsalias.org/~strombrg/loop.html

(If you look at the code, be kind. I wrote it long ago :)

There's a benefit to saving the output from each machine into a single
file for that machine. If you think some machines will produce the same
output, and you don't want to see it over and over, you can analyze the
files with something likehttp://stromberg.dnsalias.org/~strombrg/equivalence-classes.html.

On Wed, 03 Oct 2007 16:46:20 +0000, JD wrote:
Hi,
I want send my jobs over a whole bunch of machines (using ssh). The
jobs will need to be run in the following pattern:
(Machine A) (Machine B) (Machine C)
Job A1 Job B1 Job C1
Job A2 Job B2 etc
Job A3 etc
etc
Jobs runing on machine A, B, C should be in parallel, however, for
each machine, jobs should run one after another.
How can I do it with thesubprocess?
Thanks,
JD

Oct 5 '07 #11

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Marc Carter | last post by:
I am trying to rewrite a PERL automation which started a "monitoring" application on many machines, via RSH, and then multiplexed their collective outputs to stdout. In production there are lots...
2
by: Stewart Midwinter | last post by:
this has me puzzled; I've created a small test app to show the problem I'm having. I want to use subprocess to execute system commands from inside a Tkinter app running under Cygwin. When I...
3
by: Darren Dale | last post by:
I'm a developer on the matplotlib project, and I am having trouble with the subprocess module on windows (Python 2.4.2 on winXP). No trouble to report with linux. I need to use _subprocess instead...
5
by: Cameron Laird | last post by:
Question: import subprocess, StringIO input = StringIO.StringIO("abcdefgh\nabc\n") # I don't know of a compact, evocative, and # cross-platform way to exhibit this behavior. # For now, depend...
5
by: Grant Edwards | last post by:
I'm trying to use the py-gnuplot module on windows, and have been unable to get it to work reliably under Win2K and WinXP. By default, it uses popen(gnuplotcmd,'w'), but in some situations that...
9
by: Phoe6 | last post by:
Hi all, Consider this scenario, where in I need to use subprocess to execute a command like 'ping 127.0.0.1' which will have a continuous non- terminating output in Linux. # code # This...
12
by: bhunter | last post by:
Hi, I've used subprocess with 2.4 several times to execute a process, wait for it to finish, and then look at its output. Now I want to spawn the process separately, later check to see if it's...
7
by: skunkwerk | last post by:
Hi, i'm trying to call subprocess.popen on the 'rename' function in linux. When I run the command from the shell, like so: rename -vn 's/\.htm$/\.html/' *.htm it works fine... however when I...
2
by: Gabriel Genellina | last post by:
En Tue, 19 Aug 2008 16:48:26 -0300, aditya shukla <adityashukla1983@gmail.comescribi�: I don't have a Vista machine to test right now, but I'd expect the above sequence to hang indefinitely....
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.