splitting a script over two servers

#1 April 12th, 2007, 04:05 PM

can someone point me in the right direction here..

I have a script (essentially a giant for loop) that takes about 20
mins to run on my server. I want "split" this process onto a second
server by doing something like this:

system("/srv1/path/to/php /path/to/script.php ".$arg1." ".$arg2." /
dev/null &");
system("/srv2/path/to/php srv1/path/to/script.php ".$arg1." ".$arg2."

Quote:

/dev/null &");

ideas? thoughts? questions? comments? concerns? jokes?

#2 April 12th, 2007, 05:25 PM

bleen wrote:

Quote:

can someone point me in the right direction here..
>
I have a script (essentially a giant for loop) that takes about 20
mins to run on my server. I want "split" this process onto a second
server by doing something like this:
>
system("/srv1/path/to/php /path/to/script.php ".$arg1." ".$arg2." /
dev/null &");
system("/srv2/path/to/php srv1/path/to/script.php ".$arg1." ".$arg2."

Quote:

>/dev/null &");

>
ideas? thoughts? questions? comments? concerns? jokes?

Yes:
Do you know why six is afraid of seven ?

.....
.....

Because 7 8 9.

;-)

Good luck.

Regards,
Erwin Moller

#3 April 12th, 2007, 06:05 PM

"bleen" <bleen67@gmail.comwrote in message
news:1176393365.880214.123040@y80g2000hsf.googlegr oups.com...
| can someone point me in the right direction here..
|
| I have a script (essentially a giant for loop) that takes about 20
| mins to run on my server. I want "split" this process onto a second
| server by doing something like this:
|
| system("/srv1/path/to/php /path/to/script.php ".$arg1." ".$arg2." /
| dev/null &");
| system("/srv2/path/to/php srv1/path/to/script.php ".$arg1." ".$arg2."
| /dev/null &");
|
| ideas? thoughts? questions? comments? concerns? jokes?

depends on what you're doing, don't it?

i'm sure you've already tried to split the work load on the SAME server
already...assuming the nasty thing you're wanting to avoid here is the 20
minute wait time. if not, try that...let your os optimize the processing.
other than that, what are you doing that takes 20 minutes...a foreach on a
1mb array on a 100mz i386 w/ 4mb of ram?

#4 April 12th, 2007, 06:45 PM

bleen kirjoitti:

Quote:

can someone point me in the right direction here..
>
I have a script (essentially a giant for loop) that takes about 20
mins to run on my server. I want "split" this process onto a second
server by doing something like this:
>
system("/srv1/path/to/php /path/to/script.php ".$arg1." ".$arg2." /
dev/null &");
system("/srv2/path/to/php srv1/path/to/script.php ".$arg1." ".$arg2."

Quote:

>/dev/null &");

>
ideas? thoughts? questions? comments? concerns? jokes?
>

Well, here's an idea. I'm not saying it's a good idea.

<?php
if($_GET['first']) {
system("/srv1/path/to/php /path/to/script.php ".$arg1." ".$arg2." /
dev/null &");
} else if($_GET['second']) {
system("/srv2/path/to/php srv1/path/to/script.php ".$arg1." ".$arg2."

Quote:

/dev/null &");

} else {
?>
<html>
<body>
<img src="<?php echo $_SERVER['PHP_SELF'] ?>?first=1" />
<img src="<?php echo $_SERVER['PHP_SELF'] ?>?second=1" />
</body>
</html>
<?php } ?>

--
Rami.Elomaa@gmail.com

"Wikipedia on vähän niinq internetin raamattu, kukaan ei pohjimmiltaan
usko siihen ja kukaan ei tiedä mikä pitää paikkansa." -- z00ze

#5 April 12th, 2007, 09:05 PM

On Apr 12, 11:56 am, "bleen" <blee...@gmail.comwrote:

Quote:

can someone point me in the right direction here..
>
I have a script (essentially a giant for loop) that takes about 20
mins to run on my server. I want "split" this process onto a second
server by doing something like this:
>
system("/srv1/path/to/php /path/to/script.php ".$arg1." ".$arg2." /
dev/null &");
system("/srv2/path/to/php srv1/path/to/script.php ".$arg1." ".$arg2."
>

Quote:

/dev/null &");

>
ideas? thoughts? questions? comments? concerns? jokes?

What do you get when you cross a polar bear with a seal?
....
...
..
A polar bear!

And as for your question, it really depends on what you're trying to
do and if that work can be broken down into smaller pieces. You
should go through your script and try to break it out into as many
independent blocks as you can.

You say this is a giant for loop? Does the data need to be worked on
sequentially (ie: does the order of processing matter?). If not, you
can build up a queue of smaller jobs (one for each loop iteration in
your current script?), then offload one job to each server at a time.
When one server finishes, send it another job, etc, until they're all
done. Think of the way those Folding@Home/SETI@Home projects work.
They divide up work into chunks, send each chunk out for processing at
a random computer, and assemble the results into something meaningful
when they come back.

If you can't really split up the work, then you're SOL. I've got a
few of those scripts, myself and there's not much you can do about it.

#6 April 13th, 2007, 10:05 AM

bleen wrote:

Quote:

system("/srv1/path/to/php /path/to/script.php ".$arg1." ".$arg2." /
dev/null &");
system("/srv2/path/to/php srv1/path/to/script.php ".$arg1." ".$arg2."

Quote:

>/dev/null &");

Ummm.... those will still run on the same server.

What you could do would be:

<?php
$r1 = file("http://server1/path/to/script.php?arg1={$arg1}&arg2={$arg2}");
foreach ($r1 as $R) print $R;
$r2 = file("http://server2/path/to/script.php?arg1={$arg1}&arg2={$arg2}");
foreach ($r2 as $R) print $R;
?>

--
Toby A Inkster BSc (Hons) ARCS
Contact Me ~ http://tobyinkster.co.uk/contact
Geek of ~ HTML/SQL/Perl/PHP/Python*/Apache/Linux

* = I'm getting there!

#7 April 13th, 2007, 12:15 PM

Toby A Inkster wrote:

Quote:

bleen wrote:
>

Quote:

>system("/srv1/path/to/php /path/to/script.php ".$arg1." ".$arg2." /
>dev/null &");
>system("/srv2/path/to/php srv1/path/to/script.php ".$arg1." ".$arg2."

Quote:

>>/dev/null &");

>
Ummm.... those will still run on the same server.
>
What you could do would be:
>
<?php
$r1 = file("http://server1/path/to/script.php?arg1={$arg1}&arg2={$arg2}");
foreach ($r1 as $R) print $R;
$r2 = file("http://server2/path/to/script.php?arg1={$arg1}&arg2={$arg2}");
foreach ($r2 as $R) print $R;
?>
>

But they'll still be processed sequentially, not in parallel.

You would want to start the scripts as above, but have run the processes
on other machines, i.e. with rsexec or similar.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

splitting a script over two servers

Popular Articles

What is Bytes?