473,320 Members | 2,003 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Cross Server File Transfer

162 100+
I need to transfer files between a network of servers all over the world. I have looked into using cURL and ftp_fput() but am having little luck. I can get a transfer going from my local to the server or visa versa but not from server to server.

Basically im trying to replicate Akamai on a very small scale. This is the scenario. I am working from home. I search for a file on Web Server 1 in Oregon. The file doesn't exist. I search my other servers and find that it is located on Server 2 in Germany. I then need to COPY the file from Web Server 2 to Web Server 1. That's it.

I have it all in place except getting the server-to-server transfer to work. Who can help? Thanks much.
Mar 10 '09 #1
6 3390
empiresolutions
162 100+
One other thing. The connections need to be made via FTP. I have IP, USER & PASSWORDS for each account. Thanks.
Mar 10 '09 #2
Markus
6,050 Expert 4TB
What exactly are you having a problem with? Do you get any errors? What does happen? What doesn't happen?
Mar 10 '09 #3
empiresolutions
162 100+
Thanks for all your input. Let me explain a little more about what i have done. First of, each of the files I'm transferring are over 1Gb so file_get_contents() wont work. Files are not transferred until requested by the user. I would prefer to do it all in CURL, but as i said im having issues.

I have looked into SST scripts but none seems to satisfy.

My thinking is that cURL cannot directly transfer a file server to server (please correct if wrong). I thought that the way to make it work would be to call a file on Web Server 2 using cURL and add some parameters (file name to transfer) to the call so that when ran it would using cURL transfer the file to Web Server 1. Sound good? Problem, Web Server 2 doesn't have a URL that will resolve to a specific file on the server. Therefor again only cURL via FTP login can be used (as i see it).

So do further define my question, with cURL, is there a way to transfer files between servers directly using a curl statement like
Expand|Select|Wrap|Line Numbers
  1. $ch = curl_init("ftp://theuser:thepass@72.47.251.180/testfile.m4v");
  2.  
but one that works more like below with a location to get and set from/to. I know the following code is incorrect, i just would like to know how to get it right.
Expand|Select|Wrap|Line Numbers
  1. $c = curl_init("ftp://theuser:thepass@72.47.251.180/testfile.m4v","ftp://theuser:thepass@42.76.151.236/testfile.m4v");
  2.  
I know i have left out parts of the cURL process. curl_exec() and curl_close() are pretty standard. Thanks.
Mar 10 '09 #4
empiresolutions
162 100+
SOLVED -

Thanks for everyones help. I am posting my full function for those who may find it useful. To recap, this function "try's" to simulate Akamai's service.

1. GeoLoc user.
2. GeoLoc all servers
3. Calculate distance between user and each server.
4. Sort by distance.
5. Check to see if file is at closest server. If so download.
6. If file not at closest server, copy file from mother server to server missing file. Then notify user via email with download link to closest server.

Expand|Select|Wrap|Line Numbers
  1. function sst() {
  2.  
  3.     global $_GET, $_SERVER;
  4.  
  5.     /*   start geo loc   */
  6.     // http://www.imaginerc.com/software/GeoCalc/
  7.     require_once('standards/classes/geoplugin.class.php'); 
  8.     $geoplugin = new geoPlugin();
  9.     // http://www.phpclasses.org/browse/file/25516.html
  10.     include_once("standards/classes/GeoCalc.class.php"); 
  11.     $oGC = new GeoCalc();
  12.  
  13.     /*   set server ip's and FTP logins        */
  14.     $servers = array($_SERVER['REMOTE_ADDR'],'xxx.xx.xx.xxx','xxx.xx.xx.xxx');
  15.     $user = array('','usr1','usr2');
  16.     $pass = array('','pass1','pass2',);
  17.  
  18.     /*   get user location   */
  19.     $geo[0]['ip'] = $servers[0];
  20.     $geoplugin->locate($geo[0]['ip']);
  21.     $geo[0]['lat'] = $geoplugin->latitude;
  22.     $geo[0]['lon'] = $geoplugin->longitude;
  23.     $geo[0]['city'] = $geoplugin->city;
  24.     $geo[0]['countryName'] = $geoplugin->countryName;
  25.  
  26.     /*   get  servers data  */
  27.     for ($i = 1; $i < count($servers); $i++) {
  28.  
  29.         $geo[$i]['ip'] = $servers[$i];
  30.         $geo[$i]['user'] = $user[$i];
  31.         $geo[$i]['pass'] = $pass[$i];
  32.         $geoplugin->locate($geo[$i]['ip']);
  33.         $geo[$i]['lat'] = $geoplugin->latitude;
  34.         $geo[$i]['lon'] = $geoplugin->longitude;
  35.         $geo[$i]['city'] = $geoplugin->city;
  36.         $geo[$i]['countryName'] = $geoplugin->countryName;
  37.         $geo[$i]['dist'] = $oGC->GCDistance($geo[0]['lat'],$geo[0]['lon'],$geo[$i]['lat'],$geo[$i]['lon']);
  38.  
  39.     }
  40.  
  41.     /*   order by distance.   */
  42.     $geo = orderBy($geo, 'dist');
  43.  
  44.     /*   connect to closeest server   */
  45.     $conn_id = ftp_connect($geo[1]['ip']) or die("Couldn't connect to $ftp_server");
  46.     ftp_login($conn_id, $geo[1]['user'], $geo[1]['pass']);
  47.     $contents = ftp_nlist($conn_id, "tts/incoming");
  48.     ftp_close($conn_id);
  49.  
  50.     $localfile = '/user/dac15/tts/incoming/';
  51.     $remotefile = '/user/'.$geo[1]['user'].'/tts/incoming/';
  52.  
  53.     /*  if file is on the closest server.    */
  54.     if (find($_GET['name'],$contents)) {
  55.  
  56.         // http://www.awesomephp.com/?Tutorials*16/Download-file-with-resume,-stream-and-speed-options.html
  57.         downloadFile($remotefile,$_GET['name'],900,false); 
  58.  
  59.     }else{ 
  60.  
  61.         /*   push file to the closest server requesting server.*/
  62.         $ch = curl_init();
  63.         $fp = fopen($localfile.$_GET['name'], 'r');
  64.         curl_setopt($ch, CURLOPT_URL, 'ftp://'.$geo[1]['user'].':'.$geo[1]['pass'].'@'.$geo[1]['ip'].'/'.$remotefile.$_GET['name']);
  65.         curl_setopt($ch, CURLOPT_UPLOAD, 1);
  66.         curl_setopt($ch, CURLOPT_INFILE, $fp);
  67.         curl_setopt($ch, CURLOPT_INFILESIZE, filesize($localfile.$_GET['name']));
  68.         curl_exec($ch);
  69.         $error_no = curl_errno($ch);
  70.         curl_close($ch);
  71.         if ($error_no == 0) {
  72.             $error = 'File uploaded succesfully.';
  73.  
  74.             $mailto[0][0] = $_SERVER['HTTP_EMAIL'];
  75.             $mailto[0][1] = $_SERVER['HTTP_FNAME'];
  76.             $mailto[0][2] = $_SERVER['HTTP_LNAME'];
  77.  
  78.             $content="
  79.             Dear ".$mailto[0][1]."
  80.             <br>
  81.             Your file is ready for download. Go to this link in your preferred web browser.
  82.             ";
  83.  
  84.             // http://www.xpertmailer.com/
  85.             if (smtp_mail($mailto,"DoD File Download",$content)) { Return 1; }
  86.  
  87.         }else{
  88.             $error = 'File upload error.';
  89.             echo $error;
  90.         }
  91.  
  92.     }
  93.  
  94.     die;
  95.  
  96. }
  97.  
  98. function orderBy($data, $field, $reverse_sort='') {
  99.     $code = "return strnatcmp(\$a['$field'], \$b['$field']);";
  100.     usort($data, create_function('$a,$b', $code));
  101.     if ($reverse_sort==1) { krsort($data); }
  102.     return $data;
  103. }
  104.  
Mar 11 '09 #5
empiresolutions
162 100+
I have another question to go along with this post. Once the cURL transfer is started on my 1GB file the screen just sits white while doing its thing (page load bar crawls). Is there a way to finish loading the page while the transfer is in process. This way the user doesn't have to wait for the transfer to finish, they are just there to make it happen. (they are not downloading at the time so it would be okay for them to continue). Thanks.
Mar 11 '09 #6
empiresolutions
162 100+
Expand|Select|Wrap|Line Numbers
  1. /**
  2.  * Takes a needle and haystack (just like in_array()) and does a wildcard search on it's values.
  3.  *
  4.  * @param    string        $string        Needle to find
  5.  * @param    array        $array        Haystack to look through
  6.  * @result    array                    Returns the elements that the $string was found in
  7.  */
  8. function find ($string, $array = array ())
  9. {
  10.     foreach ($array as $key => $value) {
  11.         unset ($array[$key]);
  12.         if (strpos($value, $string) !== false) {
  13.             $array[$key] = $value;
  14.         }
  15.     }
  16.     return $array;
  17. }
  18.  
Mar 17 '09 #7

Sign in to post your reply or Sign up for a free account.

Similar topics

9
by: Steve Buster | last post by:
All right, I have read every forum, newsgroup etc about this issue and no one seems to know how to fix it. I am getting a "Server Application Unavailable" exception running my .NET 1.1...
6
by: Andrew | last post by:
Hello, friends, I tried to do cross-page posting, i.e., in firstpage.aspx I used action="secondpage.aspx". Since I am using asp.net 1.1, so it always goes back to firstpage.aspx aftern...
18
by: UJ | last post by:
Folks, We provide custom content for our customers. Currently we put the files on our server and people have a program we provide that will download the files. These files are usually SWF, HTML or...
8
by: bryan | last post by:
I've got a custom HttpHandler to process all requests for a given extension. It gets invoked OK, but if I try to do a Server.Transfer I get an HttpException. A Response.Redirect works, but I really...
10
by: John Salerno | last post by:
I always read about how you need to set certain file permissions (for cgi files, for example), but it's never been clear to me *how* you do this. I know you can run the line chmod 755...
1
by: Alex | last post by:
Hello, I'm trying to write a little php script to transfert some files from a server to clients (web/http). It's working fin with small files. But transfering big files (try on 1Gb) failed!...
3
by: Peter Nofelt | last post by:
On Server.Transfer(), is there a reason why the previous page's elements would appear on the new page? Note: this transfer is occuring between two pages on the same server. <Scenario> I'm using...
13
by: Jeff | last post by:
We have an intranet website that currently uses ActiveX but we need to make it cross-browser compatible and also get around the problems we've been having with making it work with IE7 and Vista. We...
4
by: CSINVA | last post by:
Trying to figure out how to do a cross post page within the code behind file. I need to do some calculations before I send a string value to another pages. Here is an example of the concept I...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.