473,544 Members | 1,955 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Cross Server File Transfer

162 New Member
I need to transfer files between a network of servers all over the world. I have looked into using cURL and ftp_fput() but am having little luck. I can get a transfer going from my local to the server or visa versa but not from server to server.

Basically im trying to replicate Akamai on a very small scale. This is the scenario. I am working from home. I search for a file on Web Server 1 in Oregon. The file doesn't exist. I search my other servers and find that it is located on Server 2 in Germany. I then need to COPY the file from Web Server 2 to Web Server 1. That's it.

I have it all in place except getting the server-to-server transfer to work. Who can help? Thanks much.
Mar 10 '09 #1
6 3402
empiresolutions
162 New Member
One other thing. The connections need to be made via FTP. I have IP, USER & PASSWORDS for each account. Thanks.
Mar 10 '09 #2
Markus
6,050 Recognized Expert Expert
What exactly are you having a problem with? Do you get any errors? What does happen? What doesn't happen?
Mar 10 '09 #3
empiresolutions
162 New Member
Thanks for all your input. Let me explain a little more about what i have done. First of, each of the files I'm transferring are over 1Gb so file_get_conten ts() wont work. Files are not transferred until requested by the user. I would prefer to do it all in CURL, but as i said im having issues.

I have looked into SST scripts but none seems to satisfy.

My thinking is that cURL cannot directly transfer a file server to server (please correct if wrong). I thought that the way to make it work would be to call a file on Web Server 2 using cURL and add some parameters (file name to transfer) to the call so that when ran it would using cURL transfer the file to Web Server 1. Sound good? Problem, Web Server 2 doesn't have a URL that will resolve to a specific file on the server. Therefor again only cURL via FTP login can be used (as i see it).

So do further define my question, with cURL, is there a way to transfer files between servers directly using a curl statement like
Expand|Select|Wrap|Line Numbers
  1. $ch = curl_init("ftp://theuser:thepass@72.47.251.180/testfile.m4v");
  2.  
but one that works more like below with a location to get and set from/to. I know the following code is incorrect, i just would like to know how to get it right.
Expand|Select|Wrap|Line Numbers
  1. $c = curl_init("ftp://theuser:thepass@72.47.251.180/testfile.m4v","ftp://theuser:thepass@42.76.151.236/testfile.m4v");
  2.  
I know i have left out parts of the cURL process. curl_exec() and curl_close() are pretty standard. Thanks.
Mar 10 '09 #4
empiresolutions
162 New Member
SOLVED -

Thanks for everyones help. I am posting my full function for those who may find it useful. To recap, this function "try's" to simulate Akamai's service.

1. GeoLoc user.
2. GeoLoc all servers
3. Calculate distance between user and each server.
4. Sort by distance.
5. Check to see if file is at closest server. If so download.
6. If file not at closest server, copy file from mother server to server missing file. Then notify user via email with download link to closest server.

Expand|Select|Wrap|Line Numbers
  1. function sst() {
  2.  
  3.     global $_GET, $_SERVER;
  4.  
  5.     /*   start geo loc   */
  6.     // http://www.imaginerc.com/software/GeoCalc/
  7.     require_once('standards/classes/geoplugin.class.php'); 
  8.     $geoplugin = new geoPlugin();
  9.     // http://www.phpclasses.org/browse/file/25516.html
  10.     include_once("standards/classes/GeoCalc.class.php"); 
  11.     $oGC = new GeoCalc();
  12.  
  13.     /*   set server ip's and FTP logins        */
  14.     $servers = array($_SERVER['REMOTE_ADDR'],'xxx.xx.xx.xxx','xxx.xx.xx.xxx');
  15.     $user = array('','usr1','usr2');
  16.     $pass = array('','pass1','pass2',);
  17.  
  18.     /*   get user location   */
  19.     $geo[0]['ip'] = $servers[0];
  20.     $geoplugin->locate($geo[0]['ip']);
  21.     $geo[0]['lat'] = $geoplugin->latitude;
  22.     $geo[0]['lon'] = $geoplugin->longitude;
  23.     $geo[0]['city'] = $geoplugin->city;
  24.     $geo[0]['countryName'] = $geoplugin->countryName;
  25.  
  26.     /*   get  servers data  */
  27.     for ($i = 1; $i < count($servers); $i++) {
  28.  
  29.         $geo[$i]['ip'] = $servers[$i];
  30.         $geo[$i]['user'] = $user[$i];
  31.         $geo[$i]['pass'] = $pass[$i];
  32.         $geoplugin->locate($geo[$i]['ip']);
  33.         $geo[$i]['lat'] = $geoplugin->latitude;
  34.         $geo[$i]['lon'] = $geoplugin->longitude;
  35.         $geo[$i]['city'] = $geoplugin->city;
  36.         $geo[$i]['countryName'] = $geoplugin->countryName;
  37.         $geo[$i]['dist'] = $oGC->GCDistance($geo[0]['lat'],$geo[0]['lon'],$geo[$i]['lat'],$geo[$i]['lon']);
  38.  
  39.     }
  40.  
  41.     /*   order by distance.   */
  42.     $geo = orderBy($geo, 'dist');
  43.  
  44.     /*   connect to closeest server   */
  45.     $conn_id = ftp_connect($geo[1]['ip']) or die("Couldn't connect to $ftp_server");
  46.     ftp_login($conn_id, $geo[1]['user'], $geo[1]['pass']);
  47.     $contents = ftp_nlist($conn_id, "tts/incoming");
  48.     ftp_close($conn_id);
  49.  
  50.     $localfile = '/user/dac15/tts/incoming/';
  51.     $remotefile = '/user/'.$geo[1]['user'].'/tts/incoming/';
  52.  
  53.     /*  if file is on the closest server.    */
  54.     if (find($_GET['name'],$contents)) {
  55.  
  56.         // http://www.awesomephp.com/?Tutorials*16/Download-file-with-resume,-stream-and-speed-options.html
  57.         downloadFile($remotefile,$_GET['name'],900,false); 
  58.  
  59.     }else{ 
  60.  
  61.         /*   push file to the closest server requesting server.*/
  62.         $ch = curl_init();
  63.         $fp = fopen($localfile.$_GET['name'], 'r');
  64.         curl_setopt($ch, CURLOPT_URL, 'ftp://'.$geo[1]['user'].':'.$geo[1]['pass'].'@'.$geo[1]['ip'].'/'.$remotefile.$_GET['name']);
  65.         curl_setopt($ch, CURLOPT_UPLOAD, 1);
  66.         curl_setopt($ch, CURLOPT_INFILE, $fp);
  67.         curl_setopt($ch, CURLOPT_INFILESIZE, filesize($localfile.$_GET['name']));
  68.         curl_exec($ch);
  69.         $error_no = curl_errno($ch);
  70.         curl_close($ch);
  71.         if ($error_no == 0) {
  72.             $error = 'File uploaded succesfully.';
  73.  
  74.             $mailto[0][0] = $_SERVER['HTTP_EMAIL'];
  75.             $mailto[0][1] = $_SERVER['HTTP_FNAME'];
  76.             $mailto[0][2] = $_SERVER['HTTP_LNAME'];
  77.  
  78.             $content="
  79.             Dear ".$mailto[0][1]."
  80.             <br>
  81.             Your file is ready for download. Go to this link in your preferred web browser.
  82.             ";
  83.  
  84.             // http://www.xpertmailer.com/
  85.             if (smtp_mail($mailto,"DoD File Download",$content)) { Return 1; }
  86.  
  87.         }else{
  88.             $error = 'File upload error.';
  89.             echo $error;
  90.         }
  91.  
  92.     }
  93.  
  94.     die;
  95.  
  96. }
  97.  
  98. function orderBy($data, $field, $reverse_sort='') {
  99.     $code = "return strnatcmp(\$a['$field'], \$b['$field']);";
  100.     usort($data, create_function('$a,$b', $code));
  101.     if ($reverse_sort==1) { krsort($data); }
  102.     return $data;
  103. }
  104.  
Mar 11 '09 #5
empiresolutions
162 New Member
I have another question to go along with this post. Once the cURL transfer is started on my 1GB file the screen just sits white while doing its thing (page load bar crawls). Is there a way to finish loading the page while the transfer is in process. This way the user doesn't have to wait for the transfer to finish, they are just there to make it happen. (they are not downloading at the time so it would be okay for them to continue). Thanks.
Mar 11 '09 #6
empiresolutions
162 New Member
Expand|Select|Wrap|Line Numbers
  1. /**
  2.  * Takes a needle and haystack (just like in_array()) and does a wildcard search on it's values.
  3.  *
  4.  * @param    string        $string        Needle to find
  5.  * @param    array        $array        Haystack to look through
  6.  * @result    array                    Returns the elements that the $string was found in
  7.  */
  8. function find ($string, $array = array ())
  9. {
  10.     foreach ($array as $key => $value) {
  11.         unset ($array[$key]);
  12.         if (strpos($value, $string) !== false) {
  13.             $array[$key] = $value;
  14.         }
  15.     }
  16.     return $array;
  17. }
  18.  
Mar 17 '09 #7

Sign in to post your reply or Sign up for a free account.

Similar topics

9
647
by: Steve Buster | last post by:
All right, I have read every forum, newsgroup etc about this issue and no one seems to know how to fix it. I am getting a "Server Application Unavailable" exception running my .NET 1.1 application. I use W2K SP4 and have applied the hot Fix KB824146 and KB824105, both IE fixes. I don't have VS installed because this is a Quality...
6
1516
by: Andrew | last post by:
Hello, friends, I tried to do cross-page posting, i.e., in firstpage.aspx I used action="secondpage.aspx". Since I am using asp.net 1.1, so it always goes back to firstpage.aspx aftern submitting firstpage.aspx. I then tried in page_load of firstpage.aspx:
18
2276
by: UJ | last post by:
Folks, We provide custom content for our customers. Currently we put the files on our server and people have a program we provide that will download the files. These files are usually SWF, HTML or JPG files. The problem as I see it - if you know the name of the file, you could download it off the server (currently we are using an HTTP/Get but...
8
3878
by: bryan | last post by:
I've got a custom HttpHandler to process all requests for a given extension. It gets invoked OK, but if I try to do a Server.Transfer I get an HttpException. A Response.Redirect works, but I really need to avoid the extra round-trip to the client. I've tried Passing the page name, the full URL, and the instance of the handler class to the...
10
2591
by: John Salerno | last post by:
I always read about how you need to set certain file permissions (for cgi files, for example), but it's never been clear to me *how* you do this. I know you can run the line chmod 755 scriptname.py but *where* do you run this? Is this done on your personal system, or on the server? What if you are using Windows? If it's done on the...
1
8068
by: Alex | last post by:
Hello, I'm trying to write a little php script to transfert some files from a server to clients (web/http). It's working fin with small files. But transfering big files (try on 1Gb) failed! The transfert is stoped randomly (sometimes at 25%, sometimes at 75%,...). And I don't understand why?! :/
3
2022
by: Peter Nofelt | last post by:
On Server.Transfer(), is there a reason why the previous page's elements would appear on the new page? Note: this transfer is occuring between two pages on the same server. <Scenario> I'm using Page.Context.Items and Server.Transfer to communicate between two pages (lets call them page1.aspx and page2.aspx). On postback to page1.aspx I...
13
3367
by: Jeff | last post by:
We have an intranet website that currently uses ActiveX but we need to make it cross-browser compatible and also get around the problems we've been having with making it work with IE7 and Vista. We decided to write an external application to do the work from the client side. We are not sure how the web page can call the app though with...
4
1904
by: CSINVA | last post by:
Trying to figure out how to do a cross post page within the code behind file. I need to do some calculations before I send a string value to another pages. Here is an example of the concept I would like to achieve. Lets start with 2 pages, page1.aspx has the text boxes and page2 is where I want to display the informaiton. Page1.aspx
0
7420
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7362
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7602
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
1
7368
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
7704
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
5904
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
0
3404
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
1837
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
983
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.