Downloading data from online site

[Archanak]

Hi,

I wanted to download some online data from some site so i used perl LWP module.

code i wrote was like this:

use LWP;
use LWP::UserAgent;
$ua=new LWP::UserAgent;
$ua->proxy(['http','ftp'],'http://group:[e-mail]:3128');
 
open(FH,">datafile.txt");
chmod 0777,"datafile.txt";
$url="http://xyz.com";
sleep(5);
$response=$ua->get($url);
$result=$response->content;
print FH $result
close FH;
 

The data is written to datafile. I am connecting 3 times with different query terms to that url. The problem i am facing is the speed i.e time taken to get the data.

Is there any other method which can reduce the time taken to get the data?

I tried fcgi but no improvements!!

Is LWP module itself is slow??

Any suggestions?

with regards
Archana

[KevinADC]

The time it takes the module to compile and connect to the remote site should not be significant. How long does it take? How long do you think it should take?

[Archanak]

@KevinADC
Hi,

For one connection i.e once fetching data its taking 1 min so total 3 mins.

I am working on linux platform.

Atleast it should take a minute for 3 connections (since 3 times i am connecting)

Any suggestions??

with regards
Archana

[KevinADC]

If your script takes a full minute to fetch a short webpage then I would assume you just have a poor internet connection, on the other hand if it is fethcing a few megabytes in a minute that doesn't seem to bad. But I have no idea how it could be speeded up.

[Archanak]

@KevinADC
Hi,

Actually the data is text data and the speed i use is about 100.0 Mbps.

Is LWP module itself is slow???

Any other alternative to speedup the process?

with regards
Archana

[numberwhun]

Even if you are using a fast connection, if the place you are fetching from is slow, then that will certainly effect the time as well so please take that into account.

You could, as an alternative, try using the WWW::Mechanize module. It is quite the little gem and I have used it to fetch files from the internet.

Regards,

Jeff

[Archanak]

@numberwhun
Hi,

I installed WWW::Mechanize module and the problem i am facing is that it is working in the command prompt and not in the browser.

Here is the code

#!/usr/bin/perl
 
use CGI;
 
use CGI qw(:standard);
 
use WWW::Mechanize;
 
print header;
 
my $m = WWW::Mechanize->new;
 
$m->get("http://www.google.com");
 
print $m->content();
 
 

It is giving error when i run in the browser.

Error is:

 
Error GETing http://www.google.com: Can't connect to www.google.com:80 (connect: timeout) at  line 13
 
 

How can i solve this problem??

Regards
Archana

[numberwhun]

I noticed in your script you didn't start off the CGI with something like this:

print "Content-type: text/html\r\n\r\n";
 

You should probably check out a CGI tutorial before continuing.

Regards,

Jeff

[KevinADC]

@numberwhun
Jeff,

They have "print header;" in the script, which essentially does what you have suggested, prints the http header.

[KevinADC]

The code as posted works for me. Could be a firewall is blocking perl from accessing the internet.