downloading from webpage

[johnperl]

Hi,

Can someone please help me with this. i am new to using modules and trying to run this script. but it displays the following message:
Can't connect to www.search.cpan.org:80 (Bad hostname 'www.search.cpan.org')

here is the script

# Create a user agent object
  use LWP::UserAgent;
use HTTP::Headers;
  $ua = LWP::UserAgent->new;
  my $newagent = 'search_rfc/1.0('.$ua->agent.')';
  $ua->agent($newagent);
 
  # Create a request
  my $req = HTTP::Request->new(POST =>'http://www.search.cpan.org/search');
  $req->content_type('application/x-www-form-urlencoded');
  $req->content('query=libwww-perl&mode=dist' );
 
  # Pass request to the user agent and get a response back
  my $res = $ua->request($req);
 
  # Check the outcome of the response
  die $res->message unless $res->is_success;
  print "echo $res->as_string\n";
 

Thank you
john

[KevinADC]

John,

http://www.search.cpan.org/search

there should be no ' www ' in the above url

[johnperl]

John,

http://www.search.cpan.org/search

there should be no ' www ' in the above url

Kevin,

Thanks for your reply, but i have everything from cpan site what i need, i just am not getting the output. it keeps on displaying "Can't connect to search.cpan.org:80". whats with the port 80? please help me.

thank you
johnperl

[numberwhun]

Kevin,

Thanks for your reply, but i have everything from cpan site what i need, i just am not getting the output. it keeps on displaying "Can't connect to search.cpan.org:80". whats with the port 80? please help me.

thank you
johnperl

What are you trying to do, download all of the modules from CPAN?

Regards,

Jeff

[johnperl]

What are you trying to do, download all of the modules from CPAN?

Regards,

Jeff

Jeff

No no that's just an example. if it works than i will apply it. well here is the site from which i just need tha month data. http://stats.world-aluminium.org/iai...er.asp?form=11 could you please help me with this.

thank you

[eWish]

Do you have any other code that you have written? Currently it only looks for the website. Do you have any code to get the data from the page that the LWP module is returning?

Have you looked into using an HTML Parser to parse the html and gather the data you wish to keep?

One way you can do this is to use the following modules:

WWW:Mechanize (to get the content of the page and save it in one step. The FAQ section can be helpful)
HTML::Parser or HTML::TokeParser (to filter out the data you want form an HTML document.)

[johnperl]

Do you have any other code that you have written? Currently it only looks for the website. Do you have any code to get the data from the page that the LWP module is returning?

Have you looked into using an HTML Parser to parse the html and gather the data you wish to keep?

One way you can do this is to use the following modules:

WWW:Mechanize (to get the content of the page and save it in one step. The FAQ section can be helpful)
HTML::Parser or HTML::TokeParser (to filter out the data you want form an HTML document.)

Hi,
I tried all the modules the problem is with the proxy setting we have a setting of webproxy. the script works fine for internal network site but doe not work for any other site due to proxy setting. the only thing is how can i set the proxy. our proxy setting has to be webproxy.
below is the script,

#!/usr/local/bin/perl_5.6.1 -w
 
$env_proxy = $ENV{"env_proxy"};
 
use LWP::UserAgent;
  $ua = LWP::UserAgent->new;
  $ua->env_proxy; # initialize from environment variables
  # or
  $ua->proxy(ftp  => 'http://proxy.bip.com');
  $ua->proxy(webproxy => "$env_proxy");
  $ua->no_proxy(qw(no se fi));
 
  my $req = HTTP::Request->new(GET => "http://bip/");
  print $ua->request($req)->as_string;
 

thank you

[numberwhun]

Hi,
I tried all the modules the problem is with the proxy setting we have a setting of webproxy. the script works fine for internal network site but doe not work for any other site due to proxy setting. the only thing is how can i set the proxy. our proxy setting has to be webproxy.
below is the script,

#!/usr/local/bin/perl_5.6.1 -w
 
$env_proxy = $ENV{"env_proxy"};
 
use LWP::UserAgent;
  $ua = LWP::UserAgent->new;
  $ua->env_proxy; # initialize from environment variables
  # or
  $ua->proxy(ftp  => 'http://proxy.bip.com');
  $ua->proxy(webproxy => "$env_proxy");
  $ua->no_proxy(qw(no se fi));
 
  my $req = HTTP::Request->new(GET => "http://bip/");
  print $ua->request($req)->as_string;
 

thank you

You have posted code into the forum without using proper code tags. It is best practice here on TSDN to wrap all code posted into the forum in code tags.

Code tags start with [code] and end with [/code]. If you need an example other than this one, the please refer to the REPLY GUIDELINES (or POSTING GUIDELINES if you are starting a discussion) to the right of the Message window. You will find the examples and suggestions for posting there.

In addition, you can also add the language to your code tags. Here is an example:

[code=perl]
<some code>
[/code]

Please know that I have fixed your posts above to include the proper code tags. Please be sure and use code tags in all of your future posts here on TSDN.

- Moderator

[eWish]

Try reading the FAQ page I linked to above for WWW::Mechanize module. It will show you how to deal with a proxy server.