Retrieving a page generated by a NPH cgi script

[AndyHunt]

Hello,

I'm having problems using LWP::UserAgent and HTTP::Request:: Common when trying to retrieve a dynamic page generated by a cgi script.

I've got it to the point where I receive a page, but it has none of the dynamic content, and it has several sections that say:
"WARNING: YOUR BROWSER DOESN'T SUPPORT THIS SERVER-PUSH TECHNOLOGY"

Now, a bit of googling has narrowed it down, and I think the problem is that the LWP::* and HTTP::* don't support nph responses. The only info I can find through google is all concerned with using the perl script on the server-side, but I want to use a perl script as the client.

For added info, I'm actually trying to query a bugzilla installation (e.g:
http://bugzilla/show_bug.cgi?qu ery_format=spec ific&bug_status =__open__&produ ct=&content=com and+injection

I've tried setting as much as I can in the userAgent
I wouldn't be surprised if I'm going about this completely the wrong way, but any pointers would be appreciated. (code sniuppet below)

Thanks,

Andy

use HTTP::Request::Common;
use LWP::UserAgent;
 
my  $ua = LWP::UserAgent->new;
 
$ua->agent('Firefox/2.0.0.6');  # Also tried  #Mozilla/5.0
push @{ $ua->requests_redirectable }, 'GET';
 
my $Outputfile = "a.htm";
open OUTPUTFILE, ">$Outputfile";
 
$res = $ua->request(  POST 'http://bugzilla/buglist.cgi', Content_Type => 'multi-part/form-data', Content =>   [ query_format => "specific", bug_status   => "__open__" , product => , content=>"command injection" ]);
 
 
  if ($res->is_success) {
      print OUTPUTFILE $res->content;
  }
  else {
      print $res->status_line, "\n";
  }
 

[miller]

Greetings Andy and welcome,

A few notes.

Adding 'GET' to request_redirec table is redundant, as the default for LWP::UserAgent is to follows GET and HEAD method redirections. I suspect that this was probably just a debugging addition.

Consecutive commas in an array definition are colapsed. Therefore the product will actually be set equal to "content", which I wouldn't be surprised might cause some problems. Put in a hard undef or empty string there instead for a null value.

I typically separate my request declaration from the actual request call. This purely a stylistic preference, but I consider it easier to debug the statement if it is expanded out on multiple lines. Also, this makes it easier to throw in a quick "use Data::Dumper; print Dumper($req);" call or "print $req->as_string;" to confirm that things are formatted as you expect.

Finally, try using a full user agent. I don't know what type of browser specific codes that bugzilla looks for, but the easiest way to ensure that the user_agent isn't the problem is to simply use your own browser user agent string. The follow code uses my own computers.

Finally ** 2. Always "use strict;";.

use HTTP::Request::Common;
use LWP::UserAgent;
 
use strict;
 
my $ua = LWP::UserAgent->new;
$ua->agent("Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6");
 
# push @{ $ua->requests_redirectable }, 'GET';
 
my $req = POST 'http://bugzilla/buglist.cgi',
        Content_Type => 'form-data',
        Content => [
            query_format => "specific",
            bug_status   => "__open__" ,
            product      => undef,
            content      => "command injection",
        ];
 
my $res = $ua->request($req);
 
if ($res->is_success) {
    my $outfile = "a.htm";
    open(OUTFILE, ">$outfile") or die "Can't open $outfile: $!";
    print OUTFILE $res->content;
    close OUTFILE;
}
else {
    print $res->status_line, "\n";
}
 

- Miller

[AndyHunt]

Hey Miller !

Thanks very much - that hits the spot.

For completeness, a couple more observations:
- When I changed the ua->agent to what you suggested, although the script pulled the entire page, I still received the warnings about "your browser doesn't support...etc".
If I didn't set out $ua->agent(..... at all, then the page is 100% perfect! I'm not sure what's going on there, but.....

- If I set

product => , 

then the retrieval of the page works, just doesn't return any results (due, as you predicted, to product being wrong.)

So it looks like a combination of the above two was giving me a page (a) with no results because the product was wrong, and (b) with the error message!


Thank you very much for you help,

Andy

[miller]

Great!

Yes, it's sometimes hard figuring out what code is intentional, and which code is just from fiddling in the effort to get things to work. Glad to hear it's working.

- Miller