Trouble with LWP::UserAgent on certain website

All,

I have the following code:

Expand|Select|Wrap|Line Numbers

 
use 5.010;

use LWP::UserAgent;

use URI::URL;

use LWP::Debug qw(+);
 
my $browser = LWP::UserAgent->new();

$browser->agent('Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322)');

my $url = url 'http://www.investway.com';

my $resp = $browser->get($url);

open my $htmlFile, '>', 'c:\\temp\\temp5.html';

print $htmlFile $resp->{_content};
 
And here is the LWP::Debug output:
 
LWP::UserAgent::new: ()

LWP::UserAgent::request: ()

LWP::UserAgent::send_request: GET http://www.investway.com/

LWP::UserAgent::_need_proxy: Not proxied

LWP::Protocol::http::request: ()

LWP::Protocol::collect: read 160 bytes

LWP::UserAgent::request: Simple response: Found

LWP::UserAgent::request: ()

LWP::UserAgent::send_request: GET http://www.investway.com/ErrorPage.aspx?aspxerrorpath=/Section.aspx

LWP::UserAgent::_need_proxy: Not proxied

LWP::Protocol::http::request: ()

LWP::Protocol::collect: read 784 bytes

LWP::Protocol::collect: read 2242 bytes

LWP::UserAgent::request: Simple response: Internal Server Error
 
And here is the temp5.html file created by the code:
 
<html>

    <head>

        <title>Runtime Error</title>

        <style>

         body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;} 

         p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}

         b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}

         H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }

         H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }

         pre {font-family:"Lucida Console";font-size: .9em}

         .marker {font-weight: bold; color: black;text-decoration: none;}

         .version {color: gray;}

         .error {margin-bottom: 10px;}

         .expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:hand; }

        </style>

    </head>
 
    <body bgcolor="white">
 
            <span><H1>Server Error in '/' Application.<hr width=100% size=1 color=silver></H1>
 
            <h2> <i>Runtime Error</i> </h2></span>
 
            <font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">
 
            <b> Description: </b>An application error occurred on the server. The current custom error settings for this application prevent the details of the application error from being viewed remotely (for security reasons). It could, however, be viewed by browsers running on the local server machine.

            <br><br>
 
            <b>Details:</b> To enable the details of this specific error message to be viewable on remote machines, please create a &lt;customErrors&gt; tag within a &quot;web.config&quot; configuration file located in the root directory of the current web application. This &lt;customErrors&gt; tag should then have its &quot;mode&quot; attribute set to &quot;Off&quot;.<br><br>
 
            <table width=100% bgcolor="#ffffcc">

               <tr>

                  <td>

                      <code><pre>
 
&lt;!-- Web.Config Configuration File --&gt;
 
&lt;configuration&gt;

    &lt;system.web&gt;

        &lt;customErrors mode=&quot;Off&quot;/&gt;

    &lt;/system.web&gt;

&lt;/configuration&gt;</pre></code>
 
                  </td>

               </tr>

            </table>
 
            <br>
 
            <b>Notes:</b> The current error page you are seeing can be replaced by a custom error page by modifying the &quot;defaultRedirect&quot; attribute of the application's &lt;customErrors&gt; configuration tag to point to a custom error page URL.<br><br>
 
            <table width=100% bgcolor="#ffffcc">

               <tr>

                  <td>

                      <code><pre>
 
&lt;!-- Web.Config Configuration File --&gt;
 
&lt;configuration&gt;

    &lt;system.web&gt;

        &lt;customErrors mode=&quot;RemoteOnly&quot; defaultRedirect=&quot;mycustompage.htm&quot;/&gt;

    &lt;/system.web&gt;

&lt;/configuration&gt;</pre></code>
 
                  </td>

               </tr>

            </table>
 
            <br>
 
    </body>

</html>

These errors only happen with this particular web-site (www.investway.com). It works just fine when I use just about any other web site. There's really not enough information put out by LWP::Debug for me to figure this out. Any help would be greatly appeciated.

Chuck

Jul 29 '08 #1

Subscribe Post Reply

3290

KevinADC

4,059

Expert 2GB

this line looks wrong:

Expand|Select|Wrap|Line Numbers

my $url = url 'http://www.investway.com';

should it be like this?

Expand|Select|Wrap|Line Numbers

my $url = 'http://www.investway.com';

Jul 29 '08 #2

cketcham

this line looks wrong:

Expand|Select|Wrap|Line Numbers

my $url = url 'http://www.investway.com';

should it be like this?

Expand|Select|Wrap|Line Numbers

my $url = 'http://www.investway.com';

Probably not. As I said, it works fine with other web sites.

Jul 29 '08 #3

KevinADC

4,059

Expert 2GB

I get the same response from that website you get after correcting that syntax error. There is no problem with LWP::UserAgent because it does connect to the website and it does get a response, its just not the response you want. Evidently that website is using something to detect access by bots, but that is out of my range of experience and I am not sure what to suggest. Try asking on www.perlmonks.com and see if anyone has a suggestion.

Jul 29 '08 #4

by: La Jesus | last post by:

Hello - I am new to Perl, so sorry if this is a silly / ultra easy question. I am using LWP::UserAgent to POST a HTTP message to another website, and it works great. However, I can't figure...

Perl

LWP with proxy problem

by: Woogie | last post by:

When running the sample code below without a proxy the GET returns the expected data. When run with the $proxy uncommented the GET returns the content of the login page for the site being...

Perl

Problem with LWP::USERAGENT

by: Chandra | last post by:

Hello gurus, I am using LWP::Useragent perl module to get a url from the www. Code snippet: #!perl -w use LWP::UserAgent; LWP::Debug::level('+'); $ua = new LWP::UserAgent || die "$!";

Perl

Tcl equivalent for Perl's Win32::OLE, LWP::UserAgent, HTTP::Request, Net::Telnet

by: Vinay Gupta | last post by:

Hi, In a Perl to Tcl conversion project, I am planning to use the following Tcl extensions as a replacement for some Perl libraries. My development environment is Windows. Win32::OLE --> "DDE"...

Perl

Problem with detecting error by LWP::UserAgent

by: Playker | last post by:

Hello I have a website on 2 servers. I want to make a script which checks is first server availbale (he often breaks down) and redirects to 1st or 2nd server. Script works well if an error is the...

Perl

LWP::UserAgent

by: haimk63 | last post by:

Hi ! I am beginner in PERL. I would like to know whether the LWP::UserAgent module? can be used in any server that supports PERL. Thanks

Perl

LWP::UserAgent Get a page after authenicating in SSL mode

by: powersakthi | last post by:

Hi All, I am using LWP::UserAgent and HTML::TreeBuilder to communicate to the below site Here is the website map: https://secure.server.com/index.htm which asks for two fields user_id...

Perl

LWP Authentication is not sticking

by: kkamath | last post by:

Hi, I use LWP::UserAgent to automate windows login process. I set the credentials using $ua->credentials("host:port","realm","id","password"); but it fetches only the current page requested. It...

Perl

Using Credentials on LWP

by: Freedolen | last post by:

Hi, I want to download a pdf file from password protected website. Using browser(IE/FireFox) when i click the pdf link it shows a window, which asks 'username' and 'password'. I want to automate...

Perl

React to native button blinking effect

by: lllomh | last post by:

Define the method first this.state = { buttonBackgroundColor: 'green', isBlinking: false, // A new status is added to identify whether the button is blinking or not } autoStart=()=>{

Software Development

Access Europe meeting on Wed 4 Oct - The future of Access

by: isladogs | last post by:

The next Access Europe meeting will be on Wednesday 4 Oct 2023 starting at 18:00 UK time (6PM UTC+1) and finishing at about 19:15 (7.15PM) The start time is equivalent to 19:00 (7PM) in Central...

Microsoft Access / VBA

VBA Class Basics

by: NeoPa | last post by:

Introduction For this article I'll be using a very simple database which has Form (clsForm) & Report (clsReport) classes that simply handle making the calling Form invisible until the Form, or all...

Microsoft Access / VBA

Report value list

by: Teri B | last post by:

Hi, I have created a sub-form Roles. In my course form the user selects the roles assigned to the course. 0ne-to-many. One course many roles. Then I created a report based on the Course form and...

Microsoft Access / VBA

Access Europe Meeting - Wed 1 Nov - Using SQL Server skills to extend Access

by: isladogs | last post by:

The next Access Europe meeting will be on Wednesday 1 Nov 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM) Please note that the UK and Europe revert to winter time on...

Microsoft Access / VBA

Finding a record, and then saving a new record after the record has been amended.

by: nia12 | last post by:

Hi there, I am very new to Access so apologies if any of this is obvious/not clear. I am creating a data collection tool for health care employees to complete. It consists of a number of...

Microsoft Access / VBA

VBA Class Basics (Report Class)

by: NeoPa | last post by:

Introduction For this article I'll be focusing on the Report (clsReport) class. This simply handles making the calling Form invisible until all of the Reports opened by it have been closed, when it...

Microsoft Access / VBA

Access Europe Meeting - Wed 6 Dec - Streamline Your Import/Export Spec Workflow with VBA Classes

by: isladogs | last post by:

The next online meeting of the Access Europe User Group will be on Wednesday 6 Dec 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, Mike...

Microsoft Access / VBA

Take a Database to the Next Level

by: GKJR | last post by:

Does anyone have a recommendation to build a standalone application to replace an Access database? I have my bookkeeping software I developed in Access that I would like to make available to other...

Microsoft Access / VBA

Trouble with LWP::UserAgent on certain website

Similar topics