By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
438,852 Members | 2,200 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 438,852 IT Pros & Developers. It's quick & easy.

Trouble with LWP::UserAgent on certain website

P: 2
All,

I have the following code:
Expand|Select|Wrap|Line Numbers
  1. use 5.010;
  2. use LWP::UserAgent;
  3. use URI::URL;
  4. use LWP::Debug qw(+);
  5.  
  6. my $browser = LWP::UserAgent->new();
  7. $browser->agent('Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322)');
  8. my $url = url 'http://www.investway.com';
  9. my $resp = $browser->get($url);
  10. open my $htmlFile, '>', 'c:\\temp\\temp5.html';
  11. print $htmlFile $resp->{_content};
  12.  
  13. And here is the LWP::Debug output:
  14.  
  15. LWP::UserAgent::new: ()
  16. LWP::UserAgent::request: ()
  17. LWP::UserAgent::send_request: GET http://www.investway.com/
  18. LWP::UserAgent::_need_proxy: Not proxied
  19. LWP::Protocol::http::request: ()
  20. LWP::Protocol::collect: read 160 bytes
  21. LWP::UserAgent::request: Simple response: Found
  22. LWP::UserAgent::request: ()
  23. LWP::UserAgent::send_request: GET http://www.investway.com/ErrorPage.aspx?aspxerrorpath=/Section.aspx
  24. LWP::UserAgent::_need_proxy: Not proxied
  25. LWP::Protocol::http::request: ()
  26. LWP::Protocol::collect: read 784 bytes
  27. LWP::Protocol::collect: read 2242 bytes
  28. LWP::UserAgent::request: Simple response: Internal Server Error
  29.  
  30.  
  31. And here is the temp5.html file created by the code:
  32.  
  33. <html>
  34.     <head>
  35.         <title>Runtime Error</title>
  36.         <style>
  37.          body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;} 
  38.          p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}
  39.          b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}
  40.          H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }
  41.          H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }
  42.          pre {font-family:"Lucida Console";font-size: .9em}
  43.          .marker {font-weight: bold; color: black;text-decoration: none;}
  44.          .version {color: gray;}
  45.          .error {margin-bottom: 10px;}
  46.          .expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:hand; }
  47.         </style>
  48.     </head>
  49.  
  50.     <body bgcolor="white">
  51.  
  52.             <span><H1>Server Error in '/' Application.<hr width=100% size=1 color=silver></H1>
  53.  
  54.             <h2> <i>Runtime Error</i> </h2></span>
  55.  
  56.             <font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">
  57.  
  58.             <b> Description: </b>An application error occurred on the server. The current custom error settings for this application prevent the details of the application error from being viewed remotely (for security reasons). It could, however, be viewed by browsers running on the local server machine.
  59.             <br><br>
  60.  
  61.             <b>Details:</b> To enable the details of this specific error message to be viewable on remote machines, please create a &lt;customErrors&gt; tag within a &quot;web.config&quot; configuration file located in the root directory of the current web application. This &lt;customErrors&gt; tag should then have its &quot;mode&quot; attribute set to &quot;Off&quot;.<br><br>
  62.  
  63.             <table width=100% bgcolor="#ffffcc">
  64.                <tr>
  65.                   <td>
  66.                       <code><pre>
  67.  
  68. &lt;!-- Web.Config Configuration File --&gt;
  69.  
  70. &lt;configuration&gt;
  71.     &lt;system.web&gt;
  72.         &lt;customErrors mode=&quot;Off&quot;/&gt;
  73.     &lt;/system.web&gt;
  74. &lt;/configuration&gt;</pre></code>
  75.  
  76.                   </td>
  77.                </tr>
  78.             </table>
  79.  
  80.             <br>
  81.  
  82.             <b>Notes:</b> The current error page you are seeing can be replaced by a custom error page by modifying the &quot;defaultRedirect&quot; attribute of the application's &lt;customErrors&gt; configuration tag to point to a custom error page URL.<br><br>
  83.  
  84.             <table width=100% bgcolor="#ffffcc">
  85.                <tr>
  86.                   <td>
  87.                       <code><pre>
  88.  
  89. &lt;!-- Web.Config Configuration File --&gt;
  90.  
  91. &lt;configuration&gt;
  92.     &lt;system.web&gt;
  93.         &lt;customErrors mode=&quot;RemoteOnly&quot; defaultRedirect=&quot;mycustompage.htm&quot;/&gt;
  94.     &lt;/system.web&gt;
  95. &lt;/configuration&gt;</pre></code>
  96.  
  97.                   </td>
  98.                </tr>
  99.             </table>
  100.  
  101.             <br>
  102.  
  103.     </body>
  104. </html>
  105.  
These errors only happen with this particular web-site (www.investway.com). It works just fine when I use just about any other web site. There's really not enough information put out by LWP::Debug for me to figure this out. Any help would be greatly appeciated.

Chuck
Jul 29 '08 #1
Share this Question
Share on Google+
3 Replies


KevinADC
Expert 2.5K+
P: 4,059
this line looks wrong:

Expand|Select|Wrap|Line Numbers
  1. my $url = url 'http://www.investway.com';
should it be like this?

Expand|Select|Wrap|Line Numbers
  1. my $url = 'http://www.investway.com';
Jul 29 '08 #2

P: 2
this line looks wrong:

Expand|Select|Wrap|Line Numbers
  1. my $url = url 'http://www.investway.com';
should it be like this?

Expand|Select|Wrap|Line Numbers
  1. my $url = 'http://www.investway.com';

Probably not. As I said, it works fine with other web sites.
Jul 29 '08 #3

KevinADC
Expert 2.5K+
P: 4,059
I get the same response from that website you get after correcting that syntax error. There is no problem with LWP::UserAgent because it does connect to the website and it does get a response, its just not the response you want. Evidently that website is using something to detect access by bots, but that is out of my range of experience and I am not sure what to suggest. Try asking on www.perlmonks.com and see if anyone has a suggestion.
Jul 29 '08 #4

Post your reply

Sign in to post your reply or Sign up for a free account.