Connecting Tech Pros Worldwide Forums | Help | Site Map
bytes > topic > perl > answers

How can I get my https file fetch working?

Member
  
Join Date: Aug 2007
Posts: 42
#1: Feb 4 '09
Hi y'all!
I'm new at perl, and I'm trying to automate a file fetch.

I have this url (in this example called 'https://GetMyFile'), which, when I paste it into a browser, gives me the pop-up "File Download" - Do you want to open or save this file?.. And clicking 'save' gives me the file I want.

I would like to achieve the same result automatically, without having to paste the url into a browser and click 'save' a specify where to save my file.

So, here's my first attempt:
---------------------------------------------------------------------
Expand|Select|Wrap|Line Numbers
  1. use strict;
  2. use WWW::Mechanize;
  3. use LWP::Debug qw(+);
  4.  
  5. my $ua = new LWP::UserAgent;
  6. $ua->proxy([qw( https http )], "myProxyAddress");
  7.  
  8. my $url = "https://GetMyFile";
  9.  
  10. my $mech = WWW::Mechanize->new();
  11.  
  12. print "Fetching $url";
  13.    $mech->get( $url, ':content_file' => 'C:\Tmp\myFile.zip' );
  14.    die "Ooops, that didn't work: ", $mech->response->status_line unless $mech->success;
--------------------------------------------------------------------
The thing is, I don't get the "oops" printout, instead myFile.zip is downloaded to the correct location, but this file is corrupted. It seems it doesn't get downloaded entirely since when I download it manually it's much bigger.

Here are some debug printouts I get

Expand|Select|Wrap|Line Numbers
  1. LWP::UserAgent::new: ()
  2. LWP::UserAgent::proxy: ARRAY(someHexNumber) myProxyAddress
  3. LWP::UserAgent::proxy: https myProxyAddress
  4. LWP::UserAgent::proxy: http myProxyAddress
  5. LWP::UserAgent::new: ()
  6. LWP::UserAgent::request: ()
  7. HTTP::Cookies::add_cookie_header: Checking GetMyFile for cookies
  8. LWP::UserAgent::send_request: GET https://GetMyFile
  9. LWP::UserAgent::_need_proxy: Not proxied
  10. LWP::Protocol::http::request: ()
  11. LWP::Protocol::collect: read 336 bytes
  12. LWP::UserAgent::request: Simple response: Found
  13. LWP::UserAgent::request: ()
  14. HTTP::Cookies::add_cookie_header: Checking GetMyFile for cookies
  15. LWP::UserAgent::send_request: GET https://GetMyFile
  16. LWP::UserAgent::_need_proxy: Not proxied
  17. LWP::Protocol::http::request: ()
  18. LWP::Protocol::collect: read 439 bytes
  19. LWP::Protocol::collect: read 176 bytes
  20. LWP::UserAgent::request: Simple response: Found
  21.  
  22. ... Then these printouts are repeated
  23.  
  24. ...
  25.  
  26. LWP::UserAgent::_need_proxy: Not proxied
  27. LWP::Protocol::http::request: ()
  28. LWP::Protocol::collect: read 869 bytes
  29. LWP::Protocol::collect: read 4096 bytes
  30. LWP::Protocol::collect: read 4096 bytes
  31. LWP::Protocol::collect: read 2395 bytes
  32. LWP::UserAgent::request: Simple response: OK
  33. Fetching https://GetMyFile
  34.  
Any help or suggestions as to why I don't get the entire file (?) would be greatly appreciated!

Cheers

KevinADC's Avatar
Expert
  
Join Date: Jan 2007
Location: Southern California USA
Posts: 4,091
#2: Feb 4 '09

re: How can I get my https file fetch working?

Looks like it should work. Don't know what the problem is.
numberwhun's Avatar
Site Moderator
  
Join Date: May 2007
Location: New Hampshire
Posts: 2,571
#3: Feb 5 '09

re: How can I get my https file fetch working?

I agree with Kevin. Right off the bat, it looks like it might work, but I haven't gone through it thoroughly. What I can say is that you want to look at the book "Spidering Hacks". Specifically, this part here:

http://books.google.com/books?id=4M2...PXiAg#PPA60,M1

That will help you with a fetch using the Mechanize module.

Regards,

Jeff
Member
  
Join Date: Aug 2007
Posts: 42
#4: Feb 5 '09

re: How can I get my https file fetch working?

Hi! Thank you so much guys, for giving me feedback quickly!
I now know though, that the problem is related to credentials...
When, in the script, I change
$mech->get( $url, ':content_file' => 'C:\Tmp\myFile.zip' );
to
$mech->get( $url, ':content_file' => 'C:\Tmp\myFile.html' );
I can see that the downloaded file is indeed a webpage; and that is, a login page..

I don't really know how to solve this though. I will have to investigate further.
There is some autologin asp session involved when fetching files from where I want to fetch them. Probably the browser handles a lot of that "behind the scenes", and I don't really know exactly what's going on, which, of course I must, in order to get my script to work.. These enterprise networks.. *sigh* :)...
numberwhun's Avatar
Site Moderator
  
Join Date: May 2007
Location: New Hampshire
Posts: 2,571
#5: Feb 5 '09

re: How can I get my https file fetch working?

Quote:

Originally Posted by MimiMi View Post

Hi! Thank you so much guys, for giving me feedback quickly!
I now know though, that the problem is related to credentials...
When, in the script, I change
$mech->get( $url, ':content_file' => 'C:\Tmp\myFile.zip' );
to
$mech->get( $url, ':content_file' => 'C:\Tmp\myFile.html' );
I can see that the downloaded file is indeed a webpage; and that is, a login page..

I don't really know how to solve this though. I will have to investigate further.
There is some autologin asp session involved when fetching files from where I want to fetch them. Probably the browser handles a lot of that "behind the scenes", and I don't really know exactly what's going on, which, of course I must, in order to get my script to work.. These enterprise networks.. *sigh* :)...

Check out the module documentation on CPAN for WWW::Mechanize. I am pretty positive that it provides options for logging in to such pages, you just have to code for it.

I don't know if it will help any, but here is a script I wrote a while ago that logs into a website (you had to log in before you could see the list of files) and then downloads everything that was there:

Expand|Select|Wrap|Line Numbers
  1. #!/usr/bin/perl
  2.  
  3. use strict;
  4. use warnings;
  5. use File::Basename;
  6. use WWW::Mechanize;
  7. use MIME::Base64;
  8.  
  9. $|++;
  10.  
  11. my $username = "username";
  12. my $password = "password";
  13. my $url = "http://www.site.com/page.asp";
  14. my $realm;
  15. my $tempfile = "temp.txt";
  16.  
  17. my $agent = WWW::Mechanize->new();
  18. my @args = (
  19.     Authorization => "Basic " . MIME::Base64::encode( $username . ':' . $password )
  20. );
  21.  
  22.  
  23. $agent->credentials( $url, $realm, $username, $password );
  24.  
  25. $agent->get( $url, @args)
  26.  
Obviously, site name, username and password have all been changed to protect the innocent and the above values for each should be replaced with whatever you are using.


Regards,

Jeff
KevinADC's Avatar
Expert
  
Join Date: Jan 2007
Location: Southern California USA
Posts: 4,091
#6: Feb 5 '09

re: How can I get my https file fetch working?

Look into Win32::IE::Mechanize which can handle a lot more things than WWW::Mechanize can
Member
  
Join Date: Aug 2007
Posts: 42
#7: Feb 27 '09

re: How can I get my https file fetch working?

Hello again!
I appreciate all your efforts to help me out here!

I've been working on other things, but now it's time to get back to this. (I still haven't got it working).

Here's my current status:

The myFile.html I get from
Expand|Select|Wrap|Line Numbers
  1. $mech->get( $url, ':content_file' => 'C:\Tmp\myFile.html' );
  2. (see previous posts if I'm unclear)
has JavaScript on it.. Here are some parts of the html-file (including the JavaScript):

Expand|Select|Wrap|Line Numbers
  1. <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
  2. <HTML>
  3.     <head>
  4.         <title>TheCompany Portal Login</title>
  5.         <link type="text/css" rel="stylesheet" href="styles.css">    
  6.         <META HTTP-EQUIV="Pragma" CONTENT="no-cache">
  7.         <META HTTP-EQUIV="Expires" CONTENT="-1">
  8.         <meta content="text/html; charset=iso-8859-1" http-equiv="Content-Type">
  9.  
  10.             <SCRIPT LANGUAGE="JavaScript">
  11.                 function resetCredFields()
  12.                 {
  13.                   document.Login.PASSWORD.value = "";
  14.                 }
  15.  
  16.                 function submitForm()
  17.                 {         
  18.                      document.Login.submit();
  19.                 }
  20.  
  21.                 function cancelLogin()
  22.                 {
  23.                         window.history.go(-1);
  24.                 }
  25.  
  26.                 if (top.frames.length > 1)
  27.                 {
  28.                     top.location.href = document.location;
  29.                 }
  30.  
  31.                 function checkEnter(event)
  32.                 {
  33.                     var code = 0;
  34.                     NS4 = (document.layers) ? true : false;
  35.                     if (NS4)
  36.                     code = event.which;
  37.                     else
  38.                     code = event.keyCode;
  39.                     if (code==13)
  40.                     document.Login.submit();
  41.                 }
  42.  
  43.             </SCRIPT>
  44.  
  45.  
  46.     </head>
  47.  
  48. <BODY topmargin="0" leftmargin="0" marginwidth="0" marginheight="0">
  49.  
  50.  
  51.  
  52.  
  53. <table height="95%" width="100%" border="0" cellspacing="0" cellpadding="0">
  54. <tr>
... And so on and so forth..

I don't know anything about how WWW::Mechanize could work with JavaScript.. is that even possible? How then can I provide the JavaScript with the right credentials?

Cheers
Member
  
Join Date: Aug 2007
Posts: 42
#8: Feb 27 '09

re: How can I get my https file fetch working?

Sorry sorry.. I don't need to waste your time by asking silly questions such as whether WWW::Mechanize works with JavaScript, that wasn't hard to find out for myself. The answer is NO. Unfortunately.

Have to figure out how to solve this then.. some other way.. :/

Cheers
KevinADC's Avatar
Expert
  
Join Date: Jan 2007
Location: Southern California USA
Posts: 4,091
#9: Feb 27 '09

re: How can I get my https file fetch working?

I guess you missed my previosu post:

Look into Win32::IE::Mechanize which can handle a lot more things than WWW::Mechanize can
Member
  
Join Date: Aug 2007
Posts: 42
#10: Mar 9 '09

re: How can I get my https file fetch working?

Hi!
KevinADC: Yes that's right I missed looking into Win32::IE::Mechanize, sorry for that!
Now I've started looking into that though, and it seems to be filling my needs somewhat better, feels like I'm almost there, but still I don't get how I can get my files downloaded without manually having to provide any user input whatsoever.

As of now I get an IE browser starting up, and I get to the download file prompt, but I don't want to manually have to click
"save" and provide location etc.. Plus, I don't want IE to show at all.. Is that possible?

This script is to be run at a server, so I want everything to be "invisible"..

Here's my current script:
Expand|Select|Wrap|Line Numbers
  1. use warnings;
  2. use Win32::IE::Mechanize;
  3.  
  4. my $ie = Win32::IE::Mechanize->new( visible => 1 );
  5.  
  6. my $username = "user";
  7. my $password = "pwd";
  8.  
  9. my $url = "http://weblink.To.TheFile";
  10. my $realm;
  11.  
  12.  $ie->credentials( 'myHostname:myPort', $realm, $username, $password );
  13.  
  14. print "Fetching $url";
  15.    $ie->get( $url, ':content_file' => 'C:\Temp\result\result.zip');
  16.    die "Ooops, this didn't work: ", $ie->response->status_line unless $ie->success;
  17.  
KevinADC's Avatar
Expert
  
Join Date: Jan 2007
Location: Southern California USA
Posts: 4,091
#11: Mar 9 '09

re: How can I get my https file fetch working?

Sorry but I don't know the answer or have any suggestions for your last questions. All I can suggest is to carefully read the modules documentation and see if there is anything that can help you solve those parts of your question.
Icecrack's Avatar
Expert
  
Join Date: Sep 2008
Location: Sydney, Australia
Posts: 173
#12: Mar 9 '09

re: How can I get my https file fetch working?

That is not a Perl issue that is a browser issue, you have too look into your browser settings or use the first version of google chrome as they started download when a file was clicked on. (this was updated in newer versions as it is a security risk, this is why they have a save option).
Reply

« previous question | next question »

Similar Perl bytes

/bytes/development

/bytes/it

Forums
Visit our community forums for general discussions and latest on Bytes

Our staff and community areas have moved to bytes.com/forums/

/bytes/about

We are a network of experts and professionals in IT and software development that help one another with answers to tough questions and share insights. Get the best answers to your questions from over 226,449 network members.

dev questions:
algorithms | asp | asp.net | c/c++ | coldfusion | c# | flash | html/css | java | javascript |
mobile dev | .net | perl | php | python | ruby | software dev | vb.net | visual basic | xml
it questions:
access | apache | careers | db2 | iis | macosx | mysql | networking | oracle | postgresql | sql server | unix | windows

About Bytes | Copyright 1995-2009 BYTES. All rights Reserved