Hi y'all!
I'm new at perl, and I'm trying to automate a file fetch.
I have this url (in this example called 'https://GetMyFile'), which, when I paste it into a browser, gives me the pop-up "File Download" - Do you want to open or save this file?.. And clicking 'save' gives me the file I want.
I would like to achieve the same result automatically, without having to paste the url into a browser and click 'save' a specify where to save my file.
So, here's my first attempt:
--------------------------------------------------------------------- - use strict;
-
use WWW::Mechanize;
-
use LWP::Debug qw(+);
-
-
my $ua = new LWP::UserAgent;
-
$ua->proxy([qw( https http )], "myProxyAddress");
-
-
my $url = "https://GetMyFile";
-
-
my $mech = WWW::Mechanize->new();
-
-
print "Fetching $url";
-
$mech->get( $url, ':content_file' => 'C:\Tmp\myFile.zip' );
-
die "Ooops, that didn't work: ", $mech->response->status_line unless $mech->success;
--------------------------------------------------------------------
The thing is, I don't get the "oops" printout, instead myFile.zip is downloaded to the correct location, but this file is corrupted. It seems it doesn't get downloaded entirely since when I download it manually it's much bigger.
Here are some debug printouts I get - LWP::UserAgent::new: ()
-
LWP::UserAgent::proxy: ARRAY(someHexNumber) myProxyAddress
-
LWP::UserAgent::proxy: https myProxyAddress
-
LWP::UserAgent::proxy: http myProxyAddress
-
LWP::UserAgent::new: ()
-
LWP::UserAgent::request: ()
-
HTTP::Cookies::add_cookie_header: Checking GetMyFile for cookies
-
LWP::UserAgent::send_request: GET https://GetMyFile
-
LWP::UserAgent::_need_proxy: Not proxied
-
LWP::Protocol::http::request: ()
-
LWP::Protocol::collect: read 336 bytes
-
LWP::UserAgent::request: Simple response: Found
-
LWP::UserAgent::request: ()
-
HTTP::Cookies::add_cookie_header: Checking GetMyFile for cookies
-
LWP::UserAgent::send_request: GET https://GetMyFile
-
LWP::UserAgent::_need_proxy: Not proxied
-
LWP::Protocol::http::request: ()
-
LWP::Protocol::collect: read 439 bytes
-
LWP::Protocol::collect: read 176 bytes
-
LWP::UserAgent::request: Simple response: Found
-
-
... Then these printouts are repeated
-
-
...
-
-
LWP::UserAgent::_need_proxy: Not proxied
-
LWP::Protocol::http::request: ()
-
LWP::Protocol::collect: read 869 bytes
-
LWP::Protocol::collect: read 4096 bytes
-
LWP::Protocol::collect: read 4096 bytes
-
LWP::Protocol::collect: read 2395 bytes
-
LWP::UserAgent::request: Simple response: OK
-
Fetching https://GetMyFile
-
Any help or suggestions as to why I don't get the entire file (?) would be greatly appreciated!
Cheers
12 10706
Looks like it should work. Don't know what the problem is.
I agree with Kevin. Right off the bat, it looks like it might work, but I haven't gone through it thoroughly. What I can say is that you want to look at the book "Spidering Hacks". Specifically, this part here: http://books.google.com/books?id=4M2...PXiAg#PPA60,M1
That will help you with a fetch using the Mechanize module.
Regards,
Jeff
Hi! Thank you so much guys, for giving me feedback quickly!
I now know though, that the problem is related to credentials...
When, in the script, I change
$mech->get( $url, ':content_file' => 'C:\Tmp\myFile.zip' );
to
$mech->get( $url, ':content_file' => 'C:\Tmp\myFile.html' );
I can see that the downloaded file is indeed a webpage; and that is, a login page..
I don't really know how to solve this though. I will have to investigate further.
There is some autologin asp session involved when fetching files from where I want to fetch them. Probably the browser handles a lot of that "behind the scenes", and I don't really know exactly what's going on, which, of course I must, in order to get my script to work.. These enterprise networks.. *sigh* :)...
@MimiMi
Check out the module documentation on CPAN for WWW::Mechanize. I am pretty positive that it provides options for logging in to such pages, you just have to code for it.
I don't know if it will help any, but here is a script I wrote a while ago that logs into a website (you had to log in before you could see the list of files) and then downloads everything that was there: -
#!/usr/bin/perl
-
-
use strict;
-
use warnings;
-
use File::Basename;
-
use WWW::Mechanize;
-
use MIME::Base64;
-
-
$|++;
-
-
my $username = "username";
-
my $password = "password";
-
my $url = "http://www.site.com/page.asp";
-
my $realm;
-
my $tempfile = "temp.txt";
-
-
my $agent = WWW::Mechanize->new();
-
my @args = (
-
Authorization => "Basic " . MIME::Base64::encode( $username . ':' . $password )
-
);
-
-
-
$agent->credentials( $url, $realm, $username, $password );
-
-
$agent->get( $url, @args)
-
Obviously, site name, username and password have all been changed to protect the innocent and the above values for each should be replaced with whatever you are using.
Regards,
Jeff
Look into Win32::IE::Mechanize which can handle a lot more things than WWW::Mechanize can
Hello again!
I appreciate all your efforts to help me out here!
I've been working on other things, but now it's time to get back to this. (I still haven't got it working).
Here's my current status:
The myFile.html I get from - $mech->get( $url, ':content_file' => 'C:\Tmp\myFile.html' );
-
(see previous posts if I'm unclear)
has JavaScript on it.. Here are some parts of the html-file (including the JavaScript): - <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
-
<HTML>
-
<head>
-
<title>TheCompany Portal Login</title>
-
<link type="text/css" rel="stylesheet" href="styles.css">
-
<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
-
<META HTTP-EQUIV="Expires" CONTENT="-1">
-
<meta content="text/html; charset=iso-8859-1" http-equiv="Content-Type">
-
-
<SCRIPT LANGUAGE="JavaScript">
-
function resetCredFields()
-
{
-
document.Login.PASSWORD.value = "";
-
}
-
-
function submitForm()
-
{
-
document.Login.submit();
-
}
-
-
function cancelLogin()
-
{
-
window.history.go(-1);
-
}
-
-
if (top.frames.length > 1)
-
{
-
top.location.href = document.location;
-
}
-
-
function checkEnter(event)
-
{
-
var code = 0;
-
NS4 = (document.layers) ? true : false;
-
if (NS4)
-
code = event.which;
-
else
-
code = event.keyCode;
-
if (code==13)
-
document.Login.submit();
-
}
-
-
</SCRIPT>
-
-
-
</head>
-
-
<BODY topmargin="0" leftmargin="0" marginwidth="0" marginheight="0">
-
-
-
-
-
<table height="95%" width="100%" border="0" cellspacing="0" cellpadding="0">
-
<tr>
... And so on and so forth..
I don't know anything about how WWW::Mechanize could work with JavaScript.. is that even possible? How then can I provide the JavaScript with the right credentials?
Cheers
Sorry sorry.. I don't need to waste your time by asking silly questions such as whether WWW::Mechanize works with JavaScript, that wasn't hard to find out for myself. The answer is NO. Unfortunately.
Have to figure out how to solve this then.. some other way.. :/
Cheers
I guess you missed my previosu post:
Look into Win32::IE::Mechanize which can handle a lot more things than WWW::Mechanize can
Hi!
KevinADC: Yes that's right I missed looking into Win32::IE::Mechanize, sorry for that!
Now I've started looking into that though, and it seems to be filling my needs somewhat better, feels like I'm almost there, but still I don't get how I can get my files downloaded without manually having to provide any user input whatsoever.
As of now I get an IE browser starting up, and I get to the download file prompt, but I don't want to manually have to click
"save" and provide location etc.. Plus, I don't want IE to show at all.. Is that possible?
This script is to be run at a server, so I want everything to be "invisible"..
Here's my current script: -
use warnings;
-
use Win32::IE::Mechanize;
-
-
my $ie = Win32::IE::Mechanize->new( visible => 1 );
-
-
my $username = "user";
-
my $password = "pwd";
-
-
my $url = "http://weblink.To.TheFile";
-
my $realm;
-
-
$ie->credentials( 'myHostname:myPort', $realm, $username, $password );
-
-
print "Fetching $url";
-
$ie->get( $url, ':content_file' => 'C:\Temp\result\result.zip');
-
die "Ooops, this didn't work: ", $ie->response->status_line unless $ie->success;
-
Sorry but I don't know the answer or have any suggestions for your last questions. All I can suggest is to carefully read the modules documentation and see if there is anything that can help you solve those parts of your question.
That is not a Perl issue that is a browser issue, you have too look into your browser settings or use the first version of google chrome as they started download when a file was clicked on. (this was updated in newer versions as it is a security risk, this is why they have a save option).
Why not use wget?
No need for a big perl script--assuming you're running *nix.
man wget
You can use it to imitate a browser, including login information and site cookies, while downloading files or webpages.
Example: - wget -m -c --convert-links --user="Mister Man" --password=PreTTyPlease --load-cookies cookie.txt --user-agent="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2)" http://www.your-special-site.com/get-that-archive.zip
You can automate the process via a cronjob on your *nix server to get those files from the remote location.
Of course, there are some security issues with putting your password into a shell command, and anyone with access to your crontab will be able to see it in plain text...but there are some other options if you are needing more security.
Sign in to post your reply or Sign up for a free account.
Similar topics
by: Wim Roffil |
last post by:
Hi,
I have a screencraper application that read webpages with fopen() and then
processes them.
Now I found a https page that does not require a password. However, I cannot
read it with...
|
by: rxl124 |
last post by:
someone please please help w/ this one.
As I been working on this on and off and it just does not want to work.
1 #!/usr/bin/perl -w
2
3 $file = "/home/user1/dothis";
4 open(FILE, ">$file");...
|
by: Drew |
last post by:
I have been looking all over the web for an example of how to accomplish this. I am trying to download a comma seperated file from a https server. I can't establish the connection - the error reads...
|
by: archana |
last post by:
Hi all
I am having oen csv file having two columns and some rows.
In my first row and second column i have some string say 'aaa'.
but for all other rows second column contains some integer...
|
by: DrNoose |
last post by:
Hi!
I may have made this post several weeks back, but not sure.
I have a project due Thurs. The problem is when I copy the file the the
local inetpub/wwwroot folder and try to run the aspx...
|
by: mmarlow |
last post by:
Hi all,
I'm trying to get the contents of a file from a URL using
file_get_contents() or file(). It doesn't work, it eventually just
times out. It used to display a 'failed to open file' message...
|
by: kmdshuaib |
last post by:
Hi,
I have created a web application, and i need the total web site to be in https. So i have written the below code in the Page Init of the Master Page to automatically redirect the site from...
|
by: kkshansid |
last post by:
external javascript file not working on server
<script type="text/javascript" src="../include/common.js"></script>
while internal wrking fine
<script language="JavaScript">
.
.
</script>
i...
|
by: George Wade |
last post by:
As many of you know, if you put an VB6.EXE.Manifest file in the same directory as the VB6.exe file then VB6 IDE will run with the XP look for many of the controls.
I got a new computer with...
|
by: MeoLessi9 |
last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
|
by: DolphinDB |
last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation.
Take...
|
by: Aftab Ahmad |
last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below.
Dim IE As Object
Set IE =...
|
by: ryjfgjl |
last post by:
ExcelToDatabase: batch import excel into database automatically...
|
by: isladogs |
last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM).
In this month's session, we are pleased to welcome back...
|
by: marcoviolo |
last post by:
Dear all,
I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
|
by: Vimpel783 |
last post by:
Hello!
Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
|
by: jfyes |
last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
|
by: PapaRatzi |
last post by:
Hello,
I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
| |