Perl for checking broken link on a website

[sweetbox]

Hi Guys!


I am new on this forum and I am hoping that I could acquire help to improve my PERL skill.

My goal is to be able to create a report of all broken links from a website. Based from my understanding, PERL can go through to server directory and search for htm/html file and put it on an array. From this array I believe PERL can search for a "http://" string and emulate to click the URL. If the URL contains "Page cannot be found" then create and append to a log file.

My question, what is the best method to do this and what is the required library and syntax?

Also for searching the "htm" file on a server, assuming that I don't have access to get in to server directory (which is unlikely) , how do I get this done? and if I have an access to the server how do I get PERL to login to the server first before PERL search for "htm" file on a specific directory? or do I have to put the PERL .pl file on the server root directory and run it there?

I really appreciate if someone could help. Thank you.

[eWish]

Welcome to TSDN!

I would suggest that you use a module like WWW::Mechanize or LWP::UserAgent to achieve that you are wanting. Also, checkout WWW::Mechanize::FAQ.

--Kevin

[eWish]

You might also check out W3C::LogValidator::LinkChecker . If I read it correctly it will use your server log to locate broken links.

--Kevin

[sweetbox]

Thanks for your feedback Kevin! anyone else?

I will try to post the sample code later on for your feedback and also for someone else to learn. Cheers!

[numberwhun]

Thanks for your feedback Kevin! anyone else?

I will try to post the sample code later on for your feedback and also for someone else to learn. Cheers!

Considering that Kevin pretty much covered it, I doubt anyone else will be throwing out the same links that he did.

Regards,

Jeff

[sweetbox]

Guys I'm a bit stuck here.

I would like to know how to capture a certain word (in this case "http://") in the file content however I don't know what the syntax is. Could anyone help? Thanks.

 #Capturing .htm or .html filename on root directory
my @root_files = <*.htm>;
#Searching through directory
for ($count=0;$count<@root_files;$count++)
{
    #opening the file one by one
    my $FILENAME = @root_files[$count];
    open(FILEPOINTER, "< $FILENAME") or die "Can't open $filename : $!";
 
    #Searching file content line by line
    do
    {
        my $LINECONTENT = <FILEPOINTER>;
                # I need to capture only the http link if there is any. the "print" below is just for testing to make sure file content is shown line by line on command prompt
        print $LINECONTENT;
    }
    until eof;
    close FILEOPEN;    
}
 

[numberwhun]

Guys I'm a bit stuck here.

I would like to know how to capture a certain word (in this case "http://") in the file content however I don't know what the syntax is. Could anyone help? Thanks.

 #Capturing .htm or .html filename on root directory
my @root_files = <*.htm>;
#Searching through directory
for ($count=0;$count<@root_files;$count++)
{
    #opening the file one by one
    my $FILENAME = @root_files[$count];
    open(FILEPOINTER, "< $FILENAME") or die "Can't open $filename : $!";
 
    #Searching file content line by line
    do
    {
        my $LINECONTENT = <FILEPOINTER>;
                # I need to capture only the http link if there is any. the "print" below is just for testing to make sure file content is shown line by line on command prompt
        print $LINECONTENT;
    }
    until eof;
    close FILEOPEN;    
}
 

Ok, I think that the best thing for you to do is grab a good Perl book and start reading.

I say this because this line:

my @root_files = <*.htm>;
 

does not do what you are expecting. You cannot just specify "*.htm" and expect all files with .htm on the end to magically be in the array.

First, what OS are you working on? Windows or Unix/Linux? That will help tell us how you are going to get your directory listing.

Regards,

Jeff

[sweetbox]

Thanks for your reply Jeff.

I'm working on Linux and the "my @root_files = <*.htm>" worked as long as I have the .htm file at the same directory as the .pl file. I guess I will also need to improve the code to automatically dig deeper into the directory if it contains more folders (need more time, my perl book is confusing, if you can give me a hint that'll be great!).

Right now I am working on the Regular Expression to grab only the http link

 
do
    {
        my $LINECONTENT = <FILEPOINTER>;
        $_ = $LINECONTENT;
       #this is where I'm stuck at this stage. The code below list the whole line instead of just the http://
 
        if (/a href="?"/) 
        {
            print $LINECONTENT;
        }
    }
    until eof;
 

[KevinADC]

Hi Guys!


I am new on this forum and I am hoping that I could acquire help to improve my PERL skill.

My goal is to be able to create a report of all broken links from a website. Based from my understanding, PERL can go through to server directory and search for htm/html file and put it on an array. From this array I believe PERL can search for a "http://" string and emulate to click the URL. If the URL contains "Page cannot be found" then create and append to a log file.

My question, what is the best method to do this and what is the required library and syntax?

Also for searching the "htm" file on a server, assuming that I don't have access to get in to server directory (which is unlikely) , how do I get this done? and if I have an access to the server how do I get PERL to login to the server first before PERL search for "htm" file on a specific directory? or do I have to put the PERL .pl file on the server root directory and run it there?

I really appreciate if someone could help. Thank you.

Can't you just use something like FrontPage to do that? I am sure there are many third party programs you can use to report broken site links.

[sweetbox]

Ummm yes I guess you can use 3rd party software to check broken link but you won't learn anything...

I have just finished a very basic script although need a lot of improvement.

 
#!/usr/bin/perl
use LWP::UserAgent;
 
 
#Capturing .htm file
#Improvement required. How if there are multiple folders
my @root_files = <*.htm>;
#Searching through folder
for ($count=0;$count<@root_files;$count++)
{
    #opening the file one by one
    my $FILENAME = (@root_files[$count]);
    open(FILEPOINTER, "< $FILENAME") or die "Can't open $filename : $!";
 
    #Searching file content line by line
    do
    {
        my $LINECONTENT = <FILEPOINTER>;
        $_ = $LINECONTENT;
        if (/href="?"/i)
        {
            #capturing quoted strings in href
            #Improvement required. How if there are multiple href in each line?
            my $beginstring = index($LINECONTENT,'"');
            my $endstring = index($LINECONTENT, '"', $beginstring + 1);
            my $finalstring = substr($LINECONTENT,($beginstring+1),($endstring-$beginstring-1));
 
            my $ua = LWP::UserAgent->new;
            $ua->timeout(3);
 
            #For each link found check for 404 error and log to text file #Improvement required. How if there are a relative URI?
            my $response = $ua->get($finalstring);
            if ($response->is_error)
            { 
                #output the link that has 404 error
                open(TEXT,">>broken_link_output.txt");
                printf TEXT $finalstring. "\n";                
            }
            close TEXT;
        }
    }
    until eof;
    close FILEPOINTER;    
}
 
 

[KevinADC]

Ummm yes I guess you can use 3rd party software to check broken link but you won't learn anything...

If the goal includes learning how to write perl code then by all means, proceed. But you did not mention that in your openning post. ;)

[KevinADC]

You will want to look into using File::Find to search in all the sub directories. But then you might not learn how to recurse through directories yourself. ;)


For internal links you might consider just checking if the file exists and is readable instead of trying to fetch files with get(). It should be much faster.

For links to external files you have to use get().

There are also bound to be links to the same file more than once, you may want to avoid checking for the same file. You can use a hash table to keep track of what links are already checked.

The "do{}until eof" construct is not necessary. You might want to look into HTML::LinkExtractor (I think thats the name) to find the links in the html documents.