Perl for checking broken link on a website

[sweetbox]

Hi Guys!


I am new on this forum and I am hoping that I could acquire help to improve my PERL skill.

My goal is to be able to create a report of all broken links from a website. Based from my understanding, PERL can go through to server directory and search for htm/html file and put it on an array. From this array I believe PERL can search for a "http://" string and emulate to click the URL. If the URL contains "Page cannot be found" then create and append to a log file.

My question, what is the best method to do this and what is the required library and syntax?

Also for searching the "htm" file on a server, assuming that I don't have access to get in to server directory (which is unlikely) , how do I get this done? and if I have an access to the server how do I get PERL to login to the server first before PERL search for "htm" file on a specific directory? or do I have to put the PERL .pl file on the server root directory and run it there?

I really appreciate if someone could help. Thank you.

[KevinADC]

Ummm yes I guess you can use 3rd party software to check broken link but you won't learn anything...

If the goal includes learning how to write perl code then by all means, proceed. But you did not mention that in your openning post. ;)

[KevinADC]

You will want to look into using File::Find to search in all the sub directories. But then you might not learn how to recurse through directories yourself. ;)


For internal links you might consider just checking if the file exists and is readable instead of trying to fetch files with get(). It should be much faster.

For links to external files you have to use get().

There are also bound to be links to the same file more than once, you may want to avoid checking for the same file. You can use a hash table to keep track of what links are already checked.

The "do{}until eof" construct is not necessary. You might want to look into HTML::LinkExtra ctor (I think thats the name) to find the links in the html documents.