Finding Broken Link in WebSite

[sristhrashguy]

Hi everyone,
i want .net(VB or C#) code for finding broken links in a website.
The requirement is that the user will be able to type the
url in a text box so once the button is clicked , it has to show
whether there are any broken links in that particular page.
Please help me out in this.


Thanks
Sridhar.S

[wimpos]

You can download the website entered into the textbox.
try using the webclient class
Now you have the source html code of the webpage.
Than find al the links in the webpages <a href="**"></a> (maybe a method in webclient, otherwise use a regex)
extract the ** and try to download this website as you did before. If it succeeds the link is alive, otherwise it is dead.

This is a guidline you can follow. try it, if you run into trouble don't hesitate to get more detailed info.

regards
W.

[kenobewan]

I find this question interesting because a link isn't really broken until you click and get the 404 error. Also an ok local link may be broken on the web. Having said that there are third party tools that may help, but my assumption is that it may be a case of 'physician heal thyself".

Having said that a site map may help, but you would need to customize to cover all dynamic links. Here is a great resource on TDD, applied mainly to software but if web programming was done this way...
My Uber-Test-Driven Development ("TDD") Links Listing

[Plater]

W3.org offers testing for broken links.
All they do is check to see if a good status is returned when they attempt to navigate to the address inside an anchor tag.

[sristhrashguy]

Hi all,

I'm storing a set of "<a href" tags in a list. Now i want to check all these links are valid or not? i.e, the list contains any broken links or not.

This is the code.....

the code will allow user to enter the url and it will render all the html code back
and display only the href tags. Now i want to check the validity of all these links
and display the result.... i.e all the links are not dead.....

// make an object of the WebClient class 
            WebClient objWebClient = new WebClient();
         // gets the HTML from the url written in the textbox
            aRequestHTML = objWebClient.DownloadData(TextBox1.Text); 
         // creates UTf8 encoding object
            UTF8Encoding utf8 = new UTF8Encoding();  
        // gets the UTF8 encoding of all the html we got in aRequestHTML
            myString = utf8.GetString(aRequestHTML); 
 
 
            ArrayList list = new ArrayList();
            int curindex = 0;
            int index = 0;
            do
            {
                index = myString.IndexOf("<a href=", curindex);
                if (index==-1) {break;}
                curindex = myString.IndexOf(">", index);
                string ancordata = myString.Substring(index, curindex - index);
                if (ancordata.ToLower().IndexOf("javascript") < 0)
                {                    
                    list.Add(ancordata);
                }
            } 
            while (index != -1);
              GridView1.DataSource = list; 
        //// binds the databind
            GridView1.DataBind();
 

[sristhrashguy]

You can download the website entered into the textbox.
try using the webclient class
Now you have the source html code of the webpage.
Than find al the links in the webpages <a href="**"></a> (maybe a method in webclient, otherwise use a regex)
extract the ** and try to download this website as you did before. If it succeeds the link is alive, otherwise it is dead.

This is a guidline you can follow. try it, if you run into trouble don't hesitate to get more detailed info.

regards
W.

Hi thanks for ur reply,

i have passed half the ocean.
Now i have all the list of "<a href" tags in a array list.
I'm totally stuck here. Please help me.

[Plater]

Try using an HttpWebRequest to see if they return a good status or not?

[jallred]

Try using an HttpWebRequest to see if they return a good status or not?

Spot on. For each url, use the code at http://msdn2.microsoft .com/en-us/library/system.net.http webrequest.getr esponse.aspx, check the status code of the response. 200 is OK, while 404 is the typical broken link. Other status codes will require some judgement.

John
http://blogs.msdn.com/usisvde/

[wimpos]

Tip:
It might be not that important but I still recommend using a regular expression to search for the <a href="" >

It's cleaner, it 's faster, more reliable

regards