Ferreting out broken links

[Dave]

Is it difficult to write a program that, given an array of URLs, will probe
each one, and return a status of Found or Not Found? How would you approach
it?

While Googling, I found utility after utility that will do something like
that for you, but I would like to write a custom program to do this. It
doesn't have to be VB. C#, javascript, etc. - whatever will run on .NET.

Thanks in advance,

Dave

[AMDRIT]

Try something like this. It is not the most intelligent nor elegant
solution, but it will get you what you want.

Dim aList As ArrayList
Dim qXML As Xml.XmlDocument
qXML = New Xml.XmlDocument

aList = New ArrayList

Dim oml As mylist
With oml
.SiteIndex = 1
.SiteURL = "http://www.microsoft.c om"
.SiteValidFlagB oolean = False
End With

aList.Add(oml)

With oml
.SiteIndex = 1
.SiteURL = "http://www.yourdomain. com"
.SiteValidFlagB oolean = False
End With

aList.Add(oml)

For Each oml In aList

Try
qXML.Load(oml.S iteURL)
oml.SiteValidFl agBoolean = True
Catch exxml As System.Xml.XmlE xception
'Page loaded, but was not parsable by xml
oml.SiteValidFl agBoolean = True
Catch exweb As System.net.WebE xception
'Page Not Found
If exweb.ToString. IndexOf("404") > 0 Then
oml.SiteValidFl agBoolean = False
Else
'Some Other Net Message, prolly domain not found.
MsgBox(exweb.To String)
End If
Catch ex As Exception
MsgBox(ex.ToStr ing)
End Try

Next
"Dave" <da************ *************** **@stic.net> wrote in message
news:%2******** ********@TK2MSF TNGP14.phx.gbl. ..

Is it difficult to write a program that, given an array of URLs, will
probe each one, and return a status of Found or Not Found? How would you
approach it?

While Googling, I found utility after utility that will do something like
that for you, but I would like to write a custom program to do this. It
doesn't have to be VB. C#, javascript, etc. - whatever will run on .NET.

Thanks in advance,

Dave

[Curtis]

Dave,

Not exactly sure what you are wanting but it might be similar to a function
I use in one of my apps. You can call this function within in a loop, and if
you don't receive a reponse it will catch the exception, it uses the MSHTML
class. It might be a little more than you need but might be what you are
looking for.

Public Function Send(ByVal URL As String, _
Optional ByVal PostData As String = "", _
Optional ByVal Method As HTTPMethod = HTTPMethod.HTTP _GET, _
Optional ByVal ContentType As String = "") As String
Dim Request As HttpWebRequest = WebRequest.Crea te(URL)
Dim Response As HttpWebResponse
Dim SW As StreamWriter
Dim SR As StreamReader
Dim ResponseData As String
Dim I As Integer
Dim RcookCon As New CookieContainer

' Prepare Request Object
Request.Method = Method.ToString ().Substring(5)
Request.KeepAli ve = True
Request.AllowAu toRedirect = True
If HldCookCon.Coun t > 0 Then
RcookCon = HldCookCon
End If
Request.CookieC ontainer = RcookCon

' Set form/post content-type if necessary
If (Method = HTTPMethod.HTTP _POST AndAlso PostData <> "" AndAlso
ContentType = "") Then
ContentType = "applicatio n/x-www-form-urlencoded"
End If

' Set Content-Type
If (ContentType <> "") Then
Request.Content Type = ContentType
Request.Content Length = PostData.Length
End If

' Send Request, If Request
If (Method = HTTPMethod.HTTP _POST) Then
Try
SW = New StreamWriter(Re quest.GetReques tStream())
SW.Write(PostDa ta)
Catch Err As WebException
MsgBox(Err.Mess age, MsgBoxStyle.Inf ormation, "Error")

Finally
Try
SW.Close()
Catch
'Don't process an error from SW not closing
End Try
End Try
End If
'Get Response
Try
Response = Request.GetResp onse()
SR = New StreamReader(Re sponse.GetRespo nseStream())
ResponseData = SR.ReadToEnd()
'Display cookies
For I = 0 To Response.Cookie s.Count - 1
HldCookCon.Add( Response.Cookie s.Item(I))
Next
Catch Err As WebException
Return False
Finally
Try
SR.Close()
Catch
'Don't process an error from SR not closing
End Try
End Try
Return ResponseData
End Function

Curtis

"Dave" <da************ *************** **@stic.net> wrote in message
news:%2******** ********@TK2MSF TNGP14.phx.gbl. ..

Is it difficult to write a program that, given an array of URLs, will
probe each one, and return a status of Found or Not Found? How would you
approach it?

While Googling, I found utility after utility that will do something like
that for you, but I would like to write a custom program to do this. It
doesn't have to be VB. C#, javascript, etc. - whatever will run on .NET.

Thanks in advance,

Dave

[Tom Shelton]

In article <#1************ **@TK2MSFTNGP14 .phx.gbl>, Dave wrote:

Is it difficult to write a program that, given an array of URLs, will probe
each one, and return a status of Found or Not Found? How would you approach
it?

While Googling, I found utility after utility that will do something like
that for you, but I would like to write a custom program to do this. It
doesn't have to be VB. C#, javascript, etc. - whatever will run on .NET.

Thanks in advance,

Dave

hmmm...

Should be fairly straight forward using the System.Net.WebC lient class.
Or better yet, would probably be the System.Net.Http WebRequest class...

Something like:

Dim request As HttpWebRequest
Dim response As HttpWebResponse

For Each url As String In urls
request = WebRequest.Crea te (url)
response = request.GetResp onse ()

If Response.Status Code = 404 Then
Console.WriteLi ne ("Not Found")
Else
Console.WriteLi ne ("Found")
End If
Next

Actually, you might want to do a more take a closer look at the
StatusCode :)
--
Tom Shelton [MVP]

[Dave]

Amdrit, Curtis, Tom,

I haven't worked all these details out yet, but I didn't want too much time
to pass before I said thanks.

I have only had time to try Curtis & Tom's solution. The
Response.GetRes ponse() line is giving me a problem when I try to use
https:// sites because they cannot establish a trust relationship. This may
not be fatal - it may be acceptable to just inform my users that they will
have to manually check those.

I'm getting a problem when I hit a 404; Tom's line "If Response.Status Code
= 404 Then" just doesn't work.

I get the crash screen with the message "The remote server returned an
error: (404) Not Found"
So, I'm going to try to work with those methods, then I'll try Amdrit's
method if I can't get them good to go.

I really do appreciate the responses. I had no clue how to approach this
problem, and now I have plenty to work with.

Thanks again.

Dave


"Dave" <da************ *************** **@stic.net> wrote in message
news:%2******** ********@TK2MSF TNGP14.phx.gbl. ..

Is it difficult to write a program that, given an array of URLs, will
probe each one, and return a status of Found or Not Found? How would you
approach it?

While Googling, I found utility after utility that will do something like
that for you, but I would like to write a custom program to do this. It
doesn't have to be VB. C#, javascript, etc. - whatever will run on .NET.

Thanks in advance,

Dave

[Tom Shelton]

In article <#$************ *@TK2MSFTNGP14. phx.gbl>, Dave wrote:

Amdrit, Curtis, Tom,

I haven't worked all these details out yet, but I didn't want too much time
to pass before I said thanks.

I have only had time to try Curtis & Tom's solution. The
Response.GetRes ponse() line is giving me a problem when I try to use
https:// sites because they cannot establish a trust relationship. This may
not be fatal - it may be acceptable to just inform my users that they will
have to manually check those.

I'm getting a problem when I hit a 404; Tom's line "If Response.Status Code
= 404 Then" just doesn't work.

I get the crash screen with the message "The remote server returned an
error: (404) Not Found"
So, I'm going to try to work with those methods, then I'll try Amdrit's
method if I can't get them good to go.

I really do appreciate the responses. I had no clue how to approach this
problem, and now I have plenty to work with.

Thanks again.

Dave


"Dave" <da************ *************** **@stic.net> wrote in message
news:%2******** ********@TK2MSF TNGP14.phx.gbl. ..

Is it difficult to write a program that, given an array of URLs, will
probe each one, and return a status of Found or Not Found? How would you
approach it?

While Googling, I found utility after utility that will do something like
that for you, but I would like to write a custom program to do this. It
doesn't have to be VB. C#, javascript, etc. - whatever will run on .NET.

Thanks in advance,

Dave

Dave,

Sounds like you might need to refine what I wrote some :) I did that
off the top, and didn't test any of that - so it may not be exactly
right. Another method that's a bit more work, but not to bad, is to
simply open a socket connection to the server and make the http request
your self. This would avoid the "non-trust" issues and exceptions :)

--
Tom Shelton [MVP]

[Curtis]

Dave,

This should fix the trusted relationship problem. Its kinda of a work around
that I found. Here is a link to an explanation of it:
http://gotdotnet.com/Community/Messa....aspx?id=40795. I have
implemented it in VB.net by creating a new class with the following code.

Imports System.Net
Imports System.Security .Cryptography.X 509Certificates
Public Class myCertificatePo licy
Implements ICertificatePol icy
Public Function CheckValidation Result(ByVal srvPoint As ServicePoint, _
ByVal cert As X509Certificate , ByVal request As
WebRequest, _
ByVal certificateProb lem As Integer) _
As Boolean Implements
ICertificatePol icy.CheckValida tionResult
'Return True to force the certificate to be accepted.
Return True
End Function
End Class

You would then call the class with this line in your application:

'force the certificate to be accepted
System.Net.Serv icePointManager .CertificatePol icy = New myCertificatePo licy

This basically overrides a "non-trusted connection" by making your
application alway accept the certificates.

Curtis

"Dave" <da************ *************** **@stic.net> wrote in message
news:%2******** *******@TK2MSFT NGP14.phx.gbl.. .

Amdrit, Curtis, Tom,

I haven't worked all these details out yet, but I didn't want too much
time to pass before I said thanks.

I have only had time to try Curtis & Tom's solution. The
Response.GetRes ponse() line is giving me a problem when I try to use
https:// sites because they cannot establish a trust relationship. This
may not be fatal - it may be acceptable to just inform my users that they
will have to manually check those.

I'm getting a problem when I hit a 404; Tom's line "If
Response.Status Code = 404 Then" just doesn't work.

I get the crash screen with the message "The remote server returned an
error: (404) Not Found"
So, I'm going to try to work with those methods, then I'll try Amdrit's
method if I can't get them good to go.

I really do appreciate the responses. I had no clue how to approach this
problem, and now I have plenty to work with.

Thanks again.

Dave


"Dave" <da************ *************** **@stic.net> wrote in message
news:%2******** ********@TK2MSF TNGP14.phx.gbl. ..

Is it difficult to write a program that, given an array of URLs, will
probe each one, and return a status of Found or Not Found? How would you
approach it?

While Googling, I found utility after utility that will do something like
that for you, but I would like to write a custom program to do this. It
doesn't have to be VB. C#, javascript, etc. - whatever will run on .NET.

Thanks in advance,

Dave

[Dave]

Tom & Curtis,

You guys are too much. Thanks.

Dave