Using WebClient to get the actual file name.

[Scampi]

Hi,

I've been working on an application to do some 'scraping' of web
content, using the WebClient class. I'm using code rather like the
following..

Dim objWebClient As New WebClient
Dim strURL As String = CType(URL, String)
Dim aRequestedHTML() As Byte
Dim objUTF8 As New UTF8Encoding
Dim strRequestedHTML As String

aRequestedHTML = objWebClient.DownloadData(strURL)
strRequestedHTML = objUTF8.GetString(aRequestedHTML)

Return strRequestedHTML

However, if I enter the url, say
http://web.archive.org/web/200306221...odhouse.co.uk/

the returned html is not the page that I requested (as viewed in the
browser), but rather a error page saying that the page has not been
found.

Does anyone know why this may be the case? My ideas so far have been
that the web browser (and server) and work out to send the correct
page, while the Web Client class isn't actually specifying a page at
the end of the url..

If anyone has an idea on this problem I would like to hear it, the
deadline for this project is coming up fast and this is a stumbling
block that I need to overcome, once this is done then its (hopefully!)
all plain sailing from here.

Thanks for your time

Chris Williams (chris (at) oxymoron-failsafe.com)

[William LaMartin]

If you are wanting the content of the page, here are two ways of getting it.

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As
System.EventArgs) Handles Button1.Click
Dim wc As New System.Net.WebClient
Dim ascii As System.Text.Encoding = System.Text.Encoding.ASCII
Dim Results As Byte() =
wc.DownloadData("http://web.archive.org/web/20030622145316/http://www.lynwoo
dhouse.co.uk/")
Dim asciiChars(Results.GetLength(0)) As Char
ascii.GetChars(Results, 0, Results.Length, asciiChars, 0)
Dim asciiString As New String(asciiChars)
MsgBox(asciiString)
End Sub
Private Sub Button2_Click(ByVal sender As System.Object, ByVal e As
System.EventArgs) Handles Button2.Click
Dim myRequest As System.Net.WebRequest =
System.Net.WebRequest.Create("http://web.archive.org/web/20030622145316/http
://www.lynwoodhouse.co.uk/")
Dim myResponse As System.Net.WebResponse = myRequest.GetResponse()
Dim myStream As System.IO.Stream = myResponse.GetResponseStream
Dim sr As System.IO.StreamReader = New
System.IO.StreamReader(myStream)
MsgBox(sr.ReadToEnd)
myStream.Close()
myResponse.Close()
End Sub

"Scampi" <sc****@lyingsackofshit.com> wrote in message
news:b7**************************@posting.google.c om...

Hi,

I've been working on an application to do some 'scraping' of web
content, using the WebClient class. I'm using code rather like the
following..

Dim objWebClient As New WebClient
Dim strURL As String = CType(URL, String)
Dim aRequestedHTML() As Byte
Dim objUTF8 As New UTF8Encoding
Dim strRequestedHTML As String

aRequestedHTML = objWebClient.DownloadData(strURL)
strRequestedHTML = objUTF8.GetString(aRequestedHTML)

Return strRequestedHTML

However, if I enter the url, say
http://web.archive.org/web/200306221...odhouse.co.uk/

the returned html is not the page that I requested (as viewed in the
browser), but rather a error page saying that the page has not been
found.

Does anyone know why this may be the case? My ideas so far have been
that the web browser (and server) and work out to send the correct
page, while the Web Client class isn't actually specifying a page at
the end of the url..

If anyone has an idea on this problem I would like to hear it, the
deadline for this project is coming up fast and this is a stumbling
block that I need to overcome, once this is done then its (hopefully!)
all plain sailing from here.

Thanks for your time

Chris Williams (chris (at) oxymoron-failsafe.com)