Using WebClient to get the actual file name.

[Scampi]

Hi,

I've been working on an application to do some 'scraping' of web
content, using the WebClient class. I'm using code rather like the
following..

Dim objWebClient As New WebClient
Dim strURL As String = CType(URL, String)
Dim aRequestedHTML( ) As Byte
Dim objUTF8 As New UTF8Encoding
Dim strRequestedHTM L As String

aRequestedHTML = objWebClient.Do wnloadData(strU RL)
strRequestedHTM L = objUTF8.GetStri ng(aRequestedHT ML)

Return strRequestedHTM L

However, if I enter the url, say
http://web.archive.org/web/200306221...odhouse.co.uk/

the returned html is not the page that I requested (as viewed in the
browser), but rather a error page saying that the page has not been
found.

Does anyone know why this may be the case? My ideas so far have been
that the web browser (and server) and work out to send the correct
page, while the Web Client class isn't actually specifying a page at
the end of the url..

If anyone has an idea on this problem I would like to hear it, the
deadline for this project is coming up fast and this is a stumbling
block that I need to overcome, once this is done then its (hopefully!)
all plain sailing from here.

Thanks for your time

Chris Williams (chris (at) oxymoron-failsafe.com)

[William LaMartin]

If you are wanting the content of the page, here are two ways of getting it.

Private Sub Button1_Click(B yVal sender As System.Object, ByVal e As
System.EventArg s) Handles Button1.Click
Dim wc As New System.Net.WebC lient
Dim ascii As System.Text.Enc oding = System.Text.Enc oding.ASCII
Dim Results As Byte() =
wc.DownloadData ("http://web.archive.org/web/20030622145316/http://www.lynwoo
dhouse.co.uk/")
Dim asciiChars(Resu lts.GetLength(0 )) As Char
ascii.GetChars( Results, 0, Results.Length, asciiChars, 0)
Dim asciiString As New String(asciiCha rs)
MsgBox(asciiStr ing)
End Sub
Private Sub Button2_Click(B yVal sender As System.Object, ByVal e As
System.EventArg s) Handles Button2.Click
Dim myRequest As System.Net.WebR equest =
System.Net.WebR equest.Create(" http://web.archive.org/web/20030622145316/http
://www.lynwoodhous e.co.uk/")
Dim myResponse As System.Net.WebR esponse = myRequest.GetRe sponse()
Dim myStream As System.IO.Strea m = myResponse.GetR esponseStream
Dim sr As System.IO.Strea mReader = New
System.IO.Strea mReader(myStrea m)
MsgBox(sr.ReadT oEnd)
myStream.Close( )
myResponse.Clos e()
End Sub

"Scampi" <sc****@lyingsa ckofshit.com> wrote in message
news:b7******** *************** ***@posting.goo gle.com...

Hi,

I've been working on an application to do some 'scraping' of web
content, using the WebClient class. I'm using code rather like the
following..

Dim objWebClient As New WebClient
Dim strURL As String = CType(URL, String)
Dim aRequestedHTML( ) As Byte
Dim objUTF8 As New UTF8Encoding
Dim strRequestedHTM L As String

aRequestedHTML = objWebClient.Do wnloadData(strU RL)
strRequestedHTM L = objUTF8.GetStri ng(aRequestedHT ML)

Return strRequestedHTM L

However, if I enter the url, say
http://web.archive.org/web/200306221...odhouse.co.uk/

the returned html is not the page that I requested (as viewed in the
browser), but rather a error page saying that the page has not been
found.

Does anyone know why this may be the case? My ideas so far have been
that the web browser (and server) and work out to send the correct
page, while the Web Client class isn't actually specifying a page at
the end of the url..

If anyone has an idea on this problem I would like to hear it, the
deadline for this project is coming up fast and this is a stumbling
block that I need to overcome, once this is done then its (hopefully!)
all plain sailing from here.

Thanks for your time

Chris Williams (chris (at) oxymoron-failsafe.com)