Incomplete WebResponse?

[Chris Coddington]

I make a request, stuff it into a stringbuffer, then append it to a
textbox. It works on many sites (e.g. yahoo.com), but not on many
others (e.g. amazon.com). Any insight that could be provided would be
greatly appreciated.

Relevant code:

Sub GetPage()
Try

Dim loRequest As HttpWebRequest
Dim loResponse As HttpWebResponse
Dim loSR As StreamReader
Dim loSB As New StringBuilder

loRequest = CType(WebRequest.Create("http://www.amazon.com"),
HttpWebRequest)
loResponse = CType(loRequest.GetResponse(), HttpWebResponse)
loSR = New StreamReader(loResponse.GetResponseStream,
Encoding.ASCII)
loSB.Append(loSR.ReadToEnd)
loSR.Close()

txtStatus.AppendText(loSB.ToString)

Catch ex As Exception
MessageBox.Show(ex.ToString)
End Try

End Sub

[Cor Ligthert]

Hi Chris,

Probably an easy one,
A lot of site pages exist as seperate documents.
A frame containerdocument and framedocuments.

So when you try to get them as you do, you get the frame container, try to
find the seperate frames one by one.

I hope this helps?

Cor

[Chris Coddington]

When I say that it doesn't work, I mean that I get a response from the
webserver but I don't get the complete page. What am I missing?

ch*************@hotmail.com (Chris Coddington) wrote in message news:<77**************************@posting.google. com>...

I make a request, stuff it into a stringbuffer, then append it to a
textbox. It works on many sites (e.g. yahoo.com), but not on many
others (e.g. amazon.com). Any insight that could be provided would be
greatly appreciated.

Relevant code:

Sub GetPage()
Try

Dim loRequest As HttpWebRequest
Dim loResponse As HttpWebResponse
Dim loSR As StreamReader
Dim loSB As New StringBuilder

loRequest = CType(WebRequest.Create("http://www.amazon.com"),
HttpWebRequest)
loResponse = CType(loRequest.GetResponse(), HttpWebResponse)
loSR = New StreamReader(loResponse.GetResponseStream,
Encoding.ASCII)
loSB.Append(loSR.ReadToEnd)
loSR.Close()

txtStatus.AppendText(loSB.ToString)

Catch ex As Exception
MessageBox.Show(ex.ToString)
End Try

End Sub

[Cor Ligthert]

Hi Chris,

What is not a complete page than?

Cor

[Chris Coddington]

I'll get a random number of bytes (never consistent) and I'll never
get the closing body and html tags. It's as though I only get half the
html. I have tinkered with the timeout but it never throws an
exception - of any kind.

"Cor Ligthert" <no**********@planet.nl> wrote in message news:<Oj*************@TK2MSFTNGP11.phx.gbl>...

Hi Chris,

What is not a complete page than?

Cor

[Cor Ligthert]

Hi Chris,

And it is sure a HTML page not something as a bytearrea?

Maybe you can download the file doing a webclient download to see the file.
\\\
Dim wbc As New System.Net.WebClient
wbc.DownloadFile(Uri, Filename)
///
Than you can see the file normaly

Otherwise I think that we are just guessing

I hope this helps?

Cor

[Chris Coddington]

Hello Cor,

Well, that worked. But I'd like to eliminate the need to use the
local filesystem. Any other ways to handle this?

thx - Chris

"Cor Ligthert" <no**********@planet.nl> wrote in message news:<OA**************@TK2MSFTNGP11.phx.gbl>...

Hi Chris,

And it is sure a HTML page not something as a bytearrea?

Maybe you can download the file doing a webclient download to see the file.
\\\
Dim wbc As New System.Net.WebClient
wbc.DownloadFile(Uri, Filename)
///
Than you can see the file normaly

Otherwise I think that we are just guessing

I hope this helps?

Cor

[Cor Ligthert]

HI Chris,

Is the file a real HTML page with a start HTML tag and an end HTML tag?

Cor

Hello Cor,

Well, that worked. But I'd like to eliminate the need to use the
local filesystem. Any other ways to handle this?

thx - Chris

[Chris Coddington]

Yes, I get the entire page. When I use any of the other methods of
pulling it down that return a stream, I don't get the entire page.
I've tried:

WebClient.OpenRead
WebClient.DownloadData
WebRequest.GetResponse

Only WebClient.DownloadFile works consistently. The other three
methods work on other sites, but for whatever reason, amazon.com gives
me a problem. The symptoms:

1. The result comes back quickly.
2. No exceptions are thrown.
3. The page is never complete.
4. The cutoff point is never in the same spot in the page.
5. The size of the stream is never the same.
6. When "sniffing" the underlying protocol using Ethereal, it cuts off
just as it does in the software.
7. The same problem occurs on other computers that I have.

Arrrrrrgh! Why just amazon.com?

thx - Chris

"Cor Ligthert" <no**********@planet.nl> wrote in message news:<uS*************@TK2MSFTNGP11.phx.gbl>...

HI Chris,

Is the file a real HTML page with a start HTML tag and an end HTML tag?

Cor

Hello Cor,

Well, that worked. But I'd like to eliminate the need to use the
local filesystem. Any other ways to handle this?

thx - Chris

[Cor Ligthert]

Hi Chris,

This did work for me (see that I have used the RichTextbox, the textbox does
not)

Will you tell if you with this keeps the problems?

Cor
\\\
Dim myReg As Net.HttpWebRequest = _
DirectCast(Net.WebRequest.Create("http://www.amazon.com"), _
Net.HttpWebRequest)
Dim myResp As Net.HttpWebResponse = _
DirectCast(myReg.GetResponse(), Net.HttpWebResponse)
Dim myStream As IO.Stream = myResp.GetResponseStream()
Dim myreader As New IO.StreamReader(myStream)
Dim mystring As String = myreader.ReadToEnd()
Me.RichTextBox1.Text = mystring
myResp.Close()
///

[Chris Coddington]

Cor - you are a freaking genious! GENIOUS! I would like to kindly
repay you for your help. Shoot me an email plz!

I swapped out my textbox for the richtext box and whammo it worked.

thx - Chris

"Cor Ligthert" <no**********@planet.nl> wrote in message news:<ub**************@TK2MSFTNGP12.phx.gbl>...

Hi Chris,

This did work for me (see that I have used the RichTextbox, the textbox does
not)

Will you tell if you with this keeps the problems?

Cor
\\\
Dim myReg As Net.HttpWebRequest = _
DirectCast(Net.WebRequest.Create("http://www.amazon.com"), _
Net.HttpWebRequest)
Dim myResp As Net.HttpWebResponse = _
DirectCast(myReg.GetResponse(), Net.HttpWebResponse)
Dim myStream As IO.Stream = myResp.GetResponseStream()
Dim myreader As New IO.StreamReader(myStream)
Dim mystring As String = myreader.ReadToEnd()
Me.RichTextBox1.Text = mystring
myResp.Close()
///