473,322 Members | 1,714 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

time consuming issue with a HttpWebRequest and StreamReader

Hi,

i have a vb.net windows application that makes HTTP requests with the framework's HttpWebReqeuest object.
i have about 10 different requests that takes between 30 to 500 for the Request.GetResponse function to execute.
this is acceptable.
problem is that for 1 of the 10 urls i request, when i call the

responseBody = reader.ReadToEnd() & ""

it takes about 11.50 seconds (!) to execute and return the responseBody.
for all the resources i request including the misbehaved one, the return content length is between 1000 and 40,000 bytes, and for the misbehaved one 8063 bytes. (should not cause any problems)

the misbehaved request is from a dot.net aspx page.
the request takes about 46 milliseconds to return a 8063 bytes of text/html (not binary)

i have other aspx resources in my list that are behaving OK. for example a 92,470 bytes response takes about 30 milliseconds to execute and for the ReadToEnd another 671 milliseconds.
the reader definition looks like that:
reader = New StreamReader(res.GetResponseStream(), System.Text.Encoding.GetEncoding("utf-8"), True)
i hope someone will have some direction to look for the problem source.

TIA, z.
Nov 19 '05 #1
5 1628
z. f. wrote:
Hi,

i have a vb.net windows application that makes HTTP requests with the
framework's HttpWebReqeuest object. i have about 10 different
requests that takes between 30 to 500 for the Request.GetResponse
function to execute. this is acceptable. problem is that for 1 of
the 10 urls i request, when i call the

responseBody = reader.ReadToEnd() & ""
Out of curiosity -- why do append an empty string?
it takes about 11.50 seconds (!) to execute and return the
responseBody. for all the resources i request including the
misbehaved one, the return content length is between 1000 and 40,000
bytes, and for the misbehaved one 8063 bytes. (should not cause any
problems)
How did you obtain this value? By measuring the number of bytes
received or by relying on the Content-Length header?
the misbehaved request is from a dot.net aspx page.
the request takes about 46 milliseconds to return a 8063 bytes of
text/html (not binary)
So that's basiscally the time it takes to execute GetReponseStream(),
right?
i have other aspx resources in my list that are behaving OK. for
example a 92,470 bytes response takes about 30 milliseconds to
execute and for the ReadToEnd another 671 milliseconds.
the reader definition looks like that:
reader = New StreamReader(res.GetResponseStream(),
System.Text.Encoding.GetEncoding("utf-8"), True)
i hope someone will have some direction to look for the problem
source.


Have a look at this thread for a discussion of decoding-on-the-fly
(what you do) versus "raw" reading -- maybe a reading first and
decoding later delivers better results:

http://groups.google.de/groups?hl=de...threadm=MPG.1c
31f46243ba52bd98bafb%40msnews.microsoft.com&rnum=1 &prev=/groups%3Fq%3Djo
oss%2Bskeet%2BStreamReader%26hl%3Dde%26lr%3D%26cli ent%3Dfirefox-a%26selm
%3DMPG.1c31f46243ba52bd98bafb%2540msnews.microsoft .com%26rnum%3D1

Cheers,
--
http://www.joergjooss.de
mailto:ne********@joergjooss.de
Nov 19 '05 #2
thanks, i think that the direction of the parsing might be the point of the problem, though 11.50 seconds to execute on a 8000 bytes is not acceptable.
other questions - see inline:

"Joerg Jooss" <ne********@joergjooss.de> wrote in message news:xn****************@msnews.microsoft.com...
z. f. wrote:
Hi,

i have a vb.net windows application that makes HTTP requests with the
framework's HttpWebReqeuest object. i have about 10 different
requests that takes between 30 to 500 for the Request.GetResponse
function to execute. this is acceptable. problem is that for 1 of
the 10 urls i request, when i call the

responseBody = reader.ReadToEnd() & ""
Out of curiosity -- why do append an empty string?

i don't remember the reason i did this a year ago, but even if i remove the empty string addition the result is the same.
it takes about 11.50 seconds (!) to execute and return the
responseBody. for all the resources i request including the
misbehaved one, the return content length is between 1000 and 40,000
bytes, and for the misbehaved one 8063 bytes. (should not cause any
problems)
How did you obtain this value? By measuring the number of bytes
received or by relying on the Content-Length header?

i look for the content-length header, if i don't have one, i check the length of the data.
usually responseBody.length

the misbehaved request is from a dot.net aspx page.
the request takes about 46 milliseconds to return a 8063 bytes of
text/html (not binary)
So that's basiscally the time it takes to execute GetReponseStream(),
right?

the 46 milliseconds value if for the request.GetResponse function to execute.

i have other aspx resources in my list that are behaving OK. for
example a 92,470 bytes response takes about 30 milliseconds to
execute and for the ReadToEnd another 671 milliseconds.


the reader definition looks like that:
reader = New StreamReader(res.GetResponseStream(),
System.Text.Encoding.GetEncoding("utf-8"), True)


i hope someone will have some direction to look for the problem
source.


Have a look at this thread for a discussion of decoding-on-the-fly
(what you do) versus "raw" reading -- maybe a reading first and
decoding later delivers better results:

http://groups.google.de/groups?hl=de...threadm=MPG.1c
31f46243ba52bd98bafb%40msnews.microsoft.com&rnum=1 &prev=/groups%3Fq%3Djo
oss%2Bskeet%2BStreamReader%26hl%3Dde%26lr%3D%26cli ent%3Dfirefox-a%26selm
%3DMPG.1c31f46243ba52bd98bafb%2540msnews.microsoft .com%26rnum%3D1

Cheers,
--
http://www.joergjooss.de
mailto:ne********@joergjooss.de
Nov 19 '05 #3
interesting....
i noticed that the missbehaved request had an exception in the asp.net code.
the page flushed some HTML comments and then the asp.net framework wrote the exception details.
for some reason this small page ( less than 8 KB) is taking about 11.50 (constantly) for the streamReader.ReadToEnd with a unicode encoding.
strange.

"z. f." <zi**@info-scopeREMSPAM.co.il> wrote in message news:ea**************@tk2msftngp13.phx.gbl...
thanks, i think that the direction of the parsing might be the point of the problem, though 11.50 seconds to execute on a 8000 bytes is not acceptable.
other questions - see inline:

"Joerg Jooss" <ne********@joergjooss.de> wrote in message news:xn****************@msnews.microsoft.com...
z. f. wrote:
Hi,

i have a vb.net windows application that makes HTTP requests with the
framework's HttpWebReqeuest object. i have about 10 different
requests that takes between 30 to 500 for the Request.GetResponse
function to execute. this is acceptable. problem is that for 1 of
the 10 urls i request, when i call the

responseBody = reader.ReadToEnd() & ""
Out of curiosity -- why do append an empty string?

i don't remember the reason i did this a year ago, but even if i remove the empty string addition the result is the same.
it takes about 11.50 seconds (!) to execute and return the
responseBody. for all the resources i request including the
misbehaved one, the return content length is between 1000 and 40,000
bytes, and for the misbehaved one 8063 bytes. (should not cause any
problems)
How did you obtain this value? By measuring the number of bytes
received or by relying on the Content-Length header?

i look for the content-length header, if i don't have one, i check the length of the data.
usually responseBody.length

the misbehaved request is from a dot.net aspx page.
the request takes about 46 milliseconds to return a 8063 bytes of
text/html (not binary)
So that's basiscally the time it takes to execute GetReponseStream(),
right?

the 46 milliseconds value if for the request.GetResponse function to execute.

i have other aspx resources in my list that are behaving OK. for
example a 92,470 bytes response takes about 30 milliseconds to
execute and for the ReadToEnd another 671 milliseconds.


the reader definition looks like that:
reader = New StreamReader(res.GetResponseStream(),
System.Text.Encoding.GetEncoding("utf-8"), True)


i hope someone will have some direction to look for the problem
source.


Have a look at this thread for a discussion of decoding-on-the-fly
(what you do) versus "raw" reading -- maybe a reading first and
decoding later delivers better results:

http://groups.google.de/groups?hl=de...threadm=MPG.1c
31f46243ba52bd98bafb%40msnews.microsoft.com&rnum=1 &prev=/groups%3Fq%3Djo
oss%2Bskeet%2BStreamReader%26hl%3Dde%26lr%3D%26cli ent%3Dfirefox-a%26selm
%3DMPG.1c31f46243ba52bd98bafb%2540msnews.microsoft .com%26rnum%3D1

Cheers,
--
http://www.joergjooss.de
mailto:ne********@joergjooss.de
Nov 19 '05 #4
z. f. wrote:
thanks, i think that the direction of the parsing might be the point
of the problem, though 11.50 seconds to execute on a 8000 bytes is
not acceptable.
Certainly not. Maybe there's a problem with the UTF8Decoder, so trying
to receive the raw bytes first should reveal quickly whether this is
true or not.
Out of curiosity -- why do append an empty string?

i don't remember the reason i did this a year ago, but even if i
remove the empty string addition the result is the same.
That's what I expected ;-)
How did you obtain this value? By measuring the number of bytes
received or by relying on the Content-Length header?

i look for the content-length header, if i don't have one, i check
the length of the data. usually responseBody.length


OK. Did you verify that the Content-Length is correct? Sending an
incorrect Content-Length header can produce weird results depending on
the receiver's implementation.

<remark>
Don't get me, wrong, but please learn to use a newsreader. Replying to
your post is a pain!
</remark>

Cheers,

--
http://www.joergjooss.de
mailto:ne********@joergjooss.de
Nov 19 '05 #5
IMO there is no problem in your vb.net code that requests aspx page through
HttpWebRequest.

In general time taken by ASPX, also consists of the time it takes to get
result from database. Check the URL in question, gets the results from the
Database or not if possible check how much time the ASPX takes when you
access it using browser, for this you can enable Trace.
Nov 19 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: scott rosa | last post by:
I am having an issue with using the httpwebrequest to perform a screenscrape. The site I am retrieving data from has forms based authentication. When I pass the correct credentials it...
0
by: TJO | last post by:
Can someone at MS please reply to this. I am trying to post data so a web form via ssl with the following code. I keep getting this error: "The underlying connection was closed: Could not...
10
by: Gregory A Greenman | last post by:
I'm trying to write a program in vb.net to automate filling out a series of forms on a website. There are three forms I need to fill out in sequence. The first one is urlencoded. My program is...
9
by: Mike Cronin via DotNetMonster.com | last post by:
Hi there, Can anyone tell me what level of encryption is used when making an HTTPS POST request through an instance of the System.Net.HttpWebRequest object? Thanks much in advance! Mike...
16
by: thomas peter | last post by:
I am building a precache engine... one that request over 100 pages on an remote server to cache them remotely... can i use the HttpWebRequest and WebResponse classes for this? or must i use the...
0
by: Veerle | last post by:
Hi, On the website of the Belgian lottery, you can download an excel sheet with lottery results (the winning numbers) over the years and an excel sheet with financial results (the winnings) over...
12
by: Jens Jensen | last post by:
I try to set the timeout property on my HttpWebRequest object before lunching the request. It seems like any value above 100 seconds is discarded. Has anyone observered malfunctional with this...
4
by: Haxan | last post by:
Hi, Im testing a web service from my windows app in C#. I create a soap body and pass to the HttpWebRequest and then get the response as below. private void TestService() { string quotes =...
1
by: beebee | last post by:
OK all I have a webservice utilizing API's in to an application. I want to call this service and load the returned information onto my aspx page. right now i am not getting any errors in my...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.