473,326 Members | 2,732 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

Http stream performance

Hi.
I'm reading a 5 meg (or so) file from the response stream of a WebRequest,
and it's pitifully slow. I mean *really* slow, in the order of 15 minutes to
download. Now obviously I'm caching this on the client (Windows app /
IsolatedStorage), but I can't have it taking this long to download in the
first instance. We've got Gigabit LAN. what's the bottleneck?

I could make the resource available via a different protocol / means if
necessary to overcome the problem.

Has anyone had similar issues or know of a better method of large network
downloads?

Cheers,
Leon
Jul 21 '05 #1
4 2935
Leon Jollans <Le**********************@xaman.com> wrote:
I'm reading a 5 meg (or so) file from the response stream of a WebRequest,
and it's pitifully slow. I mean *really* slow, in the order of 15 minutes to
download. Now obviously I'm caching this on the client (Windows app /
IsolatedStorage), but I can't have it taking this long to download in the
first instance. We've got Gigabit LAN. what's the bottleneck?


How are you reading it, exactly? (Please show some code, in other
words.) Have you tried reading it with a plain browser (or wget) to see
whether it's the client or the server that's holding it up?

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Jul 21 '05 #2
It was definitely the client all along. but I think I've sussed it now: I
was reading the stream byte by byte into a pre-initialised array using
ReadByte from the stream directly and checking for -1 each time.

now I'm doing it using a BinaryReader in chunks and the speed is as per
expectations. code below for reference.

void readStream(Stream stream, long contentLength){
int i=0;
byte[] bytes;
BinaryReader reader = new BinaryReader(stream);
MemoryStream ms = new MemoryStream((int)contentLength);
do
{
bytes = reader.ReadBytes(128);
if(bytes.Length > 0)
{
ms.Write(bytes,0,bytes.Length);
}
else{
break;
}
}while( true );
reader.Close();
stream.Close();
ms.Flush();
bytes = ms.ToArray();
ms.Close();

...

cheers

Leon

"Jon Skeet" <sk***@pobox.com> wrote in message
news:MP************************@news.microsoft.com ...
Leon Jollans <Le**********************@xaman.com> wrote:
I'm reading a 5 meg (or so) file from the response stream of a WebRequest, and it's pitifully slow. I mean *really* slow, in the order of 15 minutes to download. Now obviously I'm caching this on the client (Windows app /
IsolatedStorage), but I can't have it taking this long to download in the first instance. We've got Gigabit LAN. what's the bottleneck?


How are you reading it, exactly? (Please show some code, in other
words.) Have you tried reading it with a plain browser (or wget) to see
whether it's the client or the server that's holding it up?

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

Jul 21 '05 #3
Leon Jollans <Le**********************@xaman.com> wrote:
It was definitely the client all along. but I think I've sussed it now: I
was reading the stream byte by byte into a pre-initialised array using
ReadByte from the stream directly and checking for -1 each time.
Yes, that would definitely slow it down :)
now I'm doing it using a BinaryReader in chunks and the speed is as per
expectations. code below for reference.

void readStream(Stream stream, long contentLength){
int i=0;
byte[] bytes;
BinaryReader reader = new BinaryReader(stream);
MemoryStream ms = new MemoryStream((int)contentLength);
do
{
bytes = reader.ReadBytes(128);
if(bytes.Length > 0)
{
ms.Write(bytes,0,bytes.Length);
}
else{
break;
}
}while( true );
reader.Close();
stream.Close();
ms.Flush();
bytes = ms.ToArray();
ms.Close();


Rather than doing that, I'd do something like:
(assuming you can make contentLength an int instead of a long)
byte[] data = new byte[contentLength];

int offset=0;
int remaining = contentLength;

while (remaining > 0)
{
int read = stream.Read (data, offset, remaining);
if (read==-1)
// Up to you - throw an exception or something similar
// because we didn't get as much as we expected
offset+=read;
remaining-=read;
}

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Jul 21 '05 #4
That's much faster! thanks :)

Leon

"Jon Skeet" <sk***@pobox.com> wrote in message
news:MP************************@news.microsoft.com ...
Leon Jollans <Le**********************@xaman.com> wrote:
It was definitely the client all along. but I think I've sussed it now: I was reading the stream byte by byte into a pre-initialised array using
ReadByte from the stream directly and checking for -1 each time.


Yes, that would definitely slow it down :)
now I'm doing it using a BinaryReader in chunks and the speed is as per
expectations. code below for reference.

void readStream(Stream stream, long contentLength){
int i=0;
byte[] bytes;
BinaryReader reader = new BinaryReader(stream);
MemoryStream ms = new MemoryStream((int)contentLength);
do
{
bytes = reader.ReadBytes(128);
if(bytes.Length > 0)
{
ms.Write(bytes,0,bytes.Length);
}
else{
break;
}
}while( true );
reader.Close();
stream.Close();
ms.Flush();
bytes = ms.ToArray();
ms.Close();


Rather than doing that, I'd do something like:
(assuming you can make contentLength an int instead of a long)
byte[] data = new byte[contentLength];

int offset=0;
int remaining = contentLength;

while (remaining > 0)
{
int read = stream.Read (data, offset, remaining);
if (read==-1)
// Up to you - throw an exception or something similar
// because we didn't get as much as we expected
offset+=read;
remaining-=read;
}

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

Jul 21 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: Ian Robertson | last post by:
I am trying to write a function that takes a reference to an object as its arguement. The object is created in another function and I am trying to pass the object to another function from within...
15
by: Adam H. Peterson | last post by:
I would like to make a stream or streambuf that tracks the number of lines that have been read and stuff like that (so, for example, when I get an error message, I can ask the stream for the line...
6
by: Microsoft News | last post by:
Hi, I have been using several http server code examples from the web, include one from msdn, and I can't seem to get a simple http server thread working. I can connect the server successful...
4
by: Leon Jollans | last post by:
Hi. I'm reading a 5 meg (or so) file from the response stream of a WebRequest, and it's pitifully slow. I mean *really* slow, in the order of 15 minutes to download. Now obviously I'm caching this...
0
by: David Purton | last post by:
Hi, I have some ASP code to stream a binary file. It works well and looks something like the original ASP code shown below. I'm experimenting in creating ZIP files on the fly and am trying to...
7
by: Lee | last post by:
Hi, I'm a stream virgin and am attempting to output strings to a file. My approach is to write the string initially to a 'stringstream' and only when complete write the stringstream to the file...
10
by: m00nm0nkey | last post by:
Hello. I am trying to split a file with 334,386 lines into seperate files of 50,000 each. This is the code i am running: Dim intFragmentRawIndex As Integer Dim swRawDataFile As StreamWriter...
4
by: Scott F. Brown | last post by:
Greetings all... I was playing around with compressing streams and came across a behavior that I do not understand. I create a stream (input) from the contents of a textbox. That stream is...
7
by: iporter | last post by:
I use the code below to authorise the download of certain files. Thus, instead of linking to the file in a wwwroot directory, I link to this code with the filename as a parameter, and the script...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.