Http stream performance

[Leon Jollans]

Hi.
I'm reading a 5 meg (or so) file from the response stream of a WebRequest,
and it's pitifully slow. I mean *really* slow, in the order of 15 minutes to
download. Now obviously I'm caching this on the client (Windows app /
IsolatedStorage), but I can't have it taking this long to download in the
first instance. We've got Gigabit LAN. what's the bottleneck?

I could make the resource available via a different protocol / means if
necessary to overcome the problem.

Has anyone had similar issues or know of a better method of large network
downloads?

Cheers,
Leon

[Jon Skeet]

Leon Jollans <Le**********************@xaman.com> wrote:

I'm reading a 5 meg (or so) file from the response stream of a WebRequest,
and it's pitifully slow. I mean *really* slow, in the order of 15 minutes to
download. Now obviously I'm caching this on the client (Windows app /
IsolatedStorage), but I can't have it taking this long to download in the
first instance. We've got Gigabit LAN. what's the bottleneck?

How are you reading it, exactly? (Please show some code, in other
words.) Have you tried reading it with a plain browser (or wget) to see
whether it's the client or the server that's holding it up?

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Leon Jollans]

It was definitely the client all along. but I think I've sussed it now: I
was reading the stream byte by byte into a pre-initialised array using
ReadByte from the stream directly and checking for -1 each time.

now I'm doing it using a BinaryReader in chunks and the speed is as per
expectations. code below for reference.

void readStream(Stream stream, long contentLength){
int i=0;
byte[] bytes;
BinaryReader reader = new BinaryReader(stream);
MemoryStream ms = new MemoryStream((int)contentLength);
do
{
bytes = reader.ReadBytes(128);
if(bytes.Length > 0)
{
ms.Write(bytes,0,bytes.Length);
}
else{
break;
}
}while( true );
reader.Close();
stream.Close();
ms.Flush();
bytes = ms.ToArray();
ms.Close();

...

cheers

Leon

"Jon Skeet" <sk***@pobox.com> wrote in message
news:MP************************@news.microsoft.com ...

Leon Jollans <Le**********************@xaman.com> wrote:

I'm reading a 5 meg (or so) file from the response stream of a WebRequest, and it's pitifully slow. I mean *really* slow, in the order of 15 minutes to download. Now obviously I'm caching this on the client (Windows app /
IsolatedStorage), but I can't have it taking this long to download in the first instance. We've got Gigabit LAN. what's the bottleneck?

How are you reading it, exactly? (Please show some code, in other
words.) Have you tried reading it with a plain browser (or wget) to see
whether it's the client or the server that's holding it up?

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Jon Skeet]

Leon Jollans <Le**********************@xaman.com> wrote:

It was definitely the client all along. but I think I've sussed it now: I
was reading the stream byte by byte into a pre-initialised array using
ReadByte from the stream directly and checking for -1 each time.
Yes, that would definitely slow it down :)
now I'm doing it using a BinaryReader in chunks and the speed is as per
expectations. code below for reference.

void readStream(Stream stream, long contentLength){
int i=0;
byte[] bytes;
BinaryReader reader = new BinaryReader(stream);
MemoryStream ms = new MemoryStream((int)contentLength);
do
{
bytes = reader.ReadBytes(128);
if(bytes.Length > 0)
{
ms.Write(bytes,0,bytes.Length);
}
else{
break;
}
}while( true );
reader.Close();
stream.Close();
ms.Flush();
bytes = ms.ToArray();
ms.Close();

Rather than doing that, I'd do something like:
(assuming you can make contentLength an int instead of a long)
byte[] data = new byte[contentLength];

int offset=0;
int remaining = contentLength;

while (remaining > 0)
{
int read = stream.Read (data, offset, remaining);
if (read==-1)
// Up to you - throw an exception or something similar
// because we didn't get as much as we expected
offset+=read;
remaining-=read;
}

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Leon Jollans]

That's much faster! thanks :)

Leon

"Jon Skeet" <sk***@pobox.com> wrote in message
news:MP************************@news.microsoft.com ...

Leon Jollans <Le**********************@xaman.com> wrote:

It was definitely the client all along. but I think I've sussed it now: I was reading the stream byte by byte into a pre-initialised array using
ReadByte from the stream directly and checking for -1 each time.

Yes, that would definitely slow it down :)

now I'm doing it using a BinaryReader in chunks and the speed is as per
expectations. code below for reference.

void readStream(Stream stream, long contentLength){
int i=0;
byte[] bytes;
BinaryReader reader = new BinaryReader(stream);
MemoryStream ms = new MemoryStream((int)contentLength);
do
{
bytes = reader.ReadBytes(128);
if(bytes.Length > 0)
{
ms.Write(bytes,0,bytes.Length);
}
else{
break;
}
}while( true );
reader.Close();
stream.Close();
ms.Flush();
bytes = ms.ToArray();
ms.Close();

Rather than doing that, I'd do something like:
(assuming you can make contentLength an int instead of a long)
byte[] data = new byte[contentLength];

int offset=0;
int remaining = contentLength;

while (remaining > 0)
{
int read = stream.Read (data, offset, remaining);
if (read==-1)
// Up to you - throw an exception or something similar
// because we didn't get as much as we expected
offset+=read;
remaining-=read;
}

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too