Frustrating http connection behaviour in .Net (C#)

[Rob Levine]

(This post also available at
http://roblevine.blogspot.com/2004/1...behaviour.html
in a slightly more readable format!)
Hi All,

I seem to be having a bit of a frustrating time with http connections in
..Net at the moment.
Basically I am working on a project in which many subsystems use http to
communicate with eachother. Once in a while, my .Net client will fail with a
"System.Net.Web Exception: The underlying connection was closed: Unable to
connect to the remote server." exception.
The remote server can be an IIS server or a Jetty server running on Linux -
I see the same problem with both.

To try and reproduce this problem I wrote a very simple client app that
requests a single static page from a web server. It loops round making the
request repeatedly, up to 100000 times.
Here is the static Main method from my console application:
[STAThread]
static void Main(string[] args)
{
int i = 0;
try
{
for (i = 0; i < 10000; i++)
{

HttpWebRequest req = (HttpWebRequest )
WebRequest.Crea te("http://proxima/test.html");
using (HttpWebRespons e resp =
(HttpWebRespons e)req.GetRespon se())
{
byte[] buffer = new byte[1024*1024];
int read = 0;
Stream stream = resp.GetRespons eStream();
do
{
read = stream.Read(buf fer, 0, 1024*1024);
}
while (read !=0);
stream.Close();
resp.Close();
req = null;
}

if (i % 1000 == 0)
Console.WriteLi ne(i);
}
Console.WriteLi ne(i);
}
catch (Exception ex)
{
Console.WriteLi ne("Exception at iteration " + i);
Console.WriteLi ne( ex.ToString());
}

}

For this example I created a very small html file to put on my server - only
50 bytes.
I see some extremely peculiar behaviour when I run this application;
generally the first time it runs, it will process many thousands of
iterations before falling over with the aforementioned exception. Once it
has failed, if I attempt to run it again straight away the application fails
in a matter of a few 10's or 100's of iterations. Why?
It makes no difference to the overall behaviour if I swap my HttpWebRequest
stuff for a System.Net.WebC lient.
It makes no difference if I insert a GC.Collect() in the loop.
What is going on here and what is 'remembering' the fact that the client has
previously failed?

I have to admit to being stumped for the time being.
Does anyone else see this behaviour?

TIA for any help,

Rob Levine

Location: Bristol, UK
Email: rob <underscore> levine <at> hotmail <dot> com
Blog: http://roblevine.blogspot.com