Memory Leak with Web-Requests

[Andreas Lennartz]

Hello,

I have some problems with the class WebRequest and its childs. I am
developing a bot that visits a lot of different urls and returns the
content. But every time an URL-Request is finished there is a memory leak.
Here is a small example that reproduces this problem:

I read of a File with many different(!) URIs. The next URI to visit is saved
in the variable "uri".Then I try to connect to this uri. I do it with the
following line of codes:
// HttpWebRequest Request = null; //is declared at the Beginning of the
class
// HttpWebResponse Response = null; //is declared at the Beginning of the
class
try
{
Request = (HttpWebRequest )WebRequest.Cre ate(uri);
Request.Timeout = 1000;
Response = (HttpWebRespons e)Request.GetRe sponse();
}
catch (Exception e) {;}
finally
{
if (Response != null) Response.Close( );
if (Request != null) Request.Abort() ;
}

The connection is established without any problems. But if i do this
code-block in a
Not-Ending Loop, the memory requirement is climbing constantly.

Can anybody say me how to stop this memory leak?

(Sorry for my english, it's not my native language)

[Nicholas Paldino [.NET/C# MVP]]

Andreas,

This is not a memory leak. The CLR will allocate more memory as needed.
The CLR will continue to grab up memory until it feels that it would
constrain the system by doing so.

Hope this helps.

--
- Nicholas Paldino [.NET/C# MVP]
- ni************* *@exisconsultin g.com

"Andreas Lennartz" <al***@gmx.ne t> wrote in message
news:OP******** ******@TK2MSFTN GP09.phx.gbl...

Hello,

I have some problems with the class WebRequest and its childs. I am
developing a bot that visits a lot of different urls and returns the
content. But every time an URL-Request is finished there is a memory leak.
Here is a small example that reproduces this problem:

I read of a File with many different(!) URIs. The next URI to visit is saved in the variable "uri".Then I try to connect to this uri. I do it with the
following line of codes:
// HttpWebRequest Request = null; //is declared at the Beginning of the
class
// HttpWebResponse Response = null; //is declared at the Beginning of the
class
try
{
Request = (HttpWebRequest )WebRequest.Cre ate(uri);
Request.Timeout = 1000;
Response = (HttpWebRespons e)Request.GetRe sponse();
}
catch (Exception e) {;}
finally
{
if (Response != null) Response.Close( );
if (Request != null) Request.Abort() ;
}

The connection is established without any problems. But if i do this
code-block in a
Not-Ending Loop, the memory requirement is climbing constantly.

Can anybody say me how to stop this memory leak?

(Sorry for my english, it's not my native language)

[Andreas Lennartz]

This is not a memory leak. The CLR will allocate more memory as needed. The CLR will continue to grab up memory until it feels that it would
constrain the system by doing so.

Hallo,

first thank you for your answer. Well, that the CLR grabs as much memory as
it feels to need would explain the behaviour of my programm. The bot I'm
developing visits about 20 pages every turn, and a turn lasts about 6 sec.
So, there are many different pages which are visited in a (relative) short
time. I tried serveral methods of spidering the pages (with threads and
asynchronous I/O), with several instances of the bot or only one instance.
Unfortunately, the memory requirement of the bot is always above 100MB. Till
now I always thought that would be a memory leak. After some more test runs
I recognized that it level off around 120MB. But that seems to me very much
to fulfil such an simple task.
Running this on a webserver would be risky.
Are there any possibilities to reduce the memory my application gets?