HttpWebRequest and Multi Threaded Apps

[Mark Fletcher]

Hi,

Im in the process of writing a program that crawls a website. Im using the
HttpWebRequest and HttpWebResponse classed to get content. To make my
application more scalable, my application is multithreaded, with each thread
making a different request.

One problem Ive ran into is that when writing the response stream to the
filesystem, from the HttpWebResponse.GetResponseStream() method, if the
response is of a sufficient size, it blocks all other other executing
threads. If this exceeds the timeout, then all the other threads that are in
the process of making a Web Request time out.

Ive taken every effort to ensure that my application is not locking, I was
wondering if given the scenario that Ive listed, would using Asynchronous Web
Requests be more efficient? Ive not worked with them before so Id be grateful
of your opinions.

Cheers,

Mark

[hB]

you are using different instances of httpwebrequest-response objects in
different threads?
i mean resources should be independent of eachother in all threads.

---
hB

[Mark Fletcher]

Hi,

Im creating new instances of HttpWebRequest and HttpWebResponse in each
thread. The threads are not sharing requests and responses. Im also not using
the ThreadPool, so Im avoiding issues where Im running out of threads.
Instead Im using Stephen Toubs ManagedThreadPool.

Cheers,

Mark
"hB" wrote:

you are using different instances of httpwebrequest-response objects in
different threads?
i mean resources should be independent of eachother in all threads.

---
hB

[hB]

Request and response objects are used together.

For example:
HttpWebRequest HttpWReq =
(HttpWebRequest)WebRequest.Create("http://www.contoso.com");
HttpWebResponse HttpWResp = (HttpWebResponse)HttpWReq.GetResponse();
// Insert code that uses the response object.
HttpWResp.Close();

You get Response Object from the request one.

So for one thread doing HTTPWebRequest Object do HTTPWebResponse work
in same thread.
For multithreading do separate Requests.
Break Threads like
http://example.com/sec1 (req-resp)
http://example.com/sec2 (req-resp)
....

---
hB

[hB]

Request and response objects are used together.

For example:
HttpWebRequest HttpWReq =
(HttpWebRequest)WebRequest.Create("http://www.contoso.com");
HttpWebResponse HttpWResp = (HttpWebResponse)HttpWReq.GetResponse();
// Insert code that uses the response object.
HttpWResp.Close();

You get Response Object from the request one.

So for one thread doing HTTPWebRequest Object do HTTPWebResponse work
in same thread.
For multithreading do separate Requests.
Break Threads like
http://example.com/sec1 (req-resp)
http://example.com/sec2 (req-resp)
....

---
hB

[Mark Fletcher]

Hi,

Thanks for the response. Yep thats what Im currently implenting. Each active
thread makes one Http Request and gets the response.

I do find that I have occasional problems, when reading the response stream.
I find that when calling Stream.Read() this sometimes times out with the
following exception:

Message: The operation has timed out
Stack Trace: at System.ConnectStream.Read(...)

Now it doesnt matter if I set HttpRequest.Timeout = 10000 or 100000, I still
occasionally get these timeouts. When I debug the application its like the
thread is in the process of reading bytes part of the way through and then
hangs.

I can provide code if requested, but I was wondering if anyone else has seen
symptoms like this, and if they could suggest a remedy.

Thanks,

Mark


"hB" wrote:

Request and response objects are used together.

For example:
HttpWebRequest HttpWReq =
(HttpWebRequest)WebRequest.Create("http://www.contoso.com");
HttpWebResponse HttpWResp = (HttpWebResponse)HttpWReq.GetResponse();
// Insert code that uses the response object.
HttpWResp.Close();

You get Response Object from the request one.

So for one thread doing HTTPWebRequest Object do HTTPWebResponse work
in same thread.
For multithreading do separate Requests.
Break Threads like
http://example.com/sec1 (req-resp)
http://example.com/sec2 (req-resp)
....

---
hB