Multithreading webresponse

[RS]

I have written a utility which currently downloads xml data from a high
number (40+) of web based sources in a synchronous manner.

When attempting to spawn threads off to speed things up I run into
major problems due to blocking and number of threads available in the
thread pool.

Despite trying various methods (invoking delegates, counting threads in
order to try manage the threadpool, etc) none of these have worked as
hoped.

Basically all my code is doing is calling the following function:

internal myclass DeserializeMyClass(string url)
{
XmlSerializer serializer = new XmlSerializer(typeof(myclass));
myclass t = new myclass();

try
{
WebRequest wReq = WebRequest.Create(url);
WebResponse wRes = wReq.GetResponse(); // this is the culprit!!!
StreamReader strm = new StreamReader(wRes.GetResponseStream());
XmlTextReader xmlReader = new XmlTextReader(strm);
t = (myclass) serializer.Deserialize(xmlReader);
}
catch
{
// todo: log error
}

return t;
}

Can anyone point to how I could successfully do this in a
multi-threaded manner?

[S. Senthil Kumar]

Are you sure your bandwidth is not the bottleneck? You can spawn as
many threads as you like, but if bandwidth is the limiting factor, then
those threads will actually slow things down.

How did you attempt to spawn new threads? If the system threadpool
might not be ideal for your needs, you can try a custom one.

Regards
Senthil

[RS]

S. Senthil Kumar wrote:

Are you sure your bandwidth is not the bottleneck? You can spawn as
many threads as you like, but if bandwidth is the limiting factor, then
those threads will actually slow things down.

How did you attempt to spawn new threads? If the system threadpool
might not be ideal for your needs, you can try a custom one.

Regards
Senthil

The actual error messages (when I received them) related that there
where not enough threads in the threadpool.

I tried creating a delegate for the function and calling via
begininvoke, as well as various other methods.

In most cases the webresponse line seemed to be the problem, as the
debugger only stepped here before the next thread spawned. None of the
functions then responded with a valid class as the threads simply timed
out.

[Jon Skeet [C# MVP]]

RS <rs********@delphi-dolphin.com> wrote:

The actual error messages (when I received them) related that there
where not enough threads in the threadpool.

In that case (and anyway, frankly) I'd avoid using the system
threadpool - use either new threads, or (preferrably, IMO) a custom
threadpool. That way you don't end up with hard-to-debug problems due
to other methods using the system threadpool unexpectedly, and you have
much more control over how many threads it can use etc.

You may, however, want to consider using asynchronous IO instead. This
would be more complicated but I believe it would avoid creating quite
as many new threads. (I'm not entirely sure of the implementation.)

If you want a custom threadpool, there are plenty of free ones around.
The one I've written is at
http://www.pobox.com/~skeet/csharp/miscutil

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Yosh]

RS...

At what point are you seeing the blocking? After 2,3,4 threads have been
spawned?

It sounds like an http ConnectionLimit problem meaning you can only be
downloading 2 web requests/responses at a given time. However, this is
configurable via the ServicePoint class using the ConnectionLimit property.
Try setting this property to the number of threads you are going to spawn
and see if this fixes.

Yosh
"RS" <rs********@delphi-dolphin.com> wrote in message
news:11**********************@g47g2000cwa.googlegr oups.com...

I have written a utility which currently downloads xml data from a high
number (40+) of web based sources in a synchronous manner.

When attempting to spawn threads off to speed things up I run into
major problems due to blocking and number of threads available in the
thread pool.

Despite trying various methods (invoking delegates, counting threads in
order to try manage the threadpool, etc) none of these have worked as
hoped.

Basically all my code is doing is calling the following function:

internal myclass DeserializeMyClass(string url)
{
XmlSerializer serializer = new XmlSerializer(typeof(myclass));
myclass t = new myclass();

try
{
WebRequest wReq = WebRequest.Create(url);
WebResponse wRes = wReq.GetResponse(); // this is the culprit!!!
StreamReader strm = new StreamReader(wRes.GetResponseStream());
XmlTextReader xmlReader = new XmlTextReader(strm);
t = (myclass) serializer.Deserialize(xmlReader);
}
catch
{
// todo: log error
}

return t;
}

Can anyone point to how I could successfully do this in a
multi-threaded manner?

[Joerg Jooss]

RS wrote:

I have written a utility which currently downloads xml data from a
high number (40+) of web based sources in a synchronous manner.

When attempting to spawn threads off to speed things up I run into
major problems due to blocking and number of threads available in the
thread pool.

Despite trying various methods (invoking delegates, counting threads
in order to try manage the threadpool, etc) none of these have worked
as hoped.

Basically all my code is doing is calling the following function:

internal myclass DeserializeMyClass(string url)
{
XmlSerializer serializer = new XmlSerializer(typeof(myclass));
myclass t = new myclass();

try
{
WebRequest wReq = WebRequest.Create(url);
WebResponse wRes = wReq.GetResponse(); // this is the culprit!!!
StreamReader strm = new StreamReader(wRes.GetResponseStream());
XmlTextReader xmlReader = new XmlTextReader(strm);
t = (myclass) serializer.Deserialize(xmlReader);
}
catch
{
// todo: log error
}

return t;
}

Can anyone point to how I could successfully do this in a
multi-threaded manner?

I'd start by fixing the rather obvious resource leaks here: You never
close any request or response object.

Also note that the number of simultaneous TCP connections to a single
host is limited by ServicePoint.ConnectionLimit.

Cheers.
--
http://www.joergjooss.de
mailto:ne********@joergjooss.de

[Cor Ligthert [MVP]]

RS

In addition to Joerg.

Http 1.1

http://support.microsoft.com/default...b;en-us;183110

I hope this helps,

Cor