Very Fast Multithreaded URL Fetching

[Arsen V.]

Hello,

Does anyone know if an object or how to create one, that will allow me to
fetch up to 10 URLs (containing XML-feed data) in an extremelly fast server
side fashion?

If the request is taking longer than it should, the object would need to be
able to timeout on the spot (without waiting). The timeout value would need
to be acurate to about 50 milliseconds. Usually, I would want the timeout to
be around 1.5 seconds.

Thanks in advance,
Arsen

[joseph.inglis]

Arsen,

Suppose the first question is why does it need to be multithreaded?

Running it in a thread possibly fair enough but a simple loop should
suffice, no point complicating something that doesnt require it ;-)

Look at WebRequest class. You can set the timeout as a property. Feed the
WebRequest.Create(URI) from values in a loop from an array of 10 that holds
your urls.

Tam

"Arsen V." <ar****************@emergency24.com> wrote in message
news:eU**************@TK2MSFTNGP12.phx.gbl...

Hello,

Does anyone know if an object or how to create one, that will allow me to
fetch up to 10 URLs (containing XML-feed data) in an extremelly fast server side fashion?

If the request is taking longer than it should, the object would need to be able to timeout on the spot (without waiting). The timeout value would need to be acurate to about 50 milliseconds. Usually, I would want the timeout to be around 1.5 seconds.

Thanks in advance,
Arsen

[joseph.inglis]

Arsen,

Suppose the first question is why does it need to be multithreaded?

Running it in a thread possibly fair enough but a simple loop should
suffice, no point complicating something that doesnt require it ;-)

Look at WebRequest class. You can set the timeout as a property. Feed the
WebRequest.Create(URI) from values in a loop from an array of 10 that holds
your urls.

Tam

"Arsen V." <ar****************@emergency24.com> wrote in message
news:eU**************@TK2MSFTNGP12.phx.gbl...

Hello,

Does anyone know if an object or how to create one, that will allow me to
fetch up to 10 URLs (containing XML-feed data) in an extremelly fast server side fashion?

If the request is taking longer than it should, the object would need to be able to timeout on the spot (without waiting). The timeout value would need to be acurate to about 50 milliseconds. Usually, I would want the timeout to be around 1.5 seconds.

Thanks in advance,
Arsen

[joseph.inglis]

sorry, and i should have added...

Running 10 consecutive webrequests is considered rude by standard internet
protocol. 2 is the standard which is why XP is set to only allow two
downloads at any one time.

so a single threaded loop to go through an array of urls is your best
solution ;)

Tam
"Arsen V." <ar****************@emergency24.com> wrote in message
news:eU**************@TK2MSFTNGP12.phx.gbl...

Hello,

Does anyone know if an object or how to create one, that will allow me to
fetch up to 10 URLs (containing XML-feed data) in an extremelly fast server side fashion?

If the request is taking longer than it should, the object would need to be able to timeout on the spot (without waiting). The timeout value would need to be acurate to about 50 milliseconds. Usually, I would want the timeout to be around 1.5 seconds.

Thanks in advance,
Arsen

[joseph.inglis]

sorry, and i should have added...

Running 10 consecutive webrequests is considered rude by standard internet
protocol. 2 is the standard which is why XP is set to only allow two
downloads at any one time.

so a single threaded loop to go through an array of urls is your best
solution ;)

Tam
"Arsen V." <ar****************@emergency24.com> wrote in message
news:eU**************@TK2MSFTNGP12.phx.gbl...

Hello,

Does anyone know if an object or how to create one, that will allow me to
fetch up to 10 URLs (containing XML-feed data) in an extremelly fast server side fashion?

If the request is taking longer than it should, the object would need to be able to timeout on the spot (without waiting). The timeout value would need to be acurate to about 50 milliseconds. Usually, I would want the timeout to be around 1.5 seconds.

Thanks in advance,
Arsen

[joseph.inglis]

hey, where did my oringal reply go?... ok, here it is again.

Arsen,

Suppose the first question is why does it need to be multithreaded?

Running it in a thread possibly fair enough but a simple loop should
suffice, no point complicating something that doesnt require it ;-)

Look at WebRequest class. You can set the timeout as a property. Feed the
WebRequest.Create(URI) from values in a loop from an array of 10 that holds
your urls.

Tam

"joseph.inglis" <jo***********@btclick.com> wrote in message
news:bv**********@hercules.btinternet.com...

sorry, and i should have added...

Running 10 consecutive webrequests is considered rude by standard internet
protocol. 2 is the standard which is why XP is set to only allow two
downloads at any one time.

so a single threaded loop to go through an array of urls is your best
solution ;)

Tam
"Arsen V." <ar****************@emergency24.com> wrote in message
news:eU**************@TK2MSFTNGP12.phx.gbl...

Hello,

Does anyone know if an object or how to create one, that will allow me to fetch up to 10 URLs (containing XML-feed data) in an extremelly fast server

side fashion?

If the request is taking longer than it should, the object would need to

be

able to timeout on the spot (without waiting). The timeout value would

need

to be acurate to about 50 milliseconds. Usually, I would want the

timeout to

be around 1.5 seconds.

Thanks in advance,
Arsen

[joseph.inglis]

hey, where did my oringal reply go?... ok, here it is again.

Arsen,

Suppose the first question is why does it need to be multithreaded?

Running it in a thread possibly fair enough but a simple loop should
suffice, no point complicating something that doesnt require it ;-)

Look at WebRequest class. You can set the timeout as a property. Feed the
WebRequest.Create(URI) from values in a loop from an array of 10 that holds
your urls.

Tam

"joseph.inglis" <jo***********@btclick.com> wrote in message
news:bv**********@hercules.btinternet.com...

sorry, and i should have added...

Running 10 consecutive webrequests is considered rude by standard internet
protocol. 2 is the standard which is why XP is set to only allow two
downloads at any one time.

so a single threaded loop to go through an array of urls is your best
solution ;)

Tam
"Arsen V." <ar****************@emergency24.com> wrote in message
news:eU**************@TK2MSFTNGP12.phx.gbl...

Hello,

Does anyone know if an object or how to create one, that will allow me to fetch up to 10 URLs (containing XML-feed data) in an extremelly fast server

side fashion?

If the request is taking longer than it should, the object would need to

be

able to timeout on the spot (without waiting). The timeout value would

need

to be acurate to about 50 milliseconds. Usually, I would want the

timeout to

be around 1.5 seconds.

Thanks in advance,
Arsen

[Eric Gunnerson [MS]]

There are a couple of ways of doing this. I think the simplest is to use the
HttpWebRequest object. In your loop you would do:

foreach (string url in urls)
{
HttpWebRequest request = new HttpWebRequest(...);
request.BeginGetResponse(...);
}

that will queue up all the requests. In the call to BeginGetResponse(), you
can specify a callback that will be executed when the requests finish. In
that function you'll get EndGetResponse() which will get you a WebResponse
object with the data you want.

--
Eric Gunnerson

Visit the C# product team at http://www.csharp.net
Eric's blog is at http://weblogs.asp.net/ericgu/

This posting is provided "AS IS" with no warranties, and confers no rights.
"Arsen V." <ar****************@emergency24.com> wrote in message
news:eU**************@TK2MSFTNGP12.phx.gbl...

Hello,

Does anyone know if an object or how to create one, that will allow me to
fetch up to 10 URLs (containing XML-feed data) in an extremelly fast server side fashion?

If the request is taking longer than it should, the object would need to be able to timeout on the spot (without waiting). The timeout value would need to be acurate to about 50 milliseconds. Usually, I would want the timeout to be around 1.5 seconds.

Thanks in advance,
Arsen

[Eric Gunnerson [MS]]

There are a couple of ways of doing this. I think the simplest is to use the
HttpWebRequest object. In your loop you would do:

foreach (string url in urls)
{
HttpWebRequest request = new HttpWebRequest(...);
request.BeginGetResponse(...);
}

that will queue up all the requests. In the call to BeginGetResponse(), you
can specify a callback that will be executed when the requests finish. In
that function you'll get EndGetResponse() which will get you a WebResponse
object with the data you want.

--
Eric Gunnerson

Visit the C# product team at http://www.csharp.net
Eric's blog is at http://weblogs.asp.net/ericgu/

This posting is provided "AS IS" with no warranties, and confers no rights.
"Arsen V." <ar****************@emergency24.com> wrote in message
news:eU**************@TK2MSFTNGP12.phx.gbl...

Hello,

Does anyone know if an object or how to create one, that will allow me to
fetch up to 10 URLs (containing XML-feed data) in an extremelly fast server side fashion?

If the request is taking longer than it should, the object would need to be able to timeout on the spot (without waiting). The timeout value would need to be acurate to about 50 milliseconds. Usually, I would want the timeout to be around 1.5 seconds.

Thanks in advance,
Arsen

[Arsen V.]

Hi Eric,

Thanks for the suggestion.

Would I be able to achive performance of the Web Application Stress tool
using the Async HttpWebRequest or do I have to use sockets in multiple
threads?

Also, in your example how do I implement the timeout (needs to be able to
timeout in 500ms - 2000ms).

Also, how do I wait for either the timeout or all of the Async requests
completing?

Thanks,
Arsen

"Eric Gunnerson [MS]" <er****@online.microsoft.com> wrote in message
news:OI**************@tk2msftngp13.phx.gbl...

There are a couple of ways of doing this. I think the simplest is to use the HttpWebRequest object. In your loop you would do:

foreach (string url in urls)
{
HttpWebRequest request = new HttpWebRequest(...);
request.BeginGetResponse(...);
}

that will queue up all the requests. In the call to BeginGetResponse(), you can specify a callback that will be executed when the requests finish. In
that function you'll get EndGetResponse() which will get you a WebResponse
object with the data you want.

--
Eric Gunnerson

Visit the C# product team at http://www.csharp.net
Eric's blog is at http://weblogs.asp.net/ericgu/

This posting is provided "AS IS" with no warranties, and confers no rights. "Arsen V." <ar****************@emergency24.com> wrote in message
news:eU**************@TK2MSFTNGP12.phx.gbl...

Hello,

Does anyone know if an object or how to create one, that will allow me to fetch up to 10 URLs (containing XML-feed data) in an extremelly fast server

side fashion?

If the request is taking longer than it should, the object would need to

be

able to timeout on the spot (without waiting). The timeout value would

need

to be acurate to about 50 milliseconds. Usually, I would want the

timeout to

be around 1.5 seconds.

Thanks in advance,
Arsen

[Chad Z. Hower aka Kudzu]

"Arsen V." <ar****************@emergency24.com> wrote in news:
#b**************@TK2MSFTNGP10.phx.gbl:

Also, in your example how do I implement the timeout (needs to be able to
timeout in 500ms - 2000ms).

Multiple threads is much easier, and use blocking sockets. Sure async is
faster - but unless you have a realy dog of a machine even blocking sockets
can easily saturate a 100 MB ethernet.
--
Chad Z. Hower (a.k.a. Kudzu) - http://www.hower.org/Kudzu/
"Programming is an art form that fights back"
ELKNews - Get your free copy at http://www.atozedsoftware.com