Fetching URLs in Parallel

[Arsen V.]

Hello,

How does an application like the Web Application Stress tool work? How does
it issue requests so fast without being blocked by slow executing ones?

What technique can I use to create an object that would fetch up to 10 URLs
in parallel and be able to timeout and terminate all the requests within 1.5
seconds?

This object would be used on the server side and would be called many times
per second.

Is there a component like this for sale?

Thanks,
Arsen

[Cowboy \(Gregory A. Beamer\)]

Spawn new threads based on the number of virtual users. That way no thread
blocks another. On a single machine, you can still experience some slow down
compared to multiple machines, but you are still likely to spawn more
requests than real users. Hope this helps.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

*************************************************
Think outside the box!
*************************************************
"Arsen V." <ar***@community.nospam> wrote in message
news:OB**************@TK2MSFTNGP14.phx.gbl...

Hello,

How does an application like the Web Application Stress tool work? How
does
it issue requests so fast without being blocked by slow executing ones?

What technique can I use to create an object that would fetch up to 10
URLs
in parallel and be able to timeout and terminate all the requests within
1.5
seconds?

This object would be used on the server side and would be called many
times
per second.

Is there a component like this for sale?

Thanks,
Arsen

[Arsen V.]

Hi,

But how to instantenuously kill threads that are blocking after the 1.5
timeout?

Thanks,
Arsen

[Angelos Karantzalis]

Have another thread kill them, after the 1.5 timeout :]

Angel
O:]
"Arsen V." <ar***@community.nospam> wrote in message
news:#m**************@TK2MSFTNGP10.phx.gbl...

Hi,

But how to instantenuously kill threads that are blocking after the 1.5
timeout?

Thanks,
Arsen

[Arsen V.]

Hi Angelos,

What is the sure way to "kill" a thread even if it is currently blocking?

I know that Abort() will simply tell the thread to start terminating, but it
may not happen instantly.

Thanks,
Arsen

[Scott Allen]

If you are using the WebRequest class to fetch the URLs, then the
WebRequest class has a Timeout property.

--
Scott
http://www.OdeToCode.com/blogs/scott/

On Mon, 1 Nov 2004 21:57:11 -0600, "Arsen V." <ar***@community.nospam>
wrote:

Hi,

But how to instantenuously kill threads that are blocking after the 1.5
timeout?

Thanks,
Arsen

[Arsen V.]

Hi Scott,

The WebRequest's timeout property is not "instantenuous". In other words,
even if I specify timeout of 1500 milliseconds, the requests can take a long
time - 5 seconds or longer. I think this has to do with gracefully
terminating everything related to the thread. I just want to drop it if it's
taking too long.

Thanks,
Arsen
"Scott Allen" <bitmask@[nospam].fred.net> wrote in message
news:qn********************************@4ax.com...

If you are using the WebRequest class to fetch the URLs, then the
WebRequest class has a Timeout property.

--
Scott
http://www.OdeToCode.com/blogs/scott/

On Mon, 1 Nov 2004 21:57:11 -0600, "Arsen V." <ar***@community.nospam>
wrote:

Hi,

But how to instantenuously kill threads that are blocking after the 1.5
timeout?

Thanks,
Arsen

[Justin Rogers]

We had this problem in the .NET Terrarium. If you are lullygagging in
unmanaged code somewhere, then the thread won't abort. The only
sure fire way to make this work is to kill your thread using Win32 APIs.
I don't recommend this approach unless you really need, as in our case
we did. Instead think of models where you terminate the thread and set
some state related to the thread and let it go. It'll get cleaned up later.
If it returns after 5 seconds, then who cares, it won't affect your
application when it finally does die or completes.
--
Justin Rogers
DigiTec Web Consultants, LLC.
Blog: http://weblogs.asp.net/justin_rogers

"Arsen V." <ar***@community.nospam> wrote in message
news:uk**************@TK2MSFTNGP14.phx.gbl...

Hi Scott,

The WebRequest's timeout property is not "instantenuous". In other words,
even if I specify timeout of 1500 milliseconds, the requests can take a long
time - 5 seconds or longer. I think this has to do with gracefully
terminating everything related to the thread. I just want to drop it if it's
taking too long.

Thanks,
Arsen
"Scott Allen" <bitmask@[nospam].fred.net> wrote in message
news:qn********************************@4ax.com...

If you are using the WebRequest class to fetch the URLs, then the
WebRequest class has a Timeout property.

--
Scott
http://www.OdeToCode.com/blogs/scott/

On Mon, 1 Nov 2004 21:57:11 -0600, "Arsen V." <ar***@community.nospam>
wrote:

>Hi,
>
>But how to instantenuously kill threads that are blocking after the 1.5
>timeout?
>
>Thanks,
>Arsen
>

[Arsen V.]

Hi Justin,

The problem is that this component would need to be called from an ASP.NET
application getting dozens of hits per second. It would be pretty bad to
have all these "orphaned" threads executing.

Is there some .NET component that allows to fetch URLs in parallel and
control the timeout in a very precise way?

If it would be just a Windows Forms apps, I would not worry that much about
letting threads run and terminate. However, since this is going to be high
volume component I think it would cause performance problems.

What do you think?

Thanks,
Arsen

[Scott Allen]

Hi Arsen:

There is no built in class to do this. Sometimes you just have to put
together a prototype and stress test it to see what works best in your
environment.

--
Scott
http://www.OdeToCode.com/blogs/scott/

On Wed, 3 Nov 2004 08:38:18 -0600, "Arsen V." <ar***@community.nospam>
wrote:

Hi Justin,

The problem is that this component would need to be called from an ASP.NET
application getting dozens of hits per second. It would be pretty bad to
have all these "orphaned" threads executing.

Is there some .NET component that allows to fetch URLs in parallel and
control the timeout in a very precise way?

If it would be just a Windows Forms apps, I would not worry that much about
letting threads run and terminate. However, since this is going to be high
volume component I think it would cause performance problems.

What do you think?

Thanks,
Arsen

[Arsen V.]

I meant a thrid party component.

[Steven Cheng[MSFT]]

Hi Arsen,

I've noticed another duplicated thread in the donet.framework newsgroup,
I've posted my reply there and you may have a look there. Also, if you feel
it convenient that we contintue to discuss the problem in that thread,
please feel free to post there. Thanks.

Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

[Arsen V.]

Hi Steven,

The application must be a web ASP.NET app. It will be kind of a META-SEARCH.
It will connect to multiple sources in parallel and fetch results from each
source.

It cannot be a desktop app.

I was experimenting with using async socket communication and
ManualResetEvent with WaitAll(timeout) to guarantee a very precise timeout
control. However, I was hoping that there is a way to do this without using
sockets directly.

I also do not think that it is a good to spawn multiple threads from one
ASP.NET request. So what is the solution? Does ASYNC IO still span multiple
threads?

Thanks,
Arsen

[Adam Clauss]

"Arsen V." <ar***@community.nospam> wrote in message news:%2***************@TK2MSFTNGP12.phx.gbl...

I was experimenting with using async socket communication and
ManualResetEvent with WaitAll(timeout) to guarantee a very precise timeout
control. However, I was hoping that there is a way to do this without using
sockets directly.

Can't you do Async requests and accomplish the same thing with the HttpWebRequest?

I also do not think that it is a good to spawn multiple threads from one
ASP.NET request.

Why not?

--
Adam Clauss
ca*****@tamu.edu

[Steven Cheng[MSFT]]

Hi Arsen,

Thanks for your followup. Yes, of course the ASYNC APIS are based on
multi-thread internally , but the buildin apis have done much performance
optimization for us. Also, as Adam has suggested, you can consider the
WebRequest class's buildin Async request feature instead of manually build
sockets:

#Making Asynchronous Requests
http://msdn.microsoft.com/library/en...ngAsynchronous
Requests.asp?frame=true

In addition, since you're spaning multi-threads in asp.net web app to
stress test other web app, I'd recommed that you increase the
"minFreeThreads" or "minFreeLocalRequestFreeThreads" attributes setting in
the <httpRuntime> of the asp.net web app's web.config

#<httpRuntime> Element
http://msdn.microsoft.com/library/en...pRuntimeSectio
n.asp?frame=true

Also, the "maxWorkerThreads" and "maxIoThreads" in the <procesModel> ( in
machine.config for IIS5 or via Application Pool Setting in IIS6) should be
increased accordingly.

#<processModel> Element
http://msdn.microsoft.com/library/en...cessmodelsecti
on.asp?frame=true

Hope helps. Thanks.

Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)