By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
435,315 Members | 2,150 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 435,315 IT Pros & Developers. It's quick & easy.

ASP Thread Scheduling Anomalies

P: n/a
Hi everyone

We're experiencing some serious anomalies with the scheduling of ASP
threads. I'd be interested to hear if anyone knows what algorithm is used
(e.g. simple round-robin, or something more sophisticated), and what
situations might perturb it. Even a hint as to what would be considered
normal scheduling might help.

The root of our problem is that we observed a normally well-behaved web
application suddenly limit itself to a single ASP thread. The application
consists mainly of a VB6 DLL instantiated via an ASP page. Note that the
application does not store STA objects in Session/Application, and normally
exhibits no thread affinities at all. The previous post asking for advice
about this is at:
http://www.google.ie/gr*************...=lang_en&hl=en.

As that previous post says, we eventually found that the switch of all
incoming requests (irrespective of their Session ID) to a single ASP thread
occurred when one thread just happened to destroy a Collection of MSXML DOM
objects. Hah!, we thought, a bug in MSXML 3.0, probably connected with its
new free-threaded capability (which we don't use). We then created a much
cut-down version of our server application with a view to reporting it to
Microsoft. Unfortunately, the cut-down version didn't show the same results.
Interestingly, though, its ASP thread scheduling was markedly different from
that in our real application.

The problem has only been observed under IIS 6, and so it could well be an
IIS/ASP difference rather than the fault of MSXML. Running the same simple,
single-client (with an auto-refreshed HTML page at 1 sec intervals) test
against our 2 servers (the real one and the cut-down one), we observe the
following strange thread behaviour. Any explanations here would be very
welcome:

IIS 6 (configured with 20 ASP threads):
Real server:
All 20 threads used in perfect round-robin fashion, until one
destroys
a Collection of MSXML DOM objects. After that, all incoming
requests
are handed to this one thread (sometimes to a pair of threads if
they
happen to have cleared their Collections down at the same time)
Cut-down server:
One thread handles the initial request. All subsequent requests
oscillate
evenly between just 2 ASP threads

IIS 5 (configured with 20 ASP threads):
Real server:
All 20 threads are used. However, the first 8 are scheduled
sequentially.
After that it gets more and more random which ASP thread is
used. No
affinity every arises though.
Cut-down server:
Only 8 ASP threads are ever used, and these are scheduled in
perfect
round-robin fashion

I can't explain these observations as the 2 servers are handling the same
requests. There seems to be more involved than simple round-robin
scheduling. Hence, I though if I understood the algorithm better, I might
have a clue as to the initial problem we encountered (which is now being
considered serious)

Tony Proctor
Jul 22 '05 #1
Share this Question
Share on Google+
6 Replies


P: n/a
Tony,

This is from the IIS help files...maybe you have server-side debugging
turned ON?

Debugging Flags

Enable ASP Server-Side Script Debugging

Select this check box to enable the Web server to enter the Microsoft Script
Debugger while processing ASP pages. You can then use the debugger to
examine your scripts.

Enabling ASP server-side debugging causes ASP to run in single-threaded
mode. For performance reasons, ASP debugging is not recommended in a
production environment.

Brian

"Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in message
news:u%****************@TK2MSFTNGP09.phx.gbl...
Hi everyone

We're experiencing some serious anomalies with the scheduling of ASP
threads. I'd be interested to hear if anyone knows what algorithm is used
(e.g. simple round-robin, or something more sophisticated), and what
situations might perturb it. Even a hint as to what would be considered
normal scheduling might help.

The root of our problem is that we observed a normally well-behaved web
application suddenly limit itself to a single ASP thread. The application
consists mainly of a VB6 DLL instantiated via an ASP page. Note that the
application does not store STA objects in Session/Application, and
normally
exhibits no thread affinities at all. The previous post asking for advice
about this is at:
http://www.google.ie/gr*************...=lang_en&hl=en.

As that previous post says, we eventually found that the switch of all
incoming requests (irrespective of their Session ID) to a single ASP
thread
occurred when one thread just happened to destroy a Collection of MSXML
DOM
objects. Hah!, we thought, a bug in MSXML 3.0, probably connected with its
new free-threaded capability (which we don't use). We then created a much
cut-down version of our server application with a view to reporting it to
Microsoft. Unfortunately, the cut-down version didn't show the same
results.
Interestingly, though, its ASP thread scheduling was markedly different
from
that in our real application.

The problem has only been observed under IIS 6, and so it could well be an
IIS/ASP difference rather than the fault of MSXML. Running the same
simple,
single-client (with an auto-refreshed HTML page at 1 sec intervals) test
against our 2 servers (the real one and the cut-down one), we observe the
following strange thread behaviour. Any explanations here would be very
welcome:

IIS 6 (configured with 20 ASP threads):
Real server:
All 20 threads used in perfect round-robin fashion, until one
destroys
a Collection of MSXML DOM objects. After that, all incoming
requests
are handed to this one thread (sometimes to a pair of threads
if
they
happen to have cleared their Collections down at the same time)
Cut-down server:
One thread handles the initial request. All subsequent requests
oscillate
evenly between just 2 ASP threads

IIS 5 (configured with 20 ASP threads):
Real server:
All 20 threads are used. However, the first 8 are scheduled
sequentially.
After that it gets more and more random which ASP thread is
used. No
affinity every arises though.
Cut-down server:
Only 8 ASP threads are ever used, and these are scheduled in
perfect
round-robin fashion

I can't explain these observations as the 2 servers are handling the same
requests. There seems to be more involved than simple round-robin
scheduling. Hence, I though if I understood the algorithm better, I might
have a clue as to the initial problem we encountered (which is now being
considered serious)

Tony Proctor

Jul 22 '05 #2

P: n/a
Thanks Brian. No, that's not the reason. The application is multi-threaded
and well-behaved until one thread decides to clear something down

Tony Proctor

"Brian Staff" <brianstaff@[NoSpam]cox.net> wrote in message
news:sMjVd.43600$xt.1828@fed1read07...
Tony,

This is from the IIS help files...maybe you have server-side debugging
turned ON?

Debugging Flags

Enable ASP Server-Side Script Debugging

Select this check box to enable the Web server to enter the Microsoft Script Debugger while processing ASP pages. You can then use the debugger to
examine your scripts.

Enabling ASP server-side debugging causes ASP to run in single-threaded
mode. For performance reasons, ASP debugging is not recommended in a
production environment.

Brian

"Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in message news:u%****************@TK2MSFTNGP09.phx.gbl...
Hi everyone

We're experiencing some serious anomalies with the scheduling of ASP
threads. I'd be interested to hear if anyone knows what algorithm is used (e.g. simple round-robin, or something more sophisticated), and what
situations might perturb it. Even a hint as to what would be considered
normal scheduling might help.

The root of our problem is that we observed a normally well-behaved web
application suddenly limit itself to a single ASP thread. The application consists mainly of a VB6 DLL instantiated via an ASP page. Note that the
application does not store STA objects in Session/Application, and
normally
exhibits no thread affinities at all. The previous post asking for advice about this is at:
http://www.google.ie/gr*************...=lang_en&hl=en.
As that previous post says, we eventually found that the switch of all
incoming requests (irrespective of their Session ID) to a single ASP
thread
occurred when one thread just happened to destroy a Collection of MSXML
DOM
objects. Hah!, we thought, a bug in MSXML 3.0, probably connected with its new free-threaded capability (which we don't use). We then created a much cut-down version of our server application with a view to reporting it to Microsoft. Unfortunately, the cut-down version didn't show the same
results.
Interestingly, though, its ASP thread scheduling was markedly different
from
that in our real application.

The problem has only been observed under IIS 6, and so it could well be an IIS/ASP difference rather than the fault of MSXML. Running the same
simple,
single-client (with an auto-refreshed HTML page at 1 sec intervals) test
against our 2 servers (the real one and the cut-down one), we observe the following strange thread behaviour. Any explanations here would be very
welcome:

IIS 6 (configured with 20 ASP threads):
Real server:
All 20 threads used in perfect round-robin fashion, until one
destroys
a Collection of MSXML DOM objects. After that, all incoming
requests
are handed to this one thread (sometimes to a pair of threads
if
they
happen to have cleared their Collections down at the same time) Cut-down server:
One thread handles the initial request. All subsequent requests oscillate
evenly between just 2 ASP threads

IIS 5 (configured with 20 ASP threads):
Real server:
All 20 threads are used. However, the first 8 are scheduled
sequentially.
After that it gets more and more random which ASP thread is
used. No
affinity every arises though.
Cut-down server:
Only 8 ASP threads are ever used, and these are scheduled in
perfect
round-robin fashion

I can't explain these observations as the 2 servers are handling the same requests. There seems to be more involved than simple round-robin
scheduling. Hence, I though if I understood the algorithm better, I might have a clue as to the initial problem we encountered (which is now being
considered serious)

Tony Proctor


Jul 22 '05 #3

P: n/a
ASP.dll does not directly schedule threads (i.e. they are not fibres). ASP
established a thread pool and passes requests to the thread pool. The OS
schedules threads.

The thread scheduler will try to reuse an idle thread that was recently run
over a thread that has not run in a while (i.e. FIFO), but this is not
guaranteed. This is an attempt to preserve the L1/L2 cache and improve
performance. You cannot guarantee which thread will service a given ASP
page. Also, it is important to differentiate if a single thread is
executing the entire ASP page or if a different ASP page is starting the
executing and then marshalling over to the other thread b/c of a COM
apartment issue. One other point is that the thread pool size will grow on
demand up to the configured maximum threshold. I think by default it only
creates 4 threads. So you have to create some degree of parallel stress to
even get it to create the 20 threads you are looking for.

I would recommend that you get an iisstate log of the problem occurring and
post it. We can take a look and see if we can identify the cause of the
serialization.
Pat

"Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in message
news:%2****************@TK2MSFTNGP12.phx.gbl...
Thanks Brian. No, that's not the reason. The application is multi-threaded
and well-behaved until one thread decides to clear something down

Tony Proctor

"Brian Staff" <brianstaff@[NoSpam]cox.net> wrote in message
news:sMjVd.43600$xt.1828@fed1read07...
Tony,

This is from the IIS help files...maybe you have server-side debugging
turned ON?

Debugging Flags

Enable ASP Server-Side Script Debugging

Select this check box to enable the Web server to enter the Microsoft

Script
Debugger while processing ASP pages. You can then use the debugger to
examine your scripts.

Enabling ASP server-side debugging causes ASP to run in single-threaded
mode. For performance reasons, ASP debugging is not recommended in a
production environment.

Brian

"Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in

message
news:u%****************@TK2MSFTNGP09.phx.gbl...
> Hi everyone
>
> We're experiencing some serious anomalies with the scheduling of ASP
> threads. I'd be interested to hear if anyone knows what algorithm is used > (e.g. simple round-robin, or something more sophisticated), and what
> situations might perturb it. Even a hint as to what would be considered
> normal scheduling might help.
>
> The root of our problem is that we observed a normally well-behaved web
> application suddenly limit itself to a single ASP thread. The application > consists mainly of a VB6 DLL instantiated via an ASP page. Note that
> the
> application does not store STA objects in Session/Application, and
> normally
> exhibits no thread affinities at all. The previous post asking for advice > about this is at:
> http://www.google.ie/gr*************...=lang_en&hl=en. >
> As that previous post says, we eventually found that the switch of all
> incoming requests (irrespective of their Session ID) to a single ASP
> thread
> occurred when one thread just happened to destroy a Collection of MSXML
> DOM
> objects. Hah!, we thought, a bug in MSXML 3.0, probably connected with its > new free-threaded capability (which we don't use). We then created a much > cut-down version of our server application with a view to reporting it to > Microsoft. Unfortunately, the cut-down version didn't show the same
> results.
> Interestingly, though, its ASP thread scheduling was markedly different
> from
> that in our real application.
>
> The problem has only been observed under IIS 6, and so it could well be an > IIS/ASP difference rather than the fault of MSXML. Running the same
> simple,
> single-client (with an auto-refreshed HTML page at 1 sec intervals)
> test
> against our 2 servers (the real one and the cut-down one), we observe the > following strange thread behaviour. Any explanations here would be very
> welcome:
>
> IIS 6 (configured with 20 ASP threads):
> Real server:
> All 20 threads used in perfect round-robin fashion, until
> one
> destroys
> a Collection of MSXML DOM objects. After that, all incoming
> requests
> are handed to this one thread (sometimes to a pair of
> threads
> if
> they
> happen to have cleared their Collections down at the same time) > Cut-down server:
> One thread handles the initial request. All subsequent requests > oscillate
> evenly between just 2 ASP threads
>
> IIS 5 (configured with 20 ASP threads):
> Real server:
> All 20 threads are used. However, the first 8 are scheduled
> sequentially.
> After that it gets more and more random which ASP thread is
> used. No
> affinity every arises though.
> Cut-down server:
> Only 8 ASP threads are ever used, and these are scheduled in
> perfect
> round-robin fashion
>
> I can't explain these observations as the 2 servers are handling the same > requests. There seems to be more involved than simple round-robin
> scheduling. Hence, I though if I understood the algorithm better, I might > have a clue as to the initial problem we encountered (which is now
> being
> considered serious)
>
> Tony Proctor
>
>



Jul 22 '05 #4

P: n/a
This is the only thread executing:

Thread ID: 21
System Thread ID: 928
Kernel Time: 0:0:2.78
User Time: 0:0:5.921
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
C:\WINDOWS\system32\MSVBVM60.DLL -
Thread Type: ASP
Executing Page:

# ChildEBP RetAddr
00 0264dec0 02ca3367 msado15!CRsetField::SetupDataAccessor+0x29d
01 0264dee4 02ca2604 msado15!CStdCollection::LookupItem+0x14e
02 0264def0 02ca7eb2 msado15!CStdComObjectRoot::InternalAddRef+0x10
03 0264df20 02ca279c msado15!CRsetFields::get_Item+0x3e
04 0264df3c 110b7b6d msado15!ATL::CComObject<CCommand>::AddRef+0xc
05 0264df48 660d9a27 ACExec!ACServices::LoadObject+0x2a3d
WARNING: Stack unwind information not available. Following frames may be
wrong.
06 0264e34c 040ccd98 MSVBVM60!_vbaFreeObj
07 00000000 00000000 0x40ccd98

I can't say why or even if this is a real problem. The thread has only used
5 seconds of time. On the good news side...No XML to be found. You can try
isolating the ACExec.dll to a separate COM+ server package and see if the
problem follows or not. Also, if you build the dll with symbols and place
the symbol file in the same directory with the dll, IISState will pick it up
and give you line number and .cls file names.

Pat
"Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in message
news:Ou*************@TK2MSFTNGP15.phx.gbl...
Thanks for the response Pat

I can confirm that there is no apartment-threading issue, or cross-thread
marshalling. These are issue we were aware of before the application was
designed.

The piece about the 4-thread increments is interesting. It explains a few
of
the observations we've made. Also, it makes pretty obvious sense to try
and
use threads FIFO to take advantage of processor caches, and even minimise
paging of TLS. Hence, I've always been a bit suspicious of the scheduling
characteristics we've so far taken for granted (i.e. round-robin across
all
threads).

I've attached an IISState log from a previous failure run involving
multiple
ASP Sessions (my recent test case involved just a single Session). I did
check the log myself but it appeared OK to the "untrained eye". At the
time
of the log, only one ASP thread was being used.

Only a few moments ago, we uncovered some more clues. The one ingredient
that explained the difference between my real server and my cut-down
server
was a WithEvents on an MSXML2.Document variable in a VB class. Without it,
there is no acquired thread affinity when the DOMs are cleared down, and
only 2 ASP threads are ever used for my single Session (requests evenly
oscillating between them). With the keyword, all 20 ASP threads are used
until the DOMs are cleared down on one thread. The other threads are then
no
longer used. The handle count is also significantly greater in this
failure
case (500 as opposed to 280 in the working case).

Although this points the finger back at MSXML, could you suggest a
criterion
that would cause ASP threads to be avoided like this Pat? In the
multi-Session case, we eventually end up with many requests queued up and
only one ASP thread handling them all.

Tony Proctor

"Pat [MSFT]" <pa******@online.microsoft.com> wrote in message
news:#p**************@TK2MSFTNGP14.phx.gbl...
ASP.dll does not directly schedule threads (i.e. they are not fibres).

ASP
established a thread pool and passes requests to the thread pool. The OS
schedules threads.

The thread scheduler will try to reuse an idle thread that was recently

run
over a thread that has not run in a while (i.e. FIFO), but this is not
guaranteed. This is an attempt to preserve the L1/L2 cache and improve
performance. You cannot guarantee which thread will service a given ASP
page. Also, it is important to differentiate if a single thread is
executing the entire ASP page or if a different ASP page is starting the
executing and then marshalling over to the other thread b/c of a COM
apartment issue. One other point is that the thread pool size will grow

on
demand up to the configured maximum threshold. I think by default it
only
creates 4 threads. So you have to create some degree of parallel stress

to
even get it to create the 20 threads you are looking for.

I would recommend that you get an iisstate log of the problem occurring

and
post it. We can take a look and see if we can identify the cause of the
serialization.
Pat

"Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in

message
news:%2****************@TK2MSFTNGP12.phx.gbl...
> Thanks Brian. No, that's not the reason. The application is multi-threaded > and well-behaved until one thread decides to clear something down
>
> Tony Proctor
>
> "Brian Staff" <brianstaff@[NoSpam]cox.net> wrote in message
> news:sMjVd.43600$xt.1828@fed1read07...
>> Tony,
>>
>> This is from the IIS help files...maybe you have server-side debugging
>> turned ON?
>>
>> Debugging Flags
>>
>> Enable ASP Server-Side Script Debugging
>>
>> Select this check box to enable the Web server to enter the Microsoft
> Script
>> Debugger while processing ASP pages. You can then use the debugger to
>> examine your scripts.
>>
>> Enabling ASP server-side debugging causes ASP to run in
>> single-threaded
>> mode. For performance reasons, ASP debugging is not recommended in a
>> production environment.
>>
>> Brian
>>
>> "Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in
> message
>> news:u%****************@TK2MSFTNGP09.phx.gbl...
>> > Hi everyone
>> >
>> > We're experiencing some serious anomalies with the scheduling of ASP
>> > threads. I'd be interested to hear if anyone knows what algorithm is
> used
>> > (e.g. simple round-robin, or something more sophisticated), and what
>> > situations might perturb it. Even a hint as to what would be considered >> > normal scheduling might help.
>> >
>> > The root of our problem is that we observed a normally well-behaved web >> > application suddenly limit itself to a single ASP thread. The
> application
>> > consists mainly of a VB6 DLL instantiated via an ASP page. Note that
>> > the
>> > application does not store STA objects in Session/Application, and
>> > normally
>> > exhibits no thread affinities at all. The previous post asking for
> advice
>> > about this is at:
>> >
> http://www.google.ie/gr*************...=lang_en&hl=en. >> >
>> > As that previous post says, we eventually found that the switch of all >> > incoming requests (irrespective of their Session ID) to a single ASP
>> > thread
>> > occurred when one thread just happened to destroy a Collection of MSXML >> > DOM
>> > objects. Hah!, we thought, a bug in MSXML 3.0, probably connected with > its
>> > new free-threaded capability (which we don't use). We then created a
> much
>> > cut-down version of our server application with a view to reporting it > to
>> > Microsoft. Unfortunately, the cut-down version didn't show the same
>> > results.
>> > Interestingly, though, its ASP thread scheduling was markedly different >> > from
>> > that in our real application.
>> >
>> > The problem has only been observed under IIS 6, and so it could well be > an
>> > IIS/ASP difference rather than the fault of MSXML. Running the same
>> > simple,
>> > single-client (with an auto-refreshed HTML page at 1 sec intervals)
>> > test
>> > against our 2 servers (the real one and the cut-down one), we
>> > observe
> the
>> > following strange thread behaviour. Any explanations here would be very >> > welcome:
>> >
>> > IIS 6 (configured with 20 ASP threads):
>> > Real server:
>> > All 20 threads used in perfect round-robin fashion, until
>> > one
>> > destroys
>> > a Collection of MSXML DOM objects. After that, all incoming >> > requests
>> > are handed to this one thread (sometimes to a pair of
>> > threads
>> > if
>> > they
>> > happen to have cleared their Collections down at the same
> time)
>> > Cut-down server:
>> > One thread handles the initial request. All subsequent
> requests
>> > oscillate
>> > evenly between just 2 ASP threads
>> >
>> > IIS 5 (configured with 20 ASP threads):
>> > Real server:
>> > All 20 threads are used. However, the first 8 are scheduled >> > sequentially.
>> > After that it gets more and more random which ASP thread is >> > used. No
>> > affinity every arises though.
>> > Cut-down server:
>> > Only 8 ASP threads are ever used, and these are scheduled in >> > perfect
>> > round-robin fashion
>> >
>> > I can't explain these observations as the 2 servers are handling the
> same
>> > requests. There seems to be more involved than simple round-robin
>> > scheduling. Hence, I though if I understood the algorithm better, I
> might
>> > have a clue as to the initial problem we encountered (which is now
>> > being
>> > considered serious)
>> >
>> > Tony Proctor
>> >
>> >
>>
>>
>
>



Jul 22 '05 #5

P: n/a
My test program can now reproduce this quite easily Pat. I will polish it a
little, to ensure that only the minimal amount of code is present, and then
submit it to MS as a problem with MSXML 3.0.

It feels as though specifying an MSXML2.Document in a WithEvents variable in
a VB class actually does something to the DOM itself. In our code, the DOM
is manipulated via a wrapper class that happens to contain the WithEvents
clause. Later, they become separated so that the DOM is cached, and the
wrapper destroyed. However, it seems that whatever was done to the DOM
previously persists after that, and causes repercussions inside a
multi-threaded environment like IIS.

Thanks for the help. Much appreciated

Tony Proctor

"Pat [MSFT]" <pa******@online.microsoft.com> wrote in message
news:e8**************@TK2MSFTNGP14.phx.gbl...
This is the only thread executing:

Thread ID: 21
System Thread ID: 928
Kernel Time: 0:0:2.78
User Time: 0:0:5.921
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\system32\MSVBVM60.DLL -
Thread Type: ASP
Executing Page:

# ChildEBP RetAddr
00 0264dec0 02ca3367 msado15!CRsetField::SetupDataAccessor+0x29d
01 0264dee4 02ca2604 msado15!CStdCollection::LookupItem+0x14e
02 0264def0 02ca7eb2 msado15!CStdComObjectRoot::InternalAddRef+0x10
03 0264df20 02ca279c msado15!CRsetFields::get_Item+0x3e
04 0264df3c 110b7b6d msado15!ATL::CComObject<CCommand>::AddRef+0xc
05 0264df48 660d9a27 ACExec!ACServices::LoadObject+0x2a3d
WARNING: Stack unwind information not available. Following frames may be
wrong.
06 0264e34c 040ccd98 MSVBVM60!_vbaFreeObj
07 00000000 00000000 0x40ccd98

I can't say why or even if this is a real problem. The thread has only used 5 seconds of time. On the good news side...No XML to be found. You can try isolating the ACExec.dll to a separate COM+ server package and see if the
problem follows or not. Also, if you build the dll with symbols and place the symbol file in the same directory with the dll, IISState will pick it up and give you line number and .cls file names.

Pat
"Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in message news:Ou*************@TK2MSFTNGP15.phx.gbl...
Thanks for the response Pat

I can confirm that there is no apartment-threading issue, or cross-thread
marshalling. These are issue we were aware of before the application was
designed.

The piece about the 4-thread increments is interesting. It explains a few of
the observations we've made. Also, it makes pretty obvious sense to try
and
use threads FIFO to take advantage of processor caches, and even minimise paging of TLS. Hence, I've always been a bit suspicious of the scheduling characteristics we've so far taken for granted (i.e. round-robin across
all
threads).

I've attached an IISState log from a previous failure run involving
multiple
ASP Sessions (my recent test case involved just a single Session). I did
check the log myself but it appeared OK to the "untrained eye". At the
time
of the log, only one ASP thread was being used.

Only a few moments ago, we uncovered some more clues. The one ingredient
that explained the difference between my real server and my cut-down
server
was a WithEvents on an MSXML2.Document variable in a VB class. Without it, there is no acquired thread affinity when the DOMs are cleared down, and
only 2 ASP threads are ever used for my single Session (requests evenly
oscillating between them). With the keyword, all 20 ASP threads are used
until the DOMs are cleared down on one thread. The other threads are then no
longer used. The handle count is also significantly greater in this
failure
case (500 as opposed to 280 in the working case).

Although this points the finger back at MSXML, could you suggest a
criterion
that would cause ASP threads to be avoided like this Pat? In the
multi-Session case, we eventually end up with many requests queued up and only one ASP thread handling them all.

Tony Proctor

"Pat [MSFT]" <pa******@online.microsoft.com> wrote in message
news:#p**************@TK2MSFTNGP14.phx.gbl...
ASP.dll does not directly schedule threads (i.e. they are not fibres).

ASP
established a thread pool and passes requests to the thread pool. The OS schedules threads.

The thread scheduler will try to reuse an idle thread that was recently

run
over a thread that has not run in a while (i.e. FIFO), but this is not
guaranteed. This is an attempt to preserve the L1/L2 cache and improve
performance. You cannot guarantee which thread will service a given ASP page. Also, it is important to differentiate if a single thread is
executing the entire ASP page or if a different ASP page is starting the executing and then marshalling over to the other thread b/c of a COM
apartment issue. One other point is that the thread pool size will grow
on
demand up to the configured maximum threshold. I think by default it
only
creates 4 threads. So you have to create some degree of parallel
stress to
even get it to create the 20 threads you are looking for.

I would recommend that you get an iisstate log of the problem occurring

and
post it. We can take a look and see if we can identify the cause of
the serialization.
Pat

"Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in

message
news:%2****************@TK2MSFTNGP12.phx.gbl...
> Thanks Brian. No, that's not the reason. The application is

multi-threaded
> and well-behaved until one thread decides to clear something down
>
> Tony Proctor
>
> "Brian Staff" <brianstaff@[NoSpam]cox.net> wrote in message
> news:sMjVd.43600$xt.1828@fed1read07...
>> Tony,
>>
>> This is from the IIS help files...maybe you have server-side debugging >> turned ON?
>>
>> Debugging Flags
>>
>> Enable ASP Server-Side Script Debugging
>>
>> Select this check box to enable the Web server to enter the Microsoft > Script
>> Debugger while processing ASP pages. You can then use the debugger to >> examine your scripts.
>>
>> Enabling ASP server-side debugging causes ASP to run in
>> single-threaded
>> mode. For performance reasons, ASP debugging is not recommended in a
>> production environment.
>>
>> Brian
>>
>> "Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in
> message
>> news:u%****************@TK2MSFTNGP09.phx.gbl...
>> > Hi everyone
>> >
>> > We're experiencing some serious anomalies with the scheduling of ASP >> > threads. I'd be interested to hear if anyone knows what algorithm is > used
>> > (e.g. simple round-robin, or something more sophisticated), and what >> > situations might perturb it. Even a hint as to what would be

considered
>> > normal scheduling might help.
>> >
>> > The root of our problem is that we observed a normally well-behaved web
>> > application suddenly limit itself to a single ASP thread. The
> application
>> > consists mainly of a VB6 DLL instantiated via an ASP page. Note
that >> > the
>> > application does not store STA objects in Session/Application, and
>> > normally
>> > exhibits no thread affinities at all. The previous post asking for
> advice
>> > about this is at:
>> >
>

http://www.google.ie/gr*************...=lang_en&hl=en.
>> >
>> > As that previous post says, we eventually found that the switch of

all
>> > incoming requests (irrespective of their Session ID) to a single ASP >> > thread
>> > occurred when one thread just happened to destroy a Collection of

MSXML
>> > DOM
>> > objects. Hah!, we thought, a bug in MSXML 3.0, probably connected

with
> its
>> > new free-threaded capability (which we don't use). We then created a > much
>> > cut-down version of our server application with a view to reporting it
> to
>> > Microsoft. Unfortunately, the cut-down version didn't show the
same >> > results.
>> > Interestingly, though, its ASP thread scheduling was markedly

different
>> > from
>> > that in our real application.
>> >
>> > The problem has only been observed under IIS 6, and so it could well be
> an
>> > IIS/ASP difference rather than the fault of MSXML. Running the
same >> > simple,
>> > single-client (with an auto-refreshed HTML page at 1 sec intervals) >> > test
>> > against our 2 servers (the real one and the cut-down one), we
>> > observe
> the
>> > following strange thread behaviour. Any explanations here would be

very
>> > welcome:
>> >
>> > IIS 6 (configured with 20 ASP threads):
>> > Real server:
>> > All 20 threads used in perfect round-robin fashion, until >> > one
>> > destroys
>> > a Collection of MSXML DOM objects. After that, all

incoming
>> > requests
>> > are handed to this one thread (sometimes to a pair of
>> > threads
>> > if
>> > they
>> > happen to have cleared their Collections down at the same > time)
>> > Cut-down server:
>> > One thread handles the initial request. All subsequent
> requests
>> > oscillate
>> > evenly between just 2 ASP threads
>> >
>> > IIS 5 (configured with 20 ASP threads):
>> > Real server:
>> > All 20 threads are used. However, the first 8 are

scheduled
>> > sequentially.
>> > After that it gets more and more random which ASP thread is
>> > used. No
>> > affinity every arises though.
>> > Cut-down server:
>> > Only 8 ASP threads are ever used, and these are
scheduled in
>> > perfect
>> > round-robin fashion
>> >
>> > I can't explain these observations as the 2 servers are handling

the > same
>> > requests. There seems to be more involved than simple round-robin
>> > scheduling. Hence, I though if I understood the algorithm better, I > might
>> > have a clue as to the initial problem we encountered (which is now
>> > being
>> > considered serious)
>> >
>> > Tony Proctor
>> >
>> >
>>
>>
>
>



Jul 22 '05 #6

P: n/a
Just for the record: this issue has been reported to MS as case
SRQ050304601390, and has been related to the manipulation of an MSXML DOM in
a VB "WithEvents" variable. MS have reproduced the problem and are currently
investigating

Tony Proctor

"Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in message
news:#s**************@tk2msftngp13.phx.gbl...
My test program can now reproduce this quite easily Pat. I will polish it a little, to ensure that only the minimal amount of code is present, and then submit it to MS as a problem with MSXML 3.0.

It feels as though specifying an MSXML2.Document in a WithEvents variable in a VB class actually does something to the DOM itself. In our code, the DOM
is manipulated via a wrapper class that happens to contain the WithEvents
clause. Later, they become separated so that the DOM is cached, and the
wrapper destroyed. However, it seems that whatever was done to the DOM
previously persists after that, and causes repercussions inside a
multi-threaded environment like IIS.

Thanks for the help. Much appreciated

Tony Proctor

"Pat [MSFT]" <pa******@online.microsoft.com> wrote in message
news:e8**************@TK2MSFTNGP14.phx.gbl...
This is the only thread executing:

Thread ID: 21
System Thread ID: 928
Kernel Time: 0:0:2.78
User Time: 0:0:5.921
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
C:\WINDOWS\system32\MSVBVM60.DLL -
Thread Type: ASP
Executing Page:

# ChildEBP RetAddr
00 0264dec0 02ca3367 msado15!CRsetField::SetupDataAccessor+0x29d
01 0264dee4 02ca2604 msado15!CStdCollection::LookupItem+0x14e
02 0264def0 02ca7eb2 msado15!CStdComObjectRoot::InternalAddRef+0x10
03 0264df20 02ca279c msado15!CRsetFields::get_Item+0x3e
04 0264df3c 110b7b6d msado15!ATL::CComObject<CCommand>::AddRef+0xc
05 0264df48 660d9a27 ACExec!ACServices::LoadObject+0x2a3d
WARNING: Stack unwind information not available. Following frames may be
wrong.
06 0264e34c 040ccd98 MSVBVM60!_vbaFreeObj
07 00000000 00000000 0x40ccd98

I can't say why or even if this is a real problem. The thread has only

used
5 seconds of time. On the good news side...No XML to be found. You can

try
isolating the ACExec.dll to a separate COM+ server package and see if the
problem follows or not. Also, if you build the dll with symbols and

place
the symbol file in the same directory with the dll, IISState will pick
it up
and give you line number and .cls file names.

Pat
"Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in message
news:Ou*************@TK2MSFTNGP15.phx.gbl...
Thanks for the response Pat

I can confirm that there is no apartment-threading issue, or

cross-thread marshalling. These are issue we were aware of before the application was designed.

The piece about the 4-thread increments is interesting. It explains a few of
the observations we've made. Also, it makes pretty obvious sense to try and
use threads FIFO to take advantage of processor caches, and even minimise paging of TLS. Hence, I've always been a bit suspicious of the scheduling characteristics we've so far taken for granted (i.e. round-robin across all
threads).

I've attached an IISState log from a previous failure run involving
multiple
ASP Sessions (my recent test case involved just a single Session). I did check the log myself but it appeared OK to the "untrained eye". At the
time
of the log, only one ASP thread was being used.

Only a few moments ago, we uncovered some more clues. The one ingredient that explained the difference between my real server and my cut-down
server
was a WithEvents on an MSXML2.Document variable in a VB class. Without it,
there is no acquired thread affinity when the DOMs are cleared down,
and only 2 ASP threads are ever used for my single Session (requests evenly oscillating between them). With the keyword, all 20 ASP threads are used until the DOMs are cleared down on one thread. The other threads are
then no
longer used. The handle count is also significantly greater in this
failure
case (500 as opposed to 280 in the working case).

Although this points the finger back at MSXML, could you suggest a
criterion
that would cause ASP threads to be avoided like this Pat? In the
multi-Session case, we eventually end up with many requests queued up and only one ASP thread handling them all.

Tony Proctor

"Pat [MSFT]" <pa******@online.microsoft.com> wrote in message
news:#p**************@TK2MSFTNGP14.phx.gbl...
> ASP.dll does not directly schedule threads (i.e. they are not fibres). ASP
> established a thread pool and passes requests to the thread pool. The OS
> schedules threads.
>
> The thread scheduler will try to reuse an idle thread that was
recently run
> over a thread that has not run in a while (i.e. FIFO), but this is not> guaranteed. This is an attempt to preserve the L1/L2 cache and improve> performance. You cannot guarantee which thread will service a given
ASP> page. Also, it is important to differentiate if a single thread is
> executing the entire ASP page or if a different ASP page is starting the> executing and then marshalling over to the other thread b/c of a COM
> apartment issue. One other point is that the thread pool size will grow on
> demand up to the configured maximum threshold. I think by default it
> only
> creates 4 threads. So you have to create some degree of parallel stress to
> even get it to create the 20 threads you are looking for.
>
> I would recommend that you get an iisstate log of the problem occurring and
> post it. We can take a look and see if we can identify the cause of the> serialization.
>
>
> Pat
>
>
>
> "Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in
message
> news:%2****************@TK2MSFTNGP12.phx.gbl...
> > Thanks Brian. No, that's not the reason. The application is
multi-threaded
> > and well-behaved until one thread decides to clear something down
> >
> > Tony Proctor
> >
> > "Brian Staff" <brianstaff@[NoSpam]cox.net> wrote in message
> > news:sMjVd.43600$xt.1828@fed1read07...
> >> Tony,
> >>
> >> This is from the IIS help files...maybe you have server-side debugging> >> turned ON?
> >>
> >> Debugging Flags
> >>
> >> Enable ASP Server-Side Script Debugging
> >>
> >> Select this check box to enable the Web server to enter the Microsoft> > Script
> >> Debugger while processing ASP pages. You can then use the debugger to> >> examine your scripts.
> >>
> >> Enabling ASP server-side debugging causes ASP to run in
> >> single-threaded
> >> mode. For performance reasons, ASP debugging is not recommended in a> >> production environment.
> >>
> >> Brian
> >>
> >> "Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in> > message
> >> news:u%****************@TK2MSFTNGP09.phx.gbl...
> >> > Hi everyone
> >> >
> >> > We're experiencing some serious anomalies with the scheduling of ASP> >> > threads. I'd be interested to hear if anyone knows what algorithm is
> > used
> >> > (e.g. simple round-robin, or something more sophisticated), and what> >> > situations might perturb it. Even a hint as to what would be
considered
> >> > normal scheduling might help.
> >> >
> >> > The root of our problem is that we observed a normally well-behaved web
> >> > application suddenly limit itself to a single ASP thread. The
> > application
> >> > consists mainly of a VB6 DLL instantiated via an ASP page. Note that> >> > the
> >> > application does not store STA objects in Session/Application,
and> >> > normally
> >> > exhibits no thread affinities at all. The previous post asking for> > advice
> >> > about this is at:
> >> >
> >

http://www.google.ie/gr*************...=lang_en&hl=en.
> >> >
> >> > As that previous post says, we eventually found that the switch of all
> >> > incoming requests (irrespective of their Session ID) to a single ASP> >> > thread
> >> > occurred when one thread just happened to destroy a Collection of MSXML
> >> > DOM
> >> > objects. Hah!, we thought, a bug in MSXML 3.0, probably connected with
> > its
> >> > new free-threaded capability (which we don't use). We then created a
> > much
> >> > cut-down version of our server application with a view to reporting it
> > to
> >> > Microsoft. Unfortunately, the cut-down version didn't show the same> >> > results.
> >> > Interestingly, though, its ASP thread scheduling was markedly
different
> >> > from
> >> > that in our real application.
> >> >
> >> > The problem has only been observed under IIS 6, and so it could well be
> > an
> >> > IIS/ASP difference rather than the fault of MSXML. Running the same> >> > simple,
> >> > single-client (with an auto-refreshed HTML page at 1 sec intervals)> >> > test
> >> > against our 2 servers (the real one and the cut-down one), we
> >> > observe
> > the
> >> > following strange thread behaviour. Any explanations here would
be very
> >> > welcome:
> >> >
> >> > IIS 6 (configured with 20 ASP threads):
> >> > Real server:
> >> > All 20 threads used in perfect round-robin fashion,
until> >> > one
> >> > destroys
> >> > a Collection of MSXML DOM objects. After that, all
incoming
> >> > requests
> >> > are handed to this one thread (sometimes to a pair of
> >> > threads
> >> > if
> >> > they
> >> > happen to have cleared their Collections down at the same> > time)
> >> > Cut-down server:
> >> > One thread handles the initial request. All subsequent> > requests
> >> > oscillate
> >> > evenly between just 2 ASP threads
> >> >
> >> > IIS 5 (configured with 20 ASP threads):
> >> > Real server:
> >> > All 20 threads are used. However, the first 8 are
scheduled
> >> > sequentially.
> >> > After that it gets more and more random which ASP thread is
> >> > used. No
> >> > affinity every arises though.
> >> > Cut-down server:
> >> > Only 8 ASP threads are ever used, and these are scheduled in
> >> > perfect
> >> > round-robin fashion
> >> >
> >> > I can't explain these observations as the 2 servers are handling the> > same
> >> > requests. There seems to be more involved than simple round-robin> >> > scheduling. Hence, I though if I understood the algorithm better, I
> > might
> >> > have a clue as to the initial problem we encountered (which is

now> >> > being
> >> > considered serious)
> >> >
> >> > Tony Proctor
> >> >
> >> >
> >>
> >>
> >
> >
>
>



Jul 22 '05 #7

This discussion thread is closed

Replies have been disabled for this discussion.