Inexplicable slowdowns

[Frank Rizzo]

Hello,

I have a 32-bit .net 1.1 service that spawns many different threads and,
in general, does a lot of processing. Up to now, it's been working on
Win2003 32-bit servers fairly well against SQL Server 2000 (on the same
box).

Now, it's been installed on Win2003 R2 64-bit working against SQL Server
2005 (64-bit). At times, this happens several times a day, a single
thread in the application will inexplicably slow down to ridiculous
levels (i can see this because I log a lot of stuff to disk), like 1/100
of the regular processing speed.

I am trying to find causes for this behavior. Does it have anything to
do with the fact that it's a 32-bit app running on a 64-bit OS.

I already profiled the calls to SQL Server and it seems to have nothing
to do with the slowdown. The servers have 6GB of RAM and have 4 dual
proc AMD chips.

Any pointers are welcome.
Regards

[Kevin Frey]

So does the CPU load on the machine increase when this anomaly occurs, or it
just seems to be taking a long time to complete, but is not hogging CPU
cycles?
"Frank Rizzo" <no**@none.comwrote in message
news:O3*************@TK2MSFTNGP05.phx.gbl...

Hello,

I have a 32-bit .net 1.1 service that spawns many different threads and,
in general, does a lot of processing. Up to now, it's been working on
Win2003 32-bit servers fairly well against SQL Server 2000 (on the same
box).

Now, it's been installed on Win2003 R2 64-bit working against SQL Server
2005 (64-bit). At times, this happens several times a day, a single
thread in the application will inexplicably slow down to ridiculous levels
(i can see this because I log a lot of stuff to disk), like 1/100 of the
regular processing speed.

I am trying to find causes for this behavior. Does it have anything to do
with the fact that it's a 32-bit app running on a 64-bit OS.

I already profiled the calls to SQL Server and it seems to have nothing to
do with the slowdown. The servers have 6GB of RAM and have 4 dual proc
AMD chips.

Any pointers are welcome.
Regards

[Frank Rizzo]

Kevin Frey wrote:

So does the CPU load on the machine increase when this anomaly occurs, or it
just seems to be taking a long time to complete, but is not hogging CPU
cycles?

The CPU is pegged and is taking a long time to complete. But only one
CPU (out of 8) is pegged for a long time. Another thing I noticed is
the drop in available physical memory, which may or may not be a result
of actions taken by my application.

[Luke Zhang [MSFT]]

Hello,

What is the task of the particular thread? What code it will execute? If
you have source code on hand, you may add some debug code in it, for
example, log exact time in a file after each of steps has been finished, so
that you can know which code cause the delay.

Sincerely,

Luke Zhang

Microsoft Online Community Support
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscripti...ult.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscripti...t/default.aspx.
==================================================

This posting is provided "AS IS" with no warranties, and confers no rights.

[Frank Rizzo]

Luke Zhang [MSFT] wrote:

Hello,

What is the task of the particular thread? What code it will execute? If
you have source code on hand, you may add some debug code in it, for
example, log exact time in a file after each of steps has been finished, so
that you can know which code cause the delay.

The task is to basically schedule an entity into a time slot. So the
tasks has to consider hundreds of business rules and find a time slot.
The operation is totally memory based (and can take a bit of time). At
the end of the operation, the results are written to the database.
However, I profiled the SQL operation and it is taking no time at all.

I also noticed that the server was operating at the end of available
physical memory (typically 200 mb free out of total of 6GB available)
and as a result of perfmon monitoring, I saw that these slowdowns occur
when the available physical memory dips to 100mb or so. I am thinking
that the operation is forced into the paging file/virtual memory.
However, I still can't explain the magnitude of the slowdown, e.g.
operation that normally takes a second will take 80 seconds.

Regards

>
Sincerely,

Luke Zhang

Microsoft Online Community Support
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscripti...ult.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscripti...t/default.aspx.
==================================================

This posting is provided "AS IS" with no warranties, and confers no rights.

[Willy Denoyette [MVP]]

"Frank Rizzo" <no**@none.comwrote in message news:u2*************@TK2MSFTNGP04.phx.gbl...

Luke Zhang [MSFT] wrote:

>Hello,

What is the task of the particular thread? What code it will execute? If you have source
code on hand, you may add some debug code in it, for example, log exact time in a file
after each of steps has been finished, so that you can know which code cause the delay.

The task is to basically schedule an entity into a time slot. So the tasks has to
consider hundreds of business rules and find a time slot. The operation is totally memory
based (and can take a bit of time). At the end of the operation, the results are written
to the database. However, I profiled the SQL operation and it is taking no time at all.

I also noticed that the server was operating at the end of available physical memory
(typically 200 mb free out of total of 6GB available) and as a result of perfmon
monitoring, I saw that these slowdowns occur when the available physical memory dips to
100mb or so. I am thinking that the operation is forced into the paging file/virtual
memory. However, I still can't explain the magnitude of the slowdown, e.g. operation that
normally takes a second will take 80 seconds.

I don't get it, your service is a 32 bit process, so, can only consume 2GB (or 3GB) of
memory, so the question is - who's consuming the other 3-4GB?
What other applications are you running on this machine? Are there external SQL clients, and
how many are they? Did you configure SQL to use all available memory? Did you configure SQL
to use all processors?
Honestly, I think SQL server is the culprit.

Willy.

[Frank Rizzo]

Willy Denoyette [MVP] wrote:

"Frank Rizzo" <no**@none.comwrote in message
news:u2*************@TK2MSFTNGP04.phx.gbl...

>Luke Zhang [MSFT] wrote:

>>Hello,

What is the task of the particular thread? What code it will execute?
If you have source code on hand, you may add some debug code in it,
for example, log exact time in a file after each of steps has been
finished, so that you can know which code cause the delay.

The task is to basically schedule an entity into a time slot. So the
tasks has to consider hundreds of business rules and find a time slot.
The operation is totally memory based (and can take a bit of time).
At the end of the operation, the results are written to the database.
However, I profiled the SQL operation and it is taking no time at all.

I also noticed that the server was operating at the end of available
physical memory (typically 200 mb free out of total of 6GB available)
and as a result of perfmon monitoring, I saw that these slowdowns
occur when the available physical memory dips to 100mb or so. I am
thinking that the operation is forced into the paging file/virtual
memory. However, I still can't explain the magnitude of the slowdown,
e.g. operation that normally takes a second will take 80 seconds.

I don't get it, your service is a 32 bit process, so, can only consume
2GB (or 3GB) of memory, so the question is - who's consuming the other
3-4GB?
What other applications are you running on this machine? Are there
external SQL clients, and how many are they? Did you configure SQL to
use all available memory? Did you configure SQL to use all processors?
Honestly, I think SQL server is the culprit.

Oh, I know that SQL Server is the culprit. It's set to consume a max of
3GB. My application consumes 2.5GB (it's compiled with
LARGEADDRESSAWARE). And there is a bunch of other smaller apps that are
pushing the limit of the memory. I've asked the DBA to kick the SQL
Server to 2.5GB, but the guy and his superiors won't budge, so I am left
with a bunch of customers who are screaming bloody murder.

Anyway, the part that confounds me is that why an operation that
typically takes 1 second could take 80 seconds when only virtual memory
is available.

Regards.

>
Willy.

[Willy Denoyette [MVP]]

"Frank Rizzo" <no**@none.comwrote in message news:u9**************@TK2MSFTNGP06.phx.gbl...

Willy Denoyette [MVP] wrote:

>"Frank Rizzo" <no**@none.comwrote in message
news:u2*************@TK2MSFTNGP04.phx.gbl...

>>Luke Zhang [MSFT] wrote:
Hello,

What is the task of the particular thread? What code it will execute? If you have
source code on hand, you may add some debug code in it, for example, log exact time in
a file after each of steps has been finished, so that you can know which code cause the
delay.

The task is to basically schedule an entity into a time slot. So the tasks has to
consider hundreds of business rules and find a time slot. The operation is totally
memory based (and can take a bit of time). At the end of the operation, the results are
written to the database. However, I profiled the SQL operation and it is taking no time
at all.

I also noticed that the server was operating at the end of available physical memory
(typically 200 mb free out of total of 6GB available) and as a result of perfmon
monitoring, I saw that these slowdowns occur when the available physical memory dips to
100mb or so. I am thinking that the operation is forced into the paging file/virtual
memory. However, I still can't explain the magnitude of the slowdown, e.g. operation
that normally takes a second will take 80 seconds.

I don't get it, your service is a 32 bit process, so, can only consume 2GB (or 3GB) of
memory, so the question is - who's consuming the other 3-4GB?
What other applications are you running on this machine? Are there external SQL clients,
and how many are they? Did you configure SQL to use all available memory? Did you
configure SQL to use all processors?
Honestly, I think SQL server is the culprit.

Oh, I know that SQL Server is the culprit. It's set to consume a max of 3GB. My
application consumes 2.5GB (it's compiled with LARGEADDRESSAWARE). And there is a bunch
of other smaller apps that are pushing the limit of the memory. I've asked the DBA to
kick the SQL Server to 2.5GB, but the guy and his superiors won't budge, so I am left with
a bunch of customers who are screaming bloody murder.

Anyway, the part that confounds me is that why an operation that typically takes 1 second
could take 80 seconds when only virtual memory is available.

Regards.

>>
Willy.

If I were the DBA, I would kick your application off that server (maybe you can try to do
the inverse), this is a typical case of a customer that things that buying a big server
with a lot of memory and processors (most of the time not investing in decent but still
expensive Disk IO subsystem) can run all of it's applications. Wrong, you are better off
with two smaller boxes than one big iron who's supposed to handle it all.
The 1 to 80 seconds is not surprising at all, if applications start competing for memory and
the system starts paging, application with a lot of private pages will suffer the most, SQL
server is optimized such that he won't give up his allocated share, so the victim is your
application.

Willy.