Unusually High CPU on SQL Server the 2nd day after a reboot.

[JL]

To All,

I have a SQL2KSP3a database(<1GB) running on a 4x3GB physical CPU with
4GB of ram. It is Windows Server 2003 with hyper-threading turn on.
There are ~420 .Net users/cxns (fat client, no web/app servers) with
connection pooling and ~1 trx/sec. The database growth is neglegeable
and actually is not even relevent which I will explain in a minute.
99% of the trxs are from one SP that does a select. The resultsets are
relatively small as well 1~100 rows. Yes I have tuned it with index
tuning wizard as well, changed the SQL memory configurations, etc....

My problem is this...
The first day after a reboot, the server runs 6%CPU during peak hours.
During the non-peak hours until the next day something apparently
happens. The next day (2nd day after a reboot), it jumps to 40%CPU
during peak hours. The server will continue to run at 40%CPU during
peak hours until the next reboot. This phenomenon has been occurring
for 6 months or more and the traffic on the server is the same for day
1 as it is for day 2,3,4,... This database was on another server with
100+ dbs and exibited the same behavior, thus bringing that server to
its knees, and thus we had to move it to the server in question with no
other dbs.

I have googled my eyes out, Microsoft site, white papers, perfmon,
SQLDiag, PSSDiag, execution plans, index tuning wizard, and the list
goes on! I currently have a case open with Microsoft that has been
open for months now. I have been passed around to the 3rd "MS Tech
Specialist". I have ran PSSDiag a total of 6 times for them for hours
on end. I have changed MAXDOP. I could give more information, but I
would be here for days. I am running out of patience/ideas and
Microsoft is apparently blowing smoke.
Any ideas are greatly appreciated!
Thanks in advance!
JL

[louis]

>> The server will continue to run at 40%CPU during peak hours until
the next reboot

Taking a shot in the dark. Ensure the TempDB isn't running out of
space during operational hours. I believe TempDB is recreated each
time the MSSQL Service is restarted.

[Erland Sommarskog]

JL (jl*****@yahoo. com) writes:

My problem is this...
The first day after a reboot, the server runs 6%CPU during peak hours.
During the non-peak hours until the next day something apparently
happens. The next day (2nd day after a reboot), it jumps to 40%CPU
during peak hours. The server will continue to run at 40%CPU during
peak hours until the next reboot. This phenomenon has been occurring
for 6 months or more and the traffic on the server is the same for day
1 as it is for day 2,3,4,... This database was on another server with
100+ dbs and exibited the same behavior, thus bringing that server to
its knees, and thus we had to move it to the server in question with no
other dbs.

I have googled my eyes out, Microsoft site, white papers, perfmon,
SQLDiag, PSSDiag, execution plans, index tuning wizard, and the list
goes on!

Does that list include Profiler?

That is, the first thing I would to do if I was called in to look at
this, was to start Profiler, to see if there are any queries that runs
longer time than needed.

Ah, there is another thing to try in place of a reboot: run a DBCC
DROPCLEANBUFFER S. This flushes cache completely. If this lowers the
CPU usage just like a reboot does, then there is an issue with a cache.
Now, to further narrow it down, next time try DBCC FREEPROCCAHCE. This
only flushes the cache for query plans. If this helps too, I have a
strong suspicion. To wit, there is a problem if there are too many
similarly-looking queries being submitted, because the hashing in the
cache breaks down.

Then again, since you seem to use stored procedures, this is not likely
to be an issue - unless these procedures generates dynamic SQL?

By the way, are you using OPENXML somewhere?

--
Erland Sommarskog, SQL Server MVP, es****@sommarsk og.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techinf...2000/books.asp

[Gert-Jan Strik]

Another hint is to monitor the number of database connections. Maybe it
keeps growing and after a few days starts swamping the system.

HTH,
Gert-Jan
JL wrote:

To All,

I have a SQL2KSP3a database(<1GB) running on a 4x3GB physical CPU with
4GB of ram. It is Windows Server 2003 with hyper-threading turn on.
There are ~420 .Net users/cxns (fat client, no web/app servers) with
connection pooling and ~1 trx/sec. The database growth is neglegeable
and actually is not even relevent which I will explain in a minute.
99% of the trxs are from one SP that does a select. The resultsets are
relatively small as well 1~100 rows. Yes I have tuned it with index
tuning wizard as well, changed the SQL memory configurations, etc....

My problem is this...
The first day after a reboot, the server runs 6%CPU during peak hours.
During the non-peak hours until the next day something apparently
happens. The next day (2nd day after a reboot), it jumps to 40%CPU
during peak hours. The server will continue to run at 40%CPU during
peak hours until the next reboot. This phenomenon has been occurring
for 6 months or more and the traffic on the server is the same for day
1 as it is for day 2,3,4,... This database was on another server with
100+ dbs and exibited the same behavior, thus bringing that server to
its knees, and thus we had to move it to the server in question with no
other dbs.

I have googled my eyes out, Microsoft site, white papers, perfmon,
SQLDiag, PSSDiag, execution plans, index tuning wizard, and the list
goes on! I currently have a case open with Microsoft that has been
open for months now. I have been passed around to the 3rd "MS Tech
Specialist". I have ran PSSDiag a total of 6 times for them for hours
on end. I have changed MAXDOP. I could give more information, but I
would be here for days. I am running out of patience/ideas and
Microsoft is apparently blowing smoke.
Any ideas are greatly appreciated!
Thanks in advance!
JL

[JL]

Tempdb is not running out of space. I have actually set tempdb to 1GB
with 10% auto-grow. No OpenXML. The connections correlate to the number
of users logged in at any given time. The DROPCLEANBUFFER S &
FREEPROCCAHCE was the first thing that Microsoft had me do. No luck.
I have used profiler many times with this. I have even done a spid to
thread correlation with not luck in finding any rogue processes.
Microsoft has run the "Read 80" tool also; it apparently creates an
aggregate report on the PSSDiag output.

Another piece to the puzzle is that I have just applied some indexes
recommended by the index tuning wizard. I did some testing and found
that indeed the indexes did help, but I have been holding off applying
them to production as to not disturb my baseline. My thinking is that
even if the indexes did help, it would not explain the problem at hand.
Since I have applied the indexes, the CPU is now running at 12%
consistently. It hasn't been rebooted since then, but SQL Server has
been restarted. It doesn't appear that the new indexes have fixed the
problem because theoretically with the indexes applied, it should run
below the 6%.

(Sorry if I sounded like I was MS bashing. They have been relatively
good with responses. I am just at my wits end.)
Thanks for the replies!
JL

[Erland Sommarskog]

JL (jl*****@yahoo. com) writes:

(Sorry if I sounded like I was MS bashing. They have been relatively
good with responses. I am just at my wits end.)
Thanks for the replies!

Admittedly, it is likely that you get better and more precise answers
from MS support staff from people in the newsgroup. After all, they
have more information about the case than we have. And they even get
paid to consider the situation!

All we can offer are shots in the dark.

Good news to hear that the indexes appear to help.
--
Erland Sommarskog, SQL Server MVP, es****@sommarsk og.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techinf...2000/books.asp

[andyturk]

> My problem is this...

The first day after a reboot, the server runs 6%CPU during peak hours. During the non-peak hours until the next day something apparently
happens. The next day (2nd day after a reboot), it jumps to 40%CPU
during peak hours. The server will continue to run at 40%CPU during
peak hours until the next reboot.

JL,

The company I work for has experienced the exact same problem. We
haven't figured out *exactly* what's happening, but we have a pretty
good idea that this CPU usage jump is caused by the use of prepared
statements in our application.

We have a Java/J2EE app which uses a mixture of stored procedures and
prepared statements to access the database. What we noticed (after
several months of beating our heads against a wall) is that the
increase in CPU overall usage was *entirely* due to an increase in the
CPU cost of prepared statements. In fact, we could detect this
"failure" by noticing when the average CPU cost of a prepared statement
jumped up above 15ms for our app.

We measure this by running a trace and then extracting summary
information on prepared statements (e.g., "... textdata LIKE 'exec
sp_execute%'") vs everything else in the mix. Unmodified, our db
performance lasted 54 hours before degenerating. However, we modified
our application to replace some of the most commonly issued prepared
statements with stored procedures and this has effectively cured the
problem.

The actual root cause is still a mystery. While running around in the
app code, we noticed some bad practice where unique prepared statements
would be generated thousands of times, differing only by some small
string which really should have been a parameter in the first place.
Maybe we blew out SQL Server's buffer for holding on to these things.

My advice is to run a trace and take a look at your prepared statement
CPU cost.

[Erland Sommarskog]

(an******@gmail .com) writes:

The actual root cause is still a mystery. While running around in the
app code, we noticed some bad practice where unique prepared statements
would be generated thousands of times, differing only by some small
string which really should have been a parameter in the first place.
Maybe we blew out SQL Server's buffer for holding on to these things.

Yes, this is an issue that have been observerd by others, and there's
even a hotfix, as described in http://support.microsoft.com/?scid=884850.
Although, I believe the discussion went something like this mainly happens
when you have lots of memory. The more memory you have, the bigger the
procedure cache can get, and the more similarly-looking queries can fit
there, increasing the time it takes to find if a query alread is in
cache or not.
--
Erland Sommarskog, SQL Server MVP, es****@sommarsk og.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techinf...2000/books.asp

[JL]

Erland Sommarskog wrote:

(an******@gmail .com) writes:

The actual root cause is still a mystery. While running around in the app code, we noticed some bad practice where unique prepared statements would be generated thousands of times, differing only by some small
string which really should have been a parameter in the first place. Maybe we blew out SQL Server's buffer for holding on to these

things.
Yes, this is an issue that have been observerd by others, and there's
even a hotfix, as described in http://support.microsoft.com/?scid=884850. Although, I believe the discussion went something like this mainly happens when you have lots of memory. The more memory you have, the bigger the procedure cache can get, and the more similarly-looking queries can fit there, increasing the time it takes to find if a query alread is in
cache or not.
--
Erland Sommarskog, SQL Server MVP, es****@sommarsk og.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techinf...2000/books.asp

Thanks for the info guys! All of the SQL is in stored procedures, but
it does look promising.

BTW - I have sent the database to Microsoft for them to look at. I
noticed that the hotfix was posted on 2/4/2005... How convenient!

Thanks Again!
JL

[Erland Sommarskog]

JL (jl*****@yahoo. com) writes:

Thanks for the info guys! All of the SQL is in stored procedures, but
it does look promising.

BTW - I have sent the database to Microsoft for them to look at. I
noticed that the hotfix was posted on 2/4/2005... How convenient!

Hm, I don't think the KB article applies to your case. I reviewed the
thread on Google, and I hinted at this problem then as well, but you
told me that FREEPROCCACHE and DROPCLEANBUFFER S had no effect. (But
FREEPROCCACHE is something Andy could try.)
--
Erland Sommarskog, SQL Server MVP, es****@sommarsk og.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techinf...2000/books.asp