473,890 Members | 1,369 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Unusually High CPU on SQL Server the 2nd day after a reboot.

JL
To All,

I have a SQL2KSP3a database(<1GB) running on a 4x3GB physical CPU with
4GB of ram. It is Windows Server 2003 with hyper-threading turn on.
There are ~420 .Net users/cxns (fat client, no web/app servers) with
connection pooling and ~1 trx/sec. The database growth is neglegeable
and actually is not even relevent which I will explain in a minute.
99% of the trxs are from one SP that does a select. The resultsets are
relatively small as well 1~100 rows. Yes I have tuned it with index
tuning wizard as well, changed the SQL memory configurations, etc....

My problem is this...
The first day after a reboot, the server runs 6%CPU during peak hours.
During the non-peak hours until the next day something apparently
happens. The next day (2nd day after a reboot), it jumps to 40%CPU
during peak hours. The server will continue to run at 40%CPU during
peak hours until the next reboot. This phenomenon has been occurring
for 6 months or more and the traffic on the server is the same for day
1 as it is for day 2,3,4,... This database was on another server with
100+ dbs and exibited the same behavior, thus bringing that server to
its knees, and thus we had to move it to the server in question with no
other dbs.

I have googled my eyes out, Microsoft site, white papers, perfmon,
SQLDiag, PSSDiag, execution plans, index tuning wizard, and the list
goes on! I currently have a case open with Microsoft that has been
open for months now. I have been passed around to the 3rd "MS Tech
Specialist". I have ran PSSDiag a total of 6 times for them for hours
on end. I have changed MAXDOP. I could give more information, but I
would be here for days. I am running out of patience/ideas and
Microsoft is apparently blowing smoke.
Any ideas are greatly appreciated!
Thanks in advance!
JL

Jul 23 '05 #1
10 7313
>> The server will continue to run at 40%CPU during peak hours until
the next reboot

Taking a shot in the dark. Ensure the TempDB isn't running out of
space during operational hours. I believe TempDB is recreated each
time the MSSQL Service is restarted.

Jul 23 '05 #2
JL (jl*****@yahoo. com) writes:
My problem is this...
The first day after a reboot, the server runs 6%CPU during peak hours.
During the non-peak hours until the next day something apparently
happens. The next day (2nd day after a reboot), it jumps to 40%CPU
during peak hours. The server will continue to run at 40%CPU during
peak hours until the next reboot. This phenomenon has been occurring
for 6 months or more and the traffic on the server is the same for day
1 as it is for day 2,3,4,... This database was on another server with
100+ dbs and exibited the same behavior, thus bringing that server to
its knees, and thus we had to move it to the server in question with no
other dbs.

I have googled my eyes out, Microsoft site, white papers, perfmon,
SQLDiag, PSSDiag, execution plans, index tuning wizard, and the list
goes on!


Does that list include Profiler?

That is, the first thing I would to do if I was called in to look at
this, was to start Profiler, to see if there are any queries that runs
longer time than needed.

Ah, there is another thing to try in place of a reboot: run a DBCC
DROPCLEANBUFFER S. This flushes cache completely. If this lowers the
CPU usage just like a reboot does, then there is an issue with a cache.
Now, to further narrow it down, next time try DBCC FREEPROCCAHCE. This
only flushes the cache for query plans. If this helps too, I have a
strong suspicion. To wit, there is a problem if there are too many
similarly-looking queries being submitted, because the hashing in the
cache breaks down.

Then again, since you seem to use stored procedures, this is not likely
to be an issue - unless these procedures generates dynamic SQL?

By the way, are you using OPENXML somewhere?

--
Erland Sommarskog, SQL Server MVP, es****@sommarsk og.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techinf...2000/books.asp
Jul 23 '05 #3
Another hint is to monitor the number of database connections. Maybe it
keeps growing and after a few days starts swamping the system.

HTH,
Gert-Jan
JL wrote:

To All,

I have a SQL2KSP3a database(<1GB) running on a 4x3GB physical CPU with
4GB of ram. It is Windows Server 2003 with hyper-threading turn on.
There are ~420 .Net users/cxns (fat client, no web/app servers) with
connection pooling and ~1 trx/sec. The database growth is neglegeable
and actually is not even relevent which I will explain in a minute.
99% of the trxs are from one SP that does a select. The resultsets are
relatively small as well 1~100 rows. Yes I have tuned it with index
tuning wizard as well, changed the SQL memory configurations, etc....

My problem is this...
The first day after a reboot, the server runs 6%CPU during peak hours.
During the non-peak hours until the next day something apparently
happens. The next day (2nd day after a reboot), it jumps to 40%CPU
during peak hours. The server will continue to run at 40%CPU during
peak hours until the next reboot. This phenomenon has been occurring
for 6 months or more and the traffic on the server is the same for day
1 as it is for day 2,3,4,... This database was on another server with
100+ dbs and exibited the same behavior, thus bringing that server to
its knees, and thus we had to move it to the server in question with no
other dbs.

I have googled my eyes out, Microsoft site, white papers, perfmon,
SQLDiag, PSSDiag, execution plans, index tuning wizard, and the list
goes on! I currently have a case open with Microsoft that has been
open for months now. I have been passed around to the 3rd "MS Tech
Specialist". I have ran PSSDiag a total of 6 times for them for hours
on end. I have changed MAXDOP. I could give more information, but I
would be here for days. I am running out of patience/ideas and
Microsoft is apparently blowing smoke.
Any ideas are greatly appreciated!
Thanks in advance!
JL

Jul 23 '05 #4
JL
Tempdb is not running out of space. I have actually set tempdb to 1GB
with 10% auto-grow. No OpenXML. The connections correlate to the number
of users logged in at any given time. The DROPCLEANBUFFER S &
FREEPROCCAHCE was the first thing that Microsoft had me do. No luck.
I have used profiler many times with this. I have even done a spid to
thread correlation with not luck in finding any rogue processes.
Microsoft has run the "Read 80" tool also; it apparently creates an
aggregate report on the PSSDiag output.

Another piece to the puzzle is that I have just applied some indexes
recommended by the index tuning wizard. I did some testing and found
that indeed the indexes did help, but I have been holding off applying
them to production as to not disturb my baseline. My thinking is that
even if the indexes did help, it would not explain the problem at hand.
Since I have applied the indexes, the CPU is now running at 12%
consistently. It hasn't been rebooted since then, but SQL Server has
been restarted. It doesn't appear that the new indexes have fixed the
problem because theoretically with the indexes applied, it should run
below the 6%.

(Sorry if I sounded like I was MS bashing. They have been relatively
good with responses. I am just at my wits end.)
Thanks for the replies!
JL

Jul 23 '05 #5
JL (jl*****@yahoo. com) writes:
(Sorry if I sounded like I was MS bashing. They have been relatively
good with responses. I am just at my wits end.)
Thanks for the replies!


Admittedly, it is likely that you get better and more precise answers
from MS support staff from people in the newsgroup. After all, they
have more information about the case than we have. And they even get
paid to consider the situation!

All we can offer are shots in the dark.

Good news to hear that the indexes appear to help.
--
Erland Sommarskog, SQL Server MVP, es****@sommarsk og.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techinf...2000/books.asp
Jul 23 '05 #6
> My problem is this...
The first day after a reboot, the server runs 6%CPU during peak hours. During the non-peak hours until the next day something apparently
happens. The next day (2nd day after a reboot), it jumps to 40%CPU
during peak hours. The server will continue to run at 40%CPU during
peak hours until the next reboot.


JL,

The company I work for has experienced the exact same problem. We
haven't figured out *exactly* what's happening, but we have a pretty
good idea that this CPU usage jump is caused by the use of prepared
statements in our application.

We have a Java/J2EE app which uses a mixture of stored procedures and
prepared statements to access the database. What we noticed (after
several months of beating our heads against a wall) is that the
increase in CPU overall usage was *entirely* due to an increase in the
CPU cost of prepared statements. In fact, we could detect this
"failure" by noticing when the average CPU cost of a prepared statement
jumped up above 15ms for our app.

We measure this by running a trace and then extracting summary
information on prepared statements (e.g., "... textdata LIKE 'exec
sp_execute%'") vs everything else in the mix. Unmodified, our db
performance lasted 54 hours before degenerating. However, we modified
our application to replace some of the most commonly issued prepared
statements with stored procedures and this has effectively cured the
problem.

The actual root cause is still a mystery. While running around in the
app code, we noticed some bad practice where unique prepared statements
would be generated thousands of times, differing only by some small
string which really should have been a parameter in the first place.
Maybe we blew out SQL Server's buffer for holding on to these things.

My advice is to run a trace and take a look at your prepared statement
CPU cost.

Jul 23 '05 #7
(an******@gmail .com) writes:
The actual root cause is still a mystery. While running around in the
app code, we noticed some bad practice where unique prepared statements
would be generated thousands of times, differing only by some small
string which really should have been a parameter in the first place.
Maybe we blew out SQL Server's buffer for holding on to these things.


Yes, this is an issue that have been observerd by others, and there's
even a hotfix, as described in http://support.microsoft.com/?scid=884850.
Although, I believe the discussion went something like this mainly happens
when you have lots of memory. The more memory you have, the bigger the
procedure cache can get, and the more similarly-looking queries can fit
there, increasing the time it takes to find if a query alread is in
cache or not.
--
Erland Sommarskog, SQL Server MVP, es****@sommarsk og.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techinf...2000/books.asp
Jul 23 '05 #8
JL

Erland Sommarskog wrote:
(an******@gmail .com) writes:
The actual root cause is still a mystery. While running around in the app code, we noticed some bad practice where unique prepared statements would be generated thousands of times, differing only by some small
string which really should have been a parameter in the first place. Maybe we blew out SQL Server's buffer for holding on to these
things.
Yes, this is an issue that have been observerd by others, and there's
even a hotfix, as described in http://support.microsoft.com/?scid=884850. Although, I believe the discussion went something like this mainly happens when you have lots of memory. The more memory you have, the bigger the procedure cache can get, and the more similarly-looking queries can fit there, increasing the time it takes to find if a query alread is in
cache or not.
--
Erland Sommarskog, SQL Server MVP, es****@sommarsk og.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techinf...2000/books.asp


Thanks for the info guys! All of the SQL is in stored procedures, but
it does look promising.

BTW - I have sent the database to Microsoft for them to look at. I
noticed that the hotfix was posted on 2/4/2005... How convenient!

Thanks Again!
JL

Jul 23 '05 #9
JL (jl*****@yahoo. com) writes:
Thanks for the info guys! All of the SQL is in stored procedures, but
it does look promising.

BTW - I have sent the database to Microsoft for them to look at. I
noticed that the hotfix was posted on 2/4/2005... How convenient!


Hm, I don't think the KB article applies to your case. I reviewed the
thread on Google, and I hinted at this problem then as well, but you
told me that FREEPROCCACHE and DROPCLEANBUFFER S had no effect. (But
FREEPROCCACHE is something Andy could try.)
--
Erland Sommarskog, SQL Server MVP, es****@sommarsk og.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techinf...2000/books.asp
Jul 23 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
4459
by: Ben Allen | last post by:
Is there a way (through a variable) to show the date of the last server reboot i.e. when PHP last started running? As I am trying to show the server uptime, which of course only works from the last reboot, therefore my %ages are very low. Cheers, Ben
2
15242
by: Ken Lindner | last post by:
I have a need to become familiar with SQL Server 2000 for work. Needless to say I am new to SQL Server any version, but not IT in general. My employer has provided me with the SQL Server 2000 Personal disk from the SQL Server 2000 Enterprise kit as this is reported here on the MSDN web site to be the version that is supported on Windows XP. In fact so many of you kind people confess to having succeeded in doing it. I have tried...
19
14859
by: Thue Tuxen Sørensen | last post by:
Hi everybody ! I´m maintaining a large intranet (approx 10000 concurrent users) running on one IIS box and one DB box with sqlserver 2000. Currently there is 2,5 GB Ram, 1 1400 mhz cpu and 2 scsi disks installed on the db box. Sqlserver is set to use max 1,4 GB RAM, and the sqlserver does not seem to be using it all.
3
1548
by: Guillaume Senneville | last post by:
hi, we are having this weird problem that can be reproduced easily on 2 out of 2 servers where we've tried: - we run a SQL Server 2000 stored procedure that processes 2.5 million rows (big procedure with many cursors) - when it hits an area of procedure (not always the exact same place, but in the same area) , the server reboots I've tried to debug the problem but:
2
2155
by: Tim Nelson | last post by:
I created a simple C# .Net application to prove I can deploy it on a Win2k server running terminal services. I installed the 1.1 .Net Framework and installed the application (using a VS Setup app) on the machine. It runs perfectly when I log in as Administrator but generates an exception when I run it as any other user. Do I need a reboot or something after installing the framework? Do I need to tweak something in the .Net framework...
1
3162
by: Keith | last post by:
All, I have been told this is an ASP.NET issue and not an IIS issue, so I am posting this here. I have a problem with ASP.NET returning an HTTP 500 error when trying to run ASPX pages on Windows Server 2003 Enterprise Edition Only. This works on Windows 2000, Windows 2003 Web Edition and
0
5659
by: jsausten | last post by:
I am having this ridiculously frustrating problem, migrating an existing web app to VS.2005. Initially I was able to successfully convert my VS 2002 app, by opening the Localhost site using the new 'Local IIS' option (File -> Add Existing Web Site ->. Local IIS -> Select the Localhost site) and everything was OK. However, when I shut down and reboot, I was unable to open the same project and got the following error: "Unable to open...
5
2340
by: war_wheelan | last post by:
I created the db with the attached script and I am able to access it until I reboot the server. I've tried enabling flag 1807 via the SQL server service and the startup parameters of the instance. In all cases the database always come up suspect after a reboot. There was one instance where I was able to recover, but I am not sure how that happened. Does anyone have an idea of how I can reboot the server without the database becomming...
3
6452
by: Steve | last post by:
Hi All I downloaded Sql server 2005 express SP2 and attempted to modify the Bootstrapper package files as I did with SP1 When i try to install SQL server as part of my VS 2005 deployment app I get an error at the end of the SQL server install phase 'Invalid endpoint'. (Note SQL server express gets installed OK) If I run SQL server express SP2 setup directly it I don't get the error
0
9826
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
11234
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10829
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
10468
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9640
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
5854
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
4682
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
4276
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
3282
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.