Connecting Tech Pros Worldwide Help | Site Map
bytes > topic > unix > answers

steadily increasing instr/s from mpstat

Member
  
Join Date: Oct 2007
Posts: 110
#1: Oct 13 '08
Hi all,

I'm fairly new to Linux and am running CentOS on my web server.

I've been using mpstat to check out the processor utilization, and most of the numbers look good (as far as I can tell).

Here's my mpstat output
Expand|Select|Wrap|Line Numbers
  1. 02:31:49 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
  2. 02:31:49 PM  all    8.70    0.00    1.68    0.10    0.02    0.02    0.00   89.49    153.21
  3.  
the only thing that I think I am slightly concerened about is intr/s. I'm not sure if 153 is a big number, but I started out around 50 or so about 2 weeks ago, and it seems to be steadily climbing, though I don't have any more services running now than I did then.

Could it be that some threads are not being released or something?
Is this something I should be concerned about?
sicarie's Avatar
Moderator
  
Join Date: Nov 2006
Location: USA
Posts: 3,929
#2: Oct 14 '08

re: steadily increasing instr/s from mpstat

From an online mpstat manpage it looks like that is an aggregate number over time of the number of interrupts received, so it should be climbing the longer the box is up.

That's just my 15 seconds of Googling, though - I couldn't tell you what a good or bad level of interrupts may be - I'm guessing that would be more for application behavior, but that is just a guess.
Member
  
Join Date: Oct 2007
Posts: 110
#3: Oct 14 '08

re: steadily increasing instr/s from mpstat

Thanks Sicarie,

I saw that page, but took the meaning to be that is the number of interupts per second, not an aggregate of the total number of interupts as long as the machine was up.

I'm trying to figure out if an aggregate of total number per second makes sense, but you could be right.

If so, I won't worry.
Bit digging deeper, using ps -aux, it shows that mysql is using 153% of cpu. But mpstat shows very little cpu being used.
I'm gonna try to restart mysql and see what happens.

Thanks,
Pete
sicarie's Avatar
Moderator
  
Join Date: Nov 2006
Location: USA
Posts: 3,929
#4: Oct 14 '08

re: steadily increasing instr/s from mpstat

pedalpete-

Could you post that ps? It's very interesting that over 100% could be utilized, though I don't doubt that it could show that. (I'm really just curious to see it, personally)

Thanks,

sicarie
Member
  
Join Date: Oct 2007
Posts: 110
#5: Oct 15 '08

re: steadily increasing instr/s from mpstat

Damn, i literally JUST restarted mysql, and it is now showing 1%.

now that I've started running a few processes, mysql has jumped to 20%
Expand|Select|Wrap|Line Numbers
  1. mysql    16157 20.1  0.9 137640 19660 pts/0    Sl   17:58   1:40 
  2.  
now 46%,
I'll post here when I get over 100%
mpstat still shows
Expand|Select|Wrap|Line Numbers
  1. 06:10:14 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
  2. 06:10:14 PM  all    8.88    0.00    1.71    0.10    0.02    0.02    0.00   89.28    156.41
  3.  
Member
  
Join Date: Oct 2007
Posts: 110
#6: Oct 15 '08

re: steadily increasing instr/s from mpstat

within 20 minutes of restarting mysql, it was showing 102% cpu,
but then it dropped and it seems to be dropping albiet slowly.

currently at 96.5%
Expand|Select|Wrap|Line Numbers
  1. mysql    16157 96.5  1.3 139592 27440 pts/0    Sl   17:58  25:12 /usr/libexec/my
  2.  
if i get a chance to see it above 100% again, i'll be sure to grab a shot and post it here.
Member
  
Join Date: Oct 2007
Posts: 110
#7: Oct 15 '08

re: steadily increasing instr/s from mpstat

here it is, about 1 hour after restarting and mysql is up above 100% again

Expand|Select|Wrap|Line Numbers
  1. mysql    16157  102  1.3 139604 27492 ?        Sl   17:58  96:45 /usr/libexec/my
  2.  

here's the mpstat
Expand|Select|Wrap|Line Numbers
  1. 07:35:20 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
  2. 07:35:20 PM  all    8.89    0.00    1.71    0.10    0.02    0.02    0.00   89.27    156.58
  3.  
here's the full output of my ps -aux
Expand|Select|Wrap|Line Numbers
  1. USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  2. root         1  0.0  0.0   2032   596 ?        Ss    2007   2:34 init [3]
  3. root         2  0.0  0.0      0     0 ?        S     2007   0:04 [migration/0]
  4. root         3  0.0  0.0      0     0 ?        SN    2007   0:00 [ksoftirqd/0]
  5. root         4  0.0  0.0      0     0 ?        S     2007   0:00 [watchdog/0]
  6. root         5  0.0  0.0      0     0 ?        S     2007   0:01 [migration/1]
  7. root         6  0.0  0.0      0     0 ?        SN    2007   0:00 [ksoftirqd/1]
  8. root         7  0.0  0.0      0     0 ?        S     2007   0:00 [watchdog/1]
  9. root         8  0.0  0.0      0     0 ?        S<    2007   0:00 [events/0]
  10. root         9  0.0  0.0      0     0 ?        S<    2007   0:00 [events/1]
  11. root        10  0.0  0.0      0     0 ?        S<    2007   0:00 [khelper]
  12. root        11  0.0  0.0      0     0 ?        S<    2007   0:00 [kthread]
  13. root        15  0.0  0.0      0     0 ?        S<    2007   0:03 [kblockd/0]
  14. root        16  0.0  0.0      0     0 ?        S<    2007   0:03 [kblockd/1]
  15. root        17  0.0  0.0      0     0 ?        S<    2007   0:00 [kacpid]
  16. root        90  0.0  0.0      0     0 ?        S<    2007   0:00 [cqueue/0]
  17. root        91  0.0  0.0      0     0 ?        S<    2007   0:00 [cqueue/1]
  18. root        94  0.0  0.0      0     0 ?        S<    2007   0:00 [khubd]
  19. root        96  0.0  0.0      0     0 ?        S<    2007   0:00 [kseriod]
  20. root       166  0.0  0.0      0     0 ?        S<    2007   0:29 [kswapd0]
  21. root       167  0.0  0.0      0     0 ?        S<    2007   0:00 [aio/0]
  22. root       168  0.0  0.0      0     0 ?        S<    2007   0:00 [aio/1]
  23. root       328  0.0  0.0      0     0 ?        S<    2007   0:00 [kpsmoused]
  24. root       358  0.0  0.0      0     0 ?        S<    2007   0:00 [ata/0]
  25. root       359  0.0  0.0      0     0 ?        S<    2007   0:00 [ata/1]
  26. root       360  0.0  0.0      0     0 ?        S<    2007   0:00 [ata_aux]
  27. root       364  0.0  0.0      0     0 ?        S<    2007   0:00 [scsi_eh_0]
  28. root       365  0.0  0.0      0     0 ?        S<    2007   0:00 [scsi_eh_1]
  29. root       366  0.0  0.0      0     0 ?        S<    2007  38:02 [kjournald]
  30. root       398  0.0  0.0      0     0 ?        S<    2007   0:15 [kauditd]
  31. root       432  0.0  0.0   2664  1136 ?        S<s   2007   0:00 /sbin/udevd -d
  32. root      1275  0.0  0.0      0     0 ?        S<    2007   0:00 [kmirrord]
  33. root      1296  0.0  0.0      0     0 ?        S<    2007   0:00 [kjournald]
  34. root      1877  0.0  0.3   9672  8100 ?        Ss    2007   0:00 /usr/sbin/resto
  35. root      1893  0.0  0.0  13088   736 ?        S<sl  2007   8:35 auditd
  36. root      1895  0.0  0.1   9496  3856 ?        S<s   2007   6:52 python /sbin/au
  37. root      1913  0.0  0.0   1688   580 ?        Ss    2007  12:32 syslogd -m 0
  38. root      1916  0.0  0.0   1640   400 ?        Ss    2007   0:00 klogd -x
  39. root      1932  0.0  0.0   1684   336 ?        Ss    2007   0:16 irqbalance
  40. root      1952  0.0  0.0   2124   508 ?        Ss    2007   1:48 mcstransd
  41. rpc       1974  0.0  0.0   1772   624 ?        Ss    2007   0:00 portmap
  42. root      1991  0.0  0.7  45984 15524 ?        Ssl   2007   8:13 /usr/bin/python
  43. root      2016  0.0  0.0   1784   728 ?        Ss    2007   0:00 rpc.statd
  44. root      2059  0.0  0.0   4936   560 ?        Ss    2007   0:00 rpc.idmapd
  45. dbus      2086  0.0  0.0  13076  1308 ?        Ssl   2007   0:00 dbus-daemon --s
  46. root      2102  0.0  0.0   2120   756 ?        Ss    2007   0:00 /usr/sbin/hcid
  47. root      2132  0.0  0.0      0     0 ?        S<    2007   0:00 [krfcommd]
  48. root      2182  0.0  0.0  12688  1348 ?        Ssl   2007   1:05 pcscd
  49. root      2206  0.0  0.0   1876   444 ?        Ss    2007   0:00 /usr/bin/hidd -
  50. apache    2222  0.0  0.5  31484 11916 ?        S    Oct12   0:02 /usr/sbin/httpd
  51. root      2227  0.1  0.0      0     0 ?        RN    2007 474:24 [kipmi0]
  52. root      2279  0.0  0.0  12400  1116 ?        Ssl   2007   0:00 automount
  53. root      2302  0.0  0.0   1636   528 ?        Ss    2007   0:00 /usr/sbin/acpid
  54. root      2317  0.0  0.0   9576  2036 ?        Ss    2007   0:00 cupsd
  55. root      2335  0.0  0.0   5172   952 ?        Ss    2007  11:38 /usr/sbin/sshd
  56. root      2384  0.0  0.0   1868   472 ?        Ss    2007   0:00 gpm -m /dev/inp
  57. root      2399  0.0  0.0   5220  1180 ?        Ss    2007   0:01 crond
  58. xfs       2428  0.0  0.0   3420  1284 ?        Ss    2007   0:00 xfs -droppriv -
  59. root      2457  0.0  0.0   2204   428 ?        Ss    2007   0:00 /usr/sbin/atd
  60. avahi     2488  0.0  0.0   2532  1372 ?        Ss    2007   0:00 avahi-daemon: r
  61. avahi     2489  0.0  0.0   2532   424 ?        Ss    2007   0:00 avahi-daemon: c
  62. 68        2504  0.0  0.1   5320  3644 ?        Ss    2007   0:02 hald
  63. root      2505  0.0  0.0   3096  1092 ?        S     2007   0:00 hald-runner
  64. 68        2511  0.0  0.0   1968   816 ?        S     2007   0:00 hald-addon-acpi
  65. root      2527  0.0  0.0   1924   656 ?        S     2007   7:26 hald-addon-stor
  66. root      2542  0.0  0.1   7028  2320 ?        Ss    2007   0:00 /usr/bin/perl /
  67. root      2577  0.0  0.0   1952   512 ?        S     2007   0:01 /usr/sbin/smart
  68. root      2580  0.0  0.0   1628   468 tty1     Ss+   2007   0:00 /sbin/mingetty
  69. root      2581  0.0  0.0   1624   464 tty2     Ss+   2007   0:00 /sbin/mingetty
  70. root      2582  0.0  0.0   1624   464 tty3     Ss+   2007   0:00 /sbin/mingetty
  71. root      2583  0.0  0.0   1628   468 tty4     Ss+   2007   0:00 /sbin/mingetty
  72. root      2584  0.0  0.0   1628   468 tty5     Ss+   2007   0:00 /sbin/mingetty
  73. root      2598  0.0  0.0   1624   464 tty6     Ss+   2007   0:00 /sbin/mingetty
  74. root      2599  0.0  0.0   1640   512 ttyS0    Ss+   2007   0:00 /sbin/agetty -L
  75. apache    3524  0.0  0.6  32712 12948 ?        S    Oct12   0:03 /usr/sbin/httpd
  76. apache    3891  0.0  0.5  31756 11988 ?        S    Oct12   0:02 /usr/sbin/httpd
  77. apache    4118  0.0  0.5  31484 11888 ?        S    Oct12   0:02 /usr/sbin/httpd
  78. apache    8973  0.0  1.4  60528 31016 ?        S    Jan22   0:00 /usr/sbin/httpd
  79. nobody   10964  0.0  3.3 102236 69944 ?        Ssl  Sep05   0:01 memcached -d -m
  80. root     15027  0.0  0.6  31044 14104 ?        Ss   Oct11   0:00 /usr/sbin/httpd
  81. root     16121  0.0  0.0   4488  1208 ?        S    17:58   0:00 /bin/sh /usr/bi
  82. mysql    16157  102  1.3 139604 27492 ?        Sl   17:58  96:45 /usr/libexec/my
  83. root     16437  0.0  0.0      0     0 ?        S    18:11   0:00 [pdflush]
  84. root     16685  0.0  0.0      0     0 ?        S    18:29   0:00 [pdflush]
  85. root     17502  0.0  0.1   8028  2448 ?        Ss   19:30   0:00 sshd: root@pts/
  86. root     17504  0.0  0.0   4620  1448 pts/0    Ss   19:30   0:00 -bash
  87. root     17538  0.0  0.0   4216   948 pts/0    R+   19:33   0:00 ps -aux
  88. apache   18041  0.0  0.5  31568 12000 ?        S    Oct12   0:03 /usr/sbin/httpd
  89. apache   18042  0.0  0.5  31484 11920 ?        S    Oct12   0:28 /usr/sbin/httpd
  90. apache   18043  0.0  0.5  31484 11900 ?        S    Oct12   0:03 /usr/sbin/httpd
  91. apache   18044  0.0  0.6  32352 12828 ?        S    Oct12   0:30 /usr/sbin/httpd
  92. apache   18045  0.0  0.6  32256 13108 ?        S    Oct12   0:15 /usr/sbin/httpd
  93. apache   18046  0.0  0.5  31976 12292 ?        S    Oct12   0:03 /usr/sbin/httpd
  94. apache   18047  0.0  0.6  32772 13580 ?        S    Oct12   0:05 /usr/sbin/httpd
  95. apache   18048  0.0  0.5  31868 12300 ?        S    Oct12   0:19 /usr/sbin/httpd
  96. apache   21222  0.0  0.6  32288 13140 ?        S    Oct12   0:53 /usr/sbin/httpd
  97. apache   21532  0.0  0.5  31896 12236 ?        S    Oct12   0:26 /usr/sbin/httpd
  98. apache   21533  0.0  0.5  31548 12052 ?        S    Oct12   0:03 /usr/sbin/httpd
  99. root     24631  0.0  0.0   8628  1828 ?        Ss   Mar28   0:00 sendmail: accep
  100. smmsp    24640  0.0  0.0   7548  1460 ?        Ss   Mar28   0:00 sendmail: Queue
  101. root     24665  0.0  0.0   4988   460 ?        Ss   Mar28   0:00 /usr/sbin/sasla
  102. root     24666  0.0  0.0   4988   244 ?        S    Mar28   0:00 /usr/sbin/sasla
  103. root     24667  0.0  0.0   4988   240 ?        S    Mar28   0:00 /usr/sbin/sasla
  104. root     24668  0.0  0.0   4988   240 ?        S    Mar28   0:00 /usr/sbin/sasla
  105. root     24669  0.0  0.0   4988   240 ?        S    Mar28   0:00 /usr/sbin/sasla
  106. apache   28252  0.0  0.6  32240 13120 ?        S    Oct12   0:29 /usr/sbin/httpd
  107. apache   28378  0.0  0.6  32400 12636 ?        S    Oct12   0:05 /usr/sbin/httpd
  108. apache   28650  0.0  0.6  32668 13648 ?        S    Oct12   0:03 /usr/sbin/httpd
  109.  
Member
  
Join Date: Oct 2007
Posts: 110
#8: Oct 15 '08

re: steadily increasing instr/s from mpstat

just checked again this am, and mysql up to 139% cpu

Expand|Select|Wrap|Line Numbers
  1. mysql    16157  139  1.3 141404 27944 ?        Sl   Oct14 1445:13 /usr/libexec/m
  2.  
at the same time, queries are running quickly, so could be an error in ps
sicarie's Avatar
Moderator
  
Join Date: Nov 2006
Location: USA
Posts: 3,929
#9: Oct 15 '08

re: steadily increasing instr/s from mpstat

Quote:

Originally Posted by pedalpete

just checked again this am, and mysql up to 139% cpu

Expand|Select|Wrap|Line Numbers
  1. mysql    16157  139  1.3 141404 27944 ?        Sl   Oct14 1445:13 /usr/libexec/m
  2.  
at the same time, queries are running quickly, so could be an error in ps

Well, what I'm curious now is the specs on your machine - does this have more than one processor? I'm gonna have to look into ps now - it's interesting to me that a Linux/UNIX programmer would allow something to register as more than 100%.

Thanks for indulging me!
Expert
  
Join Date: Sep 2007
Posts: 856
#10: Oct 15 '08

re: steadily increasing instr/s from mpstat

I have a dual-core system and I've seen the load go over 1.0 with uptime, so this may be a symptom that it's calculating each processor separately and adding them together.
sicarie's Avatar
Moderator
  
Join Date: Nov 2006
Location: USA
Posts: 3,929
#11: Oct 15 '08

re: steadily increasing instr/s from mpstat

Quote:

Originally Posted by Laharl

I have a dual-core system and I've seen the load go over 1.0 with uptime, so this may be a symptom that it's calculating each processor separately and adding them together.

That's my guess as well, but I'm extremely curious on this one - now I just have to get some free time... :\
Member
  
Join Date: Oct 2007
Posts: 110
#12: Oct 16 '08

re: steadily increasing instr/s from mpstat

It is a dual core, single proc machine, as far as I can remember (it's virtual), pretty sure its a xeon 3040.

I highly doubt mysql is using that much cpu as I don't think the server is doing THAT much, but I definitely could be wrong on that.

I would think that if mysql is using that much cpu, mpstat would report it. but mpstat is low.
Something isn't adding up.
Any idea how else I can test this?
Reply

« previous question | next question »

Similar Unix / Linux / BSD bytes

/bytes/development

/bytes/it

/bytes/about

We are a network of experts and professionals in IT and software development that help one another with answers to tough questions and share insights. Get the best answers to your questions from over 226,272 network members.

dev questions:
algorithms | asp | asp.net | c/c++ | coldfusion | c# | flash | html/css | java | javascript |
mobile dev | .net | perl | php | python | ruby | software dev | vb.net | visual basic | xml
it questions:
access | apache | careers | db2 | iis | macosx | mysql | networking | oracle | postgresql | sql server | unix | windows

About Bytes | Copyright 1995-2009 BYTES. All rights Reserved