Ian wrote:
mike_dba wrote: I have recently upgraded from SuSe 8 to 9 (kernel 2.4 to kernel 2.6).
I am running DB2 8.1 FP 11 (aka 8.2 FP 4). I have had slowdowns that I
suspect are related to the fact that an online backup is being run.
Can someone help me understand the behaviour described below and offer
a suggestion as to how to tune this ? Your assistance is appreciated.
I have a system running at 95 idle before the backup starts. Once the
backup is underway, I see a split in vmsat of roughtly 45% system and
50% wait. I also see low run-q but free memory goes down significantly
and swapping in and out is occurring. Since this is an online backup,
I do have queries coming in but do not suspect that they are causing
this as it is off-hours and the snapshots do not indicate anything out
of the ordinary with them.
I have plenty of memory on the box - some 16 GB with 3 GB for
bufferpool and 3 GB for sort allocated.
Thanks.
Where are you sending the backup (i.e. local disk, NFS file system,
etc)?
Can you post the vmstat output while this is occurring?
Ian,
Thanks for the reply. This is posing a big problem as simple queries
degrade significantly while the is occurring.
The backup is going to local disk. I am am running EEE (DPF). The
script rsh the backup command to each host. It backs up each partition
to a disk local to each host.
The linux 'uptime' and vmstat on one of the hosts looks like this at
the start -
cat vmstat.njds0052.11:00
11:00am up 19 days 8:42, 2 users, load average: 0.79, 1.31, 1.17
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
0 0 445736 898452 565076 13486952 7 8 12482 753 12 9 23
4 64 9
0 2 445736 1738696 565736 12646416 0 0 648 98 1612 8913
12 36 38 14
0 2 445736 1736836 567604 12645576 0 0 1874 64 1596 3446
5 11 44 40
0 3 445736 1735192 569156 12646080 0 0 1418 620 1898 12300
9 10 37 43
0 3 445736 1733704 570676 12645588 0 0 1512 72 1749 4205
3 8 47 42
Here we are 10 minutes into it and you can see the script that runs
every 10 min actually took 2 add'l mins (11:12)
cat vmstat.njds0052.11:12
11:12am up 19 days 8:54, 2 users, load average: 30.43, 17.71, 8.49
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
5 10 2032160 35484 97064 14231460 7 8 12487 762 12 10
23 4 64 9
1 6 2033268 36680 98272 14227160 1758 244 164476 96230 1860 6059
9 59 1 32
2 4 2033956 39184 98208 14228028 466 424 164120 77460 1745 7437
9 59 2 30
1 5 2035300 36704 98636 14227800 0 528 168542 59732 1651 3887
6 53 2 39
9 4 2037012 38068 99140 14225460 0 974 170332 57634 1671 3842
6 53 3 38
And later,
cat vmstat.njds0052.11:40
11:40am up 19 days 9:22, 3 users, load average: 23.22, 23.32,
20.56
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
1 9 3288968 35356 133008 14801316 8 9 12508 796 13 2
23 4 64 9
3 8 3287552 35480 133792 14803696 1362 386 81426 88628 1739 5788
4 37 5 54
2 13 3286592 39820 134344 14801520 1834 508 46990 95744 1828 8214
3 24 8 65
1 13 3285016 38324 135120 14798768 1768 400 88354 60472 2071 6245
6 47 4 43
2 19 3280004 41216 135164 14796956 4470 6 7018 37188 1792 5374
2 6 7 85
cat vmstat.njds0052.11:50
11:50am up 19 days 9:32, 3 users, load average: 14.25, 14.76,
17.48
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
0 3 1516436 37560 135896 15174532 8 9 12511 801 0 4
23 4 64 9
0 2 1516196 36196 136452 15176536 184 4 53784 29938 7249 7595
40 28 8 24
29 2 1516000 47284 136188 15150748 208 48 29968 72668 5789 4395
69 26 0 4
45 2 1515968 40340 136056 15154360 96 6 43782 74056 2745 897
80 20 0 0
46 3 1515900 43300 136392 15146004 82 4 44694 23618 3049 1576
80 20 0 0
cat vmstat.njds0052.12:00 <--- Here ware have completed
12:00pm up 19 days 9:42, 3 users, load average: 0.07, 3.07, 10.27
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
0 0 1432332 276984 177516 14924624 8 9 12509 803 1 4
23 4 64 9
0 2 1431708 275752 177516 14924952 662 0 1656 20 1192 1648
1 0 82 18
0 1 1429304 272032 177516 14927840 1814 0 2810 144 1538 2269
1 1 50 49
1 2 1427752 269668 177532 14929072 1180 0 1750 210 1421 2394
1 1 51 46
0 2 1426076 266568 177540 14930836 1494 0 2172 144 1370 2330
0 1 50 49
I also have the following 'ps -ef' output -
cat top20.njds0052.11:12
UID PID %CPU CPU S TIME COMMAND
1001 8926 1.8 - S 00:00:07 db2med.18636.3 2
1001 8932 1.8 - S 00:00:06 db2med.18636.9 2
1001 8918 1.6 - S 00:00:06 db2bm.18636.1 2
1001 8920 1.6 - S 00:00:06 db2bm.18636.5 2
1001 8924 1.6 - S 00:00:06 db2med.18636.1 2
1001 8923 1.6 - S 00:00:06 db2med.18636.0 2
1001 8928 1.6 - S 00:00:05 db2med.18636.5 2
1001 8931 1.5 - S 00:00:05 db2med.18636.8 2
1001 8927 1.5 - S 00:00:05 db2med.18636.4 2
1001 8929 1.4 - S 00:00:05 db2med.18636.6 2
1001 8925 1.4 - S 00:00:05 db2med.18636.2 2
1001 8930 1.4 - S 00:00:05 db2med.18636.7 2
1001 15950 1.1 - S 00:01:45 db2agent (CIF) 1
1001 32014 1.1 - S 00:27:02 db2fcmdm 1
1001 10034 1.0 - S 00:00:03 db2med.15950.7 1
1001 32009 1.0 - S 00:24:54 db2fcmdm 2
1001 10031 0.8 - S 00:00:02 db2med.15950.4 1
1001 10028 0.7 - S 00:00:02 db2med.15950.1 1
1001 10036 0.7 - S 00:00:02 db2med.15950.9 1
cat top20.njds0052.11:20
UID PID %CPU CPU S TIME COMMAND
0 12048 3.0 - S 00:00:00 [pdflush]
1001 12052 2.0 - R 00:00:00 ps -e -o
uid,pid,pcpu,cpu,state,cputime,args --sort -pcpu,-time
1001 8926 1.2 - D 00:00:11 db2med.18636.3 2
1001 8932 1.2 - D 00:00:10 db2med.18636.9 2
1001 8918 1.2 - S 00:00:10 db2bm.18636.1 2
1001 8923 1.2 - D 00:00:10 db2med.18636.0 2
1001 8928 1.2 - S 00:00:10 db2med.18636.5 2
1001 8924 1.1 - D 00:00:10 db2med.18636.1 2
1001 8920 1.1 - S 00:00:10 db2bm.18636.5 2
1001 8931 1.1 - D 00:00:10 db2med.18636.8 2
1001 8930 1.1 - D 00:00:09 db2med.18636.7 2
1001 8927 1.1 - D 00:00:09 db2med.18636.4 2
1001 15950 1.1 - S 00:01:45 db2agent (CIF) 1
1001 8929 1.1 - D 00:00:09 db2med.18636.6 2
1001 32014 1.1 - D 00:27:03 db2fcmdm 1
1001 8925 1.1 - D 00:00:09 db2med.18636.2 2
1001 32009 1.0 - S 00:24:55 db2fcmdm 2
0 10871 0.8 - S 00:00:03 [pdflush]
1001 10034 0.8 - D 00:00:06 db2med.15950.7 1
cat top20.njds0052.11:31
UID PID %CPU CPU S TIME COMMAND
0 13862 3.0 - D 00:00:04 [pdflush]
0 13912 2.5 - D 00:00:01 [pdflush]
0 13904 2.4 - S 00:00:02 [pdflush]
0 13925 1.9 - D 00:00:00 [pdflush]
0 14020 1.3 - D 00:00:00 [pdflush]
0 14023 1.2 - D 00:00:00 [pdflush]
0 13942 1.2 - D 00:00:00 [pdflush]
1001 8918 1.1 - S 00:00:17 db2bm.18636.1 2
1001 8926 1.1 - D 00:00:17 db2med.18636.3 2
1001 8932 1.1 - D 00:00:17 db2med.18636.9 2
1001 8923 1.1 - S 00:00:17 db2med.18636.0 2
1001 8928 1.1 - D 00:00:16 db2med.18636.5 2
1001 8931 1.1 - D 00:00:16 db2med.18636.8 2
1001 32014 1.1 - D 00:27:04 db2fcmdm 1
1001 8930 1.1 - D 00:00:16 db2med.18636.7 2
1001 8927 1.1 - D 00:00:16 db2med.18636.4 2
1001 8920 1.1 - S 00:00:16 db2bm.18636.5 2
1001 8924 1.1 - D 00:00:16 db2med.18636.1 2
1001 8929 1.0 - D 00:00:16 db2med.18636.6 2
