Very serious problem with my DB2 V8.1 Fixpack 6 running in AIX 5.1
machine. Every one hour my DB2 instance processes are killed and it's
going down. Several trap files are generated in the db2 diag log
directory. Following the detailed entry in the db2diag.log:
2007-02-28-05.00.10.650672 Instance:****** Node:000
PID:2109578(db2 agent (dbname) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
base sys utilities sqleagnt_sigseg vh Probe:1 Database:******
Error in agent servicing application with coor_node:
0x0FFFFFFFFFFF3 600 : 0x0000 ..
2007-02-28-05.00.24.866073 Instance:****** Node:000
PID:2109578(db2 agent (MCL01P1) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
base sys utilities sqleagnt_sigseg vh Probe:2 Database:******
Error in agent servicing application with coor_agent_inde x:
0x0FFFFFFFFFFF3 602 : 0x0281 ..
2007-02-28-05.00.24.959073 Instance:****** Node:000
PID:2109578(db2 agent (dbname) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
base sys utilities sqleagnt_sigseg vh Probe:3 Database:******
Error in agent servicing application with CLIENT PID:
0x0FFFFFFFFFFF3 618 : 3636 3236 34 66264
2007-02-28-05.00.24.964498 Instance:****** Node:000
PID:2109578(db2 agent (dbname) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
base sys utilities sqleagnt_sigseg vh Probe:4 Database:******
Error in agent servicing application with INBOUND APPLICATION ID:
0x000000011003A F93 : 4741 3644 4136 3139 2E43 3433 382E 3030
GA6DA619.C438.0 0
0x000000011003A FA3 : 3244 3938 3034 3537 3137
2D98045717
2007-02-28-05.00.24.970115 Instance:****** Node:000
PID:2109578(db2 agent (dbname) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
base sys utilities sqleagnt_sigseg vh Probe:5 Database:******
Error in agent servicing application with INBOUND SEQUENCE NUMBER:
0x000000011003A FB4 : 0x0001 ..
2007-02-28-05.00.24.975732 Instance:****** Node:000
PID:2109578(db2 agent (dbname) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
base sys utilities sqleagnt_sigseg vh Probe:6 Database:******
Error in agent servicing application with OUTBOUND APPLICATION ID:
2007-02-28-05.00.24.980011 Instance:****** Node:000
PID:2109578(db2 agent (dbname) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
base sys utilities sqleagnt_sigseg vh Probe:7 Database:******
Error in agent servicing application with OUTBOUND SEQUENCE NUMBER:
2007-02-28-05.00.24.984003 Instance:****** Node:000
PID:2109578(db2 agent (dbaname) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
base sys utilities sqleagnt_sigseg vh Probe:8 Database:******
Error in agent servicing application with AUTHORIZATION ID:
0x07800000001D6 C65 : 4443 544D 5F44 4220 0000 0000 0000 0000
DCTM_DB ........
0x07800000001D6 C75 : 0000 0000 0000 0000 0000 0000 0000
00 ...............
2007-02-28-05.00.24.989338 Instance:****** Node:000
PID:2109578(db2 agent (MCL01P1) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
base sys utilities sqleagnt_sigseg vh Probe:9 Database:MCL01P 1
Error in agent servicing application with PRODUCT SIGNATURE:
0x07800000001D6 E88 : 0x53514C3038303 136
SQL08016
2007-02-28-05.00.24.994618 Instance:****** Node:000
PID:2109578(db2 agent (dbname) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
base sys utilities sqleagnt_sigseg vh Probe:10 Database:******
Error in agent servicing application with APPLICATION NAME:
0x07800000001D6 E5E : 646F 6375 6D65 6E74 756D
documentum
2007-02-28-05.00.25.014507 Instance:****** Node:000
PID:2109578(db2 agent (dbname) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
oper system services sqloEDUCodeTrap Handler Probe:10
Database:******
ADM0503C An unexpected internal processing error has occurred. ALL
DB2
PROCESSES ASSOCIATED WITH THIS INSTANCE HAVE BEEN SHUTDOWN.
Diagnostic
information has been recorded. Contact IBM Support for further
assistance.
2007-02-28-05.00.25.053871 Instance:****** Node:000
PID:2109578(db2 agent (dbname) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
oper system services sqloEDUCodeTrap Handler Probe:20
Database:******
Signal number received
0x0FFFFFFFFFFF3 730 : 0x0000000A ....
PID:2109578 TID:1 Node:000 Title: siginfo_t...
0x0FFFFFFFFFFF3 A20 : 0000 000A 0000 0000 0000 0009 0000
0000 ............... .
0x0FFFFFFFFFFF3 A30 : 0000 0000 0000 0000 0000 0000 0000
0000 ............... .
0x0FFFFFFFFFFF3 A40 : 0000 0000 0000 0000 0000 0000 0000
0000 ............... .
0x0FFFFFFFFFFF3 A50 : 0000 0000 0000 0000 0000 0000 0000
0000 ............... .
2007-02-28-05.00.25.081306 Instance:****** Node:000
PID:2109578(db2 agent (dbname) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
relation data serv sqlrr_signal_ha ndler Probe:50 Database:******
DIA0505I Execution of a component signal handling function has begun.
PID:2109578 TID:1 Node:000 Title: RDS AGENT CB
Dump File:/home/dumclp1/sqllib/db2dump/21095781.000
PID:2109578 TID:1 Node:000 Title: SQLCA
Dump File:/home/dumclp1/sqllib/db2dump/21095781.000
PID:2109578 TID:1 Node:000 Title: RDS UCINTFC
Dump File:/home/dumclp1/sqllib/db2dump/21095781.000
PID:2109578 TID:1 Node:000 Title: RDS UCINTFC: pCurrentPID =
0x0FFFFFFFFFFF3 390 :
0x000000011003A F00 ........
PID:2109578 TID:1 Node:000 Title: RDS UCINTFC: pCurrentPID-
>pkgnamcsn.rdbc olid =2007-02-28-05.00.25.110629 Instance:****** Node:000
PID:2109578(db2 agent (******) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
relation data serv sqlrr_dump_ffdc Probe:20 Database:******
DIA8516C A severe internal processing error has occurred.
ZRC=0x87120007M-^?
PID:2109578 TID:1 Node:000 Title: SQLCA
sqlcaid : SQLCA sqlcabc: 136 sqlcode: 0 sqlerrml: 0
sqlerrmc:
sqlerrp : SQL08016
sqlerrd : (1) 0x00000000 (2) 0x00000000 (3) 0x00000000
(4) 0x00000000 (5) 0x00000000 (6) 0x00000000
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate:
2007-02-28-05.00.25.303996 Instance:****** Node:000
PID:2109578(db2 agent (dbname) 0) TID:1
Appid:GA6DA619. C438.002D980457 17
relation data serv sqlrr_dump_ffdc Probe:902 Database:******
DIA8516C A severe internal processing error has occurred.
ZRC=0x87120007M-^?
This is the same error message we are getting in the diag log for
every one hour. Searched in google for different conditions no clue..
Did any one faced a similar situation like this? what is the root
cause of this error and what can be done to resolve this.
NOTE: This DB2 here works with DOCUMENTUM application which inturn
works with the SAN storage media for data/image handling.
As this's a serious error in a Production Server..looking for some
quick solutions.
Thanks,
vj_dba