Very serious problem with my DB2 V8.1 Fixpack 6 running in AIX 5.1
machine. Every one hour my DB2 instance processes are killed and it's
going down. Several trap files are generated in the db2 diag log
directory. Following the detailed entry in the db2diag.log:
2007-02-28-05.00.10.650672 Instance:****** Node:000
PID:2109578(db2agent (dbname) 0) TID:1
Appid:GA6DA619.C438.002D98045717
base sys utilities sqleagnt_sigsegvh Probe:1 Database:******
Error in agent servicing application with coor_node:
0x0FFFFFFFFFFF3600 : 0x0000 ..
2007-02-28-05.00.24.866073 Instance:****** Node:000
PID:2109578(db2agent (MCL01P1) 0) TID:1
Appid:GA6DA619.C438.002D98045717
base sys utilities sqleagnt_sigsegvh Probe:2 Database:******
Error in agent servicing application with coor_agent_index:
0x0FFFFFFFFFFF3602 : 0x0281 ..
2007-02-28-05.00.24.959073 Instance:****** Node:000
PID:2109578(db2agent (dbname) 0) TID:1
Appid:GA6DA619.C438.002D98045717
base sys utilities sqleagnt_sigsegvh Probe:3 Database:******
Error in agent servicing application with CLIENT PID:
0x0FFFFFFFFFFF3618 : 3636 3236 34 66264
2007-02-28-05.00.24.964498 Instance:****** Node:000
PID:2109578(db2agent (dbname) 0) TID:1
Appid:GA6DA619.C438.002D98045717
base sys utilities sqleagnt_sigsegvh Probe:4 Database:******
Error in agent servicing application with INBOUND APPLICATION ID:
0x000000011003AF93 : 4741 3644 4136 3139 2E43 3433 382E 3030
GA6DA619.C438.00
0x000000011003AFA3 : 3244 3938 3034 3537 3137
2D98045717
2007-02-28-05.00.24.970115 Instance:****** Node:000
PID:2109578(db2agent (dbname) 0) TID:1
Appid:GA6DA619.C438.002D98045717
base sys utilities sqleagnt_sigsegvh Probe:5 Database:******
Error in agent servicing application with INBOUND SEQUENCE NUMBER:
0x000000011003AFB4 : 0x0001 ..
2007-02-28-05.00.24.975732 Instance:****** Node:000
PID:2109578(db2agent (dbname) 0) TID:1
Appid:GA6DA619.C438.002D98045717
base sys utilities sqleagnt_sigsegvh Probe:6 Database:******
Error in agent servicing application with OUTBOUND APPLICATION ID:
2007-02-28-05.00.24.980011 Instance:****** Node:000
PID:2109578(db2agent (dbname) 0) TID:1
Appid:GA6DA619.C438.002D98045717
base sys utilities sqleagnt_sigsegvh Probe:7 Database:******
Error in agent servicing application with OUTBOUND SEQUENCE NUMBER:
2007-02-28-05.00.24.984003 Instance:****** Node:000
PID:2109578(db2agent (dbaname) 0) TID:1
Appid:GA6DA619.C438.002D98045717
base sys utilities sqleagnt_sigsegvh Probe:8 Database:******
Error in agent servicing application with AUTHORIZATION ID:
0x07800000001D6C65 : 4443 544D 5F44 4220 0000 0000 0000 0000
DCTM_DB ........
0x07800000001D6C75 : 0000 0000 0000 0000 0000 0000 0000
00 ...............
2007-02-28-05.00.24.989338 Instance:****** Node:000
PID:2109578(db2agent (MCL01P1) 0) TID:1
Appid:GA6DA619.C438.002D98045717
base sys utilities sqleagnt_sigsegvh Probe:9 Database:MCL01P1
Error in agent servicing application with PRODUCT SIGNATURE:
0x07800000001D6E88 : 0x53514C3038303136
SQL08016
2007-02-28-05.00.24.994618 Instance:****** Node:000
PID:2109578(db2agent (dbname) 0) TID:1
Appid:GA6DA619.C438.002D98045717
base sys utilities sqleagnt_sigsegvh Probe:10 Database:******
Error in agent servicing application with APPLICATION NAME:
0x07800000001D6E5E : 646F 6375 6D65 6E74 756D
documentum
2007-02-28-05.00.25.014507 Instance:****** Node:000
PID:2109578(db2agent (dbname) 0) TID:1
Appid:GA6DA619.C438.002D98045717
oper system services sqloEDUCodeTrapHandler Probe:10
Database:******
ADM0503C An unexpected internal processing error has occurred. ALL
DB2
PROCESSES ASSOCIATED WITH THIS INSTANCE HAVE BEEN SHUTDOWN.
Diagnostic
information has been recorded. Contact IBM Support for further
assistance.
2007-02-28-05.00.25.053871 Instance:****** Node:000
PID:2109578(db2agent (dbname) 0) TID:1
Appid:GA6DA619.C438.002D98045717
oper system services sqloEDUCodeTrapHandler Probe:20
Database:******
Signal number received
0x0FFFFFFFFFFF3730 : 0x0000000A ....
PID:2109578 TID:1 Node:000 Title: siginfo_t...
0x0FFFFFFFFFFF3A20 : 0000 000A 0000 0000 0000 0009 0000
0000 ................
0x0FFFFFFFFFFF3A30 : 0000 0000 0000 0000 0000 0000 0000
0000 ................
0x0FFFFFFFFFFF3A40 : 0000 0000 0000 0000 0000 0000 0000
0000 ................
0x0FFFFFFFFFFF3A50 : 0000 0000 0000 0000 0000 0000 0000
0000 ................
2007-02-28-05.00.25.081306 Instance:****** Node:000
PID:2109578(db2agent (dbname) 0) TID:1
Appid:GA6DA619.C438.002D98045717
relation data serv sqlrr_signal_handler Probe:50 Database:******
DIA0505I Execution of a component signal handling function has begun.
PID:2109578 TID:1 Node:000 Title: RDS AGENT CB
Dump File:/home/dumclp1/sqllib/db2dump/21095781.000
PID:2109578 TID:1 Node:000 Title: SQLCA
Dump File:/home/dumclp1/sqllib/db2dump/21095781.000
PID:2109578 TID:1 Node:000 Title: RDS UCINTFC
Dump File:/home/dumclp1/sqllib/db2dump/21095781.000
PID:2109578 TID:1 Node:000 Title: RDS UCINTFC: pCurrentPID =
0x0FFFFFFFFFFF3390 :
0x000000011003AF00 ........
PID:2109578 TID:1 Node:000 Title: RDS UCINTFC: pCurrentPID-
>pkgnamcsn.rdbcolid =2007-02-28-05.00.25.110629 Instance:****** Node:000
PID:2109578(db2agent (******) 0) TID:1
Appid:GA6DA619.C438.002D98045717
relation data serv sqlrr_dump_ffdc Probe:20 Database:******
DIA8516C A severe internal processing error has occurred.
ZRC=0x87120007M-^?
PID:2109578 TID:1 Node:000 Title: SQLCA
sqlcaid : SQLCA sqlcabc: 136 sqlcode: 0 sqlerrml: 0
sqlerrmc:
sqlerrp : SQL08016
sqlerrd : (1) 0x00000000 (2) 0x00000000 (3) 0x00000000
(4) 0x00000000 (5) 0x00000000 (6) 0x00000000
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate:
2007-02-28-05.00.25.303996 Instance:****** Node:000
PID:2109578(db2agent (dbname) 0) TID:1
Appid:GA6DA619.C438.002D98045717
relation data serv sqlrr_dump_ffdc Probe:902 Database:******
DIA8516C A severe internal processing error has occurred.
ZRC=0x87120007M-^?
This is the same error message we are getting in the diag log for
every one hour. Searched in google for different conditions no clue..
Did any one faced a similar situation like this? what is the root
cause of this error and what can be done to resolve this.
NOTE: This DB2 here works with DOCUMENTUM application which inturn
works with the SAN storage media for data/image handling.
As this's a serious error in a Production Server..looking for some
quick solutions.
Thanks,
vj_dba