EDU process dieing

[soccertl]

We have a program running on AIX and uses DB/2. Every once in a while,
while our program is just running along, DB/2 dies due to a signal 9
being sent to one of the EDU processes:

2006-11-14-12.26.23.442685+000 E396314C794 LEVEL: Severe
PID : 72190 TID : 1 PROC : db2ipccm 0
INSTANCE: hdbuser NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloEDUSIGCHLDHandler,
probe:50
DATA #1 : <preformatted>
Detected the death of an EDU with process id 141222
The signal number that terminated this process was 9
Look for trap files (t141222.*) in the dump directory
CALLSTCK:
[0] 0xD4141828 sqloEDUSIGCHLDHandler__FiP9siginfo_tPv + 0x220
[1] 0x00000000 ?unknown + 0x0
[2] 0x00000000 ?unknown + 0x0
[3] 0x00000000 ?unknown + 0x0
[4] 0x00000000 ?unknown + 0x0
[5] 0x00000000 ?unknown + 0x0
[6] 0x00000000 ?unknown + 0x0
[7] 0x00000000 ?unknown + 0x0
[8] 0x00000000 ?unknown + 0x0
[9] 0x00000000 ?unknown + 0x0

DB/2 support is no help on this as they just say someone is sending
them the signal. We have no code to send this signal to a DB/2 process
and nobody is killing anything from the command line. Our process is
just running along accessing the database. So, we are stuck unless we
can somehow trace who might be doing this.

Is there something we can set up in DB/2 or AIX that would help us
determine who is sending this signal?

Thanks for any help or suggestions.

[soccertl]

Also, levels are:

AIX 5.3.0.50
DB/2 8.1.1.120 FixPak 13

[Knut Stolze]

soccertl wrote:

We have a program running on AIX and uses DB/2. Every once in a while,
while our program is just running along, DB/2 dies due to a signal 9
being sent to one of the EDU processes:

2006-11-14-12.26.23.442685+000 E396314C794 LEVEL: Severe
PID : 72190 TID : 1 PROC : db2ipccm 0
INSTANCE: hdbuser NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloEDUSIGCHLDHandler,
probe:50
DATA #1 : <preformatted>
Detected the death of an EDU with process id 141222
The signal number that terminated this process was 9
Look for trap files (t141222.*) in the dump directory

Have you such trap files in the dump directory? If so, what's the content?

DB/2 support is no help on this as they just say someone is sending
them the signal.

Right. DB2 (not DB/2!) doesn't send the SIGKILL signal to its processes.
So someone else must be doing this and you should figure out what's going
on in your environment.

--
Knut Stolze
DB2 Information Integration Development
IBM Germany

[soccertl]

Knut Stolze wrote:

>
Have you such trap files in the dump directory? If so, what's the content?

No. Those files do not exist. DB2 support said they would not be there
for some reason I can't remember.

Right. DB2 (not DB/2!) doesn't send the SIGKILL signal to its processes.
So someone else must be doing this and you should figure out what's going
on in your environment.

Right. The problem is trying to figure out who that "someone else" is
since our code doesn't send SIGKILL to DB2 processes either. I was just
wondering if someone had any insight into what I could use to track
this down.

[soccertl]

Knut Stolze wrote:

Right. DB2 (not DB/2!)

And sorry. I was used to the OS/2 world where it was DB2/2 and I have a
hard time removing the '/'. :)

[Larry]

I would ask your UNIX admins whether they have a way to trace who issues
"kill -9" commands.

Larry Edelstein

soccertl wrote:

Knut Stolze wrote:

>>Have you such trap files in the dump directory? If so, what's the content?

No. Those files do not exist. DB2 support said they would not be there
for some reason I can't remember.

>>Right. DB2 (not DB/2!) doesn't send the SIGKILL signal to its processes.
So someone else must be doing this and you should figure out what's going
on in your environment.

Right. The problem is trying to figure out who that "someone else" is
since our code doesn't send SIGKILL to DB2 processes either. I was just
wondering if someone had any insight into what I could use to track
this down.

[Phil Sherman]

A quick & dirty way to do this is to replace the kill command with a
script that logs the date, time, and user then issues the kill command.
If the user doing this is using sudo to execute the kill command, then
you'll need to check the appropriate sudo logs.

Your UNIX administrator should be able to do this in less than 10 min.

Phil Sherman
Larry wrote:

I would ask your UNIX admins whether they have a way to trace who issues
"kill -9" commands.

Larry Edelstein

soccertl wrote:

>Knut Stolze wrote:

>>Have you such trap files in the dump directory? If so, what's the
content?

No. Those files do not exist. DB2 support said they would not be there
for some reason I can't remember.

>>Right. DB2 (not DB/2!) doesn't send the SIGKILL signal to its
processes.
So someone else must be doing this and you should figure out what's
going
on in your environment.

Right. The problem is trying to figure out who that "someone else" is
since our code doesn't send SIGKILL to DB2 processes either. I was just
wondering if someone had any insight into what I could use to track
this down.

[soccertl]

Phil Sherman wrote:

A quick & dirty way to do this is to replace the kill command with a
script that logs the date, time, and user then issues the kill command.
If the user doing this is using sudo to execute the kill command, then
you'll need to check the appropriate sudo logs.

Thanks for the idea. I will check with our AIX install developers.

Your UNIX administrator should be able to do this in less than 10 min.

There is no UNIX administrator. This is a development lab with
development machines. We have developers in charge of the AIX install
but that is as close as we come. I can ask them to look into doing your
suggestion.