Another Bizzare Crash...But no Recovery. SOS!!

Okonita

Hello Gurus,
At about 11:00 am this morning, tried to connect to an instance of DB2 UDB v8.
2 fixpak 14 on a Red Hat Linux environment. The message returned say DB2
alias not found. Alarmed, we researched extensively and found that all data
and files in the /sqllib is gone...vamosed!!
Called system folks for help and we are still trying to recover as I write.
Question: (1) Anyone faced this kind of situation before and how did they
recover? Please share as much detail as you have to resolve similar problem.

(2) How can I b egin to research what happened, what agent or processes wiped
out the SQLLIB library?
(3) How can I prevent this type of event from happening again?

If you need any information, please post and I will see if I can still get
the information for you...please this is a serious matter as I am being
"beaten up" with ugly stares and decidedly subdued hostility by my users!!

Okonita

Aug 21 '07 #1

Serge Rielau

Okonita wrote:

Hello Gurus,
At about 11:00 am this morning, tried to connect to an instance of DB2 UDB v8.
2 fixpak 14 on a Red Hat Linux environment. The message returned say DB2
alias not found. Alarmed, we researched extensively and found that all data
and files in the /sqllib is gone...vamosed!!
Called system folks for help and we are still trying to recover as I write.
Question: (1) Anyone faced this kind of situation before and how did they
recover? Please share as much detail as you have to resolve similar problem.

(2) How can I b egin to research what happened, what agent or processes wiped
out the SQLLIB library?
(3) How can I prevent this type of event from happening again?

I'm no expert in file systems, so I keep my mouth shut on possible
technical causes,
but could it be human error or worse.. sabotage?
We had a really nasty PMR like this years ago. DB2 files magically
disappeared every few days. In the end a keystroke-tracker was installed
and the problem was tracked to a disgruntled employee who didn't like
the switch to DB2. Quite embarassing and rather criminal....
--
Serge Rielau
DB2 Solutions Development
IBM Toronto Lab

Aug 22 '07 #2

Knut Stolze

Okonita wrote:

Hello Gurus,
At about 11:00 am this morning, tried to connect to an instance of DB2 UDB
v8. 2 fixpak 14 on a Red Hat Linux environment. The message returned say
DB2 alias not found. Alarmed, we researched extensively and found that all
data and files in the /sqllib is gone...vamosed!!
Called system folks for help and we are still trying to recover as I
write. Question:
(1) Anyone faced this kind of situation before and how
did they recover? Please share as much detail as you have to resolve
similar problem.

One way to recover would be to run "db2iupdt" for this instance. That will
reconstruct most files in the sqllib/ directory. What will be messing is
the database/DCS directory and other config files like db2cli.ini.

--
Knut Stolze
DB2 z/OS Utilities Development
IBM Germany

Aug 22 '07 #3

aj

I had a new, overzealous sysadmin notice that the home dir of the DB2
instance owner was much larger than all the rest, and more disk was
needed there - so he moved everything (including ./sqllib) someplace else.

The engine was not pleased. I got a call on vacation in Denver at
7:30AM in my hotel room. ah.....the life of a DBA...

As I recall, we moved everything back, restarted, and I think everything
was fine...

aj

PS - was ./sqllib possibly a symlink to somewhere else, and the symlink
got clobbered? Was it an NFS mount point?
Okonita wrote:

Hello Gurus,
At about 11:00 am this morning, tried to connect to an instance of DB2 UDB v8.
2 fixpak 14 on a Red Hat Linux environment. The message returned say DB2
alias not found. Alarmed, we researched extensively and found that all data
and files in the /sqllib is gone...vamosed!!
Called system folks for help and we are still trying to recover as I write.
Question: (1) Anyone faced this kind of situation before and how did they
recover? Please share as much detail as you have to resolve similar problem.

(2) How can I b egin to research what happened, what agent or processes wiped
out the SQLLIB library?
(3) How can I prevent this type of event from happening again?

If you need any information, please post and I will see if I can still get
the information for you...please this is a serious matter as I am being
"beaten up" with ugly stares and decidedly subdued hostility by my users!!

Okonita

Aug 22 '07 #4

Okonita via DBMonster.com

Knut Stolze wrote:

>Hello Gurus,
At about 11:00 am this morning, tried to connect to an instance of DB2 UDB

[quoted text clipped - 6 lines]

>did they recover? Please share as much detail as you have to resolve
similar problem.

One way to recover would be to run "db2iupdt" for this instance. That will
reconstruct most files in the sqllib/ directory. What will be messing is
the database/DCS directory and other config files like db2cli.ini.

Knutz, thanks for chipping in on my problem. I appreciate this.
...And if you are not able to invoke the CLP to run db2iupdt? We are not able
to do that either for this instance and can't use the command center because
it is not available also.

Assuming I can run db2iupdt, how then can I recover the database DCSand other
config files like db2cli.ini?

--
Message posted via http://www.dbmonster.com

Aug 23 '07 #5

Okonita via DBMonster.com

aj wrote:

>I had a new, overzealous sysadmin notice that the home dir of the DB2
instance owner was much larger than all the rest, and more disk was
needed there - so he moved everything (including ./sqllib) someplace else.

The engine was not pleased. I got a call on vacation in Denver at
7:30AM in my hotel room. ah.....the life of a DBA...

As I recall, we moved everything back, restarted, and I think everything
was fine...

aj

PS - was ./sqllib possibly a symlink to somewhere else, and the symlink
got clobbered? Was it an NFS mount point?

>Hello Gurus,
At about 11:00 am this morning, tried to connect to an instance of DB2 UDB v8.

[quoted text clipped - 14 lines]

>>
Okonita

aj, my thanks for chipping in goes to you also. I am new in this environment
and don't know all the subtleties of our Red Hat linux environment. What I
can do is look into answering your questions in the morning as I am not able
to access our production system (Linux) from home in the state that it is now.
...And if the ./sqllib is a symlink to somewhere else or it is a NFS mount
point, what nugget of technical wisdom can you share with me?
As you know, I am much oblidged for your discussion...

okonita

--
Message posted via DBMonster.com
http://www.dbmonster.com/Uwe/Forums....m-db2/200708/1

Aug 23 '07 #6

Knut Stolze

Okonita via DBMonster.com wrote:

..And if you are not able to invoke the CLP to run db2iupdt? We are not
able to do that either for this instance and can't use the command center
because it is not available also.

You have to run "db2iupdt" as root/Administrator from a regular command
shell. On Linux, you will find this tool under /opt/IBM/db2/V9.1/instance/

Assuming I can run db2iupdt, how then can I recover the database DCSand
other config files like db2cli.ini?

You will have to catalog all databases again. Once the base sqllib/
directory is reconstructed with "db2iupdt", just do:

$ db2 catalog db dcsand

(You may need additional parameters if the database doesn't reside in the
default database path.)

As for the config files, I don't know if there is any support to recover
those. If not, you have a backup, I hope?

--
Knut Stolze
DB2 z/OS Utilities Development
IBM Germany

Aug 23 '07 #7

aj

Okonita via DBMonster.com wrote:

aj, my thanks for chipping in goes to you also. I am new in this

environment

and don't know all the subtleties of our Red Hat linux environment. What I
can do is look into answering your questions in the morning as I am not able
to access our production system (Linux) from home in the state that it is now.
..And if the ./sqllib is a symlink to somewhere else or it is a NFS mount
point, what nugget of technical wisdom can you share with me?

If this was indeed the case, you'll need to figure out where that
somewhere else is, and reestablish the symlink that points at it.
Try a
find . -name sqllib -xdev
from the root dir.

I have seen some *very* weird stuff occur w/ NFS in Red Hat. Assuming
NFS, perhaps NFS file mounts were unmounted somehow? Look in
/etc/fstab. This is less likely than a symlink..

Good luck.

cheers
aj

Aug 23 '07 #8