Best way to recover from crash?

[Steve Wampler]

Immediately after finally deploying our PG database up
on our mountain-top observatory, we got hit by lightening.
(The machines were *supposed* to be installed on a RUPS,
but weren't. Sigh)

A file was lost. Now many simple commands cause the
DB to 'crash':
----------------------------------------------------
lab.devel.configdb=# vacuum;
NOTICE: Rel attributes_table: Uninitialized page 60523 - fixing
NOTICE: Rel attributes_table: Uninitialized page 60524 - fixing
NOTICE: Rel attributes_table: Uninitialized page 60525 - fixing
NOTICE: Rel attributes_table: Uninitialized page 60526 - fixing
FATAL 2: open of /var/lib/pgsql/data/pg_clog/0000 failed: No such file
or directory
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: NOTICE:
Message from PostgreSQL backend:
The Postmaster has informed me that some other backend
died abnormally and possibly corrupted shared memory.
I have rolled back the current transaction and am
going to terminate your database system connection and exit.
Please reconnect to the database system and repeat your query.
Failed.
!# \q
----------------------------------------------

I do nightly backups of the DBs so aside from observing time
lost this isn't catestrophic, but since it takes so long to
restore from backup (some of the DBs are fairly large) I was
wondering if there's a 'known procedure' for quickly recovering
from the above.

Thanks!
Steve
--
Steve Wampler -- sw******@noao.edu
The gods that smiled on your birth are now laughing out loud.

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

[Tom Lane]

Steve Wampler <sw******@noao.edu> writes:

Immediately after finally deploying our PG database up
on our mountain-top observatory, we got hit by lightening.
(The machines were *supposed* to be installed on a RUPS,
but weren't. Sigh) A file was lost. Now many simple commands cause the
DB to 'crash': I do nightly backups of the DBs so aside from observing time
lost this isn't catestrophic, but since it takes so long to
restore from backup (some of the DBs are fairly large) I was
wondering if there's a 'known procedure' for quickly recovering
from the above.

I think you'd be foolish not to initdb and reload from the backups.
You have no way to know how extensive the data damage is ... you might
work around the problems you see now, only to find something else
later after you've put more data into the corrupted DB.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

[Steve Wampler]

On Thu, 2004-07-22 at 10:59, Tom Lane wrote:

I think you'd be foolish not to initdb and reload from the backups.
You have no way to know how extensive the data damage is ... you might
work around the problems you see now, only to find something else
later after you've put more data into the corrupted DB.

Thanks, Tom. I did that and what you say makes sense. I was
just overly worried about the down time since the backup database
dump was well over 4GB and looking for a quick fix. On the
other hand, a 1-hour downtime (what it took) isn't so bad,
and knowing things are solid is worth more than that!

Thanks again,
Steve
--
Steve Wampler -- sw******@noao.edu
The gods that smiled on your birth are now laughing out loud.

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match