PANIC: btree_split_red o: lost left sibling?

[Andrew Sukow]

Greetings,

Our postgres system crashed and upon restarting it our database had the following errors. The error log was 4.5 gigs which is much larger than usual. We looked online for information about lost left siblings and how to fix the data and not lose the 400 million records we have. Anyone have an idea what's the matter and what the fix is?

LOG: database system was interrupted while in recovery at 2004-08-17 08:59:41 PDT
HINT: This probably means that some data is corrupted and you will have to use the last backup for recovery.
LOG: checkpoint record is at 326/C007B778
LOG: redo record is at 326/BD899570; undo record is at 0/0; shutdown FALSE
LOG: next transaction ID: 46922114; next OID: 133662911
LOG: database system was not properly shut down; automatic recovery in progress
LOG: redo starts at 326/BD899570
PANIC: btree_split_red o: lost left sibling
LOG: startup process (PID 9038) was terminated by signal 6
LOG: aborting startup due to startup process failure

Thanks,

Andrew

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

[Bruce Momjian]

Please provide a PostgreSQL version and operating system information.

---------------------------------------------------------------------------

Andrew Sukow wrote:

Greetings,

Our postgres system crashed and upon restarting it our database had the following errors. The error log was 4.5 gigs which is much larger than usual. We looked online for information about lost left siblings and how to fix the data and not lose the 400 million records we have. Anyone have an idea what's the matter and what the fix is?

LOG: database system was interrupted while in recovery at 2004-08-17 08:59:41 PDT
HINT: This probably means that some data is corrupted and you will have to use the last backup for recovery.
LOG: checkpoint record is at 326/C007B778
LOG: redo record is at 326/BD899570; undo record is at 0/0; shutdown FALSE
LOG: next transaction ID: 46922114; next OID: 133662911
LOG: database system was not properly shut down; automatic recovery in progress
LOG: redo starts at 326/BD899570
PANIC: btree_split_red o: lost left sibling
LOG: startup process (PID 9038) was terminated by signal 6
LOG: aborting startup due to startup process failure

Thanks,

Andrew

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

--
Bruce Momjian | http://candle.pha.pa.us
pg***@candle.ph a.pa.us | [phone]
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

[Tom Lane]

Andrew Sukow <cr***@shaw.c a> writes:

Our postgres system crashed and upon restarting it our database had the following errors. The error log was 4.5 gigs which is much larger than usual. We looked online for information about lost left siblings and how to fix the data and not lose the 400 million records we have. Anyone have an idea what's the matter and what the fix is? PANIC: btree_split_red o: lost left sibling

Looking at the code, the most probable explanation seems to be that the
WAL log contains a reference to a btree page that doesn't exist on disk
(ie, the index file on disk is too short to contain that page number).
The code is panicing because it expects that page should exist already.
I have to agree with it --- it would seem you are suffering from
filesystem misfeasance. Are you close to being out of disk space
by any chance?

What I would suggest doing is modifying the error message (it's in
src/backend/access/nbtree/nbtxlog.c, about line 256 in 7.4) to report
the index's DB/relfileno and the block number it's failing to access.
Or if you built with debug enabled, you could gdb the core dump and
extract those numbers that way. Knowing the file and the length it
needs to be, you could append zeroes to the file to make it long enough,
and then the replay should succeed.

A quicker-and-dirtier solution is to pass extend = true instead of false
to the XLogReadBuffer just above this, but I counsel doing the file
extensions manually as sketched above, so that you will know exactly
which index(es) have got this problem. If I were doing this I would
certainly want to manually REINDEX those indexes afterwards. The
specific page that's being requested will be filled in correctly from
the WAL entry, but who knows what else is wrong elsewhere in the index?

BTW, what do you mean by "the error log was 4.5 gigs"? What you showed
us was only 10 lines.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org