473,765 Members | 2,081 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

FATAL: the database system is in recovery mode

We're developing a java app and are using postgres as the database. On
our dev server I started the app, closed it, but the java process was
still open so I killed it, which caused the above error. I've had to do
this in the past but have not had this happen before. I've searched the
archives and found a message/reply from Andrew Sullivan about this which
implies that it may be more a file system problem. We're using
postgresql 7.4.1 on a debian box (and yes, ext2). And fsync is turned on.

Aside from not killing processes which hang, is there anything I can do
to avoid this problem in the future?

Thanks
Ron
---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Nov 23 '05 #1
2 15301
Doug McNaught wrote:
Ron St-Pierre <rs*******@sysc or.com> writes:
We're developing a java app and are using postgres as the database. On
our dev server I started the app, closed it, but the java process was
still open so I killed it, which caused the above error. I've had to
do this in the past but have not had this happen before. I've searched
the archives and found a message/reply from Andrew Sullivan about this
which implies that it may be more a file system problem. We're using
postgresql 7.4.1 on a debian box (and yes, ext2). And fsync is turned
on.


Killing a client process shouldn't cause fatal errors on the
server--the most you should get is "client unexpectedly closed
connection". Something else is wrong.
Aside from not killing processes which hang, is there anything I can
do to avoid this problem in the future?


What is the actual problem with Postgres? Did the server die? Is it
refusing to start up? If you can give more complete logs that would
be helpful.

-Doug

It looks as if it hung up trying to write some customer preferences to
the db. Here's the snippet from /var/log/messages:

Jul 7 09:32:58 imp postgres[32351]: [550-1] LOG: statement: UPDATE
customer.client Preferences SET preferences='<? xml version="1.0"
encoding="UTF-8"?>
Jul 7 09:32:58 imp postgres[32351]: [550-2] ^I<java version="1.4.2_ 03"
class="java.bea ns.XMLDecoder">
Jul 7 09:32:58 imp postgres[32351]: [550-3] ^I <object
class="java.uti l.HashMap">
Jul 7 09:32:58 imp postgres[32351]: [550-4] ^I <void method="put">

..... thousand(s) lines of XML ...

Jul 7 09:32:58 imp postgres[32351]: [550-1626] ^I <void
method="put">
Jul 7 09:32:58 imp postgres[26784]: [448-1] LOG: server process (PID
32349) was terminated by signal 11
Jul 7 09:32:58 imp postgres[26784]: [449-1] LOG: terminating any other
active server processes
Jul 7 09:32:58 imp postgres[32352]: [567-1] WARNING: terminating
connection because of crash of another server process
Jul 7 09:32:58 imp postgres[32350]: [482-1] WARNING: terminating
connection because of crash of another server process
Jul 7 09:32:58 imp postgres[32348]: [450-1] WARNING: terminating
connection because of crash of another server process
Jul 7 09:32:58 imp postgres[32352]: [567-2] DETAIL: The postmaster has
commanded this server process to roll back the current transaction and
exit, because another server
Jul 7 09:32:58 imp postgres[32352]: [567-3] process exited abnormally
and possibly corrupted shared memory.
Jul 7 09:32:58 imp postgres[32350]: [482-2] DETAIL: The postmaster has
commanded this server process to roll back the current transaction and
exit, because another server
Jul 7 09:32:58 imp postgres[32348]: [450-2] DETAIL: The postmaster has
commanded this server process to roll back the current transaction and
exit, because another server
Jul 7 09:32:58 imp postgres[32352]: [567-4] HINT: In a moment you
should be able to reconnect to the database and repeat your command.
Jul 7 09:32:58 imp postgres[32350]: [482-3] process exited abnormally
and possibly corrupted shared memory.
Jul 7 09:32:58 imp postgres[32350]: [482-4] HINT: In a moment you
should be able to reconnect to the database and repeat your command.
Jul 7 09:32:58 imp postgres[32348]: [450-3] process exited abnormally
and possibly corrupted shared memory.
Jul 7 09:32:58 imp postgres[32348]: [450-4] HINT: In a moment you
should be able to reconnect to the database and repeat your command.
Jul 7 09:32:58 imp postgres[32351]: [551-1] ERROR: terminating
connection because of crash of another server process
^ ^
^ ^ ^
Jul 7 09:32:58 imp postgres[32351]: [551-2] DETAIL: The postmaster has
commanded this server process to roll back the current transaction and
exit, because another server
Jul 7 09:32:58 imp postgres[32351]: [551-3] process exited abnormally
and possibly corrupted shared memory.
Jul 7 09:32:58 imp postgres[32351]: [551-4] HINT: In a moment you
should be able to reconnect to the database and repeat your command.
Jul 7 09:32:58 imp postgres[19223]: [432-1] WARNING: terminating
connection because of crash of another server process
Jul 7 09:32:58 imp postgres[19223]: [432-2] DETAIL: The postmaster has
commanded this server process to roll back the current transaction and
exit, because another server
... more ...
Jul 7 09:32:58 imp postgres[3802]: [295-3] process exited abnormally
and possibly corrupted shared memory.
Jul 7 09:32:58 imp postgres[3802]: [295-4] HINT: In a moment you
should be able to reconnect to the database and repeat your command.
Jul 7 10:02:24 imp -- MARK --
Jul 7 10:22:24 imp -- MARK --
Jul 7 10:42:24 imp -- MARK --
Jul 7 11:02:24 imp -- MARK --
Jul 7 11:22:24 imp -- MARK --
Jul 7 11:42:24 imp -- MARK --
Jul 7 12:02:24 imp -- MARK --
Jul 7 12:22:24 imp -- MARK --
Jul 7 12:42:24 imp -- MARK --
Jul 7 13:02:24 imp -- MARK --
Jul 7 13:22:24 imp -- MARK --
Jul 7 13:42:24 imp -- MARK --
Jul 7 14:00:36 imp postgres[32351]: [552-1] LOG: could not receive
data from client: Connection reset by peer
Jul 7 14:00:36 imp postgres[32351]: [553-1] LOG: unexpected EOF on
client connection
Jul 7 14:00:36 imp postgres[26784]: [450-1] LOG: all server processes
terminated; reinitializing
Jul 7 14:00:36 imp postgres[7962]: [451-1] LOG: database system was
interrupted at 2004-07-07 09:31:44 PDT
Jul 7 14:00:36 imp postgres[7962]: [452-1] LOG: checkpoint record is
at 5E/7DB31300
Jul 7 14:00:36 imp postgres[7962]: [453-1] LOG: redo record is at
5E/7DB31300; undo record is at 0/0; shutdown FALSE
Jul 7 14:00:36 imp postgres[7962]: [454-1] LOG: next transaction ID:
21787900; next OID: 125814322
Jul 7 14:00:36 imp postgres[7962]: [455-1] LOG: database system was
not properly shut down; automatic recovery in progress
Jul 7 14:00:36 imp postgres[7962]: [456-1] LOG: record with zero
length at 5E/7DB31340
Jul 7 14:00:36 imp postgres[7962]: [457-1] LOG: redo is not required
Jul 7 14:00:38 imp postgres[7962]: [458-1] LOG: database system is ready

^ ^ ^ ^
I just checked it again, and as you can see from the logs it has
restarted. Looking at the times in the logs, it crashed at 09:32:58 and
restarted at 14:00:36, but there's nothing scheduled in cron that I can
see which would cause the restart. Odd. Normally I would have restarted
it manually with pg_ctl but I wanted to see what would happen so I left it.

Anyway my original question still stands, any idea how to avoid this
problem in the future?

Thanks
Ron
---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

Nov 23 '05 #2
Ron St-Pierre <rs*******@sysc or.com> writes:
Jul 7 09:32:58 imp postgres[26784]: [448-1] LOG: server process (PID
32349) was terminated by signal 11
This is the critical event--one of the backends got a segfault. This
often means bad hardware--either your RAM is faulty, or something got
corrupted on disk and caused a bad pointer reference. Or maybe an
overheating CPU, if a fan failed.

That said, it's also possible that it's a Postgres bug, but it's more
likely to be hardware...
Anyway my original question still stands, any idea how to avoid this
problem in the future?


Test your hardware and replace any bad components.

-Doug

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ma*******@postg resql.org so that your
message can get through to the mailing list cleanly

Nov 23 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
2502
by: Officer | last post by:
I want to setup standby database which reside on the same host with my primary database -dbSA is my primary and dbSB is my standby database. I select v$log as follows: SVRMGR> select thread#,sequence#,archived,status from v$log; THREAD# SEQUENCE# ARC STATUS ---------- ---------- --- ---------------- 1 35 YES INACTIVE 1 36 YES INACTIVE
4
11613
by: serge | last post by:
Running SQL Server 2000 Enterprise Edition SP3. The database is also used by Microsoft Project Server 2002 and also has OLAP views, so the database is being used to view/run cubes in the Analysis Manager. What is the best way of shrinking the database size and its log file too? Is there an automatic way to do this with a maintenance plan or i have to manually run a SQL statement periodically? What are the best practices?
2
2109
by: Matt | last post by:
I run SQL Server 2000 and use thier database maintenance plans to backup my databases. My questions is what happens if a change is being made to a database table while a backup is running? Should I be locking the databases before the backup begins? Scenario: 1) Database Plan begins backup at 7:00 PM. 2) At 7:01 PM a web user updates their password while the backup is taking place.
1
2339
by: pginfo | last post by:
Hi, I have had two times problem with pg database corruption. My system: Dual athlon 2.4 1 GB RAM, linux red hat 7.3 reiserFS Also on the system I have cron that every 3 h. make dump and every 24 h.
7
7911
by: Network Administrator | last post by:
Looks like for the first time in 6 years, I'm experienced some database table corruption. This was due to the space filling up on a server (you don't want to know how that happened). I have 3 tables corrupt and the others are fine (which I dumped to be safe). I have a backup which I could use but then I realized that maybe there might be some "surgery" I could perform to get the table "repaired". Note that the normal recovery that the...
51
5086
by: John Baker | last post by:
Hi: Some time ago I developed a program in Access, and separated the database and the program itself (using the normal access tools).We have the db on our server and the programin the desktop system. I have modified the program a number of times, and now find that I need to change the DB slightlt. This appears to require that I REMERGE the data base and program, and I have no idea how to do that. Can someone give me some pointers,...
0
2061
by: Greg | last post by:
Hi, creating template1 database in /var/pgsql/data1/base/1... DEBUG: invoking IpcMemoryCreate(size=1081344) FATAL: shmat(id=1179648) failed: Invalid argument What's the problem why is shmat passed a wrong id? I temporarily need Postgres running on Windows98 just to write a client app and I'm stuck at this point.
2
5421
by: Scott Jones | last post by:
Hello, In an attempt to save disk space, I went in and gzipped all of the log files for my database running DB2 version 7.2 . This resulted in the database being marked bad because a log file was no longer found (I think). Here is an excerpt from db2diag.log: String Title:sqleMarkDBad: PID:7980 Node:000
5
3490
by: Roger | last post by:
backup log testdb with truncate_only DBCC SHRINKFILE (testdb_log, 100) WITH NO_INFOMSGS backup database testdb to disk = '\\DC01\Backups\DB01\testdb.bak' with init and does the shrinkfile command reduce the size of the ldf ?
0
9398
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10156
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10007
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8831
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
6649
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5275
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5419
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3924
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
3531
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.