Mysterious Death of postmaster (-9)

[Gregory S. Williamson]

Dear peoples,

We had an oddness today with one of of postgres servers (Dell 2 CPU box running linux) and postgres 7.4. The server was under heavy load (50+ for a 1 minutes spike; about 20 for the 15 minute average) with about 250 connections (we still don't understand the heavy load itself).

Looking in the logs I see:
2004-11-13 13:30:28 LOG: unexpected EOF on client connection
2004-11-13 13:30:40 LOG: unexpected EOF on client connection
2004-11-13 13:38:28 LOG: could not send data to client: Broken pipe
2004-11-13 13:42:15 LOG: server process (PID 30272) was terminated by signal 9
2004-11-13 13:42:16 LOG: terminating any other active server processes
2004-11-13 13:42:16 WARNING: terminating connection because of crash of another
server process

The EOFs are almost certainly Proxool closing connections from the client to the database.

The sysad who was on call today swears he didn't send a kill signal (or anysignal at all) -- suddenly the load dropped off and the server was down. It has restarted normally and shows no signs of being worse for the wear (this is really a read-only db so data corruption chances are minimal, I think).

Just to rule out any internal chances, is there any way this shutdown couldhave been triggered from within postgres itself ? Can anyone construct anyscenarios in which Linux, postgres or proxool could have done this withouthuman intervention ?

I have looked through manuals and some FAQs and newsgroup discussions and my gut feeling is that this can't be from postgres, but I thought I'd ask inthe chance that I am, as is often the case, Unclear On The Concept.

Thanks for any illumination,

Greg Williamson
DBA
GlobeXplorer LLC

ps if this is not the right list please let know what might be an appropriate one. gracias!

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

[Stephan Szabo]

On Sat, 13 Nov 2004, Gregory S. Williamson wrote:

Looking in the logs I see:
2004-11-13 13:30:28 LOG: unexpected EOF on client connection
2004-11-13 13:30:40 LOG: unexpected EOF on client connection
2004-11-13 13:38:28 LOG: could not send data to client: Broken pipe
2004-11-13 13:42:15 LOG: server process (PID 30272) was terminated by signal 9
2004-11-13 13:42:16 LOG: terminating any other active server processes
2004-11-13 13:42:16 WARNING: terminating connection because of crash of another
server process Just to rule out any internal chances, is there any way this shutdown
could have been triggered from within postgres itself ? Can anyone
construct any scenarios in which Linux, postgres or proxool could have
done this without human intervention ?

Is it possible that you ran into the out of memory killer? That's the
most likely thing beyond admin intervention I can think of.
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postgresql.org

[Alvaro Herrera]

On Sat, Nov 13, 2004 at 02:39:38PM -0800, Gregory S. Williamson wrote:

Gregory,

We had an oddness today with one of of postgres servers (Dell 2 CPU box
running linux) and postgres 7.4. The server was under heavy load (50+ for a 1
minutes spike; about 20 for the 15 minute average) with about 250 connections
(we still don't understand the heavy load itself).

Looking in the logs I see:
2004-11-13 13:30:28 LOG: unexpected EOF on client connection
2004-11-13 13:30:40 LOG: unexpected EOF on client connection
2004-11-13 13:38:28 LOG: could not send data to client: Broken pipe
2004-11-13 13:42:15 LOG: server process (PID 30272) was terminated by signal 9

This looks an awful lot like the Linux Out-Of-Memory killer got you.
This happens when the Linux kernel overcommits memory. There is something
about this on the documentation, and has been discussed in the past
here. Please see the archives (www.pgsql.ru; look for "OOM killer" and
"linux overcommit").

Luckily it didn't get your postmaster, as has happenned to other
people ...

--
Alvaro Herrera (<alvherre[@]dcc.uchile.cl>)
"XML!" Exclaimed C++. "What are you doing here? You're not a programming
language."
"Tell that to the people who use me," said XML.

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings