shared memory on OS X - 7.4beta4

[John DeSoi]

I get this when I try to start up a freshly compiled beta4 on OS X
10.2.6:

FATAL: could not create shared memory segment: Invalid argument

I saw a previous thread on this for beta2. It sounded like this was a
bug that was to be fixed in beta3, but I'm still having this problem
with beta4.

Thanks,

John DeSoi, Ph.D.
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ma*******@postgresql.org so that your
message can get through to the mailing list cleanly

[Adam Witney]

I don't know what this error is particularly, but just to say that 7.4beta5
compiles and starts up fine on my OSX 10.2.8 machine

Cheers

Adam

I get this when I try to start up a freshly compiled beta4 on OS X
10.2.6:

FATAL: could not create shared memory segment: Invalid argument

I saw a previous thread on this for beta2. It sounded like this was a
bug that was to be fixed in beta3, but I'm still having this problem
with beta4.

Thanks,

John DeSoi, Ph.D.
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ma*******@postgresql.org so that your
message can get through to the mailing list cleanly

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

[Vivek Khera]

>>>>> "JD" == John DeSoi <[e-mail]> writes:

JD> I get this when I try to start up a freshly compiled beta4 on OS X
JD> 10.2.6:

JD> FATAL: could not create shared memory segment: Invalid argument
How much SHM are you requesting (shared_buffers), and does the system
allow for that much? There's some bootup script you can modify in OSX
to up that limit if you need. Check google and/or Postgres' techdocs
site.
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Vivek Khera, Ph.D. Khera Communications, Inc.
Internet: kh***@kciLink.com Rockville, MD [phone]
AIM: vivekkhera Y!: vivek_khera http://www.khera.org/~vivek/

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

[John DeSoi]

On Friday, October 24, 2003, at 11:26 AM, Vivek Khera wrote:

How much SHM are you requesting (shared_buffers), and does the system
allow for that much? There's some bootup script you can modify in OSX
to up that limit if you need. Check google and/or Postgres' techdocs
site.

I did a default configure and did not change anything in the .conf
file. It requests 10444800 bytes. But even if I drop max_connections
down to 10, it still won't start. Yes, I can modify SHMMAX for the
operating system to make it work but I'm trying to understand why I
need to do this when it was not necessary under 7.3 or an earlier 7.4
development build (pre beta 1). I was hoping it is a bug, because I
think it hurts the adoption of Postgresql if I have to worry about a
lot of low level OS settings just to setup a simple database.

Best,

John DeSoi, Ph.D.
---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

[John DeSoi]

On Friday, October 24, 2003, at 10:19 AM, Adam Witney wrote:

I don't know what this error is particularly, but just to say that
7.4beta5
compiles and starts up fine on my OSX 10.2.8 machine

OK, I compiled beta 5 and put my rest SHMMAX back to the default
settings as provided by OS X. Still the same error:

FATAL: could not create shared memory segment: Invalid argument
DETAIL: Failed system call was shmget(key=5432001, size=10444800,
03600).
HINT: This error usually means that PostgreSQL's request for a shared
memory segment exceeded your kernel's SHMMAX parameter. You can either
reduce the request size or reconfigure the kernel with larger SHMMAX.
To reduce the request size (currently 10444800 bytes), reduce
PostgreSQL's shared_buffers parameter (currently 1000) and/or its
max_connections parameter (currently 100).
If the request size is already small, it's possible that it is less
than your kernel's SHMMIN parameter, in which case raising the request
size or reconfiguring SHMMIN is called for.
The PostgreSQL documentation contains more information about shared
memory configuration.

Here is how I configured it:

../configure --prefix=/Unix/postgresql --enable-locale
--without-readline --with-includes=/sw/include --with-libraries=/sw/lib
--with-openssl=/sw/lib
Next I upgraded to 10.2.8. Same issue.

So you have not modified your memory settings in
/System/Library/StartupItems/SystemTuning/SystemTuning and you don't
get this error?

I'm using a PowerBook G4 with 512M of memory.

Thanks,

John DeSoi, Ph.D.
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

[Adam Witney]

On 26/10/03 1:01 am, "John DeSoi" <[e-mail]> wrote:

On Friday, October 24, 2003, at 10:19 AM, Adam Witney wrote:

I don't know what this error is particularly, but just to say that
7.4beta5
compiles and starts up fine on my OSX 10.2.8 machine
OK, I compiled beta 5 and put my rest SHMMAX back to the default
settings as provided by OS X. Still the same error:

FATAL: could not create shared memory segment: Invalid argument
DETAIL: Failed system call was shmget(key=5432001, size=10444800,
03600).
HINT: This error usually means that PostgreSQL's request for a shared
memory segment exceeded your kernel's SHMMAX parameter. You can either
reduce the request size or reconfigure the kernel with larger SHMMAX.
To reduce the request size (currently 10444800 bytes), reduce
PostgreSQL's shared_buffers parameter (currently 1000) and/or its
max_connections parameter (currently 100).
If the request size is already small, it's possible that it is less
than your kernel's SHMMIN parameter, in which case raising the request
size or reconfiguring SHMMIN is called for.
The PostgreSQL documentation contains more information about shared
memory configuration.

Here is how I configured it:

./configure --prefix=/Unix/postgresql --enable-locale
--without-readline --with-includes=/sw/include --with-libraries=/sw/lib
--with-openssl=/sw/lib
Next I upgraded to 10.2.8. Same issue.

So you have not modified your memory settings in
/System/Library/StartupItems/SystemTuning/SystemTuning and you don't
get this error?

Ah, yes I did change these... Mine are set to

sysctl -w kern.sysv.shmmax=4194304
sysctl -w kern.sysv.shmmin=1
sysctl -w kern.sysv.shmmni=32
sysctl -w kern.sysv.shmseg=8
sysctl -w kern.sysv.shmall=1024
I'm using a PowerBook G4 with 512M of memory.

Same as me

Do you have many other applications running? Or an old version of
PostgreSQL... I found when running the regression tests I had to shut down
my 7.3.2 installation

Cheers

adam

adam
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ma*******@postgresql.org so that your
message can get through to the mailing list cleanly

[John DeSoi]

Adam,

On Sunday, October 26, 2003, at 10:16 AM, Adam Witney wrote:

Ah, yes I did change these... Mine are set to

sysctl -w kern.sysv.shmmax=4194304
sysctl -w kern.sysv.shmmin=1
sysctl -w kern.sysv.shmmni=32
sysctl -w kern.sysv.shmseg=8
sysctl -w kern.sysv.shmall=1024

OK, this is what I was after. I did not have to change any of this for
7.3, but now 7.4 does not work unless on a standard OS X installation
unless the user modifies these system settings. I can live with it, but
I was hoping it might be a bug that could be resolved before the final
7.4. Every person that installs on OS X is going to run into this.
Tracking it down and understanding what you have to change is not
simple for us OS X users that are not familiar with Unix :).

Best,

John DeSoi, Ph.D.
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postgresql.org

[Alvaro Herrera]

On Sun, Oct 26, 2003 at 04:41:16PM -0500, John DeSoi wrote:

On Sunday, October 26, 2003, at 10:16 AM, Adam Witney wrote:

Ah, yes I did change these... Mine are set to

sysctl -w kern.sysv.shmmax=4194304
sysctl -w kern.sysv.shmmin=1
sysctl -w kern.sysv.shmmni=32
sysctl -w kern.sysv.shmseg=8
sysctl -w kern.sysv.shmall=1024

OK, this is what I was after. I did not have to change any of this for
7.3, but now 7.4 does not work unless on a standard OS X installation
unless the user modifies these system settings.

Huh, this is strange. IIRC during the installation process, initdb is
supposed to determine which values actually work; those are affected by
the shmmax operating system mostly, and the idea is to enable the
installation process to pick more reasonable values out-of-the-box for
memory usage that will work on your system.

How did you install Postgres, and what values did initdb pick for
shared_buffers and max_connections?

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Puedes vivir solo una vez, pero si lo haces bien, una vez es suficiente"

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ma*******@postgresql.org)

[Adam Witney]

On 26/10/03 9:41 pm, "John DeSoi" <[e-mail]> wrote:

Adam,

On Sunday, October 26, 2003, at 10:16 AM, Adam Witney wrote:

Ah, yes I did change these... Mine are set to

sysctl -w kern.sysv.shmmax=4194304
sysctl -w kern.sysv.shmmin=1
sysctl -w kern.sysv.shmmni=32
sysctl -w kern.sysv.shmseg=8
sysctl -w kern.sysv.shmall=1024

OK, this is what I was after. I did not have to change any of this for
7.3, but now 7.4 does not work unless on a standard OS X installation
unless the user modifies these system settings. I can live with it, but
I was hoping it might be a bug that could be resolved before the final
7.4. Every person that installs on OS X is going to run into this.
Tracking it down and understanding what you have to change is not
simple for us OS X users that are not familiar with Unix :).

Well I had to change this for 7.3 to run and I never set it back to try 7.4
with the defaults

But I think the 7.4 installation process checks for the best values of
max_connections and shared_buffers that for the current settings shmmax
settings

But someone else on the list is probably better to answer the technical
details of this

adam
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ma*******@postgresql.org so that your
message can get through to the mailing list cleanly

[John DeSoi]

On Sunday, October 26, 2003, at 05:51 PM, Alvaro Herrera wrote:

Huh, this is strange. IIRC during the installation process, initdb is
supposed to determine which values actually work; those are affected by
the shmmax operating system mostly, and the idea is to enable the
installation process to pick more reasonable values out-of-the-box for
memory usage that will work on your system.

How did you install Postgres, and what values did initdb pick for
shared_buffers and max_connections?

Here were my steps to install and attempt to run it:

Install (clean install to a new directory)

../configure --prefix=/Unix/postgresql --enable-locale
--without-readline --with-includes=/sw/include --with-libraries=/sw/lib
--with-openssl=/sw/lib
make
make install

Initdb:

initdb -D /data/pgtest --no-locale

Startup:

pg_ctl start -D /data/pgtest -l /data/pgtest/logfile -o '-i -p 5432'
the postgresql.conf has
max_connections = 100
shared_buffers = 1000

And here are are the default settings for OS X 10.2.6:

sysctl -w kern.sysv.shmmax=4194304
sysctl -w kern.sysv.shmmin=1
sysctl -w kern.sysv.shmmni=32
sysctl -w kern.sysv.shmseg=8
sysctl -w kern.sysv.shmall=1024
---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

[Alvaro Herrera Munoz]

On Sun, Oct 26, 2003 at 10:17:50PM -0500, John DeSoi wrote:

On Sunday, October 26, 2003, at 05:51 PM, Alvaro Herrera wrote:

How did you install Postgres, and what values did initdb pick for
shared_buffers and max_connections?

the postgresql.conf has
max_connections = 100
shared_buffers = 1000

And here are are the default settings for OS X 10.2.6:

sysctl -w kern.sysv.shmmax=4194304

Well, I can only say that if I lower shmmax to that value, initdb picks
max_connections = 50
shared_buffers = 300

I remember there was a subtle bug in earlier versions corrected after
beta4. Can you try beta5 and see if it works for you? But with beta4 I
get 400/20 instead of 300/50, so this is probably not the same problem.

1000/100 are actually the largest probed values, so maybe initdb is
failing to probe or the postgresql.conf is not written correctly. Does
the initdb output actually say that it is going to use 1000/100? And is
the generated postgresql.conf by that initdb run the file that gets
actually used?

Maybe this is useless anyway, because I'm on Linux. Someone with an
actual OS X system could probably be a lot more helpful.

--
Alvaro Herrera (<alvherre[@]dcc.uchile.cl>)
"Cuando no hay humildad las personas se degradan" (A. Christie)

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

[John DeSoi]

On Sunday, October 26, 2003, at 11:13 PM, Alvaro Herrera Munoz wrote:

I remember there was a subtle bug in earlier versions corrected after
beta4. Can you try beta5 and see if it works for you? But with beta4
I
get 400/20 instead of 300/50, so this is probably not the same problem.
I mentioned earlier in the thread -- I did try beta5 -- same result.

1000/100 are actually the largest probed values, so maybe initdb is
failing to probe or the postgresql.conf is not written correctly. Does
the initdb output actually say that it is going to use 1000/100? And
is
the generated postgresql.conf by that initdb run the file that gets
actually used?

I don't recall seeing the initdb output, but I'm sure the generated
postgresql.conf is the one being used. I lowered max_connections and
saw lower numbers in the failed request when I tried to start the
server.

Best,

John DeSoi, Ph.D.


#-----------------------------------------------------------------------
----
# CONNECTIONS AND AUTHENTICATION
#-----------------------------------------------------------------------
----

# - Connection Settings -

#tcpip_socket = false
max_connections = 100
# note: increasing max_connections costs about 500 bytes of shared
# memory per connection slot, in addition to costs from shared_buffers
# and max_locks_per_transaction.
#superuser_reserved_connections = 2
#port = 5432
#unix_socket_directory = ''
#unix_socket_group = ''
#unix_socket_permissions = 0777 # octal
#virtual_host = '' # what interface to listen on; defaults to any
#rendezvous_name = '' # defaults to the computer name

# - Security & Authentication -

#authentication_timeout = 60 # 1-600, in seconds
#ssl = false
#password_encryption = true
#krb_server_keyfile = ''
#db_user_namespace = false
#-----------------------------------------------------------------------
----
# RESOURCE USAGE (except WAL)
#-----------------------------------------------------------------------
----

# - Memory -

shared_buffers = 1000 # min 16, at least max_connections*2, 8KB each
#sort_mem = 1024 # min 64, size in KB
#vacuum_mem = 8192
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

[Adam Witney]

On 27/10/03 5:46 am, "John DeSoi" <[e-mail]> wrote:

On Sunday, October 26, 2003, at 11:13 PM, Alvaro Herrera Munoz wrote:

I remember there was a subtle bug in earlier versions corrected after
beta4. Can you try beta5 and see if it works for you? But with beta4
I
get 400/20 instead of 300/50, so this is probably not the same problem.

I mentioned earlier in the thread -- I did try beta5 -- same result.

1000/100 are actually the largest probed values, so maybe initdb is
failing to probe or the postgresql.conf is not written correctly. Does
the initdb output actually say that it is going to use 1000/100? And
is
the generated postgresql.conf by that initdb run the file that gets
actually used?

I don't recall seeing the initdb output, but I'm sure the generated
postgresql.conf is the one being used. I lowered max_connections and
saw lower numbers in the failed request when I tried to start the
server.

My current settings are

max_connections = 32
shared_buffers = 100

But the settings picked automatically by the installation were

max_connections = 50
shared_buffers = 300

But I found this would not always start as I often have lots of applications
open (this is my laptop) and I had trouble running both 7.3 and 7.4beta5 at
the same time

Cheers

Adam
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postgresql.org

[Jim Crate]

>> And here are are the default settings for OS X 10.2.6:

sysctl -w kern.sysv.shmmax=4194304

Well, I can only say that if I lower shmmax to that value, initdb picks
max_connections = 50
shared_buffers = 300

I just did a fresh install of 7.4b5 on a fresh install of MacOS 10.3, and initdb
picked those same values for postgresql.conf. PostgreSQL started up the first
time successfully, but then refused to start again even after restarting the
computer until I reduced shared_buffers.

Now to figure out why sysctl -w kern.sysv.shmmax=<some_reasonable_value> doesn't
seem to work on 10.3...

Jim Crate

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

[Eric Soroos]

Slightly off topic confirmations...

And here are are the default settings for OS X 10.2.6:

sysctl -w kern.sysv.shmmax=4194304

Seems to be the default on 10.2.8 as well.

10.1.5 doesn't seem to have a kern.sysv.* section:

[broccoli:~] erics% sysctl kern.sysv.shmmax
second level name sysv in kern.sysv.shmmax is invalid

Well, I can only say that if I lower shmmax to that value, initdb
picks
max_connections = 50
shared_buffers = 300

I just did a fresh install of 7.4b5 on a fresh install of MacOS 10.3,
and initdb
picked those same values for postgresql.conf. PostgreSQL started up
the first
time successfully, but then refused to start again even after
restarting the
computer until I reduced shared_buffers.

I've had cases in 10.1.x where after hammering on the database for a
while and having to restart it, there was no longer enough shared
memory to start postgres, requiring a restart of the computer. On 10.1,
I couldn't even get it to change the shmmax value, necessitating some
very low buffer numbers (nearly out of the box for pg 7.2, yes, that
bad).

eric
---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

[Vivek Khera]

>>>>> "JC" == Jim Crate <jc****@deepskytech.com> writes:

JC> Now to figure out why sysctl -w
JC> kern.sysv.shmmax=<some_reasonable_value> doesn't seem to work on
JC> 10.3...

I believe you must do it at boot time, not after. ie, it must be done
by one of the startup scripts.
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Vivek Khera, Ph.D. Khera Communications, Inc.
Internet: kh***@kciLink.com Rockville, MD [phone]
AIM: vivekkhera Y!: vivek_khera http://www.khera.org/~vivek/

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

[Tom Lane]

John DeSoi <[e-mail]> writes:

OK, I compiled beta 5 and put my rest SHMMAX back to the default
settings as provided by OS X. Still the same error: FATAL: could not create shared memory segment: Invalid argument
DETAIL: Failed system call was shmget(key=5432001, size=10444800,
03600).
HINT: This error usually means that PostgreSQL's request for a shared
memory segment exceeded your kernel's SHMMAX parameter. You can either
reduce the request size or reconfigure the kernel with larger SHMMAX.
To reduce the request size (currently 10444800 bytes), reduce
PostgreSQL's shared_buffers parameter (currently 1000) and/or its
max_connections parameter (currently 100).

There is something very wrong here. On my OS X machine, initdb selects
non-default shared_buffers and max_connections (300 and 50 respectively,
when using 7.4beta5). Why is that failing to happen on yours? I can
see two possibilities: either initdb is failing to notice that it needs
smaller-than-default values, or it's somehow failing to insert those
values into the constructed postgresql.conf file. You could determine
which is happening by looking at the output of initdb to see what it
reports.

[ thinks for a bit... ] Hmm. You *are* using the postgresql.conf file
that initdb built, aren't you?

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

[John DeSoi]

On Tuesday, October 28, 2003, at 03:13 PM, Tom Lane wrote:

There is something very wrong here. On my OS X machine, initdb selects
non-default shared_buffers and max_connections (300 and 50
respectively,
when using 7.4beta5). Why is that failing to happen on yours? I can
see two possibilities: either initdb is failing to notice that it needs
smaller-than-default values, or it's somehow failing to insert those
values into the constructed postgresql.conf file. You could determine
which is happening by looking at the output of initdb to see what it
reports.
I set the memory parameters back to their defaults, and now: it works.
I'm getting 300 and 50 with initdb. I don't recall what the output form
initdb was from previous attempts, but the .conf file always had 1000
and 100.


[ thinks for a bit... ] Hmm. You *are* using the postgresql.conf file
that initdb built, aren't you?

I have never changed anything in this file (or replaced it). I did not
even know what the shared_buffers parameter was for until I ran into
this problem.

What version of OS X are you running? I thought I replicated the
problem after upgrading to 10.2.8, but now I'm not certain I ran initdb
again (I may have just rebuilt beta 5). So possibly it could be a
difference between 10.2.6 and 10.2.8. If not, I have no clue. I'll
report back if I find anything.

Best,

John DeSoi, Ph.D.
---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

[Tom Lane]

John DeSoi <[e-mail]> writes:

What version of OS X are you running? I thought I replicated the
problem after upgrading to 10.2.8, but now I'm not certain I ran initdb
again (I may have just rebuilt beta 5). So possibly it could be a
difference between 10.2.6 and 10.2.8. If not, I have no clue. I'll
report back if I find anything.

I'm running 10.2.6 (a pretty fresh install, see prior bellyaching about
hardware problems with my laptop ;-)). IIRC, someone else reported
success with a clean 10.2.8 installation in this thread. I have also
checked PG against a 10.3 beta recently, and so have other people.
It's fairly likely that there are problems with 10.1, if anyone still
uses that, but I have no reason to think that PG 7.4 will fail with
either 10.2.* or 10.3.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster