Swap space size in C

[Otto Meijer]

Hi everyone,

for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do this
on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
OpenVMS and Windows.

This seems much more difficult than I thought origionaly.
Some platforms don't seem to have any API to query this information.

Any hints how to tack this would be very welcome

Otto

[Grumble]

Otto Meijer wrote:

for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do this
on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
OpenVMS and Windows.

This seems much more difficult than I thought origionaly.
Some platforms don't seem to have any API to query this information.

Any hints how to tack this would be very welcome

AFAICT, your best bet is the Portable Operating System Interface for
UNIX, aka POSIX.

[Dan Pop]

In <cd**************************@posting.google.com > cw*******@hotmail.com (Otto Meijer) writes:

for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do this
on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
OpenVMS and Windows.

This seems much more difficult than I thought origionaly.
Some platforms don't seem to have any API to query this information.

Any hints how to tack this would be very welcome

On a platform by platform basis. Post the question to newsgroups
dedicated to programming on each of the platforms of interest to you.
If you get the desired information, write a function for each platform.

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Da*****@ifh.de

[Dan Pop]

In <br**********@news-rocq.inria.fr> Grumble <in*****@kma.eu.org> writes:

Otto Meijer wrote:

for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do this
on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
OpenVMS and Windows.

This seems much more difficult than I thought origionaly.
Some platforms don't seem to have any API to query this information.

Any hints how to tack this would be very welcome

AFAICT, your best bet is the Portable Operating System Interface for
UNIX, aka POSIX.

Which, AFAICT, blissfully ignores the issue.

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Da*****@ifh.de

[CBFalconer]

Otto Meijer wrote:

for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do
this on a host of platforms like: HP_UX, Solaris, Linux, AIX,
OSF, OpenVMS and Windows.

This seems much more difficult than I thought origionaly. Some
platforms don't seem to have any API to query this information.

Any hints how to tack this would be very welcome

Yes. Ask on newsgroups that deal with HP_UX, Solaris, Linux, AIX,
OSF, OpenVMS, and Windows. This group deals with ISO standard
portable C only. If it isn't in the C standard, it doesn't exist
(here).

--
Chuck F (cb********@yahoo.com) (cb********@worldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home.att.net> USE worldnet address!

[Grumble]

Dan Pop wrote:

Grumble wrote:

AFAICT, your best bet is the Portable Operating System Interface
for UNIX, aka POSIX.

Which, AFAICT, blissfully ignores the issue.

Then, it seems, I could not tell far enough ;-)

[Keith Thompson]

cw*******@hotmail.com (Otto Meijer) writes:

Hi everyone,

for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do this
on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
OpenVMS and Windows.

This seems much more difficult than I thought origionaly.
Some platforms don't seem to have any API to query this information.

Any hints how to tack this would be very welcome

Otto

As others have mentioned, this is extremely platform-specific. It may
not even be possible on some platforms. Some platforms may not have a
concept of a "swap file" (it could be a swap partition, for example);
others could have a "swap file" whose size can vary over time, or that
doesn't have a well-defined size at all.

You might want to reconsider whether knowing the "size" of the "swap
file" is even useful. What are you going to do with the information?
(That's a rhetorical question; any answer is probably off-topic here.)

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://www.sdsc.edu/~kst>
Schroedinger does Shakespeare: "To be *and* not to be"
(Note new e-mail address)

[glen herrmannsfeldt]

Otto Meijer wrote:

for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do this
on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
OpenVMS and Windows.

I have done it with an exponentially decreasing malloc() loop,

for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;

There is no guarantee that it gives meaningful results
on any system, and may crash the system.

I did once crash a unix system with a malloc() loop, but
not with this one.

The i-8 is a guess of the malloc() overhead, which may or may
not be related to any real malloc() overhead.

-- glen

[Dan Pop]

In <tOBCb.102332$_M.539115@attbi_s54> glen herrmannsfeldt <ga*@ugcs.caltech.edu> writes:

Otto Meijer wrote:

for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do this
on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
OpenVMS and Windows.

I have done it with an exponentially decreasing malloc() loop,

for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;

There is no guarantee that it gives meaningful results
on any system, and may crash the system.

In particular, it's useless on systems with lazy swap space allocation.

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Da*****@ifh.de

[glen herrmannsfeldt]

Dan Pop wrote:

(snip on determining swap space)

for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j; There is no guarantee that it gives meaningful results
on any system, and may crash the system.

In particular, it's useless on systems with lazy swap space allocation.

It might still tell you the maximum for one task...

My favorite lazy swap story was someone trying to determine how fast the
memory system was, something like:

int i;
char *a,*b;
a=malloc(SIZE);
b=malloc(SIZE);
for(i=0;i<1000000;i++) memcpy(a,b,SIZE);

assuming that SIZE was enough bigger than the cache, so that it would
have to go out to memory.

It turned out that in a lazy swap allocation, all the allocated pages
referenced the same physical page, which was smaller than the cache,
so that it only tested the cache.

-- glen

[Otto Meijer]

glen herrmannsfeldt <ga*@ugcs.caltech.edu> wrote in message news:<C0sDb.401963$ao4.1321249@attbi_s51>...

Dan Pop wrote:

(snip on determining swap space)

for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;There is no guarantee that it gives meaningful results
on any system, and may crash the system.

In particular, it's useless on systems with lazy swap space allocation.

It might still tell you the maximum for one task...

My favorite lazy swap story was someone trying to determine how fast the
memory system was, something like:

int i;
char *a,*b;
a=malloc(SIZE);
b=malloc(SIZE);
for(i=0;i<1000000;i++) memcpy(a,b,SIZE);

assuming that SIZE was enough bigger than the cache, so that it would
have to go out to memory.

It turned out that in a lazy swap allocation, all the allocated pages
referenced the same physical page, which was smaller than the cache,
so that it only tested the cache.

-- glen

Okay everyone,

thanks for some of the pointers, I'm not there yet but getting closer.

Let me first tell you why I wanted this information in the first place.
Our application uses very large files (100G +) and these have to be
procesed in the fastest possible way. We do this my placing a lot of
data in memory and then run our tricks on it. However there are a number
of restrictions that stops us of allocation more and more memory.
We think the following:

1) Address Space of the CPU. (32 bit only allows for 4G range).
2) Process limits.
3) Size of the available swap/page file.

When a malloc fails I would like to print an apropriate error message
why it fails. So we can give the user a meaningfull sugestion how to
improve the job.

Now 1 and 2 are fairly easy to check. Bit 3 is as you already know
quite tricky. So just allocating some memory is not a good option
because we are talking about big systems here and allocates can take
quite some time. Also at the time I need the info there is none left
anyway.

Anway thanks for the info sofar.
Otto

[Dan Pop]

In <C0sDb.401963$ao4.1321249@attbi_s51> glen herrmannsfeldt <ga*@ugcs.caltech.edu> writes:

Dan Pop wrote:

(snip on determining swap space)

for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;There is no guarantee that it gives meaningful results
on any system, and may crash the system.

In particular, it's useless on systems with lazy swap space allocation.

It might still tell you the maximum for one task...

My favorite lazy swap story was someone trying to determine how fast the
memory system was, something like:

int i;
char *a,*b;
a=malloc(SIZE);
b=malloc(SIZE);
for(i=0;i<1000000;i++) memcpy(a,b,SIZE);

assuming that SIZE was enough bigger than the cache, so that it would
have to go out to memory.

It turned out that in a lazy swap allocation, all the allocated pages
referenced the same physical page, which was smaller than the cache,
so that it only tested the cache.

Nonsense. Such a system would be irremediably broken and perfectly
useless. Once you write a value to a memory location, that memory
location must be mapped into your program's address space. If this is
not possible (it can happen in a system with lazy swap space allocation,
due to lack of physical swap space), the behaviour is system specific.
Typical examples are sending a signal to the program or killing *other*
programs, chosen more or less randomly, to obtain the missing swap space.

Although such behaviour may sound crazy, the redeeming merits of lazy
swap space allocation make it the preferred choice of many people.

Back when using a system with only 128 MB of swap space (eager swap
allocation would have rendered it next to useless), I knew that it was
time to kill netscape when I was starting to lose xterms ;-)

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Da*****@ifh.de

[CBFalconer]

Dan Pop wrote:

glen herrmannsfeldt <ga*@ugcs.caltech.edu> writes:

Dan Pop wrote:

(snip on determining swap space)

for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;

There is no guarantee that it gives meaningful results
on any system, and may crash the system.

In particular, it's useless on systems with lazy swap space allocation.

It might still tell you the maximum for one task...

My favorite lazy swap story was someone trying to determine how fast the
memory system was, something like:

int i;
char *a,*b;
a=malloc(SIZE);
b=malloc(SIZE);
for(i=0;i<1000000;i++) memcpy(a,b,SIZE);

assuming that SIZE was enough bigger than the cache, so that it would
have to go out to memory.

It turned out that in a lazy swap allocation, all the allocated pages
referenced the same physical page, which was smaller than the cache,
so that it only tested the cache.

Nonsense. Such a system would be irremediably broken and perfectly
useless. Once you write a value to a memory location, that memory
location must be mapped into your program's address space. If this is
not possible (it can happen in a system with lazy swap space allocation,
due to lack of physical swap space), the behaviour is system specific.
Typical examples are sending a signal to the program or killing *other*
programs, chosen more or less randomly, to obtain the missing swap space.

Although such behaviour may sound crazy, the redeeming merits of lazy
swap space allocation make it the preferred choice of many people.

That system may have built some hidden assumptions in, such as
freshly allocated space contains zeroes, copying zeroes to lazy
allocations does not constitute dirtying, etc. This could result
in some blinding speeds. :-)

--
Chuck F (cb********@yahoo.com) (cb********@worldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home.att.net> USE worldnet address!

[Keith Thompson]

cw*******@hotmail.com (Otto Meijer) writes:

glen herrmannsfeldt <ga*@ugcs.caltech.edu> wrote in message
news:<C0sDb.401963$ao4.1321249@attbi_s51>... [...] Let me first tell you why I wanted this information in the first place.
Our application uses very large files (100G +) and these have to be
procesed in the fastest possible way. We do this my placing a lot of
data in memory and then run our tricks on it. However there are a number
of restrictions that stops us of allocation more and more memory.
We think the following:

1) Address Space of the CPU. (32 bit only allows for 4G range).
2) Process limits.
3) Size of the available swap/page file.

When a malloc fails I would like to print an apropriate error message
why it fails. So we can give the user a meaningfull sugestion how to
improve the job.

Just a thought: as long as you can't read the whole thing into memory
at once, you'll probably get the best performance by reading only as
much as will fit into real memory, avoiding the swap file. It
probably depends on your access patterns and half a dozen other things
that are off-topic here.

<OT>Something like mmap() might be a possibility, apart from the
address space limitation (though it might be worth migrating to a
64-bit CPU).</OT>

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://www.sdsc.edu/~kst>
Schroedinger does Shakespeare: "To be *and* not to be"
(Note new e-mail address)

[glen herrmannsfeldt]

Dan Pop wrote:

In <C0sDb.401963$ao4.1321249@attbi_s51> glen herrmannsfeldt writes:
(snip)

My favorite lazy swap story was someone trying to determine how fast the
memory system was, something like: int i;
char *a,*b;
a=malloc(SIZE);
b=malloc(SIZE);
for(i=0;i<1000000;i++) memcpy(a,b,SIZE); assuming that SIZE was enough bigger than the cache, so that it would
have to go out to memory. It turned out that in a lazy swap allocation, all the allocated pages
referenced the same physical page, which was smaller than the cache,
so that it only tested the cache.

Nonsense. Such a system would be irremediably broken and perfectly
useless. Once you write a value to a memory location, that memory
location must be mapped into your program's address space. If this is
not possible (it can happen in a system with lazy swap space allocation,
due to lack of physical swap space), the behaviour is system specific.
Typical examples are sending a signal to the program or killing *other*
programs, chosen more or less randomly, to obtain the missing swap space. Although such behaviour may sound crazy, the redeeming merits of lazy
swap space allocation make it the preferred choice of many people. Back when using a system with only 128 MB of swap space (eager swap
allocation would have rendered it next to useless), I knew that it was
time to kill netscape when I was starting to lose xterms ;-)

Note that the above program does not write anything to b.

I may not have remembered the actual example right, though.
This one would have to actually allocate a, but it doesn't
ever need to allocate memory for b.

I don't know if they actually overcommit, or just don't bother
actually clearing the memory (usually required for security),
and setting up the tables to describe it.

-- glen