By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
446,238 Members | 1,804 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 446,238 IT Pros & Developers. It's quick & easy.

Swap space size in C

P: n/a
Hi everyone,

for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do this
on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
OpenVMS and Windows.

This seems much more difficult than I thought origionaly.
Some platforms don't seem to have any API to query this information.

Any hints how to tack this would be very welcome

Otto
Nov 14 '05 #1
Share this Question
Share on Google+
14 Replies


P: n/a
Otto Meijer wrote:
for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do this
on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
OpenVMS and Windows.

This seems much more difficult than I thought origionaly.
Some platforms don't seem to have any API to query this information.

Any hints how to tack this would be very welcome


AFAICT, your best bet is the Portable Operating System Interface for
UNIX, aka POSIX.

Nov 14 '05 #2

P: n/a
In <cd**************************@posting.google.com > cw*******@hotmail.com (Otto Meijer) writes:
for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do this
on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
OpenVMS and Windows.

This seems much more difficult than I thought origionaly.
Some platforms don't seem to have any API to query this information.

Any hints how to tack this would be very welcome


On a platform by platform basis. Post the question to newsgroups
dedicated to programming on each of the platforms of interest to you.
If you get the desired information, write a function for each platform.

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Da*****@ifh.de
Nov 14 '05 #3

P: n/a
In <br**********@news-rocq.inria.fr> Grumble <in*****@kma.eu.org> writes:
Otto Meijer wrote:
for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do this
on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
OpenVMS and Windows.

This seems much more difficult than I thought origionaly.
Some platforms don't seem to have any API to query this information.

Any hints how to tack this would be very welcome


AFAICT, your best bet is the Portable Operating System Interface for
UNIX, aka POSIX.


Which, AFAICT, blissfully ignores the issue.

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Da*****@ifh.de
Nov 14 '05 #4

P: n/a
Otto Meijer wrote:

for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do
this on a host of platforms like: HP_UX, Solaris, Linux, AIX,
OSF, OpenVMS and Windows.

This seems much more difficult than I thought origionaly. Some
platforms don't seem to have any API to query this information.

Any hints how to tack this would be very welcome


Yes. Ask on newsgroups that deal with HP_UX, Solaris, Linux, AIX,
OSF, OpenVMS, and Windows. This group deals with ISO standard
portable C only. If it isn't in the C standard, it doesn't exist
(here).

--
Chuck F (cb********@yahoo.com) (cb********@worldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home.att.net> USE worldnet address!
Nov 14 '05 #5

P: n/a
Dan Pop wrote:
Grumble wrote:
AFAICT, your best bet is the Portable Operating System Interface
for UNIX, aka POSIX.


Which, AFAICT, blissfully ignores the issue.


Then, it seems, I could not tell far enough ;-)

Nov 14 '05 #6

P: n/a
cw*******@hotmail.com (Otto Meijer) writes:
Hi everyone,

for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do this
on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
OpenVMS and Windows.

This seems much more difficult than I thought origionaly.
Some platforms don't seem to have any API to query this information.

Any hints how to tack this would be very welcome

Otto


As others have mentioned, this is extremely platform-specific. It may
not even be possible on some platforms. Some platforms may not have a
concept of a "swap file" (it could be a swap partition, for example);
others could have a "swap file" whose size can vary over time, or that
doesn't have a well-defined size at all.

You might want to reconsider whether knowing the "size" of the "swap
file" is even useful. What are you going to do with the information?
(That's a rhetorical question; any answer is probably off-topic here.)

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://www.sdsc.edu/~kst>
Schroedinger does Shakespeare: "To be *and* not to be"
(Note new e-mail address)
Nov 14 '05 #7

P: n/a
Otto Meijer wrote:

for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do this
on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
OpenVMS and Windows.


I have done it with an exponentially decreasing malloc() loop,

for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;

There is no guarantee that it gives meaningful results
on any system, and may crash the system.

I did once crash a unix system with a malloc() loop, but
not with this one.

The i-8 is a guess of the malloc() overhead, which may or may
not be related to any real malloc() overhead.

-- glen

Nov 14 '05 #8

P: n/a
In <tOBCb.102332$_M.539115@attbi_s54> glen herrmannsfeldt <ga*@ugcs.caltech.edu> writes:
Otto Meijer wrote:

for one of my projects I need to figure out the size of the
swap file(s) of a certain system. The problem is I need to do this
on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
OpenVMS and Windows.


I have done it with an exponentially decreasing malloc() loop,

for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;

There is no guarantee that it gives meaningful results
on any system, and may crash the system.


In particular, it's useless on systems with lazy swap space allocation.

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Da*****@ifh.de
Nov 14 '05 #9

P: n/a
Dan Pop wrote:

(snip on determining swap space)
for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j; There is no guarantee that it gives meaningful results
on any system, and may crash the system.
In particular, it's useless on systems with lazy swap space allocation.


It might still tell you the maximum for one task...

My favorite lazy swap story was someone trying to determine how fast the
memory system was, something like:

int i;
char *a,*b;
a=malloc(SIZE);
b=malloc(SIZE);
for(i=0;i<1000000;i++) memcpy(a,b,SIZE);

assuming that SIZE was enough bigger than the cache, so that it would
have to go out to memory.

It turned out that in a lazy swap allocation, all the allocated pages
referenced the same physical page, which was smaller than the cache,
so that it only tested the cache.

-- glen

Nov 14 '05 #10

P: n/a
glen herrmannsfeldt <ga*@ugcs.caltech.edu> wrote in message news:<C0sDb.401963$ao4.1321249@attbi_s51>...
Dan Pop wrote:

(snip on determining swap space)
for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;There is no guarantee that it gives meaningful results
on any system, and may crash the system.

In particular, it's useless on systems with lazy swap space allocation.


It might still tell you the maximum for one task...

My favorite lazy swap story was someone trying to determine how fast the
memory system was, something like:

int i;
char *a,*b;
a=malloc(SIZE);
b=malloc(SIZE);
for(i=0;i<1000000;i++) memcpy(a,b,SIZE);

assuming that SIZE was enough bigger than the cache, so that it would
have to go out to memory.

It turned out that in a lazy swap allocation, all the allocated pages
referenced the same physical page, which was smaller than the cache,
so that it only tested the cache.

-- glen

Okay everyone,

thanks for some of the pointers, I'm not there yet but getting closer.

Let me first tell you why I wanted this information in the first place.
Our application uses very large files (100G +) and these have to be
procesed in the fastest possible way. We do this my placing a lot of
data in memory and then run our tricks on it. However there are a number
of restrictions that stops us of allocation more and more memory.
We think the following:

1) Address Space of the CPU. (32 bit only allows for 4G range).
2) Process limits.
3) Size of the available swap/page file.

When a malloc fails I would like to print an apropriate error message
why it fails. So we can give the user a meaningfull sugestion how to
improve the job.

Now 1 and 2 are fairly easy to check. Bit 3 is as you already know
quite tricky. So just allocating some memory is not a good option
because we are talking about big systems here and allocates can take
quite some time. Also at the time I need the info there is none left
anyway.

Anway thanks for the info sofar.
Otto
Nov 14 '05 #11

P: n/a
In <C0sDb.401963$ao4.1321249@attbi_s51> glen herrmannsfeldt <ga*@ugcs.caltech.edu> writes:
Dan Pop wrote:

(snip on determining swap space)
for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;There is no guarantee that it gives meaningful results
on any system, and may crash the system.

In particular, it's useless on systems with lazy swap space allocation.


It might still tell you the maximum for one task...

My favorite lazy swap story was someone trying to determine how fast the
memory system was, something like:

int i;
char *a,*b;
a=malloc(SIZE);
b=malloc(SIZE);
for(i=0;i<1000000;i++) memcpy(a,b,SIZE);

assuming that SIZE was enough bigger than the cache, so that it would
have to go out to memory.

It turned out that in a lazy swap allocation, all the allocated pages
referenced the same physical page, which was smaller than the cache,
so that it only tested the cache.


Nonsense. Such a system would be irremediably broken and perfectly
useless. Once you write a value to a memory location, that memory
location must be mapped into your program's address space. If this is
not possible (it can happen in a system with lazy swap space allocation,
due to lack of physical swap space), the behaviour is system specific.
Typical examples are sending a signal to the program or killing *other*
programs, chosen more or less randomly, to obtain the missing swap space.

Although such behaviour may sound crazy, the redeeming merits of lazy
swap space allocation make it the preferred choice of many people.

Back when using a system with only 128 MB of swap space (eager swap
allocation would have rendered it next to useless), I knew that it was
time to kill netscape when I was starting to lose xterms ;-)

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Da*****@ifh.de
Nov 14 '05 #12

P: n/a
Dan Pop wrote:
glen herrmannsfeldt <ga*@ugcs.caltech.edu> writes:
Dan Pop wrote:

(snip on determining swap space)
for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;

There is no guarantee that it gives meaningful results
on any system, and may crash the system.

In particular, it's useless on systems with lazy swap space allocation.


It might still tell you the maximum for one task...

My favorite lazy swap story was someone trying to determine how fast the
memory system was, something like:

int i;
char *a,*b;
a=malloc(SIZE);
b=malloc(SIZE);
for(i=0;i<1000000;i++) memcpy(a,b,SIZE);

assuming that SIZE was enough bigger than the cache, so that it would
have to go out to memory.

It turned out that in a lazy swap allocation, all the allocated pages
referenced the same physical page, which was smaller than the cache,
so that it only tested the cache.


Nonsense. Such a system would be irremediably broken and perfectly
useless. Once you write a value to a memory location, that memory
location must be mapped into your program's address space. If this is
not possible (it can happen in a system with lazy swap space allocation,
due to lack of physical swap space), the behaviour is system specific.
Typical examples are sending a signal to the program or killing *other*
programs, chosen more or less randomly, to obtain the missing swap space.

Although such behaviour may sound crazy, the redeeming merits of lazy
swap space allocation make it the preferred choice of many people.


That system may have built some hidden assumptions in, such as
freshly allocated space contains zeroes, copying zeroes to lazy
allocations does not constitute dirtying, etc. This could result
in some blinding speeds. :-)

--
Chuck F (cb********@yahoo.com) (cb********@worldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home.att.net> USE worldnet address!
Nov 14 '05 #13

P: n/a
cw*******@hotmail.com (Otto Meijer) writes:
glen herrmannsfeldt <ga*@ugcs.caltech.edu> wrote in message
news:<C0sDb.401963$ao4.1321249@attbi_s51>... [...] Let me first tell you why I wanted this information in the first place.
Our application uses very large files (100G +) and these have to be
procesed in the fastest possible way. We do this my placing a lot of
data in memory and then run our tricks on it. However there are a number
of restrictions that stops us of allocation more and more memory.
We think the following:

1) Address Space of the CPU. (32 bit only allows for 4G range).
2) Process limits.
3) Size of the available swap/page file.

When a malloc fails I would like to print an apropriate error message
why it fails. So we can give the user a meaningfull sugestion how to
improve the job.


Just a thought: as long as you can't read the whole thing into memory
at once, you'll probably get the best performance by reading only as
much as will fit into real memory, avoiding the swap file. It
probably depends on your access patterns and half a dozen other things
that are off-topic here.

<OT>Something like mmap() might be a possibility, apart from the
address space limitation (though it might be worth migrating to a
64-bit CPU).</OT>

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://www.sdsc.edu/~kst>
Schroedinger does Shakespeare: "To be *and* not to be"
(Note new e-mail address)
Nov 14 '05 #14

P: n/a
Dan Pop wrote:
In <C0sDb.401963$ao4.1321249@attbi_s51> glen herrmannsfeldt writes:
(snip)
My favorite lazy swap story was someone trying to determine how fast the
memory system was, something like: int i;
char *a,*b;
a=malloc(SIZE);
b=malloc(SIZE);
for(i=0;i<1000000;i++) memcpy(a,b,SIZE); assuming that SIZE was enough bigger than the cache, so that it would
have to go out to memory. It turned out that in a lazy swap allocation, all the allocated pages
referenced the same physical page, which was smaller than the cache,
so that it only tested the cache.

Nonsense. Such a system would be irremediably broken and perfectly
useless. Once you write a value to a memory location, that memory
location must be mapped into your program's address space. If this is
not possible (it can happen in a system with lazy swap space allocation,
due to lack of physical swap space), the behaviour is system specific.
Typical examples are sending a signal to the program or killing *other*
programs, chosen more or less randomly, to obtain the missing swap space. Although such behaviour may sound crazy, the redeeming merits of lazy
swap space allocation make it the preferred choice of many people. Back when using a system with only 128 MB of swap space (eager swap
allocation would have rendered it next to useless), I knew that it was
time to kill netscape when I was starting to lose xterms ;-)


Note that the above program does not write anything to b.

I may not have remembered the actual example right, though.
This one would have to actually allocate a, but it doesn't
ever need to allocate memory for b.

I don't know if they actually overcommit, or just don't bother
actually clearing the memory (usually required for security),
and setting up the tables to describe it.

-- glen

Nov 14 '05 #15

This discussion thread is closed

Replies have been disabled for this discussion.