473,225 Members | 1,313 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,225 software developers and data experts.

vector::resize(): Why no fixed memory usage?

Hi,

why does the memory consumption of the program attached below increase
steadily during execution? Shouldn't vector::reserve() allocate one
large memory chunk of memory that doesn't change anymore?

CPU: Pentium III Coppermine (Celeron)
OS: Slackware LINUX 9.1 with kernel 2.4.22
Compiler: 3.2.3
Compile command: g++ -O0 -o foo foo.cpp
Tools used to check memory consumption: top and xosview

Felix

foo.cpp:
#include <iostream>
#include <vector>
#include <cmath>
using namespace std;

int main() {
vector<double> vect;
vect.reserve(50l*1000000l);

for (int i = 0; i < 50; ++i) {
for (long j = 0; j < 1000000; ++j)
vect.push_back(sin(j*0.34543));
cout << "vect.capacity()=" << vect.capacity()
<< "vect.size()=" << vect.size()
<< endl;
}
return 0;
}

PS: To contact me off list don't reply but send mail to "felix.klee" at
the domain "inka.de". Otherwise your email to me might get automatically
deleted!
Jul 22 '05 #1
7 2246
Felix E. Klee wrote in
news:20*************************************@gmx.n et:
Hi,

why does the memory consumption of the program attached below increase
steadily during execution? Shouldn't vector::reserve() allocate one
large memory chunk of memory that doesn't change anymore?
[snip] #include <iostream>
#include <vector>
#include <cmath>
using namespace std;

int main() {
vector<double> vect;
vect.reserve(50l*1000000l);

for (int i = 0; i < 50; ++i) {
for (long j = 0; j < 1000000; ++j)
vect.push_back(sin(j*0.34543));
cout << "vect.capacity()=" << vect.capacity()
<< "vect.size()=" << vect.size()
<< endl;
}
return 0;
}


This is an OS issue not a C++ issue, what is (probably) happening
is that when you call reserve, reserve asks the OS for 400MB of
memory, however the OS does this "virtualy". So the "real" memory
is only given to your app when its first accessed. This occurs
during your loop, so you see a steady increase in memory usage by
your programme.

Rob.
--
http://www.victim-prime.dsl.pipex.com/
Jul 22 '05 #2
On Thu, 19 Feb 2004 15:11:13 +0100, "Felix E. Klee"
<fe*************@gmx.net> wrote:
Hi,

why does the memory consumption of the program attached below increase
steadily during execution? Shouldn't vector::reserve() allocate one
large memory chunk of memory that doesn't change anymore?

CPU: Pentium III Coppermine (Celeron)
OS: Slackware LINUX 9.1 with kernel 2.4.22
Compiler: 3.2.3
Compile command: g++ -O0 -o foo foo.cpp
Tools used to check memory consumption: top and xosview

Felix

foo.cpp:
#include <iostream>
#include <vector>
#include <cmath>
using namespace std;

int main() {
vector<double> vect;
vect.reserve(50l*1000000l);

for (int i = 0; i < 50; ++i) {
for (long j = 0; j < 1000000; ++j)
vect.push_back(sin(j*0.34543));
cout << "vect.capacity()=" << vect.capacity()
<< "vect.size()=" << vect.size()
<< endl;
}
return 0;
}


I suspect your OS isn't actually allocating the space when you make
the reserve call, but only when it gets page faults from accessing
pages of virtual memory that haven't yet been backed up with physical
memory. Read up on virtual memory.

This isn't conforming, since bad_alloc probably won't be correctly
thrown under some circumstances, but low memory conditions is a
complex topic on modern OSes.

Tom

C++ FAQ: http://www.parashift.com/c++-faq-lite/
C FAQ: http://www.eskimo.com/~scs/C-faq/top.html
Jul 22 '05 #3
On Thu, 19 Feb 2004 15:37:19 +0000, tom_usenet
<to********@hotmail.com> wrote:
I suspect your OS isn't actually allocating the space when you make
the reserve call,


I mean to say that it is allocating virtual address space but not
physical memory.

Tom

C++ FAQ: http://www.parashift.com/c++-faq-lite/
C FAQ: http://www.eskimo.com/~scs/C-faq/top.html
Jul 22 '05 #4
tom_usenet <to********@hotmail.com> wrote in message news:<ds********************************@4ax.com>. ..

I suspect your OS isn't actually allocating the space when you make
the reserve call, but only when it gets page faults from accessing
pages of virtual memory that haven't yet been backed up with physical
memory. Read up on virtual memory.

This isn't conforming, since bad_alloc probably won't be correctly
thrown under some circumstances, but low memory conditions is a
complex topic on modern OSes.

Are you saying that after reserving memory, if you later try to access
that memory, it's possible that the OS might not be able to provide
physical memory, and hence necessarily crash your software? Surely
OS's must make some attempt to ensure that reserved blocks of virtual
memory will be available when required, or it would be basically
impossible to write software stable enough to cope with low-memory
conditions. To me it seems to be more an issue of the OS memory usage
reporting: I can understand distinguishing between "reserved but not
yet accessed" and "reserved and accessed/allocated" memory, but I
would have though both figures would be available.

Dylan
Jul 22 '05 #5
tom_usenet <to********@hotmail.com> wrote in message news:<ds********************************@4ax.com>. ..

I suspect your OS isn't actually allocating the space when you make
the reserve call, but only when it gets page faults from accessing
pages of virtual memory that haven't yet been backed up with physical
memory. Read up on virtual memory.

This isn't conforming, since bad_alloc probably won't be correctly
thrown under some circumstances, but low memory conditions is a
complex topic on modern OSes.


FWIW, I did a little test under Win2000 using:

char* p = (char*)malloc(10000000);
p[0] = 1;
p[10000000 - 1] = 2;
p[5000000] = 3;
p[2000000] = 4;
p[7000000] = 5;

The first call sets the 'VM size'* for the process to 10M, but each
subsequent access causes the 'Mem usage' to go up (although never to
10M). It's the same with calloc() too, which obviously means it uses
OS to provide default 0-initialized memory (HeapAlloc with
HEAP_ZERO_MEMORY).
Of course in _DEBUG build, because the CRT initializes the area with
'landfill', the 'Mem usage' immediately jumps to 10M, hence if what
you say is true regarding bad_alloc, you'd get quite different
behaviour depending on whether you were using debug-enabled memory
allocators (you'd except some slight difference perhaps, as most debug
allocators use extra space, but not that much).

Dylan
* Columns under Task Manager.
Jul 22 '05 #6
Dylan Nicholson wrote in
news:7d**************************@posting.google.c om:
tom_usenet <to********@hotmail.com> wrote in message
news:<ds********************************@4ax.com>. ..

I suspect your OS isn't actually allocating the space when you make
the reserve call, but only when it gets page faults from accessing
pages of virtual memory that haven't yet been backed up with physical
memory. Read up on virtual memory.

This isn't conforming, since bad_alloc probably won't be correctly
thrown under some circumstances, but low memory conditions is a
complex topic on modern OSes.

FWIW, I did a little test under Win2000 using:

char* p = (char*)malloc(10000000);
p[0] = 1;
p[10000000 - 1] = 2;
p[5000000] = 3;
p[2000000] = 4;
p[7000000] = 5;

The first call sets the 'VM size'* for the process to 10M, but each
subsequent access causes the 'Mem usage' to go up (although never to
10M). It's the same with calloc() too, which obviously means it uses
OS to provide default 0-initialized memory (HeapAlloc with
HEAP_ZERO_MEMORY).


This seems fine to me, the OS has reserved 10M of memory/page file
for the process, but only gives the process real memory when it
uses it. This leaves the real memory available for other processes
until its actually used. Presumably at this point the OS can then
swap out the real memory used by the other process('s) into the
page file (VM) that its reserved, and give this process real memory.

Of course in _DEBUG build, because the CRT initializes the area with
'landfill', the 'Mem usage' immediately jumps to 10M, hence if what
you say is true regarding bad_alloc, you'd get quite different
behaviour depending on whether you were using debug-enabled memory
allocators (you'd except some slight difference perhaps, as most debug
allocators use extra space, but not that much).


The only difference should be that the debug build is slower. and
is also slowing down the OS/other apps by over commiting memory.
Rob.
--
http://www.victim-prime.dsl.pipex.com/
Jul 22 '05 #7
On 19 Feb 2004 14:11:38 -0800, wi******@hotmail.com (Dylan Nicholson)
wrote:
tom_usenet <to********@hotmail.com> wrote in message news:<ds********************************@4ax.com>. ..

I suspect your OS isn't actually allocating the space when you make
the reserve call, but only when it gets page faults from accessing
pages of virtual memory that haven't yet been backed up with physical
memory. Read up on virtual memory.

This isn't conforming, since bad_alloc probably won't be correctly
thrown under some circumstances, but low memory conditions is a
complex topic on modern OSes.
Are you saying that after reserving memory, if you later try to access
that memory, it's possible that the OS might not be able to provide
physical memory, and hence necessarily crash your software?


Yup. Normally paging will slow the system to a crawl first though.

SurelyOS's must make some attempt to ensure that reserved blocks of virtual
memory will be available when required, or it would be basically
impossible to write software stable enough to cope with low-memory
conditions.
On modern OSes, you can't rely on bad_alloc being thrown from new, you
have to keep track of memory usage some other way.

To me it seems to be more an issue of the OS memory usagereporting: I can understand distinguishing between "reserved but not
yet accessed" and "reserved and accessed/allocated" memory, but I
would have though both figures would be available.


With paging and virtual memory, it doesn't necessarily make sense to
back up all allocations with physical memory at the time of the
allocation, since that prevents other processes from using it.

All this exception safety/new throwing bad_alloc is actually less of
an issue than it seems, since normally new won't throw unless you run
out of virtual memory (which is near-impossible on a 64-bit machine),
but some kind of platform specific signal or exception may be
generated when attempting to access the memory if you've run out of
physical memory and page file space.

On the Windows test I just did, I got a message box telling me that
virtual memory was getting low. I don't recall seeing that in the spec
for new!

Tom

C++ FAQ: http://www.parashift.com/c++-faq-lite/
C FAQ: http://www.eskimo.com/~scs/C-faq/top.html
Jul 22 '05 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: solosnake | last post by:
Hello, and merry Christmas! When trying to compile some code that had specific alignment requirements, I found that the Visual Studio .NET compiler gave me compile time errors. It forbids...
4
by: Peter Mrosek | last post by:
Hello, I have the following declaration in an header file (abbreviated version): typedef struct { unsigned char rgbBlue; unsigned char rgbGreen; unsigned char rgbRed; unsigned char...
9
by: Alex Vinokur | last post by:
--------- #include <vector> using namespace std; struct Foo { Foo (int) {} }; int main () { vector<Foo> v1;
10
by: vsgdp | last post by:
On Page 67 of Effective STL, Meyers writes about resize: "If n is smaller than the current size, elements at the end of the container will be destroyed." What does "destroyed" mean? It seems to...
2
by: mj | last post by:
Hi, I recently have found it necessary to move from fortran to c++ for scientific programming... I'm working on a program that needs to resize a 2d vector of vectors within a function... This...
2
by: mrbrightsidestolemymoney | last post by:
Hi, I'm having a problem resizing a (very big) nested vector. It's not the most streamlined piece of code ever but I need this array to avoid having to recalculate the same quantity millions of...
3
by: Jim Langston | last post by:
I really am not sure if this question belongs in this newsgroup, but not sure where else to ask it. There is someone working on a game that I tested, and it was taking >30 seconds to load. He...
6
by: Jeff.Boeker | last post by:
I'm learning a lesson in how I need to be more specific :) In C++ I can resize a vector and it will allocate memory and it will call the default constructor if necessary (or I can supply an...
3
by: Bram Kuijper | last post by:
Hi all, I am trying to resize a vector of objects (MyObj below), which contain references to other objects (OtherObj, see below). However, apparently somewhere in the resize operation an...
0
by: veera ravala | last post by:
ServiceNow is a powerful cloud-based platform that offers a wide range of services to help organizations manage their workflows, operations, and IT services more efficiently. At its core, ServiceNow...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
0
by: mar23 | last post by:
Here's the situation. I have a form called frmDiceInventory with subform called subfrmDice. The subform's control source is linked to a query called qryDiceInventory. I've been trying to pick up the...
2
by: jimatqsi | last post by:
The boss wants the word "CONFIDENTIAL" overlaying certain reports. He wants it large, slanted across the page, on every page, very light gray, outlined letters, not block letters. I thought Word Art...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.