Illogical std::vector size?

[simon]

Hi,

First some background.

I have a structure,

struct sFileData
{
char*sSomeStrin g1;
char*sSomeStrin g2;
int iSomeNum1;
int iSomeNum2;
sFileData(){... };
~sFileData(){.. .};
sFileData(const sFileData&){... };
const sFileData operator=( const sFileData &s ){...}
};

std::vector< sFileData, std::allocator< sFileData>> address_;

for the sake of simplicity I remove the body of the 'tors
I have no memory leaks as far as I can tell.

Then I read a file, (each line is 190 chars mostly blank spaces).
In each line I 'read' info to fill in the structure.

Because there are some many blank spaces in the line I make sure that my
data is 'trimmed'.

So in effect sSomeString1 and sSomeString2 are never more than 10 chars,
(although in the file they could be up to 40 chars).

I chose vectors because after reading the file I need to do searches of
sSomeString1 and sSomeString2, (no other reasons really).

But my problem is the size of address_ is not consistent with the size of
the file.

The file is around 13Mb with around 100000 'lines' of 190 chars each.
Because I remove blank spaces and I convert 2 numbers to int, (from char). I
guess I should not use more than half, 5Mb.

But after loading I see that I used around 40Mb, (3 times more than the
original size).

as far as I can tell you cannot really tell the size of a vector, but I use
windows and the task manager and I can see the size of my app before and
after reading the file, (I do nothing else).

So what could be the reason for those inconsistencies ?
How could I optimize my code to compress those 40mb even more?

Many thanks

Simon

[msalters]

simon schreef:

Hi,

First some background.

I have a structure,

struct sFileData
{
char*sSomeStrin g1;
char*sSomeStrin g2;
int iSomeNum1;
int iSomeNum2;
sFileData(){... };
~sFileData(){.. .};
sFileData(const sFileData&){... };
const sFileData operator=( const sFileData &s ){...}
};

std::vector< sFileData, std::allocator< sFileData>> address_;

for the sake of simplicity I remove the body of the 'tors
I have no memory leaks as far as I can tell.

Then I read a file, (each line is 190 chars mostly blank spaces).
In each line I 'read' info to fill in the structure.

Because there are some many blank spaces in the line I make sure that my
data is 'trimmed'.

So in effect sSomeString1 and sSomeString2 are never more than 10 chars,
(although in the file they could be up to 40 chars).

I chose vectors because after reading the file I need to do searches of
sSomeString1 and sSomeString2, (no other reasons really).

But my problem is the size of address_ is not consistent with the size of
the file.

The file is around 13Mb with around 100000 'lines' of 190 chars each.
Because I remove blank spaces and I convert 2 numbers to int, (from char). I
guess I should not use more than half, 5Mb.

But after loading I see that I used around 40Mb, (3 times more than the
original size).

as far as I can tell you cannot really tell the size of a vector, but I use
windows and the task manager and I can see the size of my app before and
after reading the file, (I do nothing else).

1) Windows Task Manager is not suited for this
2) vector only stores sFileData objects, not the strings themselves
3) Even when vector has excess size (which is common, don't want to
reallocate after each pusch_back) it won't include the strings
4) Many implementations of new[] allocate at least 16 bytes, plus
the overhead needed for delete[]
5) So what? 40MB is not a lot. Worry when it exceeds 1.5Gb. Memory
is cheap. Writing a custom string class is not. BTDT.

HTH,
Michiel Salters

[simon]

>

1) Windows Task Manager is not suited for this
yea, but it was what raised suspicion in the first place.
What might be better?
2) vector only stores sFileData objects, not the strings themselves
3) Even when vector has excess size (which is common, don't want to
reallocate after each pusch_back) it won't include the strings
4) Many implementations of new[] allocate at least 16 bytes, plus
the overhead needed for delete[]
Are you saying that std::string might actually be better in that case?
What might be a better way?
5) So what? 40MB is not a lot. Worry when it exceeds 1.5Gb. Memory
is cheap. Writing a custom string class is not. BTDT.
40Mb or 4Gb, there is still something not quite right, and i would preffer
to know what it is rather than brushing it under the rug.

HTH,
Michiel Salters

[velthuijsen]

> > 1) Windows Task Manager is not suited for this

yea, but it was what raised suspicion in the first place.
What might be better?

What suspicion? It only tells that the total program is now using 40 MB
not that this particular part of your program is using 40 MB.
Or if it is a change in the total amount of memory used there could be
a near infinite number of other reasons that the program is now using
40 MB instead of the expected 5 MB increase.
You'd need a profiler to check if it is indeed the vector of structs
that is the problem.

[simon]

> 1) Windows Task Manager is not suited for this
yea, but it was what raised suspicion in the first place.
What might be better?

What suspicion?

That i was doing something wrong or that i did not understand something
else.
It only tells that the total program is now using 40 MB
not that this particular part of your program is using 40 MB.
Or if it is a change in the total amount of memory used there could be
a near infinite number of other reasons that the program is now using
40 MB instead of the expected 5 MB increase.
You'd need a profiler to check if it is indeed the vector of structs
that is the problem.

I placed a break point before reading the file, check the memory, and one
after reading the file.
I then compared the before and after.
The odds of it been something else are fairly small, IMO.

Simon

[velthuijsen]

20 MB of file in memory is a good starter to explain away the
discrepancy.
You need something better then two brwakpoints and the taskmanager to
make statements about what uses the memory.

[John Carson]

"simon" <sp********@myo ddweb.com> wrote in message
news:3h******** ****@individual .net

Hi,

First some background.

I have a structure,

struct sFileData
{
char*sSomeStrin g1;
char*sSomeStrin g2;
int iSomeNum1;
int iSomeNum2;
sFileData(){... };
~sFileData(){.. .};
sFileData(const sFileData&){... };
const sFileData operator=( const sFileData &s ){...}
};

std::vector< sFileData, std::allocator< sFileData>> address_;

for the sake of simplicity I remove the body of the 'tors
I have no memory leaks as far as I can tell.

Then I read a file, (each line is 190 chars mostly blank spaces).
In each line I 'read' info to fill in the structure.

Because there are some many blank spaces in the line I make sure that
my data is 'trimmed'.

So in effect sSomeString1 and sSomeString2 are never more than 10
chars, (although in the file they could be up to 40 chars).

I chose vectors because after reading the file I need to do searches
of sSomeString1 and sSomeString2, (no other reasons really).

But my problem is the size of address_ is not consistent with the
size of the file.

The file is around 13Mb with around 100000 'lines' of 190 chars each.
Because I remove blank spaces and I convert 2 numbers to int, (from
char). I guess I should not use more than half, 5Mb.

But after loading I see that I used around 40Mb, (3 times more than
the original size).

as far as I can tell you cannot really tell the size of a vector, but
I use windows and the task manager and I can see the size of my app
before and after reading the file, (I do nothing else).

I think your problem has nothing to do with the vector. As has already been
pointed out, the vector doesn't store the characters, only the pointer. With
VC++, sizeof(sFileDat a) is 16. The memory used by the vector should be
16*address_.cap acity() plus a small amount of overhead, which we can
approximate with sizeof(address_ ). Try this:

#include <vector>
#include <iostream>
using namespace std;

struct sFileData
{
char*sSomeStrin g1;
char*sSomeStrin g2;
int iSomeNum1;
int iSomeNum2;
sFileData(){}
~sFileData(){}
sFileData(const sFileData&){}
const sFileData operator=( const sFileData &s ){ return *this;}
};

std::vector< sFileData, std::allocator< sFileData> > address_;
int main()
{
sFileData sfd;
for(int i=0; i<100000; ++i)
address_.push_b ack(sfd);
cout << "storage size of vector is approx ";
cout << sizeof(address_ )+sizeof(sFileD ata)*address_.c apacity() << endl;
return 0;
}

When I run this, I get

storage size of vector is approx 2212100

and task manager similarly shows about a 2Mb increase in memory useage.
Accordingly, it seems that the other 38Mb is due to whatever else you are
doing to allocate memory for the characters --- unless, as someone else
suggested, you are reading the whole file into memory and not taking that
into account.
--
John Carson

[Carlos Martinez Garcia]

simon wrote:

1) Windows Task Manager is not suited for this

yea, but it was what raised suspicion in the first place.
What might be better?

2) vector only stores sFileData objects, not the strings themselves
3) Even when vector has excess size (which is common, don't want to
reallocate after each pusch_back) it won't include the strings
4) Many implementations of new[] allocate at least 16 bytes, plus
the overhead needed for delete[]

Are you saying that std::string might actually be better in that case?
What might be a better way?

No, std::string have the same problem. That isn't the cause.

5) So what? 40MB is not a lot. Worry when it exceeds 1.5Gb. Memory
is cheap. Writing a custom string class is not. BTDT.

40Mb or 4Gb, there is still something not quite right, and i would preffer
to know what it is rather than brushing it under the rug.

40 Mb is about (42000000 bytes)
42000000 bytes / 100000 records = 420 bytes/record.

Even if each new char* uses at least 16 bytes, one record uses about
40 bytes.
Definetively I think you have a problem elsewhere probably in loop

[simon]

> 20 MB of file in memory is a good starter to explain away the

discrepancy.
I don't read the whole file in memory.

I use fopen(...)
read each chunk of data using fread(...) and then close the file using
fclose(...).
You need something better then two brwakpoints and the taskmanager to
make statements about what uses the memory.

Well i am sorry, but i can see that before i read the file i use x amount of
memory and that just after i finish reading the file, (after the fclose), i
used x+40mb.

Simon

[simon]

>> Hi,

First some background.

I have a structure,

I think your problem has nothing to do with the vector. As has already
been pointed out, the vector doesn't store the characters, only the
pointer. With VC++, sizeof(sFileDat a) is 16. The memory used by the vector
should be 16*address_.cap acity() plus a small amount of overhead, which we
can approximate with sizeof(address_ ). Try this:
<snip code >

When I run this, I get

storage size of vector is approx 2212100
I will try that, but i should get the same thing myself.

and task manager similarly shows about a 2Mb increase in memory useage.
Accordingly, it seems that the other 38Mb is due to whatever else you are
doing to allocate memory for the characters --- unless, as someone else
suggested, you are reading the whole file into memory and not taking that
into account.

I don't read the whole file in memory.

I use fopen(...)
read each chunk of data using fread(...) and then close the file using
fclose(...).

Simon