fasteste way to fill a structure.

[simon]

From my previous post...

If I have a structure,

struct sFileData
{
char*sSomeStrin g1;
char*sSomeStrin g2;
int iSomeNum1;
int iSomeNum2;
sFileData(){... };
~sFileData(){.. .};
sFileData(const sFileData&){... };
const sFileData operator=( const sFileData &s ){...}
};

I read the file as follows

FILE *f = fopen( szPath, "rb" );

int nLineSize = 190;
BYTE b[nLineSize+1];

fread( b, sizeof(BYTE), nLineSize, f );
int numofrecords = atoi( b ); // first line is num of records only,

// read the data itself.
while( fread( b, sizeof(BYTE), nLineSize, f ) == nLineSize )
{
// fill data
// The locations of each items is known
// sString1 = 0->39, with blank spaces filler after data
// sString2 = 40->79, with blank spaces filler after data
// iNum1 = 80->99, with blank spaces filler after data
// iNum2 = 100->end, with blank spaces filler after data
}

what would be the best way to fill the data into an array, (vector)?

Many thanks.

Simon.

[Victor Bazarov]

simon wrote:

If I have a structure,

struct sFileData
{
char*sSomeStrin g1;
char*sSomeStrin g2;
int iSomeNum1;
int iSomeNum2;
sFileData(){... };
~sFileData(){.. .};
sFileData(const sFileData&){... };
const sFileData operator=( const sFileData &s ){...}
};

I read the file as follows

FILE *f = fopen( szPath, "rb" );

int nLineSize = 190;
BYTE b[nLineSize+1];

fread( b, sizeof(BYTE), nLineSize, f );
int numofrecords = atoi( b ); // first line is num of records only,

// read the data itself.
while( fread( b, sizeof(BYTE), nLineSize, f ) == nLineSize )
{
// fill data
// The locations of each items is known
// sString1 = 0->39, with blank spaces filler after data
// sString2 = 40->79, with blank spaces filler after data
// iNum1 = 80->99, with blank spaces filler after data
// iNum2 = 100->end, with blank spaces filler after data
}

what would be the best way to fill the data into an array, (vector)?

I presume nLineSize is greater than 100. Then, something in line with

// as soon as you know the number of structures
yourvector.rese rve(numofrecord s);

// read the data themselves
while (fread(... )
{
yourvector.push _back(
sFileData(
std::string(b, b+40).c_str(),
std::string(b+4 0, b+80).c_str(),
strtol(std::str ing(b+80,b+100) .c_str(),10,0),
strtol(std::str ing(b+100,b+nLi neSize).c_str() ,10,0)
)
);
}

You will need to create another constructor for your 'sFileData',
which will take two pointers to const char, and two ints (or longs):

sFileData(char const*, char const*, int, int);

Take those pointers and extract the C strings from them to create your
members.

In general, I think it's better to have 'std::string' as members instead
of 'char*'. You may need to fix the rest of your class if you make that
switch.

V

[titancipher]

This way is _not _ fast as there are loads of unnecessary memory
allocations. Simon, you had the right idea from the start, but the
data structure can be modified to:

struct sFileData
{
char sSomeString1[40];
char sSomeString2[40];
int iSomeNum1;
int iSomeNum2;
....
};

Then, you can use either an array or a vector. Since you know the size
ahead of time, you can create an array:

struct sFileData array[ numofrecords ];
// read the data itself.
int i = 0;
while( fread( b, sizeof(BYTE), nLineSize, f ) == nLineSize )
{
array[ i ] = *(struct sFileData * )&b;
++i;
}

[titancipher]

I failed to see that the file format is most-likely ascii.

[Simon]

>>

struct sFileData
{
char*sSomeStrin g1;
char*sSomeStrin g2;
int iSomeNum1;
int iSomeNum2;
sFileData(){... };
~sFileData(){.. .};
sFileData(const sFileData&){... };
const sFileData operator=( const sFileData &s ){...}
};

I presume nLineSize is greater than 100. Then, something in line with
Why would it have to be > 100? or are you saying that because of my
definition?

// as soon as you know the number of structures
yourvector.rese rve(numofrecord s);
Ok, it does speed things up a bit.

// read the data themselves
while (fread(... )
{
yourvector.push _back(
sFileData(
std::string(b, b+40).c_str(),
std::string(b+4 0, b+80).c_str(),
strtol(std::str ing(b+80,b+100) .c_str(),10,0),
strtol(std::str ing(b+100,b+nLi neSize).c_str() ,10,0)
)
);
}

I still think that I am doing something wrong here.
To read a file with 100000 lines takes 0.66 sec, (windows machine).

But filling the structure above takes +28 seconds.

Is that normal?

Simon

[Jeff Flinn]

"Simon" <sp********@exa mple.com> wrote in message
news:3i******** ****@individual .net...

struct sFileData
{
char*sSomeStrin g1;
char*sSomeStrin g2;
int iSomeNum1;
int iSomeNum2;
sFileData(){... };
~sFileData(){.. .};
sFileData(const sFileData&){... };
const sFileData operator=( const sFileData &s ){...}
};

I presume nLineSize is greater than 100. Then, something in line with

Why would it have to be > 100? or are you saying that because of my
definition?

// as soon as you know the number of structures
yourvector.rese rve(numofrecord s);

Ok, it does speed things up a bit.

// read the data themselves
while (fread(... )
{
yourvector.push _back(
sFileData(
std::string(b, b+40).c_str(),
std::string(b+4 0, b+80).c_str(),
strtol(std::str ing(b+80,b+100) .c_str(),10,0),
strtol(std::str ing(b+100,b+nLi neSize).c_str() ,10,0)
)
);
}

I still think that I am doing something wrong here.
To read a file with 100000 lines takes 0.66 sec, (windows machine).

But filling the structure above takes +28 seconds.

Is that normal?

You won't know until you profile and see where the time is spent.

Jeff Flinn

[Victor Bazarov]

Simon wrote:

struct sFileData
{
char*sSomeStrin g1;
char*sSomeStrin g2;
int iSomeNum1;
int iSomeNum2;
sFileData(){... };
~sFileData(){.. .};
sFileData(const sFileData&){... };
const sFileData operator=( const sFileData &s ){...}
};

I presume nLineSize is greater than 100. Then, something in line with

Why would it have to be > 100? or are you saying that because of my
definition?

// as soon as you know the number of structures
yourvector.rese rve(numofrecord s);

Ok, it does speed things up a bit.

// read the data themselves
while (fread(... )
{
yourvector.push _back(
sFileData(
std::string(b, b+40).c_str(),
std::string(b+4 0, b+80).c_str(),
strtol(std::str ing(b+80,b+100) .c_str(),10,0),
strtol(std::str ing(b+100,b+nLi neSize).c_str() ,10,0)
)
);
}

I still think that I am doing something wrong here.
To read a file with 100000 lines takes 0.66 sec, (windows machine).

But filling the structure above takes +28 seconds.

Is that normal?

May not be. You may want to change the structure and make it contain
arrays of char instead of pointers to dynamically allocated arrays.

Then the construction will be a bit faster, you could simply drop the
'string' thing there. Also, if you're sure about the source of the
data, and their format, you could avoid constructing temporaries. Play
with making 'sFileData' look like

char s1[41]; // if it's a C string, reserve the room for the null char
char s2[41];
int one, two;

and then you could construct it a bit faster. You will still need to
convert the third and the fourth fields since they can't be memcpy'ed.

V

[Simon]

>

May not be. You may want to change the structure and make it contain
arrays of char instead of pointers to dynamically allocated arrays.

Then the construction will be a bit faster, you could simply drop the
'string' thing there. Also, if you're sure about the source of the
data, and their format, you could avoid constructing temporaries. Play
with making 'sFileData' look like

char s1[41]; // if it's a C string, reserve the room for the null char
char s2[41];
int one, two;

I know I am going to be told I am too difficult, but the reason why I
dynamically create the string is because they are almost never longer than 5
chars.
So by declaring s1[41] I know that I am wasting around 36 chars, (The sizes
are different, there could be a string of 40 chars).

I know that we are only talking about 36 chars here, but I load 100000's of
lines and the waste really seems unnecessary to me, (and I don't like
wasting memory).
It seems to defeat the object dynamic memory allocations.

Simon

[Steven T. Hatton]

Simon wrote:

I know I am going to be told I am too difficult, but the reason why I
dynamically create the string is because they are almost never longer than
5 chars.
So by declaring s1[41] I know that I am wasting around 36 chars, (The
sizes are different, there could be a string of 40 chars).

I know that we are only talking about 36 chars here, but I load 100000's
of lines and the waste really seems unnecessary to me, (and I don't like
wasting memory).
It seems to defeat the object dynamic memory allocations.

Simon

What about using std::string with std::string::re serve(5); Or something
close to the maximum "normal" value? That way, you have a minimum
preallocated, but it can still grow.
--
If our hypothesis is about anything and not about some one or more
particular things, then our deductions constitute mathematics. Thus
mathematics may be defined as the subject in which we never know what we
are talking about, nor whether what we are saying is true.-Bertrand Russell

[Victor Bazarov]

Simon wrote:

May not be. You may want to change the structure and make it contain
arrays of char instead of pointers to dynamically allocated arrays.

Then the construction will be a bit faster, you could simply drop the
'string' thing there. Also, if you're sure about the source of the
data, and their format, you could avoid constructing temporaries. Play
with making 'sFileData' look like

char s1[41]; // if it's a C string, reserve the room for the null char
char s2[41];
int one, two;

I know I am going to be told I am too difficult, but the reason why I
dynamically create the string is because they are almost never longer than 5
chars.
So by declaring s1[41] I know that I am wasting around 36 chars, (The sizes
are different, there could be a string of 40 chars).

I know that we are only talking about 36 chars here, but I load 100000's of
lines and the waste really seems unnecessary to me, (and I don't like
wasting memory).
It seems to defeat the object dynamic memory allocations.

Perhaps then you need to invent a smarter scheme for storing those strings
than keeping a pointer to a dynamic array of chars. Do you know that most
heap managers when you need to allocate 1 char would slap 2*sizeof(void*)
on top of it to make a dynamic array? So, you're still wasting enough
memory (not to say all the CPU cycles to allocate and then deallocate them
along with other objects).

Imagine that your 'sFileData' class has a static storage for all its
strings, from which all individual strings are cut out (or, rather, in
which all individual strings are stacked up). If your objects never
change, and only get allocated once and deallocated together at some
point, then it might be the simple custom memory manager you need. You
can allocate that static storage in large chunks and give your class some
mechanism to account for allocations... Well, as you can see, all you may
need to improve the performance is a custom memory manager. You can
probably use an open source one, if you can find it.

V