Very big arrays

Hello,

I'm trying to make a dll with some stored values, looking to store
several million doubles in a static array like this:

double delta[x_size][y_size][z_size];
void delta_init() {
delta[0][0][0] = 8.65;
delta[0][0][1] = 1.35;
delta[0][0][2] = 2.11;
delta[0][0][3] = 3.29;
delta[0][0][4] = 5.147;

etc ...

I'm trying to compile the code under Visual C++ Pro 2005, and the code
just does not compile - it runs over 10 hours and just hands there. I
was able to compile smaller arrays though.

How can I make the code compile? What is a better way to do this?
Thank you for your advice.

re: Very big arrays

victorg...@gmail.com skrev:It must be a program with millions of lines, then?
Technically it haven't failed yet, but I don't want to bet that the
compile will eventually succeed. But your question is platform
specific, so it is off-topic here. Go visit some group where you would
be on topic - perhaps microsoft.public.vc.language?

/Peter

re: Very big arrays

victorgrnt@gmail.com wrote:lol, 10 hours? Why would you store values in a dll? Wouldn't that eat
away at memory available to your program - dll must be loaded in mem?
Have you considered storing these in a file or several files? Or
perhaps develop an algorithm to initialize the values? Do you realize
that an array requires contiguous memory? Why don't you store related
values together in a struct/class and store instances of the
struct/class instead?

If i had to store points, for example, with an x, y and a z:

#include <iostream>
#include <ostream>
#include <vector>
#include <deque>

template< typename T >
struct Point
{
// universal ctor
Point(const T& x_ = T(),
const T& y_ = T(),
const T& z_ = T())
: x(x_), y(y_), z(z_) { }
// copy ctor
Point(const Point& copy)
{
x = copy.x;
y = copy.y;
z = copy.z;
}
private:
T x, y, z;
};

int main() {
// one million Points as doubles, all set to 0.0
std::deque< Point< double dcontainer(1000000); // done

std::cout << "double container's size = ";
std::cout << dcontainer.size() * sizeof(Point< double >);
std::cout << std::endl;

// one million Points as floats, all set to 0.0
std::deque< Point< float fcontainer(1000000);

std::cout << "float container's size = ";
std::cout << fcontainer.size() * sizeof(Point< float >);
std::cout << std::endl;

return 0;
}

/* output on this platform:

double container's size = 24000000 // 24 MB
float container's size = 12000000 // 12 MB

*/

re: Very big arrays

victorgrnt@gmail.com wrote:Assuming sizeof(double)==8, several million doubles will
likely occupy several tens of MB. You really want a dll that
big?
Take a look at the HDD light, perhaps the compiler ran out of
memory and started thrashing?
What exactly disallows you to read this data from a file?

HTH,
- J.

re: Very big arrays

Thank you Peter & Peter.

I have a function that is slow to compute, so I pre-compute the values
and store them in a dll to retrieve in real-time. Obviously computing
the values in real-time would make the dll too slow. Few megs of RAM
that I'll need to use for it are hardly an issue on most desktops. Am I
missing something?

re: Very big arrays

victorgrnt@gmail.com wrote:Why don't you use static initialization?

double delta[x_size][y_size][z_size] =
{
{
{
8.65,
1.35,
2.11,

re: Very big arrays

Salt_Peter wrote:Especially if you store your numbers in a binary file, with fixed
record length, you may not even have to load the entire file into
memory (assuming disk access is fast enough for your purposes). You
could just seek to the values in the file and read them in, as you need
them. This way you could get away with using far less memory than a few
megabytes, or a few tens of megabytes.

Regards,
Markus.

re: Very big arrays

victorgrnt@gmail.com skrev:Yes. If the purpose is simply to keep some precomputed values, it would
be much better to write those values to a file and read them back when
you need them (using a std::vector for flexibility).

/Peter

re: Very big arrays

peter koch escreveu:
I think you should store the precomputed values into a file and map
that file to memory. This is an OS feature, and by doing this, you
won´t have to worry about lots of the memory management specific
details.

Regards,
Henrique

re: Very big arrays

Henrique Bastos skrev:Apart from non-portability and (probably) some faster load-time, I do
not see your method having an advantage.

/Peter

re: Very big arrays

Hi,
I have the similar problem. It looks only occur on VC 2005
I have to compile a an 47K size array
UINT8 UIRES_FONTLIB_DATA_000[2774*16+5] =
{
0x00,0x00,0xad,0x61,/* file size of fontfile */
0x00,0x01,0x00,0x00,0x00,0x0b,0x00...}

I compile OK on all old VC build(6.0, or 7.0). but recently, I upgrade
my VC to VC2005 Express, then it failed to compile for this array.
takes too long to compile, I have to ctrl+break it.
if I cut the array shorter, then It compile OK too.

who can tell what's the problem? a bug of VC2005?



Thanks,


On 12ÔÂ13ÈÕ, ÏÂÎç10ʱ26·Ö, "peter koch"
<peter.koch.lar...@gmail.comwrote:

re: Very big arrays

ouyangsonghua@gmail.com skrev:I believe it is. I recommend you take it to a microsoft group
(microsoft.public.vc.language if my memory serves me). There is a
service pack for the VC2005 Express - did you apply it? It just might
solve your problem.

/Peter