malloc problem

[vjsarthy]

Hi All,

I am trying to use malloc function to allocate memory for 580,000,000 numbers (long int).

valid = 580000000;
infoBuffer = malloc((valid)*sizeof(long));

if(!infoBuffer)
printf("Allocating memory failed\n");
else
printf("Successful memory allocation\n");

I have all these numbers in a file.

I am using my C code to load these numbers to the buffer so that I can use them to calculate something.

The memory allocation fails, but when I lowered the number to 520,000,000 it successfully loads them to memory, which tells me the limitation is 520,000,000. (i tried 521,000,000, it fails)

I tried this on a 4GB RAM machine and also on a 64 bit machine but all in vain (limitation 520,000,000 for all of them).
By the way my machine is 1GB.

I am using Visual C++.

Is this the limitation of VC++? or the malloc() ? Anything else?

Please advise me on any other function I can use to get this done.

[weaknessforcats]

There arte two questions I have:
1)Why do you need all 580 million longs in the machine at once???
It looks like you may have a design weakness.

2) Why are you not using SQL Server or Oracle where these problems have already been solved and coded???

[vjsarthy]

There arte two questions I have:
1)Why do you need all 580 million longs in the machine at once???
It looks like you may have a design weakness.

2) Why are you not using SQL Server or Oracle where these problems have already been solved and coded???

Each of the number is a node and it carries vital information (when decoded). Given any two nodes, I should connect those nodes by finding the "in-between" link nodes (which are again in these 580M numbers).

I cannot use SQL or Oracle as these are not supported by the client.

[weaknessforcats]

Then you will need to design differently. Probably, you will use a disc file precisely formatted so the 580 million longs are divided into manageable sections that can be floated in and out of memory.

This will be a huge coding effort where you re-create code that has already been written. I stirngly suggest you get yourself a DBMS for this.

[vjsarthy]

Then you will need to design differently. Probably, you will use a disc file precisely formatted so the 580 million longs are divided into manageable sections that can be floated in and out of memory.

This will be a huge coding effort where you re-create code that has already been written. I stirngly suggest you get yourself a DBMS for this.

Thank you very much for your reply.
So, is this the limitation of malloc or the computer? or VC++?
What do you think?

[weaknessforcats]

I expect the limit is your OS.

For example, on Windiows XP, a long has a max value of 2,147,483,647.

580 million longs needs 2,320,000,000 bytes of memory.

If malloc() attempts to use a long for the number of bytes to allocate, it will overflow that long.

This means you will need to design your code to use allocated segments. You will truly be at this for a long time. Seriously consider a DBMS making sure your table size and row limits are within your requirements.

Of course, you could go to a 64-bit OS where you have 64-bit integers.

[RRick]

I don't think the problem is with the compiler. Malloc takes a size_t parameter which is usually an unsigned int. On most 32 bit OSes this would be a 32 bit value which is plenty big for your value. If you really want to see what the max value is, then take a look at the system's malloc.h. You'll find size_t defined there, somewhere.

Also, where your program gives up is not at a 2**31 boundary. To cross that boundary, you would need 537,000,000 four byte numbers, not 521.

Soooo, it looks like a system memory issue and this is where the fun begins. Windows has a habit of using up the address space for its own uses (remember 640K?). For example, Windows 2000 could only used 2GB of memory for an application and I hear that 32 bit XP and Vista can use 3GB max. If you want to go higher you need a 64 bit system.

As for accessing that number of values, I agree with W4cats. Your are going beyond the system memory into virtual memory and that translates into slow, very slow processing (the issue is called thrashing). I would definitely suggest a design change to get around this limitation, or else get a machine big and bad enough to deal with that amount of data.

[vjsarthy]

I don't think the problem is with the compiler. Malloc takes a size_t parameter which is usually an unsigned int. On most 32 bit OSes this would be a 32 bit value which is plenty big for your value. If you really want to see what the max value is, then take a look at the system's malloc.h. You'll find size_t defined there, somewhere.

Also, where your program gives up is not at a 2**31 boundary. To cross that boundary, you would need 537,000,000 four byte numbers, not 521.

Soooo, it looks like a system memory issue and this is where the fun begins. Windows has a habit of using up the address space for its own uses (remember 640K?). For example, Windows 2000 could only used 2GB of memory for an application and I hear that 32 bit XP and Vista can use 3GB max. If you want to go higher you need a 64 bit system.

As for accessing that number of values, I agree with W4cats. Your are going beyond the system memory into virtual memory and that translates into slow, very slow processing (the issue is called thrashing). I would definitely suggest a design change to get around this limitation, or else get a machine big and bad enough to deal with that amount of data.

Thank you very much Rick and Cats.

To see if this works on a 64 bit machine, I ran the .exe file in a 64 bit machine. It gives me the same error.

The reason could be that this executable was created in a 32 bit machine. Since the 64 bit machine did nt have VC++, I was nt able to create the exe in that machine.

Do you think its worth installing VC++ in the 64 bit machine and try executing this file. That machine has got 4G RAM and I feel its powerful.

Please advise.

[weaknessforcats]

I say again: The total array size cannot exceed 0x7fffffff, which is 2,147,483,647.

2,147,483,647 is LONG_MAX.

malloc() needs 580000000 X 4 bytes and that number does not fit in a long.

I repeat, you will have to allocate this in segments.

Truly, check out using a DBMS where these problems have already been worked out.

Using a 64-bit OS will work if the long is 64-bits. So, you could switch to 64-bit Unix or Linux. 64-bit Windows retains the sizes of the 32-bit built-in types and adds new types for 64 bits. malloc() uses the built-in types so 64-bit Windows won't work for you.