By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
424,963 Members | 1,046 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 424,963 IT Pros & Developers. It's quick & easy.

Big arrays (confusion over FAQ 6.14)

P: n/a
I have a need to create an array that is larger than size_t - 1, which
I believe is the largest guaranteed array. I though I'd found
salvation in FAQ 6.14, but it appears I am not understanding
something.

The array is an array of pointers to structs. Here is an example
using a small array:

/* checks that malloc succeeded, main's return, etc. removed for
brevity */

#include <stdio.h>
#include <stdlib.h>

typedef struct {
int foo;
int bar;
} example_struct;

int main (void) {
example_struct * small_array[1000];
example_struct * a_single_struct;

a_single_struct = malloc ( sizeof(example_struct *) );
a_single_struct->foo = 4000;
printf ("a_single_struct foo: %i \n", a_single_struct->foo);

small_array[900] = a_single_struct;
printf ("small_array 900 foo: %i\n",small_array[900]->foo);

}

This works fine:

a_single_struct foo: 4000
small_array 900 foo: 4000

So far, so good. Now I want to do an array of, say, 3,000,000
members, too large for a simple big_array[3000000] declaration. So I
use the code from faq 6.14:

/* checks that malloc succeeded, main's return, etc. removed for
brevity */

#include <stdio.h>
#include <stdlib.h>

typedef struct {
int foo;
int bar;
} example_struct;

int main (void) {
example_struct * big_array;
example_struct * a_single_struct;

a_single_struct = malloc ( sizeof(example_struct *) );
a_single_struct->foo = 4000;
printf ("a_single_struct foo: %i \n", a_single_struct->foo);

big_array = malloc ( 3000000 * sizeof ( example_struct * ) );
big_array[1000000] = a_single_struct; /* line 20, error below */
printf ("big_array foo: %i\n",big_array[1000000]->foo);

}

No go. The compiler (gcc in this case) returns:

test.c: In function 'main':
test.c:20: error: incompatible types in assignment
test.c:21: error: invalid type argument of '->'

So the example in FAQ 6.14 is:

"int *dynarray;
dynarray = malloc(10 * sizeof(int));

....you can reference dynarray[i] (for i from 0 to 9) almost as if
dynarray were a conventional, statically-allocated array (int a[10])"

Given that, I'm confused why

small_array[900] = a_simple_struct;

works, while

big_array[1000000] = a_simple_struct;

doesn't.

Is there some obvious error in my code? Have I just embarrassed
myself by displaying my obtusity before the comp.lang.c gods? Or is
there some better method of creating arrays (size_t - 1) ?

Thanks.

Aug 12 '07 #1
Share this Question
Share on Google+
35 Replies


P: n/a
Andrew Fabbro wrote:
>
I have a need to create an array that is larger than size_t - 1,
which I believe is the largest guaranteed array. I though I'd
found salvation in FAQ 6.14, but it appears I am not
understanding something.
You can't have it. No object whose size cannot be measured in a
size_t can exist. What are you trying to do?

--
Chuck F (cbfalconer at maineline dot net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home.att.net>

--
Posted via a free Usenet account from http://www.teranews.com

Aug 12 '07 #2

P: n/a

"CBFalconer" <cb********@yahoo.comwrote in message
news:46***************@yahoo.com...
Andrew Fabbro wrote:
>>
I have a need to create an array that is larger than size_t - 1,
which I believe is the largest guaranteed array. I though I'd
found salvation in FAQ 6.14, but it appears I am not
understanding something.

You can't have it. No object whose size cannot be measured in a
size_t can exist. What are you trying to do?
It can exist in the backing store.
As long as you don't need too many random accesses to your array, store it
on disk and read in chunks as you need.
Unfortunately the fseek() function quite often takes a long, which might not
be big enough. The fsetpos() takes yet another data type, fpos_t, and may
not be available.

--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

Aug 12 '07 #3

P: n/a
On Sat, 11 Aug 2007 22:09:57 -0700, Andrew Fabbro wrote:
I have a need to create an array that is larger than size_t - 1, which
I believe is the largest guaranteed array. I though I'd found
salvation in FAQ 6.14, but it appears I am not understanding
something.
On modern PCs, (size_t)(-1) is one byte less than 4 GB (on 32-bit
processors), or about 2.8 GB times the human population of the
Earth (on 64-bit processors). You are very likely *not* to have
enough memory, and if you have I think is impossible (or almost so)
to access it. What the hell are you trying to do? You might store
all those structs on a disk file.
(If you're programming for MS-DOS or some other system where it is
not absurd to have more memory than (size_t)(-1), go to
comp.os.msdos.programming or whatever.)
--
Army1987 (Replace "NOSPAM" with "email")
No-one ever won a game by resigning. -- S. Tartakower

Aug 12 '07 #4

P: n/a
Andrew Fabbro <an***********@gmail.comwrote:
I have a need to create an array that is larger than size_t - 1, which
I believe is the largest guaranteed array. I though I'd found
salvation in FAQ 6.14, but it appears I am not understanding
something.
The array is an array of pointers to structs. Here is an example
using a small array:
/* checks that malloc succeeded, main's return, etc. removed for
brevity */
#include <stdio.h>
#include <stdlib.h>
typedef struct {
int foo;
int bar;
} example_struct;
int main (void) {
example_struct * small_array[1000];
example_struct * a_single_struct;
a_single_struct = malloc ( sizeof(example_struct *) );
You already have a severe problem here: you don't allocate
enough memory. What you need to to allocate is not just a
pointer to an 'example_struct' (that you already have with
'a_single_struct'), but enough memory for an 'example_struct'.
So you need here

a_single_struct = malloc) sizeof( example_struct ) );
a_single_struct->foo = 4000;
With your alloction this only works by pure luck - if the
size of an 'example_struct' is larger than that of a pointer
to such a structure then you may be overwriting memory you
didn't allocate (since on most machines the size of a pointer
is at least as large as that of an int you may just have
avoided disaster since you only wrote to the 'foo' member,
but if you also write to the 'bar' member then things will
look worse....)
printf ("a_single_struct foo: %i \n", a_single_struct->foo);
small_array[900] = a_single_struct;
printf ("small_array 900 foo: %i\n",small_array[900]->foo);
}
This works fine:
a_single_struct foo: 4000
small_array 900 foo: 4000
Unfortunately, it only looks like it works fine.
So far, so good. Now I want to do an array of, say, 3,000,000
members, too large for a simple big_array[3000000] declaration. So I
use the code from faq 6.14:
/* checks that malloc succeeded, main's return, etc. removed for
brevity */
#include <stdio.h>
#include <stdlib.h>
typedef struct {
int foo;
int bar;
} example_struct;
int main (void) {
example_struct * big_array;
example_struct * a_single_struct;
a_single_struct = malloc ( sizeof(example_struct *) );
a_single_struct->foo = 4000;
printf ("a_single_struct foo: %i \n", a_single_struct->foo);
You have the same problem here as above...
big_array = malloc ( 3000000 * sizeof ( example_struct * ) );
'big_array' is a pointer to an 'example_struct', but you
allocate memory for 3000000 pointers to 'example_struct'
which you then assign to 'big_array'.
big_array[1000000] = a_single_struct; /* line 20, error below */
And here you get caught: since 'big_array' is of type
pointer to 'example_struct' big_array[1000000] is of
type 'example_struct', but what you try to assign it
is of type pointer to 'example_struct' and that's why
the compiler tells you
test.c:20: error: incompatible types in assignment
Since you want an array of pointers to 'example_struct'
you must define 'big_array' to be a pointer to pointer
to 'example_struct', i.e.

example_struct ** big_array;

to get things to work.
printf ("big_array foo: %i\n",big_array[1000000]->foo);
And since 'big_array[1000000]' is of type 'example_struct'
(and not pointer to such a structure) you can't use '->'
with that as the compiler tells you with
test.c:21: error: invalid type argument of '->'
Or is
there some better method of creating arrays (size_t - 1) ?
I don't think you really have a problem with 'size_t' here
and, BTW, size_t is a type and not a value, the maximum
value that can be stored in a variable of type size_t is
SIZE_MAX as defined in <stdint.h>, at least if your com-
piler supports this C99-ism (gcc does), which typically
is much larger than just 3 millions. I guess the problem
is rather that you want an array that's larger than the
amount of memory that the system allows you for local
"real" (fixed sized) array. But no, there's no better way
for creating large arrays (or arrays that have a variable
size) than using memory allocation at run-time.

Regards, Jens
--
\ Jens Thoms Toerring ___ jt@toerring.de
\__________________________ http://toerring.de
Aug 12 '07 #5

P: n/a
Malcolm McLean wrote, On 12/08/07 08:25:
>
"CBFalconer" <cb********@yahoo.comwrote in message
news:46***************@yahoo.com...
>Andrew Fabbro wrote:
>>>
I have a need to create an array that is larger than size_t - 1,
which I believe is the largest guaranteed array. I though I'd
found salvation in FAQ 6.14, but it appears I am not
understanding something.

You can't have it. No object whose size cannot be measured in a
size_t can exist. What are you trying to do?
It can exist in the backing store.
I C terms that is not an object.
As long as you don't need too many random accesses to your array, store
it on disk and read in chunks as you need.
Unfortunately the fseek() function quite often takes a long,
Actually, it *always* takes a long on a conforming implementation.
which might
not be big enough.
True.
The fsetpos() takes yet another data type, fpos_t,
No, it takes a pointer to fpos_t.
and may not be available.
It is not optional on hosted implementations so it will be available on
any hosted implementation claiming conformance to any version of the C
standard. It does, however, have the problem that the only value you can
portably pass to it is one obtained with fgetpos.

You also have the problem that the file system might not support a
sufficiently large file.
--
Flash Gordon
Aug 12 '07 #6

P: n/a
On Sun, 12 Aug 2007 11:13:38 +0200, in comp.lang.c , Army1987
<ar******@NOSPAM.itwrote:
>On modern PCs, (size_t)(-1) is one byte less than 4 GB (on 32-bit
processors), or about 2.8 GB times the human population of the
Earth (on 64-bit processors). You are very likely *not* to have
enough memory,
These days, blade PCs with 8+GB aren't that rare, especially in the
commercial sector.
>and if you have I think is impossible (or almost so)
to access it.
4 arrays of 2 GB ?

--
Mark McIntyre

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it."
--Brian Kernighan
Aug 12 '07 #7

P: n/a
Andrew Fabbro wrote:
I have a need to create an array that is larger than size_t - 1, which
I believe is the largest guaranteed array.
If you mean `(size_t)-1', you're out of luck: No object can
have more than SIZE_MAX bytes. Also, it's likely that the O/S
and the rest of your program will chew up some amount of memory,
so you won't even be able to get that high. It's a "theoretical"
upper limit, not a practical one.

Note that if an array's elements are bigger than one byte
each, then the theoretical upper limit on element count will be
even lower. For example, an array of eight-byte elements cannot
possibly have more than `SIZE_MAX / 8' array slots to stay
within the allowable limit on overall byte count.
I though I'd found
salvation in FAQ 6.14, but it appears I am not understanding
something.

The array is an array of pointers to structs. Here is an example
using a small array:

/* checks that malloc succeeded, main's return, etc. removed for
brevity */

#include <stdio.h>
#include <stdlib.h>

typedef struct {
int foo;
int bar;
} example_struct;

int main (void) {
example_struct * small_array[1000];
example_struct * a_single_struct;

a_single_struct = malloc ( sizeof(example_struct *) );
This line requests enough memory for one struct pointer,
not for one struct. In this sample program you only make
use of the first int in the struct, and that's probably why
you seem to get away with the error. But if you tried to
use the second int you'd be very likely to get into trouble.

Avoiding this and similar errors is the big advantage of
the c.l.c. Approved Allocation Idiom:

a_single_struct = malloc(sizeof *a_single_struct);

.... or more generally

pointer = malloc(count * sizeof *pointer);

One thing to be wary of when allocating "huge" arrays, even
with this improved form, is that the multiplication may not give
the right answer. If the mathematical product of the two terms
is larger than SIZE_MAX, the result is reduced modulo that
amount plus one and you'll wind up requesting a much smaller
block of memory. Your struct is probably about eight bytes
long; if `count' were `SIZE_MAX / 3', say, the mathematical
product would be about two and two-thirds times SIZE_MAX, and
the eventual result would be about two-thirds SIZE_MAX, or
roughly one-quarter the size you intended.

(Pedantry alert: The above description does not hold on
"exotic" machines where SIZE_MAX < INT_MAX. I've never heard
of such a machine, but the C Standard doesn't forbid them.)
a_single_struct->foo = 4000;
printf ("a_single_struct foo: %i \n", a_single_struct->foo);

small_array[900] = a_single_struct;
printf ("small_array 900 foo: %i\n",small_array[900]->foo);

}

This works fine:

a_single_struct foo: 4000
small_array 900 foo: 4000

So far, so good. Now I want to do an array of, say, 3,000,000
members, too large for a simple big_array[3000000] declaration. So I
use the code from faq 6.14:

/* checks that malloc succeeded, main's return, etc. removed for
brevity */

#include <stdio.h>
#include <stdlib.h>

typedef struct {
int foo;
int bar;
} example_struct;

int main (void) {
example_struct * big_array;
example_struct * a_single_struct;

a_single_struct = malloc ( sizeof(example_struct *) );
Same error as above.
a_single_struct->foo = 4000;
printf ("a_single_struct foo: %i \n", a_single_struct->foo);

big_array = malloc ( 3000000 * sizeof ( example_struct * ) );
This might be the same error as above, but if I understand
your purpose I think the actual error is that big_array is not
declared correctly. You may want

example_struct **big_array;

(The c.l.c. AAI works correctly with this declaration, too.)
big_array[1000000] = a_single_struct; /* line 20, error below */
printf ("big_array foo: %i\n",big_array[1000000]->foo);
These would be correct with the revised big_array declaration.
As things stand, big_array[1000000] is not a struct pointer, but
a struct instance. If that's what you want, you could use

big_array[1000000] = *a_single_struct;
printf (... big_array[1000000].foo);
>
}

No go. The compiler (gcc in this case) returns:

test.c: In function 'main':
test.c:20: error: incompatible types in assignment
test.c:21: error: invalid type argument of '->'

So the example in FAQ 6.14 is:

"int *dynarray;
dynarray = malloc(10 * sizeof(int));

...you can reference dynarray[i] (for i from 0 to 9) almost as if
dynarray were a conventional, statically-allocated array (int a[10])"

Given that, I'm confused why

small_array[900] = a_simple_struct;

works, while

big_array[1000000] = a_simple_struct;

doesn't.

Is there some obvious error in my code? Have I just embarrassed
myself by displaying my obtusity before the comp.lang.c gods? Or is
there some better method of creating arrays (size_t - 1) ?

Thanks.

--
Eric Sosman
es*****@ieee-dot-org.invalid
Aug 12 '07 #8

P: n/a
On Sun, 12 Aug 2007 13:47:21 +0100, Mark McIntyre wrote:
On Sun, 12 Aug 2007 11:13:38 +0200, in comp.lang.c , Army1987
<ar******@NOSPAM.itwrote:
>>On modern PCs, (size_t)(-1) is one byte less than 4 GB (on 32-bit
processors), or about 2.8 GB times the human population of the
Earth (on 64-bit processors). You are very likely *not* to have
enough memory,

These days, blade PCs with 8+GB aren't that rare, especially in the
commercial sector.
What kind of processor do they have? How wide is their size_t?
I still think that having more than 4GB on a computer with a 32
bit processor is *very* silly.

--
Army1987 (Replace "NOSPAM" with "email")
No-one ever won a game by resigning. -- S. Tartakower

Aug 12 '07 #9

P: n/a
On Sun, 12 Aug 2007 15:34:35 +0200, in comp.lang.c , Army1987
<ar******@NOSPAM.itwrote:
>>
These days, blade PCs with 8+GB aren't that rare, especially in the
commercial sector.
>What kind of processor do they have?
Itanium and Amd-64, typically.
>I still think that having more than 4GB on a computer with a 32
bit processor is *very* silly.
Its just trickier to use.
--
Mark McIntyre

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it."
--Brian Kernighan
Aug 12 '07 #10

P: n/a
Mark McIntyre <ma**********@spamcop.netwrites:
On Sun, 12 Aug 2007 15:34:35 +0200, in comp.lang.c , Army1987
<ar******@NOSPAM.itwrote:
>>These days, blade PCs with 8+GB aren't that rare, especially in the
commercial sector.
>>What kind of processor do they have?

Itanium and Amd-64, typically.
Right, and such systems almost certainly have a 64-bit size_t, which
makes them irrelevant to the current discussion unless they have more
than 16 exabytes of memory.

(On the other hand, both Itanium and x86-64 systems are capable of
running x86-32 executables.)
>>I still think that having more than 4GB on a computer with a 32
bit processor is *very* silly.

Its just trickier to use.
True. Leaving aside any specific architectures, the range of size_t
limits the size of a single object, not the size of memory. A system
with a 32-bit size_t can't have an object bigger than 4 gigabytes
(ignoring the old debate about calloc()), but there's no reason as far
as standard C is concerned why many such objects couldn't exist within
a single program, or within multiple programs.

If the objects in a single program can exceed 4 gigabytes, then void*
has to be bigger than 32 bits, since each byte of each object must
have a unique address. If no one object can exceed 4 gigabytes, then
size_t can be just 32 bits. (Note that I'm using the term "gigabyte"
here to refer to 2**32 bytes, not necessarily 2**32 octets.)

We've gotten so used to flat address spaces that we sometimes tend to
conflate these things.

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <* <http://users.sdsc.edu/~kst>
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"
Aug 12 '07 #11

P: n/a
You can't have it. No object whose size cannot be measured in a
size_t can exist. What are you trying to do?
Thank you all for your learned responses. But now it appears I'm
slightly more confused.

Without getting too platform-specific, on this platform, size_t is 32-
bit...so a declaration of some_array[3000000] should work, right? The
limit would be 4,294,967,295 (2^32-1), right?

My intention was to:

1. declare a 3000000-member array of struct pointers. This is because
I'm looking up individual structs by index number ("give me the info
for # 17272") and an array makes it simple. The real code has a small
linked list for each member and accessing data in this way has been
very fast.
2. malloc space for each pointer (I see what you all are saying - my
example was sloppy)
3. initialize each member
4. process data, etc.

However, this code fails, and I'm confused as to why:

#include <stdio.h>
#include <stdlib.h>

typedef struct {
int foo;
int bar;

} example_struct;

int main (void) {
example_struct big_array[3000000];
int i;

for (i = 0; i <= 2999999; i++ ) {
big_array[i].foo = i;
}

printf ("number 200 is %i\n",big_array[200].foo);
}

I get a segmentation fault (it's gcc under 32-bit linux, though that
shouldn't matter). BUT...if I change it to big_array[300] and change
the for loop limit to <=299, it works fine. So does that mean I'm
hitting some sort of limit?

It's not a memory limit, I don't think...the struct is 8 bytes. 3
million * 8 bytes = 24 million bytes. The box has 2GB of RAM. I
wrote code that mallocs 50,000,000 bytes as a test and the malloc call
succeeded (indeed, I have code that works with over 1GB of malloc'd
RAM with no problem).

So is there a limit to how many members you can have in an array? If
so, how do I determine what that limit is? Googling c.l.c says it's
size_t - 1, but....???

Aug 12 '07 #12

P: n/a
On Aug 12, 4:52 pm, Andrew Fabbro <andrew.fab...@gmail.comwrote:
I get a segmentation fault (it's gcc under 32-bit linux, though that
shouldn't matter). BUT...if I change it to big_array[300] and change
the for loop limit to <=299, it works fine. So does that mean I'm
hitting some sort of limit?

It's not a memory limit, I don't think...the struct is 8 bytes. 3
million * 8 bytes = 24 million bytes. The box has 2GB of RAM. I
wrote code that mallocs 50,000,000 bytes as a test and the malloc call
succeeded (indeed, I have code that works with over 1GB of malloc'd
RAM with no problem).
The limit lies not with the heap, but the stack space of your program.
The stack is generally used for small data sets (such as variables
with automatic storage duration and small to medium-sized arrays) and
often has a limit on the size it can grow to. Is there some reason you
don't wish to malloc() 3,000,000 example_struct objects instead?

Aug 12 '07 #13

P: n/a
Andrew Fabbro <an***********@gmail.comwrites:
int main (void) {
example_struct big_array[3000000];
[...]
}
You may be encountering a limitation on the size of an automatic
array, rather than a limitation on the size of an object.
Allocate the array statically or dynamically instead <OT>or try
"ulimit -s unlimited"</OT>.
--
"C has its problems, but a language designed from scratch would have some too,
and we know C's problems."
--Bjarne Stroustrup
Aug 12 '07 #14

P: n/a
Andrew Fabbro <an***********@gmail.comwrote:
You can't have it. No object whose size cannot be measured in a
size_t can exist. What are you trying to do?
Thank you all for your learned responses. But now it appears I'm
slightly more confused.
Without getting too platform-specific, on this platform, size_t is 32-
bit...so a declaration of some_array[3000000] should work, right? The
limit would be 4,294,967,295 (2^32-1), right?
No, that's rather unlikely. The maximum number you can store
in a variable if type size_t is typically not the maximum size
of an array you can create. Normally the amount of memory avai-
lable for auntomatic variables is much lower (and depends on the
compiler and platform). I don't remember the exact numbers but
if I am not completely off the mark the C89 standard doesn't re-
quire from the implementation that you can obtain more 2^14 bytes
(and the C99 standard more than 2^16 bytes) for all automatic
variables existing at once.
My intention was to:
1. declare a 3000000-member array of struct pointers. This is because
I'm looking up individual structs by index number ("give me the info
for # 17272") and an array makes it simple. The real code has a small
linked list for each member and accessing data in this way has been
very fast.
2. malloc space for each pointer (I see what you all are saying - my
example was sloppy)
3. initialize each member
4. process data, etc.
However, this code fails, and I'm confused as to why:
#include <stdio.h>
#include <stdlib.h>
typedef struct {
int foo;
int bar;
} example_struct;
int main (void) {
example_struct big_array[3000000];
It's rather likely this line that gets you into the mess since
you're probably not able to get that much memory for an automatic
variable (while it's not mandated by the C standard on many imple-
mentations automatic variables get created on the stack and the
size of the stack is usually much too small for that). Thus if
you need more than a few kB of memory you better use memory you
get from malloc().
int i;
for (i = 0; i <= 2999999; i++ ) {
big_array[i].foo = i;
}
printf ("number 200 is %i\n",big_array[200].foo);
}
I get a segmentation fault (it's gcc under 32-bit linux, though that
shouldn't matter). BUT...if I change it to big_array[300] and change
the for loop limit to <=299, it works fine. So does that mean I'm
hitting some sort of limit?
It's not a memory limit, I don't think...the struct is 8 bytes. 3
million * 8 bytes = 24 million bytes. The box has 2GB of RAM. I
wrote code that mallocs 50,000,000 bytes as a test and the malloc call
succeeded (indeed, I have code that works with over 1GB of malloc'd
RAM with no problem).
So is there a limit to how many members you can have in an array? If
so, how do I determine what that limit is? Googling c.l.c says it's
size_t - 1, but....???
It's not the number of elements of an automatic array that is
limited, it is the total amount of memory for all automatic
variables currently existing that is limited. And the limit is
typically quite a bit lower than what you are asking for. The
maximum value of a variable of type size_t is just a theoretical
upper limit for the maximum number of elements an array can have.
But on most systems you will hardly have enough memory to create
even a char array with memory obtained via malloc() that's that
large. But if you obtain the memory via malloc() you at least
have a chance to check if you could create the array since you
can test malloc()s return value - with too large automatic arrays
you often end up with a segmentation fault.

Regards, Jens
--
\ Jens Thoms Toerring ___ jt@toerring.de
\__________________________ http://toerring.de
Aug 12 '07 #15

P: n/a
On Sun, 12 Aug 2007 14:29:09 -0700, in comp.lang.c , Keith Thompson
<ks***@mib.orgwrote:
>Mark McIntyre <ma**********@spamcop.netwrites:
>On Sun, 12 Aug 2007 15:34:35 +0200, in comp.lang.c , Army1987
<ar******@NOSPAM.itwrote:
>>>These days, blade PCs with 8+GB aren't that rare, especially in the
commercial sector.
>>>What kind of processor do they have?

Itanium and Amd-64, typically.

Right, and such systems almost certainly have a 64-bit size_t, which
makes them irrelevant to the current discussion
I disagree, since the point I was responding to was where someone said
that putting more than 4GB in a PC was pointless.
--
Mark McIntyre

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it."
--Brian Kernighan
Aug 12 '07 #16

P: n/a
In article <11**********************@q75g2000hsh.googlegroups .com>
Andrew Fabbro <an***********@gmail.comwrote:
>Without getting too platform-specific, on this platform, size_t is 32-
bit...so a declaration of some_array[3000000] should work, right? The
limit would be 4,294,967,295 (2^32-1), right?
If SIZE_MAX (or (size_t)-1) is 4294967295, then, yes, there is an
absolute upper limit of that many bytes ("char"s) for each object.

That does not mean you can have an object that big, though.

Consider, as an analogy, the total number of cubic feet of cabin
space in your car. If your car's interior were all squared-off
(to make calculation easy) and were exactly 4 ft high by 6 ft long
by 5 ft wide, this would be 120 cubic feet of space. We could
then be sure that a 200-cubic-foot object would *not* fit.

This does not mean that any given 100-cubic-foot object *will* fit.
If that object happens to one foot high, one foot wide, and 100
feet long, it is not going to fit. :-)

Even if all your objects "pack nicely", if the cabin of the car
already contains something else -- such as yourself, driving --
this reduces the amount of cargo space.

(The way C objects work in the abstract, you can have "multiple
separate cars" each with however many cubic feet of cargo space,
too -- separate address spaces for separate objects -- but most
*implementations* just have one big space, or sometimes two or
three pre-carved spaces they may call "stack", "heap", and so
on.)

Ultimately, the actual limit on each object's size is "SIZE_MAX,
or whatever fits, whichever is *smaller*". (To call itself a C
implementation -- at least a "hosted" one -- the Standards-writers
say that this limit must be at least 32767 bytes in C89, or 65535
bytes in C99. Even this minimal requirement is a bit weasel-worded,
though, allowing implementors to give you less in practice.)
--
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (4039.22'N, 11150.29'W) +1 801 277 2603
email: forget about it http://web.torek.net/torek/index.html
Reading email is like searching for food in the garbage, thanks to spammers.
Aug 12 '07 #17

P: n/a
Keith Thompson wrote:
Mark McIntyre <ma**********@spamcop.netwrites:
>On Sun, 12 Aug 2007 15:34:35 +0200, in comp.lang.c , Army1987
<ar******@NOSPAM.itwrote:
>>>These days, blade PCs with 8+GB aren't that rare, especially in the
commercial sector.
What kind of processor do they have?
Itanium and Amd-64, typically.

Right, and such systems almost certainly have a 64-bit size_t, which
makes them irrelevant to the current discussion unless they have more
than 16 exabytes of memory.

(On the other hand, both Itanium and x86-64 systems are capable of
running x86-32 executables.)
>>I still think that having more than 4GB on a computer with a 32
bit processor is *very* silly.
Its just trickier to use.

True. Leaving aside any specific architectures, the range of size_t
limits the size of a single object, not the size of memory. A system
with a 32-bit size_t can't have an object bigger than 4 gigabytes
(ignoring the old debate about calloc()), but there's no reason as far
as standard C is concerned why many such objects couldn't exist within
a single program, or within multiple programs.

If the objects in a single program can exceed 4 gigabytes, then void*
has to be bigger than 32 bits, since each byte of each object must
have a unique address. If no one object can exceed 4 gigabytes, then
size_t can be just 32 bits. (Note that I'm using the term "gigabyte"
here to refer to 2**32 bytes, not necessarily 2**32 octets.)
Well, gigabyte would be 2**30 of course.
We've gotten so used to flat address spaces that we sometimes tend to
conflate these things.
--
Joe Wright
"Everything should be made as simple as possible, but not simpler."
--- Albert Einstein ---
Aug 13 '07 #18

P: n/a
Mark McIntyre <ma**********@spamcop.netwrites:
On Sun, 12 Aug 2007 14:29:09 -0700, in comp.lang.c , Keith Thompson
<ks***@mib.orgwrote:
>>Mark McIntyre <ma**********@spamcop.netwrites:
>>On Sun, 12 Aug 2007 15:34:35 +0200, in comp.lang.c , Army1987
<ar******@NOSPAM.itwrote:
These days, blade PCs with 8+GB aren't that rare, especially in the
commercial sector.

What kind of processor do they have?

Itanium and Amd-64, typically.

Right, and such systems almost certainly have a 64-bit size_t, which
makes them irrelevant to the current discussion

I disagree, since the point I was responding to was where someone said
that putting more than 4GB in a PC was pointless.
The actual statement, from Army1987, was:
| What kind of processor do they have? How wide is their size_t?
| I still think that having more than 4GB on a computer with a 32
| bit processor is *very* silly.

Looking upthread, I don't see any statement that putting more than 4BG
on a PC is pointless.

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <* <http://users.sdsc.edu/~kst>
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"
Aug 13 '07 #19

P: n/a
Joe Wright <jo********@comcast.netwrites:
Keith Thompson wrote:
[...]
>If the objects in a single program can exceed 4 gigabytes, then void*
has to be bigger than 32 bits, since each byte of each object must
have a unique address. If no one object can exceed 4 gigabytes, then
size_t can be just 32 bits. (Note that I'm using the term "gigabyte"
here to refer to 2**32 bytes, not necessarily 2**32 octets.)
Well, gigabyte would be 2**30 of course.
D'oh! You're right, of course. (It's 4 gigabytes that's 2**32.)

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <* <http://users.sdsc.edu/~kst>
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"
Aug 13 '07 #20

P: n/a
On Mon, 13 Aug 2007 00:05:33 +0100, Mark McIntyre wrote:
On Sun, 12 Aug 2007 14:29:09 -0700, in comp.lang.c , Keith Thompson
<ks***@mib.orgwrote:
>>Mark McIntyre <ma**********@spamcop.netwrites:
>>On Sun, 12 Aug 2007 15:34:35 +0200, in comp.lang.c , Army1987
<ar******@NOSPAM.itwrote:
These days, blade PCs with 8+GB aren't that rare, especially in the
commercial sector.

What kind of processor do they have?

Itanium and Amd-64, typically.

Right, and such systems almost certainly have a 64-bit size_t, which
makes them irrelevant to the current discussion

I disagree, since the point I was responding to was where someone said
that putting more than 4GB in a PC was pointless.
I didn't say exactly that. I said that putting more than 4 GB on
a computer with a 32-bit processor is silly. (The case of more
than 8 exabytes on a 64-bit processor applies, too, but this time
I don't think it is silly, I think it is absurd.)
--
Army1987 (Replace "NOSPAM" with "email")
No-one ever won a game by resigning. -- S. Tartakower

Aug 13 '07 #21

P: n/a
On Sun, 12 Aug 2007 21:52:20 +0000, Andrew Fabbro wrote:
>You can't have it. No object whose size cannot be measured in a
size_t can exist. What are you trying to do?

Thank you all for your learned responses. But now it appears I'm
slightly more confused.

Without getting too platform-specific, on this platform, size_t is 32-
bit...so a declaration of some_array[3000000] should work, right? The
limit would be 4,294,967,295 (2^32-1), right?

My intention was to:

1. declare a 3000000-member array of struct pointers. This is because
I'm looking up individual structs by index number ("give me the info
for # 17272") and an array makes it simple. The real code has a small
linked list for each member and accessing data in this way has been
very fast.
2. malloc space for each pointer (I see what you all are saying - my
example was sloppy)
3. initialize each member
4. process data, etc.

However, this code fails, and I'm confused as to why:

#include <stdio.h>
#include <stdlib.h>

typedef struct {
int foo;
int bar;

} example_struct;

int main (void) {
example_struct big_array[3000000];
int i;

for (i = 0; i <= 2999999; i++ ) {
big_array[i].foo = i;
}

printf ("number 200 is %i\n",big_array[200].foo);
}

I get a segmentation fault (it's gcc under 32-bit linux, though that
shouldn't matter). BUT...if I change it to big_array[300] and change
the for loop limit to <=299, it works fine. So does that mean I'm
hitting some sort of limit?
Maybe you can't declare an automatic object that big. Try
declaring it as static. If it is still too big, allocate it with
malloc:
example_struct *bigarray = malloc(3000000 * sizeof *bigarray);
HTH.
--
Army1987 (Replace "NOSPAM" with "email")
No-one ever won a game by resigning. -- S. Tartakower

Aug 13 '07 #22

P: n/a
On Sun, 12 Aug 2007 17:51:12 -0700, in comp.lang.c , Keith Thompson
<ks***@mib.orgwrote:
>
The actual statement, from Army1987, was:
| What kind of processor do they have? How wide is their size_t?
| I still think that having more than 4GB on a computer with a 32
| bit processor is *very* silly.
In response to my point about having access to _PCs_ with 8+GB memory.
>Looking upthread, I don't see any statement that putting more than
4BG
>on a PC is pointless.
*shrug*. Read it any way you prefer. IMHO someone here is being being
gratuitously pedantic.
--
Mark McIntyre

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it."
--Brian Kernighan
Aug 13 '07 #23

P: n/a
On Mon, 13 Aug 2007 12:00:40 +0200, in comp.lang.c , Army1987
<ar******@NOSPAM.itwrote:
>a computer with a 32-bit processor is silly. (The case of more
than 8 exabytes on a 64-bit processor applies, too, but this time
I don't think it is silly, I think it is absurd.)
:-)
I'm curiously reminded of the famous "640MB" comment...

By the way, imagine you had a multiprocessor x86 box, with suitable
h/w or firmware to permit each CPU to use 2GB of independent memory,
plus a shared 2GB pool....

--
Mark McIntyre

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it."
--Brian Kernighan
Aug 13 '07 #24

P: n/a
Army1987 <ar******@NOSPAM.itwrote:

# I didn't say exactly that. I said that putting more than 4 GB on
# a computer with a 32-bit processor is silly. (The case of more
# than 8 exabytes on a 64-bit processor applies, too, but this time
# I don't think it is silly, I think it is absurd.)

Depends on the system. Multi-tasking systems like Unix can
easily have aggregate memory needs exceeding the address
space of any one process.

# No-one ever won a game by resigning. -- S. Tartakower

Wrong, actually. Successful generals and businessmen know
to abandon a losing position to preserve their resources
so those are available to exploit future opportunities.

--
SM Ryan http://www.rawbw.com/~wyrmwif/
What kind of convenience store do you run here?
Aug 13 '07 #25

P: n/a
Mark McIntyre <ma**********@spamcop.netwrites:
On Sun, 12 Aug 2007 17:51:12 -0700, in comp.lang.c , Keith Thompson
<ks***@mib.orgwrote:
>>The actual statement, from Army1987, was:
| What kind of processor do they have? How wide is their size_t?
| I still think that having more than 4GB on a computer with a 32
| bit processor is *very* silly.

In response to my point about having access to _PCs_ with 8+GB memory.
>>Looking upthread, I don't see any statement that putting more than
4BG
>>on a PC is pointless.

*shrug*. Read it any way you prefer. IMHO someone here is being being
gratuitously pedantic.
The statement was explicitly about 32-bit systems. You responded with
a point about 64-bit systems. As usual, you refuse to admit your
mistake.

You might consider worrying less about "pedantry" and more about
simple correctness.

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <* <http://users.sdsc.edu/~kst>
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"
Aug 13 '07 #26

P: n/a
On Mon, 13 Aug 2007 13:21:41 -0700, in comp.lang.c , Keith Thompson
<ks***@mib.orgwrote:
>
The statement was explicitly about 32-bit systems. You responded with
a point about 64-bit systems.
My statement was explicitly about PCs as you;d see if you had actually
bothered to read what I wrote, instead of assuming you were right and
I was wrong.
>As usual, you refuse to admit your mistake.
"As usual"? You pompous..... .

I've nothing more to say on this topic thats polite. You've insulted
me gratuitously, made false claims about my behaviour, and generally
been unpleasant.

--
I refuse to have a battle of wits with an unarmed person.
Aug 13 '07 #27

P: n/a
Mark McIntyre <ma**********@spamcop.netwrites:
On Mon, 13 Aug 2007 13:21:41 -0700, in comp.lang.c , Keith Thompson
<ks***@mib.orgwrote:
>>
The statement was explicitly about 32-bit systems. You responded with
a point about 64-bit systems.

My statement was explicitly about PCs as you;d see if you had actually
bothered to read what I wrote, instead of assuming you were right and
I was wrong.
It is a general consensus on your posts it seems.
>
>>As usual, you refuse to admit your mistake.

"As usual"? You pompous..... .

I've nothing more to say on this topic thats polite. You've insulted
me gratuitously, made false claims about my behaviour, and generally
been unpleasant.
LOL. You can dish it, but not take it. Grow up.
Aug 13 '07 #28

P: n/a
On Aug 12, 3:06 pm, Justin Spahr-Summers
<Justin.SpahrSumm...@gmail.comwrote:
The limit lies not with the heap, but the stack space of your program.
The stack is generally used for small data sets (such as variables
with automatic storage duration and small to medium-sized arrays) and
often has a limit on the size it can grow to. Is there some reason you
don't wish to malloc() 3,000,000 example_struct objects instead?
If I malloc the 3,000,000 structs, I would then want an array of
pointers to point to them, but...the array can't be 3,000,000 pointers
long :-(

Yes, there are other structures, but I was hoping for something that
was index-based (like an array), since that's how I work with the
data.
Aug 13 '07 #29

P: n/a
Andrew Fabbro <an***********@gmail.comwrote:
On Aug 12, 3:06 pm, Justin Spahr-Summers
<Justin.SpahrSumm...@gmail.comwrote:
The limit lies not with the heap, but the stack space of your program.
The stack is generally used for small data sets (such as variables
with automatic storage duration and small to medium-sized arrays) and
often has a limit on the size it can grow to. Is there some reason you
don't wish to malloc() 3,000,000 example_struct objects instead?
If I malloc the 3,000,000 structs, I would then want an array of
pointers to point to them, but...the array can't be 3,000,000 pointers
long :-(
Yes, there are other structures, but I was hoping for something that
was index-based (like an array), since that's how I work with the
data.
Allocating 3,000,000 structures inividually probably isn't a
very good idea, it's going to take quite a bit of CPU time.
Why not allocate them all at once like in

example_struct *sa = malloc( 3000000 * sizeof *sa );

Then you don't need another array of pointers to all of them
since the addresses are easy to obtain: the address of the Nth
structure (where the first is at N = 0) is simple 'sa + N'.

But if you still think you need such an array of pointers just
get it also via malloc().
Regards, Jens
--
\ Jens Thoms Toerring ___ jt@toerring.de
\__________________________ http://toerring.de
Aug 13 '07 #30

P: n/a
Jens Thoms Toerring wrote:
Andrew Fabbro <an***********@gmail.comwrote:
>On Aug 12, 3:06 pm, Justin Spahr-Summers
<Justin.SpahrSumm...@gmail.comwrote:
>>The limit lies not with the heap, but the stack space of your program.
The stack is generally used for small data sets (such as variables
with automatic storage duration and small to medium-sized arrays) and
often has a limit on the size it can grow to. Is there some reason you
don't wish to malloc() 3,000,000 example_struct objects instead?
>If I malloc the 3,000,000 structs, I would then want an array of
pointers to point to them, but...the array can't be 3,000,000 pointers
long :-(
>Yes, there are other structures, but I was hoping for something that
was index-based (like an array), since that's how I work with the
data.

Allocating 3,000,000 structures inividually probably isn't a
very good idea, it's going to take quite a bit of CPU time.
Why not allocate them all at once like in

example_struct *sa = malloc( 3000000 * sizeof *sa );

Then you don't need another array of pointers to all of them
since the addresses are easy to obtain: the address of the Nth
structure (where the first is at N = 0) is simple 'sa + N'.
A suspicion: Maybe he has only a (relatively) small
number of (relatively) large structs, and either the array
of pointers holds mostly NULLs or else a single struct
instance may be pointed to by many array elements. Am I
near the mark?

Perhaps it's time to take a step back and explain more
about the overall problem you're trying to solve, rather
than asking about the particular tactic you've decided to
try to solve it with. Somebody may well have an idea for
an approach that doesn't bog down so badly.

--
Eric Sosman
es*****@ieee-dot-org.invalid
Aug 14 '07 #31

P: n/a
Allocating 3,000,000 structures inividually probably isn't a
very good idea, it's going to take quite a bit of CPU time.
In another application, I allocated a total of 100,000,000 8-byte
structs, reading from an 800MB input file, and it only took about 20
seconds. It consumed a whole lotta rAM, but since the application was
doing nothing but computing correlations and other stats on those
structures and writing the output, it was much faster to do it all in
memory than to fetch parts from disk as needed.
Aug 14 '07 #32

P: n/a
Andrew Fabbro <an***********@gmail.comwrote:

Please don't remove attributions.
Allocating 3,000,000 structures inividually probably isn't a
very good idea, it's going to take quite a bit of CPU time.
In another application, I allocated a total of 100,000,000 8-byte
structs, reading from an 800MB input file, and it only took about 20
seconds. It consumed a whole lotta rAM, but since the application was
doing nothing but computing correlations and other stats on those
structures and writing the output, it was much faster to do it all in
memory than to fetch parts from disk as needed.
I didn't mean that you shouldn't allocate 3,000,000 structures, I was
just recommending not to call malloc() 3,000,000 times instead of
calling it just once, requesting memory for all of them at the same
time (at least if it's possible).
Regards, Jens
--
\ Jens Thoms Toerring ___ jt@toerring.de
\__________________________ http://toerring.de
Aug 14 '07 #33

P: n/a
[snips]

On Mon, 13 Aug 2007 12:00:40 +0200, Army1987 wrote:
I didn't say exactly that. I said that putting more than 4 GB on
a computer with a 32-bit processor is silly. (The case of more
than 8 exabytes on a 64-bit processor applies, too, but this time
I don't think it is silly, I think it is absurd.)
Not to mention barking expensive. :)
Aug 14 '07 #34

P: n/a
Kelsey Bjarnason <kb********@gmail.comwrote:
On Mon, 13 Aug 2007 12:00:40 +0200, Army1987 wrote:
I didn't say exactly that. I said that putting more than 4 GB on
a computer with a 32-bit processor is silly. (The case of more
than 8 exabytes on a 64-bit processor applies, too, but this time
I don't think it is silly, I think it is absurd.)

Not to mention barking expensive. :)
Patience, patience...

Richard
Aug 15 '07 #35

P: n/a
On Wed, 15 Aug 2007 06:10:34 +0000, Richard Bos wrote:
Kelsey Bjarnason <kb********@gmail.comwrote:
>On Mon, 13 Aug 2007 12:00:40 +0200, Army1987 wrote:
I didn't say exactly that. I said that putting more than 4 GB on
a computer with a 32-bit processor is silly. (The case of more
than 8 exabytes on a 64-bit processor applies, too, but this time
I don't think it is silly, I think it is absurd.)

Not to mention barking expensive. :)

Patience, patience...
We will get 128-bit processors much sooner than that.
--
Army1987 (Replace "NOSPAM" with "email")
No-one ever won a game by resigning. -- S. Tartakower

Aug 20 '07 #36

This discussion thread is closed

Replies have been disabled for this discussion.