handling large data

[a]

Would anybody please tell me how to handle array data of int 2000 * 2000?
Even make file passes by using a supercomputer, it seems calloc fails and
segmentation fault results during a trial run.

[a]

"a" <[e-mail] in message news:fi**********@justice.itsc.cuhk.edu.hk...

Would anybody please tell me how to handle array data of int 2000 * 2000?
Even make file passes by using a supercomputer, it seems calloc fails and
segmentation fault results during a trial run.

Make error:

gcc -O3 -Wall -I. vote.c datamanager.c -o vote -L. -lhungarian
gcc: Internal error: Segmentation fault (program cc1)
Please submit a full bug report.
See <URL:http://bugzilla.redhat.com/bugzilla/for instructions.
make: *** [vote] Error 1
Run-time (made in a supercomputer) error:
Segmentation fault
The program snippet:
int r[2000*2000]={
123,188,158,197,184,180,146,129,117,192,120,148,17 5,198,153,190,135,175,158,155,157,192,108,187,148, 187,199,139,158,120,174,168,184,162,132,191,166,11 3,167,148,131,159,107,120,164,115,115,110,117,105, 131,1
....};

[Ian Collins]

a wrote:

"a" <[e-mail] in message news:fi**********@justice.itsc.cuhk.edu.hk...

>Would anybody please tell me how to handle array data of int 2000 * 2000?
Even make file passes by using a supercomputer, it seems calloc fails and
segmentation fault results during a trial run.

Make error:

gcc -O3 -Wall -I. vote.c datamanager.c -o vote -L. -lhungarian
gcc: Internal error: Segmentation fault (program cc1)
Please submit a full bug report.
See <URL:http://bugzilla.redhat.com/bugzilla/for instructions.
make: *** [vote] Error 1

Maybe it ran out of space?

>
The program snippet:
int r[2000*2000]={
123,188,158,197,184,180,146,129,117,192,120,148,17 5,198,153,190,135,175,158,155,157,192,108,187,148, 187,199,139,158,120,174,168,184,162,132,191,166,11 3,167,148,131,159,107,120,164,115,115,110,117,105, 131,1
....};

Put the data in a files and read it in at runtime instead of using
static initialisation.

--
Ian Collins.

[CBFalconer]

a wrote:

>
Would anybody please tell me how to handle array data of int 2000
* 2000? Even make file passes by using a supercomputer, it seems
calloc fails and segmentation fault results during a trial run.

Assuming you have a fairly normal system with sizeof int == 4, that
array requires 4,000,000 * 4, or 16 gigabytes of memory. Note that
the guaranteed available size is 65k (for C99) and 32k (for C90).
Many systems will give you 2 gigabytes, but only some 64 bit
systems will allow 16 gigs.

Look for another mechanism.

--
Chuck F (cbfalconer at maineline dot net)
<http://cbfalconer.home.att.net>
Try the download section.

--
Posted via a free Usenet account from http://www.teranews.com

[Ian Collins]

CBFalconer wrote:

a wrote:

>Would anybody please tell me how to handle array data of int 2000
* 2000? Even make file passes by using a supercomputer, it seems
calloc fails and segmentation fault results during a trial run.

Assuming you have a fairly normal system with sizeof int == 4, that
array requires 4,000,000 * 4, or 16 gigabytes of memory.

s/giga/mega/ which leave a manageable answer, but maybe not for static
initialisation.

--
Ian Collins.

[Richard Heathfield]

CBFalconer said:

a wrote:

>>
Would anybody please tell me how to handle array data of int 2000
* 2000? Even make file passes by using a supercomputer, it seems
calloc fails and segmentation fault results during a trial run.

Assuming you have a fairly normal system with sizeof int == 4, that
array requires 4,000,000 * 4, or 16 gigabytes of memory.

ITYM "megabytes", which isn't all *that* big really (although it's pretty
hefty for a local array).

One malloc for the row pointers, and one malloc per row, should do it
nicely on modern desktop systems with not too much else running.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999

[CBFalconer]

Richard Heathfield wrote:

CBFalconer said:

>a wrote:

>>>
Would anybody please tell me how to handle array data of int 2000
* 2000? Even make file passes by using a supercomputer, it seems
calloc fails and segmentation fault results during a trial run.

Assuming you have a fairly normal system with sizeof int == 4, that
array requires 4,000,000 * 4, or 16 gigabytes of memory.

ITYM "megabytes", which isn't all *that* big really (although it's
pretty hefty for a local array).

One malloc for the row pointers, and one malloc per row, should do
it nicely on modern desktop systems with not too much else running.

I keep doing that - missing a factor of 1000, or 10 cubed. Maybe I
need three less fingers. However, I did assume that he wanted a
single array, which is way over the minimum guarantee.

--
Chuck F (cbfalconer at maineline dot net)
<http://cbfalconer.home.att.net>
Try the download section.

--
Posted via a free Usenet account from http://www.teranews.com

[Philip Potter]

CBFalconer wrote:

a wrote:

>Would anybody please tell me how to handle array data of int 2000
* 2000? Even make file passes by using a supercomputer, it seems
calloc fails and segmentation fault results during a trial run.

Assuming you have a fairly normal system with sizeof int == 4, that
array requires 4,000,000 * 4, or 16 gigabytes of memory. Note that
the guaranteed available size is 65k (for C99) and 32k (for C90).

Er, you mean megabytes. As such, the argument below is irrelevant.

Many systems will give you 2 gigabytes, but only some 64 bit
systems will allow 16 gigs.

Look for another mechanism.

[Philip Potter]

Ian Collins wrote:

a wrote:

>"a" <[e-mail] in message news:fi**********@justice.itsc.cuhk.edu.hk...

>>Would anybody please tell me how to handle array data of int 2000 * 2000?
Even make file passes by using a supercomputer, it seems calloc fails and
segmentation fault results during a trial run.

Make error:

gcc -O3 -Wall -I. vote.c datamanager.c -o vote -L. -lhungarian
gcc: Internal error: Segmentation fault (program cc1)
Please submit a full bug report.
See <URL:http://bugzilla.redhat.com/bugzilla/for instructions.
make: *** [vote] Error 1

Maybe it ran out of space?

Even if it does, it should fail cleanly rather than segfault.

[santosh]

In article <fi**********@justice.itsc.cuhk.edu.hk>, a <[e-mail] on
Saturday 24 Nov 2007 7:28 am:

Would anybody please tell me how to handle array data of int 2000 *
2000? Even make file passes by using a supercomputer, it seems calloc
fails and segmentation fault results during a trial run.

If you do not have sufficient primary storage the only option is to
operate on smaller chunks of the data. From the standpoint of Standard
C, you have no recourse upon a *alloc() failure, but to reduce the
requirements and try again. However a specific system may provide a
considerable amount of additional strategies and methods to cope with
large data.

With modern operating systems the system will do a lot of
behind-the-scenes work for you, so that often, you can simply do the
allocation and proceed. If however this does not work, you might try
memory-mapped files.

[Malcolm McLean]

"Richard Heathfield" <rj*@see.sig.invalidwrote in message

>

2000 * 2000 * int

ITYM "megabytes", which isn't all *that* big really (although it's pretty
hefty for a local array).

One malloc for the row pointers, and one malloc per row, should do it
nicely on modern desktop systems with not too much else running.

That's just a fairly normal sized image. How did we use to manage with
little 64k systems?

--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

[Richard Tobin]

In article <fi**********@justice.itsc.cuhk.edu.hk>, a <[e-mail]:

>Would anybody please tell me how to handle array data of int 2000 * 2000?
Even make file passes by using a supercomputer, it seems calloc fails and
segmentation fault results during a trial run.

If calloc() fails trying to allocate 2000*2000 ints, you almost
certainly have a system limit set too low. Ask your system
administrator how to change it (presumably you have a system
administrator if you have a supercomputer). You should of course be
checking for calloc() failing so that you don't get a segmentation
fault.

Or else there's some other bug in your program and it's nothing to do
with the amount of memory. To check for that possibility use a
program such as valgrind.

-- Richard

--
"Consideration shall be given to the need for as many as 32 characters
in some alphabets" - X3.4, 1963.

[a]

it seems that big static *2D* array creates the problem, e.g.
a[1024][1024] but not a[1024*1024] makes the problem.

i try gcc -static to see if this helps but of no hope.

there will be too many modifications to take care when I switch to malloc or
related solutions. Is there any workaround?

[a]

Which file is to be changed? I also remember there should be a config file
to set many parameters...
"Richard Tobin" <ri*****@cogsci.ed.ac.ukwrote in message
news:fi***********@pc-news.cogsci.ed.ac.uk...

In article <fi**********@justice.itsc.cuhk.edu.hk>, a <[e-mail]:

>>Would anybody please tell me how to handle array data of int 2000 * 2000?
Even make file passes by using a supercomputer, it seems calloc fails and
segmentation fault results during a trial run.

If calloc() fails trying to allocate 2000*2000 ints, you almost
certainly have a system limit set too low. Ask your system
administrator how to change it (presumably you have a system
administrator if you have a supercomputer). You should of course be
checking for calloc() failing so that you don't get a segmentation
fault.

Or else there's some other bug in your program and it's nothing to do
with the amount of memory. To check for that possibility use a
program such as valgrind.

-- Richard

--
"Consideration shall be given to the need for as many as 32 characters
in some alphabets" - X3.4, 1963.

[Vitor Py]

On 23 nov, 23:58, "a" <[e-mail]:

Would anybody please tell me how to handle array data of int 2000 * 2000?
Even make file passes by using a supercomputer, it seems calloc fails and
segmentation fault results during a trial run.

If you're handling matrices, you may want study them use more
apropriate storage. Let's say, if you have a three diagonal matrix,
use 3 arrays for storage (it's going to make your code uglier, and
require more work - but it's gonna be really faster). Diagonal
matrices are fairly common from problems arising from finite
differences or finite elements. Allocating them as full matrices
usually is not worth it's price.

Vitor.
PS: I'm not a native speaker of english, so, pardon for my errors.

[Barry Schwarz]

On Sat, 24 Nov 2007 22:32:37 +0800, "a" <[e-mail]:

>it seems that big static *2D* array creates the problem, e.g.
a[1024][1024] but not a[1024*1024] makes the problem.

i try gcc -static to see if this helps but of no hope.

there will be too many modifications to take care when I switch to malloc or
related solutions. Is there any workaround?

What is the actual type that size_t is a typedef for? What is the
maximum value for that type?

Since both a[1024][1024] and a[1024*1024] require the exact same
space, if one succeeds and the other fails then something else is
going on. A likely candidate is that SIZE_MAX is less than one
million and the multiplication in the latter declaration is wrapping
to some tolerable value. You should print the size of a in this case.

In any event, changing from a defined array to an allocated one
requires replacing only the one line declaration
int a[1024][1024];
with the small loop
int *a[1024];
for (i = 0; i < 1024; i++)
a[i] = malloc(1024 * sizeof *a[i]);
From then on, all references in your code to a[i][j] should remain
valid without modification.

Remove del for email

[Gordon]

On Nov 24, 9:58 am, "a" <[e-mail]:

Would anybody please tell me how to handle array data of int 2000 * 2000?
Even make file passes by using a supercomputer, it seems calloc fails and
segmentation fault results during a trial run.