100k X 100k data processing summary

[a]

By previous replies, it seems that the following method somehow solves the
problem up to 1000 * 1000 2D data, but when I try 10k * 10k, the
segmentation fault problem appears again.

Richard Tobin told me there is a system limit that can be changed. But I
don't know which file is to be changed.

I have modified again and again and hope to find out a solution that can
handle 100k * 100k data.

float** array_to_matrix (float* m, int rows, int cols) {
int i,j;

float** r;

r = (float**)calloc (rows,sizeof(fl oat*));

for(i=0;i<rows; i++)
{
r[i] = (float*)calloc( cols,sizeof(flo at));

for(j=0;j<cols; j++)
r[i][j] = m[i*cols+j];
}
return r;

}

[Ian Collins]

a wrote:

By previous replies, it seems that the following method somehow solves the
problem up to 1000 * 1000 2D data, but when I try 10k * 10k, the
segmentation fault problem appears again.

System memory is finite, if you attempt to allocate more than there is
available, you will fail.

Richard Tobin told me there is a system limit that can be changed. But I
don't know which file is to be changed.

I have modified again and again and hope to find out a solution that can
handle 100k * 100k data.

Which is 10GB*sizeof(flo at), do you have that much (virtual) memory to
play with?

It sound like you have more of an algorithm problem than a C one.

float** array_to_matrix (float* m, int rows, int cols) {
int i,j;

float** r;

r = (float**)calloc (rows,sizeof(fl oat*));

Drop the cast.

--
Ian Collins.

[Ian Collins]

Ian Collins wrote:

a wrote:

>By previous replies, it seems that the following method somehow solves the
problem up to 1000 * 1000 2D data, but when I try 10k * 10k, the
segmentation fault problem appears again.

System memory is finite, if you attempt to allocate more than there is
available, you will fail.

>Richard Tobin told me there is a system limit that can be changed. But I
don't know which file is to be changed.

I have modified again and again and hope to find out a solution that can
handle 100k * 100k data.

Which is 10GB*sizeof(flo at), do you have that much (virtual) memory to
play with?

It sound like you have more of an algorithm problem than a C one.

>float** array_to_matrix (float* m, int rows, int cols) {
int i,j;

float** r;

r = (float**)calloc (rows,sizeof(fl oat*));

Drop the cast.

And always check the return of [mc]alloc isn't null.

--
Ian Collins.

[Richard Tobin]

In article <fi**********@j ustice.itsc.cuh k.edu.hk>, a <[e-mail] :

>Richard Tobin told me there is a system limit that can be changed. But I
don't know which file is to be changed.

As I said, ask your system administrator. Or read the manual.
We can't tell you, because you haven't even told us what system
you're using.

>I have modified again and again and hope to find out a solution that can
handle 100k * 100k data.

100k * 100k is 10g, and if floats are 4 bytes that's 40 gigabytes.
You really will need a supercomputer for that. Perhaps you should
reconsider your algorithm, or wait several years.

-- Richard
--
"Considerat ion shall be given to the need for as many as 32 characters
in some alphabets" - X3.4, 1963.

[Ian Collins]

>
100k * 100k is 10g, and if floats are 4 bytes that's 40 gigabytes.
You really will need a supercomputer for that. Perhaps you should
reconsider your algorithm, or wait several years.

<OTWorkstatio n boards with support for 64GB of RAM are available from
several vendors! All it takes is a spare $10K for the RAM... </OT>

--
Ian Collins.

[Johannes Bauer]

Richard Tobin schrieb:

100k * 100k is 10g, and if floats are 4 bytes that's 40 gigabytes.
You really will need a supercomputer for that. Perhaps you should
reconsider your algorithm, or wait several years.

It might work if enough virtual memory is available - serious thrashing
implied. Suitable for *some* problems, however, if access pattern to
this matrix are not arbitrary (which they aren't for most algorithms).

Usually data in such huge matrices is sparse anyways - so, I fully
second your statement the OP should reconsider his algorithm. If it
fails in the early stage of memory allocation he/she probably hasn't
thourhgt about it at all.

Greetings,
Johannes

--
"Viele der Theorien der Mathematiker sind falsch und klar
Gotteslästerli ch. Ich vermute, dass diese falschen Theorien genau
deshalb so geliebt werden." -- Prophet und Visionär Hans Joss aka
HJP in de.sci.mathemat ik <47************ **********@news .sunrise.ch>

[Gordon Burditt]

>100k * 100k is 10g, and if floats are 4 bytes that's 40 gigabytes.

>You really will need a supercomputer for that. Perhaps you should
reconsider your algorithm, or wait several years.

It might work if enough virtual memory is available - serious thrashing
implied.

On a 32-bit machine (say, Pentium with PAE36) you could have 64G
of physical memory, and a lot more physical swap/page space, but
with a 32-bit address space for an individual process, you're limited
to 4G (and sometimes much less). So you need a machine with 64-bit
addressing and an OS that supports it for individual processes.
Simply adding lots of memory and swap/page space isn't enough.

>Suitable for *some* problems, however, if access pattern to
this matrix are not arbitrary (which they aren't for most algorithms).

Usually data in such huge matrices is sparse anyways - so, I fully
second your statement the OP should reconsider his algorithm. If it
fails in the early stage of memory allocation he/she probably hasn't
thourhgt about it at all.

[a]

Thanks Richard. It's red hat enterprise, or Fedora related. The biggest
problem I'm having is that I don't know the keyword because compilation
memory program memory alike doesn't give me good results in google search.
"Richard Tobin" <ri*****@cogsci .ed.ac.ukwrote in message
news:fi******** **@pc-news.cogsci.ed. ac.uk...

In article <fi**********@j ustice.itsc.cuh k.edu.hk>, a <[e-mail] :

>>Richard Tobin told me there is a system limit that can be changed. But I
don't know which file is to be changed.

As I said, ask your system administrator. Or read the manual.
We can't tell you, because you haven't even told us what system
you're using.

-- Richard
--
"Considerat ion shall be given to the need for as many as 32 characters
in some alphabets" - X3.4, 1963.

[a]

Modifying in /etc/security/limits.conf seems doesn't help solve the problem

* soft data 100000
* soft stack 100000

[a]

"a" <[e-mail] in message news:fi******** **@justice.itsc .cuhk.edu.hk...

Modifying in /etc/security/limits.conf seems doesn't help solve the
problem

* soft data 100000
* soft stack 100000

Furthermore, running as root should be unlimited....