Sort Problem in C ( For Data > 3 GB )

[praveenholal]

Hi Friends

I am writing a program in c where i am sorting more than 3 GB of data (data is in a file , Input from a file) , i am using the sort command provided by unix . I am running the program in Fedora Core 5.

But the Sort Command is not sorting this enormous amount of data (Greater than 3 GB) .

First it was giving Error " Insufficient Space Left on Device "
Then i freed some space, but again sort failed after creating intermediate files of big size .

So i need help from you all.

Thank You in Advance

[Savage]

Hi,

Did you try to break data on smaller packages and then sort it?


Savage.

[JosAH]

That reminds me of the GOD (Good Old Days (tm)) of a lack of memory, slow
devices and even slower processors ;-)

If your file has a size of 3GB you need at least another 3GB of free diskspace
to do the following:

1) read chunks of that file, the chunk size is, say 128MB; you can sort such
a chunk internally. Write each chunk to a temporary file, say t_0, t_1 ... t_23.

2) throw away your original file.

3) open each file t_i and merge them together (that's easy and it doesn't take
much internal memory). Write the result to the new sorted file.

4) close everything and throw away the temp files t_0, t_1 ... t_23

kind regards,

Jos