safely reading large files

[byte8bits]

How does C++ safely open and read very large files? For example, say I
have 1GB of physical memory and I open a 4GB file and attempt to read
it like so:

#include <iostream>
#include <fstream>
#include <string>
using namespace std;

int main () {
string line;
ifstream myfile ("example.tx t", ios::binary);
if (myfile.is_open ())
{
while (! myfile.eof() )
{
getline (myfile,line);
cout << line << endl;
}
myfile.close();
}

else cout << "Unable to open file";

return 0;
}

In particular, what if a line in the file is more than the amount of
available physical memory? What would happen? Seems getline() would
cause a crash. Is there a better way. Maybe... check amount of free
memory, then use 10% or so of that amount for the read. So if 1GB of
memory is free, then take 100MB for file IO. If only 10MB is free,
then just read 1MB at a time. Repeat this step until the file has been
read completely. Is something built into standard C++ to handle this?
Or is there a accepted way to do this?

Thanks,

Brad

[Victor Bazarov]

by*******@gmail .com wrote:

How does C++ safely open and read very large files? For example, say I
have 1GB of physical memory and I open a 4GB file and attempt to read
it like so:

#include <iostream>
#include <fstream>
#include <string>
using namespace std;

int main () {
string line;
ifstream myfile ("example.tx t", ios::binary);
if (myfile.is_open ())
{
while (! myfile.eof() )
{
getline (myfile,line);
cout << line << endl;
}
myfile.close();
}

else cout << "Unable to open file";

return 0;
}

In particular, what if a line in the file is more than the amount of
available physical memory? What would happen? Seems getline() would
cause a crash. Is there a better way. Maybe... check amount of free
memory, then use 10% or so of that amount for the read. So if 1GB of
memory is free, then take 100MB for file IO. If only 10MB is free,
then just read 1MB at a time. Repeat this step until the file has been
read completely. Is something built into standard C++ to handle this?
Or is there a accepted way to do this?

Actually, performing operations that can lead to running out of memory
is not a simple thing at all. Yes, if you can estimate the amount of
memory you will need over what you right now want to allocate and you
know the size of available memory somehow, then you can allocate a chunk
and operate on that chunk until done and move over to the next chunk.
In the good ol' days that's how we solved large systems of linear
equations, one piece of the matrix at a time (or two if the algorithm
called for it).

Unfortunately there is no single straightforward solution. In most
cases you don't even know that you're going to run out of memory until
it's too late. You can write the program to handle those situations
using C++ exceptions. The pseudo-code might look like this:

std::size_t chunk_size = 1024*1024*1024;
MyAlgorithgm algo;

do {
try {
algo.prepare_th e_operation(chu nk_size);
// if I am here, the chunk_size is OK
algo.perform_th e_operation();
algo.wrap_it_up ();
}
catch (std::bad_alloc & e) {
chunk_size /= 2; // or any other adjustment
}
}
while (chunk_size 1024*1024); // or some other threshold

That way if your preparation fails, you just restart it using a smaller
chunk, until you either complete the operation or your chunk is too
small and you can't really do anything...

V
--
Please remove capital 'A's when replying by e-mail
I do not respond to top-posted replies, please don't ask

[Sam]

by*******@gmail .com writes:

while (! myfile.eof() )
{
getline (myfile,line);
cout << line << endl;
}
myfile.close();
}

else cout << "Unable to open file";

return 0;
}

In particular, what if a line in the file is more than the amount of
available physical memory?

The C++ library will fail to allocate sufficient memory, throw an exception,
and terminate the process.

What would happen? Seems getline() would
cause a crash. Is there a better way. Maybe... check amount of free
memory, then use 10% or so of that amount for the read.

And where exactly would you propose to store the remaining 90% of
std::string?

read completely. Is something built into standard C++ to handle this?

No. std::getline() reads as much as necessary, until the end-of-line
character, and the resulting std::string has to be big enough to store the
entire line.

Or is there a accepted way to do this?

If you need a way to handle this situation, would not use std::getline(),
but some different approach, like std::istream::g et or std::istream::r ead.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iD8DBQBIM4apx9p 3GYHlUOIRAj5GAJ 44mxrzR4cPoup+r W7WzL94IMzAyQCe NHRL
nfDgBM3Esmf5had FB9LuptE=
=nO85
-----END PGP SIGNATURE-----

[red floyd]

by*******@gmail .com wrote:

How does C++ safely open and read very large files? For example, say I
have 1GB of physical memory and I open a 4GB file and attempt to read
it like so:

Others have already answered your question, so I'm going to address
something else.

#include <iostream>
#include <fstream>
#include <string>
using namespace std;

int main () {
string line;
ifstream myfile ("example.tx t", ios::binary);
if (myfile.is_open ())
{

This while loop does not do what you think it does. See FAQ 15.5
(http://parashift.com/c++-faq-lite/in....html#faq-15.5)

while (! myfile.eof() )
{
getline (myfile,line);
cout << line << endl;
}
myfile.close();
}

else cout << "Unable to open file";

return 0;
}

[James Kanze]

On May 21, 4:11 am, Victor Bazarov <v.Abaza...@com Acast.netwrote:

byte8b...@gmail .com wrote:

How does C++ safely open and read very large files? For example, say I
have 1GB of physical memory and I open a 4GB file and attempt to read
it like so:

#include <iostream>
#include <fstream>
#include <string>
using namespace std;

int main () {
string line;
ifstream myfile ("example.tx t", ios::binary);
if (myfile.is_open ())
{
while (! myfile.eof() )
{
getline (myfile,line);
cout << line << endl;
}
myfile.close();
}

else cout << "Unable to open file";

return 0;
}

In particular, what if a line in the file is more than the
amount of available physical memory? What would happen?
Seems getline() would cause a crash. Is there a better way.
Maybe... check amount of free memory, then use 10% or so of
that amount for the read. So if 1GB of memory is free, then
take 100MB for file IO. If only 10MB is free, then just read
1MB at a time. Repeat this step until the file has been read
completely. Is something built into standard C++ to handle
this? Or is there a accepted way to do this?

Actually, performing operations that can lead to running out
of memory is not a simple thing at all.

I'm sure you don't mean what that literally says. There's
certainly nothing difficult about running out of memory. Doing
something reasonable (other than just aborting) when it happens
is difficult, however.

Yes, if you can estimate the amount of memory you will need
over what you right now want to allocate and you know the size
of available memory somehow, then you can allocate a chunk and
operate on that chunk until done and move over to the next
chunk. In the good ol' days that's how we solved large
systems of linear equations, one piece of the matrix at a time
(or two if the algorithm called for it).

And you'd manually manage overlays, as well, so that only part
of the program was in memory at a time. (I once saw a PL/1
compiler which ran in 16 KB real memory, using such techniques.
Took something like three hours to compile a 500 line program,
but it did work.)

Unfortunately there is no single straightforward solution. In
most cases you don't even know that you're going to run out of
memory until it's too late. You can write the program to
handle those situations using C++ exceptions. The pseudo-code
might look like this:

std::size_t chunk_size = 1024*1024*1024;
MyAlgorithgm algo;

do {
try {
algo.prepare_th e_operation(chu nk_size);
// if I am here, the chunk_size is OK
algo.perform_th e_operation();
algo.wrap_it_up ();
}
catch (std::bad_alloc & e) {
chunk_size /= 2; // or any other adjustment
}
}
while (chunk_size 1024*1024); // or some other threshold

Shouldn't the condition here be "while ( operation not done )",
something like:

bool didIt = false ;
do {
try {
// your code from the try block
didIt = true ;
}
// ... your catch
} while ( ! didIt ) ;

That way if your preparation fails, you just restart it using
a smaller chunk, until you either complete the operation or
your chunk is too small and you can't really do anything...

Just a note, but that isn't allways reliable. Not all OS's will
tell you when there isn't enough memory: they'll return an
address, then crash or suspend your program when you try to
access it. (I've seen this happen on at least three different
systems: Windows, AIX and Linux. At least in the case of AIX
and Linux, and probably Windows as well, it depends on the
version, and some configuration parameters, but most Linux are
still configured so that you cannot catch allocation errors: if
the command "/sbin/sysctl vm.overcommit_m emory" displays any
value other than 2, then a reliably conforming implementation of
C or C++ is impossible.)

--
James Kanze (GABI Software) email:ja******* **@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]

[brad]

by*******@gmail .com wrote:

How does C++ safely open and read very large files?

How about this mate? It's a start.

// read a file into memory
// read in chunks if the file is larger than 16MB
#include <iostream>
#include <fstream>

using namespace std;

int main ()
{

int max = 16384000;

ifstream is;
is.open ("test.txt") ;

// get length of file:
is.seekg (0, ios::end);
int file_size = is.tellg();
is.seekg (0, ios::beg);

if (file_size max)
{
// allocate memory
char* buffer = new char[16384000];

cout << file_size << " bytes... break up to read" << endl;
while (!is.eof())
{
//read data as a block
is.read (buffer, max);

//write the read data to stdout
cout << buffer;
}
delete[] buffer;
}
else
{
// allocate memory:
char* buffer = new char[file_size];

cout << file_size << " bytes" << endl;
is.read (buffer, file_size);
cout << buffer;
delete[] buffer;
}

is.close();

return 0;
}

[ltcmelo]

On May 21, 5:33*am, James Kanze <james.ka...@gm ail.comwrote:

On May 21, 4:11 am, Victor Bazarov <v.Abaza...@com Acast.netwrote:

byte8b...@gmail .com wrote:

How does C++ safely open and read very large files? For example, say I
have 1GB of physical memory and I open a 4GB file and attempt to read
it like so:
#include <iostream>
#include <fstream>
#include <string>
using namespace std;
int main () {
* string line;
* ifstream myfile ("example.tx t", ios::binary);
* if (myfile.is_open ())
* {
* * while (! myfile.eof() )
* * {
* * * getline (myfile,line);
* * * cout << line << endl;
* * }
* * myfile.close();
* }
* else cout << "Unable to open file";

* return 0;
}
In particular, what if a line in the file is more than the
amount of available physical memory? What would happen?
Seems getline() would cause a crash. Is there a better way.
Maybe... check amount of free memory, then use 10% or so of
that amount for the read. So if 1GB of memory is free, then
take 100MB for file IO. If only 10MB is free, then just read
1MB at a time. Repeat this step until the file has been read
completely. Is something built into standard C++ to handle
this? *Or is there a accepted way to do this?

Actually, performing operations that can lead to running out
of memory is not a simple thing at all.

I'm sure you don't mean what that literally says. *There's
certainly nothing difficult about running out of memory. *Doing
something reasonable (other than just aborting) when it happens
is difficult, however.

Yes, if you can estimate the amount of memory you will need
over what you right now want to allocate and you know the size
of available memory somehow, then you can allocate a chunk and
operate on that chunk until done and move over to the next
chunk. *In the good ol' days that's how we solved large
systems of linear equations, one piece of the matrix at a time
(or two if the algorithm called for it).

And you'd manually manage overlays, as well, so that only part
of the program was in memory at a time. *(I once saw a PL/1
compiler which ran in 16 KB real memory, using such techniques.
Took something like three hours to compile a 500 line program,
but it did work.)

Unfortunately there is no single straightforward solution. *In
most cases you don't even know that you're going to run out of
memory until it's too late. *You can write the program to
handle those situations using C++ exceptions. *The pseudo-code
might look like this:
* * *std::size_t chunk_size = 1024*1024*1024;
* * *MyAlgorithgm algo;
* * *do {
* * * * *try {
* * * * * * *algo.prepare_t he_operation(ch unk_size);
* * * * * * *// if I am here, the chunk_size is OK
* * * * * * *algo.perform_t he_operation();
* * * * * * *algo.wrap_it_u p();
* * * * *}
* * * * *catch (std::bad_alloc & e) {
* * * * * * *chunk_size /= 2; // or any other adjustment
* * * * *}
* * *}
* * *while (chunk_size 1024*1024); // or some other threshold

Shouldn't the condition here be "while ( operation not done )",
something like:

* * bool didIt = false ;
* * do {
* * * * try {
* * * * * * // *your code from the try block
* * * * * * didIt = true ;
* * * * }
* * * * // *... your catch
* * } while ( ! didIt ) ;

That way if your preparation fails, you just restart it using
a smaller chunk, until you either complete the operation or
your chunk is too small and you can't really do anything...

Just a note, but that isn't allways reliable. *Not all OS's will
tell you when there isn't enough memory: they'll return an
address, then crash or suspend your program when you try to
access it. *(I've seen this happen on at least three different
systems: Windows, AIX and Linux. *At least in the case of AIX
and Linux, and probably Windows as well, it depends on the
version, and some configuration parameters, but most Linux are
still configured so that you cannot catch allocation errors: if
the command "/sbin/sysctl vm.overcommit_m emory" displays any
value other than 2, then a reliably conforming implementation of
C or C++ is impossible.)

Even assuming you have memory, there's another concern. If you're
running on a 32-bit platform, you can have problems working with such
large files. Basically, the file pointer might not be large enough to
traverse, for example, a file with more than 4GB. (There are "non-
standard" solutions for this.)

--
Leandro T. C. Melo

[Victor Bazarov]

ltcmelo wrote:

[..]
Even assuming you have memory, there's another concern. If you're
running on a 32-bit platform, you can have problems working with such
large files. Basically, the file pointer might not be large enough to
traverse, for example, a file with more than 4GB. (There are "non-
standard" solutions for this.)

Why do you say it's non-standard? The Standard defines 'streampos' for
stream positioning, which any decent library implementation on a system
that allows its files to have more than 32-bit size, should be the next
larger integral type (whatever that might be on that system). Also,
there is the 'streamoff' for offsetting in a stream buffer. Both are
implementation-defined.

V
--
Please remove capital 'A's when replying by e-mail
I do not respond to top-posted replies, please don't ask

[ltcmelo]

On May 21, 4:19*pm, Victor Bazarov <v.Abaza...@com Acast.netwrote:

ltcmelo wrote:

[..]
Even assuming you have memory, there's another concern. If you're
running on a 32-bit platform, you can have problems working with such
large files. Basically, the file pointer might not be large enough to
traverse, for example, a file with more than 4GB. (There are "non-
standard" solutions for this.)

Why do you say it's non-standard? *The Standard defines 'streampos' for
stream positioning, which any decent library implementation on a system
that allows its files to have more than 32-bit size, should be the next
larger integral type (whatever that might be on that system). *Also,
there is the 'streamoff' for offsetting in a stream buffer. *Both are
implementation-defined.

Maybe "non-portable" (across systems) would have been a better
choice... I say that for particular cases like in the GNU C Library
where you can use macros to enable support for large files.

--
Leandro T. C. Melo

[Victor Bazarov]

ltcmelo wrote:

On May 21, 4:19 pm, Victor Bazarov <v.Abaza...@com Acast.netwrote:

>ltcmelo wrote:

>>[..]
Even assuming you have memory, there's another concern. If you're
running on a 32-bit platform, you can have problems working with such
large files. Basically, the file pointer might not be large enough to
traverse, for example, a file with more than 4GB. (There are "non-
standard" solutions for this.)

Why do you say it's non-standard? The Standard defines 'streampos' for
stream positioning, which any decent library implementation on a system
that allows its files to have more than 32-bit size, should be the next
larger integral type (whatever that might be on that system). Also,
there is the 'streamoff' for offsetting in a stream buffer. Both are
implementati on-defined.

Maybe "non-portable" (across systems) would have been a better
choice... I say that for particular cases like in the GNU C Library
where you can use macros to enable support for large files.

I am still confused. What's non-portable if you use standard types?
Perhaps you're using some special meaning of the word "non-portable" (or
the word "portable") ... Care to elaborate?

V
--
Please remove capital 'A's when replying by e-mail
I do not respond to top-posted replies, please don't ask