How to open VERY large file using std::ifstream

[Charlie]

Dear all,

I am currently writting a trace analyzer in C++.
It always fails to open a very large input file (3.7Gb).

I tried on a simple program, same thing happens:
==============================
#include <iostream>
#include <fstream>
using namespace std;

int main(int argc, char *argv[])
{
std::ifstream loop_file(argv[1]);

if (!loop_file)
cout<<"cannot open file: "<<argv[1];
else
cout<<"successfully opened"<<endl;
}
==================
# ./tracer input.data
cannot open file input.data
But it works well on smaller input file, (e.g 200Mb)

I guess there is a limit of size on "std::ifstream" object.
I wonder if there is method I could open this large input file
in C++?

Any suggestion or pointer is highly appreciated!

Many thanks!
Charlie

[Ian McCulloch]

Charlie wrote:

Dear all,

I am currently writting a trace analyzer in C++.
It always fails to open a very large input file (3.7Gb).

I tried on a simple program, same thing happens:
==============================
#include <iostream>
#include <fstream>
using namespace std;

int main(int argc, char *argv[])
{
std::ifstream loop_file(argv[1]);

if (!loop_file)
cout<<"cannot open file: "<<argv[1];
else
cout<<"successfully opened"<<endl;
}
==================
# ./tracer input.data
cannot open file input.data
But it works well on smaller input file, (e.g 200Mb)

I guess there is a limit of size on "std::ifstream" object.
I wonder if there is method I could open this large input file
in C++?

There is no limit imposed by C++, but on 32-bit machines you will need what
is called "large file support" in the operating system to open files larger
than 2^31 bytes. Linux definitely supports this and surely Windows does
too, but you will have to consult your compiler documentation as to how to
make this work for you. It is probably a compiler switch of some sort.

HTH,
Ian McCulloch

[eed132]

I don't know if this will fix your problem, but you could try to go
back to the older C style input/output routines fopen, fscanf, fprintf,
etc. If that works you could give it a try.

One thing to be aware of is that I've heard that mixing stdio and
iostream can lead to bad results, but I doubt it'd affect anythnig in
this case so you would probably be okay. But it's sorta playing with
fire, at least without knowing more about how much they interact.
Perhaps someone else can say something about that. It probably wouldn't
cause any problems (in fact I wouldn't be surprised if one is usually
implemented in terms of the other), but it might be wise to stay away
from iostream if you are using these functions. Perhaps Google could
say more.

[msalters]

Charlie wrote:

Dear all,

I am currently writting a trace analyzer in C++.
It always fails to open a very large input file (3.7Gb).

I tried on a simple program, same thing happens: (OK) I guess there is a limit of size on "std::ifstream" object.
I wonder if there is method I could open this large input file
in C++?

The problem isn't C++, but your particular implementation. A 64
bits implementation is unlikely to choke on it. A good 32 bits
implementation might also be able to handle it. Try MSVC 8, IIRC
it can handle these sizes. I do suspect that gcc can too, but I'm
less sure about that.

HTH,
Michiel Salters