STL Stream maximum filesize?

[Michel Rosien]

Hello All,

I am trying to read data from a textfile using std::ifstream.
The problem is that I have a very huge text file, approximately 4.8 GB.
When I use this file, the program I wrote refuses to read in every line of
the file,
after a while it just stops, it doesn't reach the last line of the file.
When I use smaller files it works perfectly.

Is there a maximum filesize limit for std::ifstream?
Or could this be an operating system or compiler issue?
(I use Borland C++Builder5 and Windows2000 with NTFS)

--Michel

[Clemens Auer]

maybe there is a 4gig limit .. (i think) they use int as an counter ..
could try around the 4gb mark and then you'll see
On Thu, 18 Sep 2003 10:06:31 +0200
"Michel Rosien" <ro****@REMOVEcs.utwente.nl> wrote:

The problem is that I have a very huge text file, approximately 4.8
GB. When I use this file, the program I wrote refuses to read in every
line of the file,

[Rob Williscroft]

Michel Rosien wrote in news:bk**********@ares.cs.utwente.nl:

Hello All,

I am trying to read data from a textfile using std::ifstream.
The problem is that I have a very huge text file, approximately 4.8
GB. When I use this file, the program I wrote refuses to read in every
line of the file,
after a while it just stops, it doesn't reach the last line of the
file. When I use smaller files it works perfectly.

There is probably a 2GB limit ( pow( 2, 31 ) ).
Is there a maximum filesize limit for std::ifstream?
Yes, what it is and how you find out (other than trying) with any
given implementation I don't know.
Or could this be an operating system or compiler issue?
(I use Borland C++Builder5 and Windows2000 with NTFS)

Rob.
--
http://www.victim-prime.dsl.pipex.com/

[Rob Williscroft]

Michel Rosien wrote in news:bk**********@ares.cs.utwente.nl:

[repost - I sent the previous by accident before I was finnished]

I am trying to read data from a textfile using std::ifstream.
The problem is that I have a very huge text file, approximately 4.8
GB. When I use this file, the program I wrote refuses to read in every
line of the file,
after a while it just stops, it doesn't reach the last line of the
file. When I use smaller files it works perfectly.

Is there a maximum filesize limit for std::ifstream?
Yes, but its implementaion ( C++Builder5 in your case ) defined.
Its probably 2GB ( pow( 2, 31 ) ).
Or could this be an operating system or compiler issue?
(I use Borland C++Builder5 and Windows2000 with NTFS)

A simple solution may be to change the way your programme works
from:

yourprog bigfile

to:

yourprog < bigfile

And read you data in from std::cin, do write a small test programme
first, just to check std::cin doesn't have the same problem.

If that doesn't work you'll need a platform specific solution, goto
MSDN http://msdn.microsoft.com/ and look for CreateFile to start
with.

HTH

Rob.
--
http://www.victim-prime.dsl.pipex.com/

[Michel Rosien]

"Rob Williscroft" <rt*@freenet.REMOVE.co.uk> wrote in message
news:Xn**********************************@195.129. 110.130...

A simple solution may be to change the way your programme works
from:

yourprog bigfile

to:

yourprog < bigfile

And read you data in from std::cin, do write a small test programme
first, just to check std::cin doesn't have the same problem.

Thank you, I will try this

--Michel

[Michel Rosien]

"Rob Williscroft" <rt*@freenet.REMOVE.co.uk> wrote in message
news:Xn**********************************@195.129. 110.130...

A simple solution may be to change the way your programme works
from:

yourprog bigfile

to:

yourprog < bigfile

And read you data in from std::cin, do write a small test programme
first, just to check std::cin doesn't have the same problem.

I rewrote the program as you suggested, but it still has the same problem
after a while the program just stops reading from the file (the stream
fails)
It has read 7864320 lines of 81(+2) bytes each, after that it stops.
This is the exact same number as with the original approach.
The total number of lines it should read is 58720256

But the point where it stops reading is no where near 2GB or 4GB (622.5 MB)
I don't know what's going on here

I think I'll just try to split the file into multiple smaller files.
I've tried other large text files of up to 72 MB and they have no problems.

--Michel

[Kris]

Interesting where it stops

7864320 = 780000h exactly

"Michel Rosien" <ro****@REMOVEcs.utwente.nl> wrote in message
news:bk**********@ares.cs.utwente.nl...

"Rob Williscroft" <rt*@freenet.REMOVE.co.uk> wrote in message
news:Xn**********************************@195.129. 110.130...

A simple solution may be to change the way your programme works
from:

yourprog bigfile

to:

yourprog < bigfile

And read you data in from std::cin, do write a small test programme
first, just to check std::cin doesn't have the same problem.
I rewrote the program as you suggested, but it still has the same problem
after a while the program just stops reading from the file (the stream
fails)
It has read 7864320 lines of 81(+2) bytes each, after that it stops.
This is the exact same number as with the original approach.
The total number of lines it should read is 58720256

But the point where it stops reading is no where near 2GB or 4GB (622.5

MB) I don't know what's going on here

I think I'll just try to split the file into multiple smaller files.
I've tried other large text files of up to 72 MB and they have no problems.
--Michel

[Fraser Ross]

This will give you the size and is the correct way to discover it at
runtime.

#include <streambuf>
__int64 temp=std::numeric_limits<std::streamsize>::max();

Fraser.

[Mike Wahler]

"Michel Rosien" <ro****@REMOVEcs.utwente.nl> wrote in message
news:bk**********@ares.cs.utwente.nl...

Hello All,

I am trying to read data from a textfile using std::ifstream.
The problem is that I have a very huge text file, approximately 4.8 GB.
When I use this file, the program I wrote refuses to read in every line of
the file,
after a while it just stops, it doesn't reach the last line of the file.
When I use smaller files it works perfectly.

Is there a maximum filesize limit for std::ifstream?
Yes. It's

std::numeric_limits<std::streamsize>::max()
Or could this be an operating system or compiler issue?

The actual value is implementation dependent, yes.

-Mike