STL Stream maximum filesize?

Hello All,

I am trying to read data from a textfile using std::ifstream.
The problem is that I have a very huge text file, approximately 4.8 GB.
When I use this file, the program I wrote refuses to read in every line of
the file,
after a while it just stops, it doesn't reach the last line of the file.
When I use smaller files it works perfectly.

Is there a maximum filesize limit for std::ifstream?
Or could this be an operating system or compiler issue?
(I use Borland C++Builder5 and Windows2000 with NTFS)

--Michel

re: STL Stream maximum filesize?

maybe there is a 4gig limit .. (i think) they use int as an counter ..
could try around the 4gb mark and then you'll see


On Thu, 18 Sep 2003 10:06:31 +0200
"Michel Rosien" <rosien@REMOVEcs.utwente.nl> wrote:
[color=blue]
> The problem is that I have a very huge text file, approximately 4.8
> GB. When I use this file, the program I wrote refuses to read in every
> line of the file,[/color]

re: STL Stream maximum filesize?

Michel Rosien wrote in news:bkbp28$njq$1@ares.cs.utwente.nl:
[color=blue]
> Hello All,
>
> I am trying to read data from a textfile using std::ifstream.
> The problem is that I have a very huge text file, approximately 4.8
> GB. When I use this file, the program I wrote refuses to read in every
> line of the file,
> after a while it just stops, it doesn't reach the last line of the
> file. When I use smaller files it works perfectly.
>[/color]

There is probably a 2GB limit ( pow( 2, 31 ) ).
[color=blue]
> Is there a maximum filesize limit for std::ifstream?[/color]

Yes, what it is and how you find out (other than trying) with any
given implementation I don't know.
[color=blue]
> Or could this be an operating system or compiler issue?
> (I use Borland C++Builder5 and Windows2000 with NTFS)
>[/color]




Rob.
--
http://www.victim-prime.dsl.pipex.com/

re: STL Stream maximum filesize?

Michel Rosien wrote in news:bkbp28$njq$1@ares.cs.utwente.nl:

[repost - I sent the previous by accident before I was finnished]
[color=blue]
> I am trying to read data from a textfile using std::ifstream.
> The problem is that I have a very huge text file, approximately 4.8
> GB. When I use this file, the program I wrote refuses to read in every
> line of the file,
> after a while it just stops, it doesn't reach the last line of the
> file. When I use smaller files it works perfectly.
>
> Is there a maximum filesize limit for std::ifstream?[/color]

Yes, but its implementaion ( C++Builder5 in your case ) defined.
Its probably 2GB ( pow( 2, 31 ) ).
[color=blue]
> Or could this be an operating system or compiler issue?
> (I use Borland C++Builder5 and Windows2000 with NTFS)
>[/color]

A simple solution may be to change the way your programme works
from:

yourprog bigfile

to:

yourprog < bigfile

And read you data in from std::cin, do write a small test programme
first, just to check std::cin doesn't have the same problem.

If that doesn't work you'll need a platform specific solution, goto
MSDN http://msdn.microsoft.com/ and look for CreateFile to start
with.

HTH



Rob.
--
http://www.victim-prime.dsl.pipex.com/

re: STL Stream maximum filesize?

"Rob Williscroft" <rtw@freenet.REMOVE.co.uk> wrote in message
news:Xns93FA67696BC8CukcoREMOVEfreenetrtw@195.129. 110.130...
[color=blue]
> A simple solution may be to change the way your programme works
> from:
>
> yourprog bigfile
>
> to:
>
> yourprog < bigfile
>
> And read you data in from std::cin, do write a small test programme
> first, just to check std::cin doesn't have the same problem.[/color]

Thank you, I will try this

--Michel

re: STL Stream maximum filesize?

"Rob Williscroft" <rtw@freenet.REMOVE.co.uk> wrote in message
news:Xns93FA67696BC8CukcoREMOVEfreenetrtw@195.129. 110.130...
[color=blue]
> A simple solution may be to change the way your programme works
> from:
>
> yourprog bigfile
>
> to:
>
> yourprog < bigfile
>
> And read you data in from std::cin, do write a small test programme
> first, just to check std::cin doesn't have the same problem.[/color]

I rewrote the program as you suggested, but it still has the same problem
after a while the program just stops reading from the file (the stream
fails)
It has read 7864320 lines of 81(+2) bytes each, after that it stops.
This is the exact same number as with the original approach.
The total number of lines it should read is 58720256

But the point where it stops reading is no where near 2GB or 4GB (622.5 MB)
I don't know what's going on here

I think I'll just try to split the file into multiple smaller files.
I've tried other large text files of up to 72 MB and they have no problems.

--Michel

re: STL Stream maximum filesize?

Interesting where it stops

7864320 = 780000h exactly

"Michel Rosien" <rosien@REMOVEcs.utwente.nl> wrote in message
news:bkc683$71t$1@ares.cs.utwente.nl...[color=blue]
> "Rob Williscroft" <rtw@freenet.REMOVE.co.uk> wrote in message
> news:Xns93FA67696BC8CukcoREMOVEfreenetrtw@195.129. 110.130...
>[color=green]
> > A simple solution may be to change the way your programme works
> > from:
> >
> > yourprog bigfile
> >
> > to:
> >
> > yourprog < bigfile
> >
> > And read you data in from std::cin, do write a small test programme
> > first, just to check std::cin doesn't have the same problem.[/color]
>
> I rewrote the program as you suggested, but it still has the same problem
> after a while the program just stops reading from the file (the stream
> fails)
> It has read 7864320 lines of 81(+2) bytes each, after that it stops.
> This is the exact same number as with the original approach.
> The total number of lines it should read is 58720256
>
> But the point where it stops reading is no where near 2GB or 4GB (622.5[/color]
MB)[color=blue]
> I don't know what's going on here
>
> I think I'll just try to split the file into multiple smaller files.
> I've tried other large text files of up to 72 MB and they have no[/color]
problems.[color=blue]
>
> --Michel
>
>[/color]

re: STL Stream maximum filesize?

This will give you the size and is the correct way to discover it at
runtime.

#include <streambuf>
__int64 temp=std::numeric_limits<std::streamsize>::max();

Fraser.

re: STL Stream maximum filesize?

"Michel Rosien" <rosien@REMOVEcs.utwente.nl> wrote in message
news:bkbp28$njq$1@ares.cs.utwente.nl...[color=blue]
> Hello All,
>
> I am trying to read data from a textfile using std::ifstream.
> The problem is that I have a very huge text file, approximately 4.8 GB.
> When I use this file, the program I wrote refuses to read in every line of
> the file,
> after a while it just stops, it doesn't reach the last line of the file.
> When I use smaller files it works perfectly.
>
> Is there a maximum filesize limit for std::ifstream?[/color]

Yes. It's

std::numeric_limits<std::streamsize>::max()
[color=blue]
> Or could this be an operating system or compiler issue?[/color]

The actual value is implementation dependent, yes.

-Mike