473,573 Members | 2,870 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

How to open VERY large file using std::ifstream

Dear all,

I am currently writting a trace analyzer in C++.
It always fails to open a very large input file (3.7Gb).

I tried on a simple program, same thing happens:
=============== ===============
#include <iostream>
#include <fstream>
using namespace std;

int main(int argc, char *argv[])
{
std::ifstream loop_file(argv[1]);

if (!loop_file)
cout<<"cannot open file: "<<argv[1];
else
cout<<"successf ully opened"<<endl;
}
=============== ===
# ./tracer input.data
cannot open file input.data
But it works well on smaller input file, (e.g 200Mb)

I guess there is a limit of size on "std::ifstr eam" object.
I wonder if there is method I could open this large input file
in C++?

Any suggestion or pointer is highly appreciated!

Many thanks!
Charlie
Jul 23 '05 #1
3 6793
Charlie wrote:
Dear all,

I am currently writting a trace analyzer in C++.
It always fails to open a very large input file (3.7Gb).

I tried on a simple program, same thing happens:
=============== ===============
#include <iostream>
#include <fstream>
using namespace std;

int main(int argc, char *argv[])
{
std::ifstream loop_file(argv[1]);

if (!loop_file)
cout<<"cannot open file: "<<argv[1];
else
cout<<"successf ully opened"<<endl;
}
=============== ===
# ./tracer input.data
cannot open file input.data
But it works well on smaller input file, (e.g 200Mb)

I guess there is a limit of size on "std::ifstr eam" object.
I wonder if there is method I could open this large input file
in C++?


There is no limit imposed by C++, but on 32-bit machines you will need what
is called "large file support" in the operating system to open files larger
than 2^31 bytes. Linux definitely supports this and surely Windows does
too, but you will have to consult your compiler documentation as to how to
make this work for you. It is probably a compiler switch of some sort.

HTH,
Ian McCulloch

Jul 23 '05 #2
I don't know if this will fix your problem, but you could try to go
back to the older C style input/output routines fopen, fscanf, fprintf,
etc. If that works you could give it a try.

One thing to be aware of is that I've heard that mixing stdio and
iostream can lead to bad results, but I doubt it'd affect anythnig in
this case so you would probably be okay. But it's sorta playing with
fire, at least without knowing more about how much they interact.
Perhaps someone else can say something about that. It probably wouldn't
cause any problems (in fact I wouldn't be surprised if one is usually
implemented in terms of the other), but it might be wise to stay away
from iostream if you are using these functions. Perhaps Google could
say more.

Jul 23 '05 #3

Charlie wrote:
Dear all,

I am currently writting a trace analyzer in C++.
It always fails to open a very large input file (3.7Gb).

I tried on a simple program, same thing happens: (OK) I guess there is a limit of size on "std::ifstr eam" object.
I wonder if there is method I could open this large input file
in C++?


The problem isn't C++, but your particular implementation. A 64
bits implementation is unlikely to choke on it. A good 32 bits
implementation might also be able to handle it. Try MSVC 8, IIRC
it can handle these sizes. I do suspect that gcc can too, but I'm
less sure about that.

HTH,
Michiel Salters

Jul 23 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

15
17561
by: solartimba | last post by:
Why can I open a file using a C-string but not a C++ string? //C-strings works char path; strcpy(path,"c:\test.txt"); ifstream infile(path); //C++ string class does not string path("c:\test.txt"); ifstream infile(path);
3
6025
by: Anne-Marte | last post by:
Hi I simply don't understand how to read a simple file using std::istream. How do I open a file for reading with istream?? Anne-Marte
12
11638
by: Steven T. Hatton | last post by:
I know of a least one person who believes std::ifstream::read() and std::ofstream::write() are "mistakes". They seem to do the job I want done. What's wrong with them. This is the code I currently have as a test for using std::ifstream::read(). Is there anything wrong with the way I'm getting the file? #include <vector> #include...
9
13407
by: Charles F McDevitt | last post by:
I'm trying to upgrade some old code that used old iostreams. At one place in the code, I have a path/filename in a wchar_t string (unicode utf-16). I need to open an ifstream to that file. But the open() on ifstream only takes char * strings (mbcs?). In old iostreams, I could _wopen() the file, get the filedesc, and call attach() on...
10
2360
by: Gunnar G | last post by:
I'm having problem reading from the beginning of a file. Here is the code (more or less) ifstream codefin; ofstream codefout; while (not_annoyed)
2
10567
by: Assertor | last post by:
Hi, All. (VC++6.0) I found some strange thins when using getline() and seekg() of std::ifstream. After the file position of an open file was shift to the end of the file, seekg() did not work rightly. (i.e. I could not move the file position to the begin or some position using seekg())
2
14029
by: Karl | last post by:
Hey everyone! I've got a quick question on whether std::ifstream is buffered or not. The reason is that I have a homework assignment that requires me to benchmark copying files using different buffer sizes. I ended up doing this using std::istream::readsome() and std::ostream::write(): // now try writing to the destination file...
5
12375
by: Assertor | last post by:
Hi, all. Is there any way to create an instance of std::ifstream using std::string. (through std::ifstream's constructor or assignment operator or iterator, etc...) i.e. std::string str = "this is a string"; std::ifstream ifs = str; //<----- this is a just pseudo code.
2
15220
by: mpalomas | last post by:
Hi C++ folks, I have trouble to open files whose path contains non-ascii characters with std::ifstream. For instance let's say i just have a file which has Japanese characters either in the file path or the file name : 疑問.dat. The file itself does not contains unicode characters or whatever, it is a binary file, but the file name, or...
0
7755
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, well explore What is ONU, What Is Router, ONU & Routers main...
0
7679
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7992
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8048
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
1
5571
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupr who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
3722
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
1
2183
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1284
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
1027
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.