473,508 Members | 2,460 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Read large data files

2 New Member
Hi everybody,
Please can you tell me how can I read large data file (> 15 millions line) in C/C++ languge.

Thanks for your help,
Claire
Aug 13 '07 #1
3 2963
Meetee
931 Recognized Expert Moderator Contributor
Hi everybody,
Please can you tell me how can I read large data file (> 15 millions line) in C/C++ languge.

Thanks for your help,
Claire
You can use vectors for this:

1. declare/open a file-stream for reading (using std::ifstream )
2. declare a vector of strings (std::vector<std::string>)
3. if you haven't already reached the end of file, then do the following, else goto 8:
4. declare a temporary string that will hold the next line (std::string temp;).
5. fill this temporary string with the current line of your file (std::getline)
6. add the string to your string-vector (using push_back function)
7. go back to 3
8. close the file

If you are not aware of vectors then http://www.cppreference.com/cppvector/index.html

Regrads
Aug 13 '07 #2
gpraghuram
1,275 Recognized Expert Top Contributor
Gi,
If the file size is huge then reading it at a short wont be very effective.
Try to split the file into smaller sizes and then have a logic to to open these files in a loop on by one and then read it

Raghuram
Aug 13 '07 #3
clairePuj
2 New Member
Thanks a lot.
best,[

QUOTE=zodilla58]You can use vectors for this:

1. declare/open a file-stream for reading (using std::ifstream )
2. declare a vector of strings (std::vector<std::string>)
3. if you haven't already reached the end of file, then do the following, else goto 8:
4. declare a temporary string that will hold the next line (std::string temp;).
5. fill this temporary string with the current line of your file (std::getline)
6. add the string to your string-vector (using push_back function)
7. go back to 3
8. close the file

If you are not aware of vectors then http://www.cppreference.com/cppvector/index.html

Regrads[/quote]
Aug 14 '07 #4

Sign in to post your reply or Sign up for a free account.

Similar topics

11
8987
by: Sebastian Krause | last post by:
Hello, I tried to read in some large ascii files (200MB-2GB) in Python using scipy.io.read_array, but it did not work as I expected. The whole idea was to find a fast Python routine to read in...
3
3698
by: Albert Tu | last post by:
Dear there, We have an x-ray CT system. The acquisition computer acquires x-ray projections and outputs multiple data files in binary format (2-byte unsigned integer) such as projection0.raw,...
6
6127
by: shailesh kumar | last post by:
Hi, I need to design data interfaces for accessing files of very large sizes efficiently. The data will be accessed in chunks of fixed size ... My data interface should be able to do a random...
6
2633
by: Greg | last post by:
I am working on a project that will have about 500,000 records in an XML document. This document will need to be queried with XPath, and records will need to be updated. I was thinking about...
6
8443
by: Rolf Schroedter | last post by:
(Sorry for cross-posting). I need to access large files > 2GByte (Linux, WinXP/NTFS) using the standard C-library calls. Till today I thought I know how to do it, namely for Win32: Use open(),...
3
6298
by: Buddy Ackerman | last post by:
I'm trying to write files directly to the client so that it forces the client to open the Save As dialog box rather than display the file. On some occasions the files are very large (100MB+). On...
6
2303
by: comp.lang.php | last post by:
if (!function_exists('bigfile')) { /** * Works like file() in PHP except that it will work more efficiently with very large files * * @access public * @param mixed $fullFilePath * @return...
3
3371
by: KWienhold | last post by:
I'm currently writing an application (using Visual Studio 2003 SP1 and C#) that stores files and additional information in a single compound file using IStorage/IStream. Since files in a compound...
3
2925
by: =?Utf-8?B?ZGF2aWQ=?= | last post by:
I try to follow Steve's paper to build a database, and store a small text file into SQL Server database and retrieve it later. Only difference between my table and Steve's table is that I use NTEXT...
2
5439
by: Kevin Ar18 | last post by:
I posted this on the forum, but nobody seems to know the solution: http://python-forum.org/py/viewtopic.php?t=5230 I have a zip file that is several GB in size, and one of the files inside of it...
0
7223
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
7377
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
1
7034
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
7488
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
5623
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
1
5045
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...
0
4702
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and...
0
1544
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated ...
0
412
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.