By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
424,950 Members | 989 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 424,950 IT Pros & Developers. It's quick & easy.

How to Read very large file with formatting (file size > 1 GB)

P: 14
Dear All,

Can anybody help me to read a very large file? and I need to maintain the formatting. The file size starts from 1 GB.

Is there any built-in library?

Thanks in advance
Dec 14 '10 #1
Share this Question
Share on Google+
4 Replies


weaknessforcats
Expert Mod 5K+
P: 9,197
As long as you read the file onto the heap there should be no problem unless your system restricts maximum heap size per process. Just read the file.
Dec 14 '10 #2

Expert 100+
P: 2,396
You may need to open it as a binary file. It depends on the file format.

If you can't allocate a single heap buffer large enough for the entire file then you will be forced to look through the file a block at a time, trading reduced memory usage for more complicated software and increased execution time.
Dec 14 '10 #3

P: 14
Thank you for your guidance.
One more question...
my 1GB file has 5 columns. first column contains integer and second onwards float or double or integer.

e.g
32, 5.12759588393658e-030, 45.0, 0, 50.0

Please tell an efficient way to access each data. Could you share any code?

Thanks in advance
Dec 15 '10 #4

Expert 100+
P: 2,396
A 1GB file formatted as you describe is approximately 200,000 lines. Would you rather end up with a 1GB buffer holding the text of the file or four 200,000-element arrays holding the numeric values pulled from the file?

If you want the numeric values then you only need to repeatedly read a single line, parse it into the four numbers, and add that set of numbers to the arrays.

You should use a small test file with only a few lines to test your parsing software.
Dec 15 '10 #5

Post your reply

Sign in to post your reply or Sign up for a free account.