473,320 Members | 1,828 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

ofstream limitations?

mikejfe
12
I wrote a program to read a file, store that information into a 3d array, sort that array, and then write to a file. Should be straightforward. The problem I am running into is within the writing part.

I've taken the sorting function out.

For debugging purposes- when the file is written, a counter tells me that all pieces of information have been written, however, by visual inspection the data is jumbled and missing quite a few pieces of data. I have also tried displaying the data, supposedly being written, to the screen. I can't see anything funky going on. This is partly because I am trying to work with files on the order of 512-1000MB.

If I load a small file, ~10MB, no problems. I just have problems with larger file sizes.

So, here's my question (sorry for my long windedness): Are there limitations to the size of file that can be written using ofstream?

I can post my code if that would be helpful. I know this should be so simple. I am pretty inexperienced with C++.

Thanks in advance!
Apr 10 '07 #1
8 5368
Banfa
9,065 Expert Mod 8TB
So, here's my question (sorry for my long windedness): Are there limitations to the size of file that can be written using ofstream?
It should only be limited by the underlying filing system. You haven't said what you are using but the most common filing systems today (NTFS and whatever Linux uses) should be OK with that size of file.
Apr 10 '07 #2
RedSon
5,000 Expert 4TB
FYI

NTFS:
File size limit = 16 TiB with current implementation

ext3: (common Linux filesystem)
File size limit = 16GiB – 2TiB

And for those who do not know what at TiB is:

A tebibyte (a contraction of tera binary byte) is a unit of information or computer storage, abbreviated TiB.

1 tebibyte = 2^40 bytes = 1,099,511,627,776 bytes = 1,024 gibibytes
Apr 10 '07 #3
RedSon
5,000 Expert 4TB
And for anyone who is interested:

http://en.wikipedia.org/wiki/Compari...systems#Limits
Apr 10 '07 #4
mikejfe
12
Thanks for your replies.

I apologize for not putting this in my original post. I'm using XP, Dev-C++, and set up as NTFS. I know this is such a beginners question..ugh.

Please don't laugh at me but I figured out that when I analyzed the data, my analysis was incorrect. My files are being saved "correctly."

However, I do have a related side question: are there any theories why when I save the file, the file size drops from 128 MB to 32 MB? The data is the same (eg 65,536x256 pieces of information) and the files both stay .asc. It was this drastic change in file size that made me think I had an error in the first place. :S
Apr 11 '07 #5
Banfa
9,065 Expert Mod 8TB
However, I do have a related side question: are there any theories why when I save the file, the file size drops from 128 MB to 32 MB? The data is the same (eg 65,536x256 pieces of information) and the files both stay .asc. It was this drastic change in file size that made me think I had an error in the first place. :S
1. You are not saving all the original data

or

2. The original file contained a lot of redundant data that is not required

65,536x256 = 16M implying your pieces of information are each 2 bytes long to make a 32M file. Is this correct?
Apr 11 '07 #6
RedSon
5,000 Expert 4TB
Also some NTFS disks are setup to optimize their sectors so they compress the data.
Apr 11 '07 #7
mikejfe
12
Banfa, you must be correct about the redundant data. Apparently each piece of information in the original file is 8 bytes long. What kind of redundant data could be in the file that would change the information from 8 bytes to 2 bytes? In my array I store all the information as floats before writing to the file.

Thanks for your responses. This is definitely a "learning thread" for me.
Apr 11 '07 #8
Banfa
9,065 Expert Mod 8TB
Banfa, you must be correct about the redundant data. Apparently each piece of information in the original file is 8 bytes long. What kind of redundant data could be in the file that would change the information from 8 bytes to 2 bytes? In my array I store all the information as floats before writing to the file.

Thanks for your responses. This is definitely a "learning thread" for me.
OK well

Is the original file text while the one you are writing binary?

Is the original data stored as binary doubles, but you are saving a floats (??? see comment below)?

If your data items are floats and therefore 4 bytes long your file is too short, it should be 64M = 65536 * 256 * 4.


You probably have the information to answer this question (where as we don't) because you know what you are reading, and what you are writing.
Apr 11 '07 #9

Sign in to post your reply or Sign up for a free account.

Similar topics

1
by: red floyd | last post by:
Is there any way to retrieve the filename given to a std::ofstream (passed in constructor or in ofstream::open())? Or, should I derive from ofstream (should probably be a template to handle...
3
by: Chase Bradford | last post by:
Hey all I have a class Foo, and I'm trying to overload the << operator for both the ostream and ofstream for it. This way I should have two seperate formats for output, one for files and another...
2
by: Marina | last post by:
I get an "access violation" when I use someting like this: @@@@@@@@@@@@@@ string tempo; const char *output; vector <ofstream> outs(3); .... .... open_output=(const char *)tempo.c_str();...
2
by: slyphiad | last post by:
i'm kinda new at c++ so be patient ^_^ i was just wondering if u guys could help me to solve this problem that i had. i'm trying to create 5 sequential files using ofstream. this is what i...
5
by: Squid Seven | last post by:
I'm trying to use a pointer to an ofstream object and having problems: ofstream *sessionFile = NULL; if( directory == "" ) sessionFile = new ofstream( fileName.c_str(), ios::out ); else {
1
by: shyam | last post by:
Hi All I have a application which is basically a log manager and accepts log messages from other processes and logs them in files ( one file per process per day ). The application manages the...
5
by: Gary Wessle | last post by:
Hi I have a map<string, doublem_temperatures which gets updated often. I need to save the data to files corresponding to each string each time the map is updated, I am expecting about 80 files...
5
by: Joe Hesse | last post by:
Hi, I have a C++ function that writes to an ofstream object. I would like to sometimes use it to write to cout. I realize that cout is of type ostream which is not ofstream. Since cout is "kind...
15
by: aaragon | last post by:
Hello, does anyone have a clue about this error? and how to solve it? It seems to be trivial to me, but not for the compiler. I'm using g++ 4.2 on an Ubuntu Linux system: // main() .......
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.