I wrote a program to read a file, store that information into a 3d array, sort that array, and then write to a file. Should be straightforward. The problem I am running into is within the writing part.
I've taken the sorting function out.
For debugging purposes- when the file is written, a counter tells me that all pieces of information have been written, however, by visual inspection the data is jumbled and missing quite a few pieces of data. I have also tried displaying the data, supposedly being written, to the screen. I can't see anything funky going on. This is partly because I am trying to work with files on the order of 512-1000MB.
If I load a small file, ~10MB, no problems. I just have problems with larger file sizes. So, here's my question (sorry for my long windedness): Are there limitations to the size of file that can be written using ofstream?
I can post my code if that would be helpful. I know this should be so simple. I am pretty inexperienced with C++.
Thanks in advance!
8 5368 Banfa 9,065
Expert Mod 8TB So, here's my question (sorry for my long windedness): Are there limitations to the size of file that can be written using ofstream?
It should only be limited by the underlying filing system. You haven't said what you are using but the most common filing systems today (NTFS and whatever Linux uses) should be OK with that size of file.
FYI
NTFS:
File size limit = 16 TiB with current implementation
ext3: (common Linux filesystem)
File size limit = 16GiB – 2TiB
And for those who do not know what at TiB is:
A tebibyte (a contraction of tera binary byte) is a unit of information or computer storage, abbreviated TiB.
1 tebibyte = 2^40 bytes = 1,099,511,627,776 bytes = 1,024 gibibytes
Thanks for your replies.
I apologize for not putting this in my original post. I'm using XP, Dev-C++, and set up as NTFS. I know this is such a beginners question..ugh.
Please don't laugh at me but I figured out that when I analyzed the data, my analysis was incorrect. My files are being saved "correctly."
However, I do have a related side question: are there any theories why when I save the file, the file size drops from 128 MB to 32 MB? The data is the same (eg 65,536x256 pieces of information) and the files both stay .asc. It was this drastic change in file size that made me think I had an error in the first place. :S
Banfa 9,065
Expert Mod 8TB
However, I do have a related side question: are there any theories why when I save the file, the file size drops from 128 MB to 32 MB? The data is the same (eg 65,536x256 pieces of information) and the files both stay .asc. It was this drastic change in file size that made me think I had an error in the first place. :S
1. You are not saving all the original data
or
2. The original file contained a lot of redundant data that is not required
65,536x256 = 16M implying your pieces of information are each 2 bytes long to make a 32M file. Is this correct?
Also some NTFS disks are setup to optimize their sectors so they compress the data.
Banfa, you must be correct about the redundant data. Apparently each piece of information in the original file is 8 bytes long. What kind of redundant data could be in the file that would change the information from 8 bytes to 2 bytes? In my array I store all the information as floats before writing to the file.
Thanks for your responses. This is definitely a "learning thread" for me.
Banfa 9,065
Expert Mod 8TB
Banfa, you must be correct about the redundant data. Apparently each piece of information in the original file is 8 bytes long. What kind of redundant data could be in the file that would change the information from 8 bytes to 2 bytes? In my array I store all the information as floats before writing to the file.
Thanks for your responses. This is definitely a "learning thread" for me.
OK well
Is the original file text while the one you are writing binary?
Is the original data stored as binary doubles, but you are saving a floats (??? see comment below)?
If your data items are floats and therefore 4 bytes long your file is too short, it should be 64M = 65536 * 256 * 4.
You probably have the information to answer this question (where as we don't) because you know what you are reading, and what you are writing.
Sign in to post your reply or Sign up for a free account.
Similar topics
by: red floyd |
last post by:
Is there any way to retrieve the filename given to a std::ofstream
(passed in constructor or in ofstream::open())?
Or, should I derive from ofstream (should probably be a template to
handle...
|
by: Chase Bradford |
last post by:
Hey all
I have a class Foo, and I'm trying to overload the << operator for both
the ostream and ofstream for it. This way I should have two seperate
formats for output, one for files and another...
|
by: Marina |
last post by:
I get an "access violation" when I use someting like this:
@@@@@@@@@@@@@@
string tempo;
const char *output;
vector <ofstream> outs(3);
....
....
open_output=(const char *)tempo.c_str();...
|
by: slyphiad |
last post by:
i'm kinda new at c++ so be patient ^_^
i was just wondering if u guys could help me to solve this problem
that i had.
i'm trying to create 5 sequential files using ofstream.
this is what i...
|
by: Squid Seven |
last post by:
I'm trying to use a pointer to an ofstream object and having problems:
ofstream *sessionFile = NULL;
if( directory == "" )
sessionFile = new ofstream( fileName.c_str(), ios::out );
else
{
|
by: shyam |
last post by:
Hi All
I have a application which is basically a log manager and accepts log
messages from other processes and logs them in files ( one file per
process per day ). The application manages the...
|
by: Gary Wessle |
last post by:
Hi
I have a map<string, doublem_temperatures which gets updated
often.
I need to save the data to files corresponding to each string each
time the map is updated, I am expecting about 80 files...
|
by: Joe Hesse |
last post by:
Hi,
I have a C++ function that writes to an ofstream object.
I would like to sometimes use it to write to cout.
I realize that cout is of type ostream which is not ofstream.
Since cout is "kind...
|
by: aaragon |
last post by:
Hello, does anyone have a clue about this error? and how to solve it?
It seems to be trivial to me, but not for the compiler. I'm using g++
4.2 on an Ubuntu Linux system:
// main()
.......
|
by: DolphinDB |
last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation.
Take...
|
by: isladogs |
last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM).
In this month's session, we are pleased to welcome back...
|
by: Vimpel783 |
last post by:
Hello!
Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
|
by: jfyes |
last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
|
by: PapaRatzi |
last post by:
Hello,
I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
|
by: Defcon1945 |
last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
|
by: Shællîpôpï 09 |
last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
|
by: af34tf |
last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
|
by: Faith0G |
last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
| |