473,672 Members | 3,445 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

basic_fstream and large files

Hello,

I have code written using the basic fstream object on a Win32 system. This
is of course simply a typedef for basic_fstream<c har, char_traits<cha r.

This object can not be used to write or read files that are larger than 2Gb
in size, as it would appear (from looking at the iosfwd header) that the
underlying streamoff and streamsize types are typedef'd to long and int. On
Win64 systems, these are typedef'd to __int64, so it seems that on Win64
systems fstream can in fact be used to read and write large files.

However, is there a way I can define a new type and define a char_traits for
this type so that I can use basic_fstream to read and write large files on
Win32? Looking into the std c++ iostream headers, it is not clear that this
is in fact doable, as many things seem to depend on these basic streamoff
and streamsize typedefs, which are not mutable in my own class.

I would like to know whether I am reading this correctly, or whether people
have had success in doing this, i.e. creating a char_traits type that when
used with basic_fstream will allow for large (>2Gb) file reading and
writing.

Thanks,

-Eric Twietmeyer

(MSDN no-spam alias: MS***********@n ospam.nospam)
Mar 27 '07 #1
3 3075
Eric Twietmeyer wrote:
:: Hello,
::
:: I have code written using the basic fstream object on a Win32
:: system. This is of course simply a typedef for basic_fstream<c har,
:: char_traits<cha r.
::
:: This object can not be used to write or read files that are larger
:: than 2Gb in size, as it would appear (from looking at the iosfwd
:: header) that the underlying streamoff and streamsize types are
:: typedef'd to long and int. On Win64 systems, these are typedef'd to
:: __int64, so it seems that on Win64 systems fstream can in fact be
:: used to read and write large files.
::
:: However, is there a way I can define a new type and define a
:: char_traits for this type so that I can use basic_fstream to read
:: and write large files on Win32? Looking into the std c++ iostream
:: headers, it is not clear that this is in fact doable, as many things
:: seem to depend on these basic streamoff and streamsize typedefs,
:: which are not mutable in my own class.
::
:: I would like to know whether I am reading this correctly, or whether
:: people have had success in doing this, i.e. creating a char_traits
:: type that when used with basic_fstream will allow for large (>2Gb)
:: file reading and writing.
::

You CAN read and write larger files, as long as you do it sequentially. The
size of streamoff limits the amount of positioning you can do. Do you really
do seeks with offsets >2GB?
Bo Persson
Mar 27 '07 #2

"Bo Persson" <bo*@gmb.dkwrot e in message
news:56******** *****@mid.indiv idual.net...
Eric Twietmeyer wrote:
:: Hello,
::
:: I have code written using the basic fstream object on a Win32
:: system. This is of course simply a typedef for basic_fstream<c har,
:: char_traits<cha r.
::
:: This object can not be used to write or read files that are larger
:: than 2Gb in size, as it would appear (from looking at the iosfwd
:: header) that the underlying streamoff and streamsize types are
:: typedef'd to long and int. On Win64 systems, these are typedef'd to
:: __int64, so it seems that on Win64 systems fstream can in fact be
:: used to read and write large files.
::
:: However, is there a way I can define a new type and define a
:: char_traits for this type so that I can use basic_fstream to read
:: and write large files on Win32? Looking into the std c++ iostream
:: headers, it is not clear that this is in fact doable, as many things
:: seem to depend on these basic streamoff and streamsize typedefs,
:: which are not mutable in my own class.
::
:: I would like to know whether I am reading this correctly, or whether
:: people have had success in doing this, i.e. creating a char_traits
:: type that when used with basic_fstream will allow for large (>2Gb)
:: file reading and writing.
::

You CAN read and write larger files, as long as you do it sequentially.
The size of streamoff limits the amount of positioning you can do. Do you
really do seeks with offsets >2GB?
Bo Persson

Yes, unfortunately, I do in fact need to be able to offset in the stream to
locations greater than 2Gb from the current point.

I'm also just curious how one may specialize the basic stream / buf objects
using different char_traits, whether that is actually doable given this
implementation provided with DevStudio, or whether it is expected that only
char_traits<cha rand char_traits<wch ar_twould be used, and no others.

-Eric
Mar 28 '07 #3
Eric Twietmeyer wrote:
:: "Bo Persson" <bo*@gmb.dkwrot e in message
:: news:56******** *****@mid.indiv idual.net...
::: Eric Twietmeyer wrote:
::::: Hello,
:::::
::::: I have code written using the basic fstream object on a Win32
::::: system. This is of course simply a typedef for
::::: basic_fstream<c har, char_traits<cha r.
:::::
::::: This object can not be used to write or read files that are larger
::::: than 2Gb in size, as it would appear (from looking at the iosfwd
::::: header) that the underlying streamoff and streamsize types are
::::: typedef'd to long and int. On Win64 systems, these are typedef'd
::::: to __int64, so it seems that on Win64 systems fstream can in fact
::::: be used to read and write large files.
:::::
::::: However, is there a way I can define a new type and define a
::::: char_traits for this type so that I can use basic_fstream to read
::::: and write large files on Win32? Looking into the std c++ iostream
::::: headers, it is not clear that this is in fact doable, as many
::::: things seem to depend on these basic streamoff and streamsize
::::: typedefs, which are not mutable in my own class.
:::::
::::: I would like to know whether I am reading this correctly, or
::::: whether people have had success in doing this, i.e. creating a
::::: char_traits type that when used with basic_fstream will allow for
::::: large (>2Gb) file reading and writing.
:::::
:::
::: You CAN read and write larger files, as long as you do it
::: sequentially. The size of streamoff limits the amount of
::: positioning you can do. Do you really do seeks with offsets >2GB?
:::
:::
::: Bo Persson
:::
:::
::
:: Yes, unfortunately, I do in fact need to be able to offset in the
:: stream to locations greater than 2Gb from the current point.

Too bad.

The Band-Aid solution is to do multiple seeks, 2 GB at a time. Works for
moderate file sizes.

::
:: I'm also just curious how one may specialize the basic stream / buf
:: objects using different char_traits, whether that is actually doable
:: given this implementation provided with DevStudio, or whether it is
:: expected that only char_traits<cha rand char_traits<wch ar_twould
:: be used, and no others.

It is possible, but hard. It will probably not help you here, anyway.

Eventually, all the seeks will probably come down to a C level fseek(),
which also takes a long parameter. The real solution is to have a 'long'
type that is actually sort-of-long, and not just a different name for 'int'.
A bit late for that now, unfortunately.
Bo Persson
Mar 28 '07 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
1512
by: Avner | last post by:
I have a C++ code which I want to upgrade to the standard C++ The code constructs a file and sets the file permission The new iostream library does not include filebuf::openprot The old ofstream constructor (<fstream.h>) accepts a third parameter (int nProt = filebuf::openprot) i.e. ofstream( const char* szName, int nMode = ios::out, int nProt = filebuf::openprot );
6
2648
by: Greg | last post by:
I am working on a project that will have about 500,000 records in an XML document. This document will need to be queried with XPath, and records will need to be updated. I was thinking about splitting up the XML into several XML documents (perhaps 50,000 per document) to be more efficient but this will make things a lot more complex because the searching needs to go accross all 500,000 records. Can anyone point me to some best practices...
3
6344
by: Buddy Ackerman | last post by:
I'm trying to write files directly to the client so that it forces the client to open the Save As dialog box rather than display the file. On some occasions the files are very large (100MB+). On these files teh time that it takes until the client displays the Save As dialog can be extrordinarily long (3+ minutes). I don't understand why. I was initiall using the format: Respnse.writefile("filepath", offset, length) but that simply...
3
2332
by: A.M-SG | last post by:
Hi, I have a ASP.NET aspx file that needs to pass large images from a network storage to client browser. The requirement is that users cannot have access to the network share. The aspx file must be the only method that users receive image files.
2
1963
by: jdev8080 | last post by:
We are looking at creating large XML files containing binary data (encoded as base64) and passing them to transformers that will parse and transform the data into different formats. Basically, we have images that have associated metadata and we are trying to develop a unified delivery mechanism. Our XML documents may be as large as 1GB and contain up to 100,000 images. My question is, has anyone done anything like this before?
1
6313
by: Lars B | last post by:
Hey guys, I have written a C++ program that passes data from a file to an FPGA board and back again using software and DMA buffers. In my program I need to compare the size of a given file against a software buffer of size 3MB. This is needed so as to see which function to use to read from the file. As the files used range from very large (>30GB) to very small (<3MB), I have enabled large file support and I obtain the file size by using the...
8
6388
by: theCancerus | last post by:
Hi All, I am not sure if this is the right place to ask this question but i am very sure you may have faced this problem, i have already found some post related to this but not the answer i am looking for. My problem is that i have to upload images and store them. I am using filesystem for that. setup is something like this, their will be items/groups/user each can
1
3885
by: =?Utf-8?B?UVNJRGV2ZWxvcGVy?= | last post by:
Using .NET 2.0 is it more efficient to copy files to a single folder versus spreading them across multiple folders. For instance if we have 100,000 files to be copied, Do we copy all of them to a single folder called 'All Files' Do we spread them out and copy them to multiple folders like Folder 000 - Copy files from 0 to 1000 Folder 001 - Copy files from 1000 to 2000 Folder 002 - Copy files from 2000 to 2999
17
9908
by: byte8bits | last post by:
How does C++ safely open and read very large files? For example, say I have 1GB of physical memory and I open a 4GB file and attempt to read it like so: #include <iostream> #include <fstream> #include <string> using namespace std; int main () {
0
8504
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
1
8643
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8697
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7475
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
5720
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4439
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2837
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2093
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
2
1837
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.