473,320 Members | 1,949 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

streampos difficulties, tellg() > numeric_limits<unsigned long int>::max()


Hello!

This is my first time dealing with Very Large Files.

I have vector of strings representing numbers and I need
to extract bytes in binary mode from a Large File that
correspond to ranges specified by the strings. For example
for an input of
"0", "100", "500", "700"
I need to create three files, the first would contain
bytes 0..99 of the original file, the second: bytes 100..499
of the original file and the third: bytes 500..699 of the
original file.

All is fine, except the original file is 14 GB long and
most of the offsets represent numbers that are larger
than numeric_limits<unsigned long int>::max() which is
2^32-1 on my system, since sizeof(unsigned long int)==4 here.

So it seems I cannot use any of the portable builtin integer
types to calculate offsets, right? Still, seekg() takes
an argument of type streampos, so I figure such large
files should be supported by my implementation <OT>(g++ 4.1.2)</OT>.

How does one use this streampos type? sizeof(streampos)==16
on my system, so at first I thought it is a Large Integer
that can hold numbers up to 2^128-1. I tried outputting
numeric_limits<streampos>::max() but it says 0.

I am now confused -- it this numeric limit not supported
for streampos or is it cout that is unable to deal with
a number that large?

I also noticed that arithmetic operations are not defined
for this type, at least ++ was not defined. How do I calculate
differences between file offsets stored in streampos then?

I tried this:
ifstream in("myfile");
in.seekg(0,ios_base::end);
streampos sp = in.tellg();
cout << sp << endl;

but this throws (!) an exception with what() of:
"basic_string::_S_construct NULL not valid".

Does that mean that cout cannot be used with this streampos type?

So, to summarize my doubts. How do I portably use values
stored in a streampos to calculate differences between offsets?
I would like to tell the user that "I am now extracting
bytes 7000000000..12000000000, a total of 5000000000 bytes".
I don't want to use any _int64 or such. How do I output these
large values to an ostream? How do I tell what is the largest
file size supported on my platform?

TIA,
- J.
Feb 6 '07 #1
1 8199
Jacek Dziedzic wrote:
>
Hello!

This is my first time dealing with Very Large Files.

I have vector of strings representing numbers and I need
to extract bytes in binary mode from a Large File that
correspond to ranges specified by the strings. For example
for an input of
"0", "100", "500", "700"
I need to create three files, the first would contain
bytes 0..99 of the original file, the second: bytes 100..499
of the original file and the third: bytes 500..699 of the
original file.

All is fine, except the original file is 14 GB long and
most of the offsets represent numbers that are larger
than numeric_limits<unsigned long int>::max() which is
2^32-1 on my system, since sizeof(unsigned long int)==4 here.

So it seems I cannot use any of the portable builtin integer
types to calculate offsets, right? Still, seekg() takes
an argument of type streampos, so I figure such large
files should be supported by my implementation <OT>(g++ 4.1.2)</OT>.

How does one use this streampos type? sizeof(streampos)==16
on my system, so at first I thought it is a Large Integer
that can hold numbers up to 2^128-1. I tried outputting
numeric_limits<streampos>::max() but it says 0.

I am now confused -- it this numeric limit not supported
for streampos or is it cout that is unable to deal with
a number that large?

I also noticed that arithmetic operations are not defined
for this type, at least ++ was not defined. How do I calculate
differences between file offsets stored in streampos then?

I tried this:
ifstream in("myfile");
in.seekg(0,ios_base::end);
streampos sp = in.tellg();
cout << sp << endl;

but this throws (!) an exception with what() of:
"basic_string::_S_construct NULL not valid".

Does that mean that cout cannot be used with this streampos type?

So, to summarize my doubts. How do I portably use values
stored in a streampos to calculate differences between offsets?
I would like to tell the user that "I am now extracting
bytes 7000000000..12000000000, a total of 5000000000 bytes".
I don't want to use any _int64 or such. How do I output these
large values to an ostream? How do I tell what is the largest
file size supported on my platform?

TIA,
- J.
I just found out about streamoff. After a bit of testing,
numeric_limits<streamoff>::max() successfully prints
a Very Large Number, which is good. However this
still

ifstream in("myfile");
in.seekg(0,ios_base::end);
streampos sp = in.tellg();
cout << (streamoff)sp << endl;

throws, as earlier.

- J.
Feb 6 '07 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: Bengt Richter | last post by:
>>> hex(-5) __main__:1: FutureWarning: hex()/oct() of negative int will return a signed string in Python 2.4 and up '0xfffffffb' >>> hex(-5) '0xfffffffb' >>> hex(-5L) '-0x5L' That is sooo...
16
by: ^_^ | last post by:
conversion from: a="a"; to a=0x????; If there are many unicode strings to convert, how can I do batch-conversion?
3
by: Piotre Ugrumov | last post by:
I have done the overload on the operator >> and << in the class Attore. These 2 overload work correctly. I have done the overload of the same overload in the class Film. The class film ha inside...
5
by: Gianni Mariani | last post by:
The spirit of this arguably pointless exercise, is that the numeric_limits<T> class could be replaced with a totally generic template of compile-time, template computed constants. The problem is...
4
by: Simon | last post by:
Hi, I am trying to create a round kind of function, (to round float and double to int). In the H file I can do the following with not too much trouble... // #include <limits> #include...
1
by: liam_herron | last post by:
Does anyone know why this is? Is there a compile flag to have this return 'inf' as opposed to zero? Any help would be much appreciated. I have included a sample program with sample output: ...
2
by: Peter Lee | last post by:
ASSUME unsigned int = 32 bits #q1 unsigned int u1 = 3u << 0u valid? #q2 unsigned int u2 = 3u >0u valid? #q3 unsigned int u3 = 3u << 32u valid? #q4 unsigned int u4 = 3u >32u valid?
4
by: john | last post by:
The code: #include <iostream> #include <limits> int main() { using namespace std;
42
by: barcaroller | last post by:
In the boost::program_options tutorial, the author included the following code: cout << "Input files are: " << vm.as< vector<string() << "\n"; Basically, he is trying to print a vector...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.