No buffered I/O in the Standard Library?

[Steven T. Hatton]

I haven't looked very closely, but from what I'm seeing, it looks like there
are no buffered I/O streams in the Standard Library. There are stream
buffers, but not buffered streams. I don't have an excellent definition of
what a buffered stream is right off the top of my head, but it's something
like a cache that can hold data from the source, or for the destination
that can't be immediately processed. Say, for example, if you are
streaming to a network socket that sends data in larger chunks than your
process produces per output operation. You don't want to send an almost
empty packet every time you write to the stream, then sit around an wait
till the output stream becomes available again when the network driver
finishes its transmission. So you write to a buffer from which the network
driver reads.

Does the standard library not provide a generic buffered stream?

--
If our hypothesis is about anything and not about some one or more
particular things, then our deductions constitute mathematics. Thus
mathematics may be defined as the subject in which we never know what we
are talking about, nor whether what we are saying is true.-Bertrand Russell

[Victor Bazarov]

Steven T. Hatton wrote:

I haven't looked very closely, but from what I'm seeing, it looks like there
are no buffered I/O streams in the Standard Library. There are stream
buffers, but not buffered streams. [...]

Does the standard library not provide a generic buffered stream?

Uh... So, let me get this straight.... If a stream has a buffer, you still
do not consider it a "buffered stream"? Huh... Interesting...

[James Daughtry]

> I haven't looked very closely

I don't have an excellent definition of what a buffered stream is

If you haven't looked hard for something you don't understand, how can
you tell if it's there or not?

[Steven T. Hatton]

Victor Bazarov wrote:

Steven T. Hatton wrote:

I haven't looked very closely, but from what I'm seeing, it looks like
there
are no buffered I/O streams in the Standard Library. There are stream
buffers, but not buffered streams. [...]

Does the standard library not provide a generic buffered stream?

Uh... So, let me get this straight.... If a stream has a buffer, you still
do not consider it a "buffered stream"? Huh... Interesting...

It just wasn't what I had expected. Fact of the matter is, it was far
simpler than I thought it would be. What I wanted to do was read a file in
to an in-memory buffer, and then process it as if it were being read from a
stream. I'm still not sure how to do that with binary data, but then I
haven't attempted that either. The trick was `ss << in.rdbuf();'. All the
examples I was seeing were reading one character at a time from cin, or one
line at at time from a file. I suspect that is not the best way to read
from system resources.

#include <fstream>
#include <iostream>
#include <sstream>
#include <string>
#include <map>

using namespace std;

ostream& printPair(ostream& out, const pair<const string, string>& p) {
return out << "key=" << p.first << ", value=" << p.second << endl;
}

template<typename T>
istream& setPair(istream& in, map<const T,T>& m) {
string key;
string value;
in >> key >> value;
if(key=="" || value=="") return in;
m[key] = value;
return in;
}
int main(int argc, char* argv[]) {
if(!argc > 1) {
cerr << "rss filename: " << endl;
return -1;
}

ifstream in(argv[1]);

if(!in.is_open()) {
cerr << "Failed to open file: " << argv[1] << endl;
return -1;
}

stringstream ss;
ss << in.rdbuf();
typedef map<const string, string> Map_T;
Map_T smap;
while(setPair(ss, smap));

for(Map_T::iterator it = smap.begin();it != smap.end(); it++ )
printPair(cout, *it);
}

--
If our hypothesis is about anything and not about some one or more
particular things, then our deductions constitute mathematics. Thus
mathematics may be defined as the subject in which we never know what we
are talking about, nor whether what we are saying is true.-Bertrand Russell

[Phil Endecott]

Steven T. Hatton wrote:

it looks like there
are no buffered I/O streams in the Standard Library.

No, streams are buffered by default. e.g.

ifstream f("...");
for(int i=0; i<100; ++i) {
char c;
f >> c;
}

does NOT do 100 calls to read(), it probably does 1.
--Phil.

[Rolf Magnus]

Phil Endecott wrote:

Steven T. Hatton wrote:

it looks like there
are no buffered I/O streams in the Standard Library.

No, streams are buffered by default. e.g.

ifstream f("...");
for(int i=0; i<100; ++i) {
char c;
f >> c;
}

does NOT do 100 calls to read(), it probably does 1.

As an example, on my linux system, a trace of the system calls (strace)
shows for the above code:

open("...", O_RDONLY) = 3
fstat64(0x3, 0xbffff62c) = 0
old_mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40018000
read(3, "lakzsdfjlzadfkjaslzdfkjasdflzkja"..., 131072) = 614
read(3, "", 131072) = 0
close(3) = 0

That means that the implementation reads and buffers the data from files
in blocks of 128kB. The old_mmap is the allocation of the buffer.

[Steven T. Hatton]

Rolf Magnus wrote:

Phil Endecott wrote:

Steven T. Hatton wrote:

it looks like there
are no buffered I/O streams in the Standard Library.

No, streams are buffered by default. e.g.

ifstream f("...");
for(int i=0; i<100; ++i) {
char c;
f >> c;
}

does NOT do 100 calls to read(), it probably does 1.

As an example, on my linux system, a trace of the system calls (strace)
shows for the above code:

open("...", O_RDONLY) = 3
fstat64(0x3, 0xbffff62c) = 0
old_mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x40018000 read(3, "lakzsdfjlzadfkjaslzdfkjasdflzkja"..., 131072)
= 614
read(3, "", 131072) = 0
close(3) = 0

That means that the implementation reads and buffers the data from files
in blocks of 128kB. The old_mmap is the allocation of the buffer.

I'd have to read the manpages on the system to understand the output, but
I'll take your word for it. This behavior is typically what a person would
want. I'm fairly confident it is not specified by the Standard(though I
could well be wrong). There are times when this is not what a programmer
would want. For example using embedded systems with limited memory, or
very large AV files. I suspect the same might apply when working with
databases that hold all of their data in a single file. I am certain that
there were (significant) performance differences between using input
streams, and buffered input streams in Java when transferring data over a
network connection. I'm fairly confident the same applies to file access.

That seems to suggest that Java offers finer grained control over I/O than
does C++ at the same level of abstraction. That finer control comes at the
cost of more complexity in the API. Typically we would expect just the
opposite when conparing Java and C++. I have less than 24 hours experience
dealing with this topic in C++ (and many of them were spent sleeping) so I
have much to learn before I can make a meaningful assessment.

This, BTW, is where I found out how to push file contents from an ifstream
to a stringstream with one ">>".

http://www.mindview.net/Books/TICPP/...ngInCPP2e.html

I believe it was Bob Tisdale who suggested Eckel's books as a good source on
I/O.
--
If our hypothesis is about anything and not about some one or more
particular things, then our deductions constitute mathematics. Thus
mathematics may be defined as the subject in which we never know what we
are talking about, nor whether what we are saying is true.-Bertrand Russell

[Rolf Magnus]

Steven T. Hatton wrote:

Rolf Magnus wrote:

Phil Endecott wrote:

Steven T. Hatton wrote:
it looks like there
are no buffered I/O streams in the Standard Library.

No, streams are buffered by default. e.g.

ifstream f("...");
for(int i=0; i<100; ++i) {
char c;
f >> c;
}

does NOT do 100 calls to read(), it probably does 1.
As an example, on my linux system, a trace of the system calls (strace)
shows for the above code:

open("...", O_RDONLY) = 3
fstat64(0x3, 0xbffff62c) = 0
old_mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x40018000
read(3, "lakzsdfjlzadfkjaslzdfkjasdflzkja"..., 131072) = 614
read(3, "", 131072) = 0
close(3) = 0

That means that the implementation reads and buffers the data from files
in blocks of 128kB. The old_mmap is the allocation of the buffer.

I'd have to read the manpages on the system to understand the output, but
I'll take your word for it.

I thought it mostly speaks for itself, but forgot that it might not be so
obvious if you're not so familiar with this kind of system.
Some explanation:

The first line says that the file was opened in read-only mode. The returned
value (the 3 at the end) is the file id.

The second line just retrieves some information about the file (first
parameter is the file id, the second a pointer to the struct that the
function fills). The return value of 0 means success. I'm not sure what the
stream does with that information.

The old_mmap in the 3rd line allocates 131072 bytes of memory. The returned
value is its address. This is the low-level OS interface that malloc and
new on my system use.

The next two lines do the actual reading from the file. They each try to
read 131072 bytes from the file (first parameter is the id, the second one
a pointer to the memory that the function reads to - the output shows the
first few bytes actually read, and parameter 3 is the number of bytes to
read). The returned value of 614 is the number of bytes actually read. The
second read() returns 0, which indicates nothing more to read, i.e. end of
file.

Well, and the last line finally closes the file.
This behavior is typically what a person would want. I'm fairly confident
it is not specified by the Standard(though I could well be wrong). There
are times when this is not what a programmer would want. For example
using embedded systems with limited memory, or very large AV files.
Right, but in such a situation, people typically resort to system specific
functions. Especially in the second case, you might not only want to switch
off the in-program buffering, but also open the file with another OS hard
disk cache strategy. And you have to read in reasonably sized chunks
yourself then, and the strategy you have to use for is often system
specific too. So I don't think that a standard stream would help in such a
situation.
OTOH, you can implement your own streambuf and use the standard streams with
it.
I suspect the same might apply when working with databases that hold all
of their data in a single file. I am certain that there were
(significant) performance differences between using input streams, and
buffered input streams in Java when transferring data over a network
connection. I'm fairly confident the same applies to file access.

That seems to suggest that Java offers finer grained control over I/O than
does C++ at the same level of abstraction. That finer control comes at
the cost of more complexity in the API. Typically we would expect just
the opposite when conparing Java and C++. I have less than 24 hours
experience dealing with this topic in C++ (and many of them were spent
sleeping) so I have much to learn before I can make a meaningful
assessment.

The C++ standard streams and streambuffers don't offer much flexibility in
this direction.