Sanity check output streambuf scheme?

[Fred Ma]

Hello,

I posted previously under the thread:

How to break this up into streambuf/ostream

I've asked our library to get "C++ IOStreams and Locales..." by A.
Langer et al. Meantime, I've looked at the C++ standard (ISO/IEC
14882) from the web to get a grip on the relationship between ostream
methods and streambuf methods: flush(), operator<<(), sputc(),
sputn(), xsputn(), overflow(), sync(), and pubsync().

My problem is that I don't want to use a fixed size buffer in my
output streambuf derivative (mostreambuf), and I don't want the buffer
to flush automatically. I want the buffer to continue amassing data
until the user explicitly invokes ostream::flush(), possibly using
endl. This means mostreambuf will not use the pbase(), pptr(), or
epptr(). Instead, it will use a member ostringstream object "oss" to
buffer output. This way, the buffer can grow indefinitely as
required, until the user does a flush.

Before going further, I should point out that the standard does not
say anything about whether sync calls overload or vice-versa, so I
can't presume either.

My approach to realizing the above scheme is based on Josuttis's
example code in "The C++ Standard Library", as well as seeing that
flush() calls pubsync(), which calls the virtual (overloadable)
sync(). First, I set all 3 buffer pointers are set to NULL so that
sputc(c) calls overflow(), which I will overload so that c is written
to oss. Likewise, xsputn() will be overloaded to write to oss. I
will also overload sync() so that ostream::flush() causes the contents
of oss to be sent to its destination, followed by nulling oss to "".

Is this a reasonable approach?

Thanks for any feedback.

Fred
--
Fred Ma
Dept. of Electronics, Carleton University
1125 Colonel By Drive, Ottawa, Ontario
Canada, K1S 5B6

[John Harrison]

"Fred Ma" <fm*@doe.carleton.ca> wrote in message
news:40***************@doe.carleton.ca...

Hello,

I posted previously under the thread:

How to break this up into streambuf/ostream

I've asked our library to get "C++ IOStreams and Locales..." by A.
Langer et al. Meantime, I've looked at the C++ standard (ISO/IEC
14882) from the web to get a grip on the relationship between ostream
methods and streambuf methods: flush(), operator<<(), sputc(),
sputn(), xsputn(), overflow(), sync(), and pubsync().

My problem is that I don't want to use a fixed size buffer in my
output streambuf derivative (mostreambuf), and I don't want the buffer
to flush automatically. I want the buffer to continue amassing data
until the user explicitly invokes ostream::flush(), possibly using
endl. This means mostreambuf will not use the pbase(), pptr(), or
epptr(). Instead, it will use a member ostringstream object "oss" to
buffer output. This way, the buffer can grow indefinitely as
required, until the user does a flush.

Before going further, I should point out that the standard does not
say anything about whether sync calls overload or vice-versa, so I
overflow
can't presume either.
Well it the code you write that does this, you decide if sync calls overflow
or vice versa. Normally I would do neither.

My approach to realizing the above scheme is based on Josuttis's
example code in "The C++ Standard Library", as well as seeing that
flush() calls pubsync(), which calls the virtual (overloadable)
sync(). First, I set all 3 buffer pointers are set to NULL so that
sputc(c) calls overflow(), which I will overload so that c is written
to oss. Likewise, xsputn() will be overloaded to write to oss. I
will also overload sync() so that ostream::flush() causes the contents
of oss to be sent to its destination, followed by nulling oss to "".

Is this a reasonable approach?

Seems fine to me.

john

[Fred Ma]

John Harrison wrote:

Before going further, I should point out that the standard does not
say anything about whether sync calls overload or vice-versa, so I

overflow

Yes, my bad.

can't presume either.

Well it the code you write that does this, you decide if sync calls overflow
or vice versa. Normally I would do neither.

Duh. Yeah, you're right. I was thinking about the default
behaviour, but it doesn't matter in my case.

Is this a reasonable approach?

Seems fine to me.

Thanks for the sanity check, John. Time to code it up.

Fred

[Kevin Goodsell]

Fred Ma wrote:

I want the buffer to continue amassing data
until the user explicitly invokes ostream::flush(), possibly using
endl. This means mostreambuf will not use the pbase(), pptr(), or
epptr(). Instead, it will use a member ostringstream object "oss" to
buffer output. This way, the buffer can grow indefinitely as
required, until the user does a flush.

I would suggest using a std::vector or maybe std::deque for your buffer,
instead of an ostringstream. Using ostringstream requires rather a lot
of overhead for simple buffering (including construction of the stream
and buffer objects, the std::string used internally, and all the
overhead involved in the actual I/O, such as creating and destroying
sentry objects).

-Kevin
--
My email address is valid, but changes periodically.
To contact me please use the address from a recent posting.

[Fred Ma]

Kevin Goodsell wrote:

I want the buffer to continue amassing data
until the user explicitly invokes ostream::flush(), possibly using
endl. This means mostreambuf will not use the pbase(), pptr(), or
epptr(). Instead, it will use a member ostringstream object "oss" to
buffer output. This way, the buffer can grow indefinitely as
required, until the user does a flush.

I would suggest using a std::vector or maybe std::deque for your buffer,
instead of an ostringstream. Using ostringstream requires rather a lot
of overhead for simple buffering (including construction of the stream
and buffer objects, the std::string used internally, and all the
overhead involved in the actual I/O, such as creating and destroying
sentry objects).

Thanks for the cautionary note, Kevin. I'm going to hang on to
it as an additional migration step. Right now, I'm trying to
get broken code to work again, so the smaller step above would
be a bit of a godsend in the short term. I'll have to see
whether my use of the my ostream's operator<<() relies on
the characteristics of my ostringstream buffer when the data is
sent to my streambuf [I've got operator<<() code sprinkled
throughout my sprawling programing]. See how much modification
(if any) is needed to use a vector buffer instead. Probably not
much, since I now actually derive from ostream, which takes care
of the manipulators. Prior to that, I had created my own
abomination: ostream/streambuf hybrid. Before our compiler
was upgraded, it worked (probably because the manipulators were
being taken care of by the ostringstream object.

Fred
--
Fred Ma
Dept. of Electronics, Carleton University
1125 Colonel By Drive, Ottawa, Ontario
Canada, K1S 5B6

[Fred Ma]

Kevin Goodsell wrote:

I want the buffer to continue amassing data
until the user explicitly invokes ostream::flush(), possibly using
endl. This means mostreambuf will not use the pbase(), pptr(), or
epptr(). Instead, it will use a member ostringstream object "oss" to
buffer output. This way, the buffer can grow indefinitely as
required, until the user does a flush.

I would suggest using a std::vector or maybe std::deque for your buffer,
instead of an ostringstream. Using ostringstream requires rather a lot
of overhead for simple buffering (including construction of the stream
and buffer objects, the std::string used internally, and all the
overhead involved in the actual I/O, such as creating and destroying
sentry objects).

So I've revamped all my ostream derivatives to use my streambuf
derivatives. Instead of using ostringstream for buffering, I'm
using a vector<char> (still debugging). Part of the debugging
led me to read up on traits for strings and streams. It's not
all clear yet, but I did find mention of reasons to use strings
rather than vector<char>, related to traits. The professed
reason was that traits encapsulate possibly platform-dependent
optimized implementations of string processing functions. So
I looked again at the the above reasoning for using vector<char>.
It seems to be one of saving memory. Am I right in concluding
that it isn't directly a speed issue to use ostringstream for
buffering, and that's only a memory issue? I will only create
these ostring objects once when the program starts.

Fred
--
Fred Ma
Dept. of Electronics, Carleton University
1125 Colonel By Drive, Ottawa, Ontario
Canada, K1S 5B6

[Fred Ma]

Fred Ma wrote:

Kevin Goodsell wrote:

I want the buffer to continue amassing data
until the user explicitly invokes ostream::flush(), possibly using
endl. This means mostreambuf will not use the pbase(), pptr(), or
epptr(). Instead, it will use a member ostringstream object "oss" to
buffer output. This way, the buffer can grow indefinitely as
required, until the user does a flush.

I would suggest using a std::vector or maybe std::deque for your buffer,
instead of an ostringstream. Using ostringstream requires rather a lot
of overhead for simple buffering (including construction of the stream
and buffer objects, the std::string used internally, and all the
overhead involved in the actual I/O, such as creating and destroying
sentry objects).

I did find mention of reasons to use strings
rather than vector<char>, related to traits. The professed
reason was that traits encapsulate possibly platform-dependent
optimized implementations of string processing functions. So
I looked again at the the above reasoning for using vector<char>.
It seems to be one of saving memory. Am I right in concluding
that it isn't directly a speed issue to use ostringstream for
buffering, and that's only a memory issue? I will only create
these ostring objects once when the program starts.

I'm answering my own question (correctly, I hope). It seems like
a good way around the overhead of ostringstream as a buffer within
streambuf is to simply use a string. In contrast to a vector, this
would benefit from any platform-optimizations in string handling.

Fred
--
Fred Ma
Dept. of Electronics, Carleton University
1125 Colonel By Drive, Ottawa, Ontario
Canada, K1S 5B6

[Kevin Goodsell]

Fred Ma wrote:

So I've revamped all my ostream derivatives to use my streambuf
derivatives. Instead of using ostringstream for buffering, I'm
using a vector<char> (still debugging). Part of the debugging
led me to read up on traits for strings and streams. It's not
all clear yet, but I did find mention of reasons to use strings
rather than vector<char>, related to traits. The professed
reason was that traits encapsulate possibly platform-dependent
optimized implementations of string processing functions.
I don't see any reason that using a vector<char> would prevent you from
using traits.
So
I looked again at the the above reasoning for using vector<char>.
It seems to be one of saving memory. Am I right in concluding
that it isn't directly a speed issue to use ostringstream for
buffering, and that's only a memory issue? I will only create
these ostring objects once when the program starts.

The main issue I had in mind is speed. The job of a streambuf is to
store bytes and dump them to or read them from some other location when
appropriate. A ostringstream is a very round about way of doing this.
Essentially you are making your class a wrapper for a (rather
inefficient) wrapper for stringbuf. Every byte you write will require a
redundant call to a formatting function, which includes considerable
overhead.

Implementing a streambuf (which is the sports car of the C++ I/O world,
fast but not terribly functional) in terms of a stream (which is the van
of the C++ I/O world - it's full of all the tools that provide advanced
functionality, but it's not exactly speedy) is silly. It's like building
a sports-car-shaped shell around your van. It looks like a sports car...
sort of... but it's big and bulky, and as slow as a van. Plus, you can't
get to the parts of the van with all your tools anymore. So you've got
something that's as slow as a van (actually a bit slower due to the
extra bulk of the shell), and has the functionality of the sports car
(which is to say, not much).

Think about it. Once you're done with this, you'll be using a stream
that's implemented using a streambuf that's implemented using a stream
that's implemented using a streambuf! There's two extra steps there that
make no sense!

You could make your streambuf a direct wrapper for stringbuf. This would
make more sense, and it would be faster and probably easier to write.
Implementing a new streambuf using a vector is probably a bit more
difficult, but you get the functionality you need without a lot of extra
stuff you don't need, and it would probably be a bit faster still.

-Kevin
--
My email address is valid, but changes periodically.
To contact me please use the address from a recent posting.

[Kevin Goodsell]

Fred Ma wrote:

I'm answering my own question (correctly, I hope). It seems like
a good way around the overhead of ostringstream as a buffer within
streambuf is to simply use a string. In contrast to a vector, this
would benefit from any platform-optimizations in string handling.

What string handling? This is a streambuf, all it does is store bytes.

-Kevin
--
My email address is valid, but changes periodically.
To contact me please use the address from a recent posting.

[Fred Ma]

Kevin Goodsell wrote:

I'm answering my own question (correctly, I hope). It seems like
a good way around the overhead of ostringstream as a buffer within
streambuf is to simply use a string. In contrast to a vector, this
would benefit from any platform-optimizations in string handling.

What string handling? This is a streambuf, all it does is store
bytes.

Here's an example "mostreambuf" of what I mean. I'm still debugging,
but it is the basic idea. The only difference with a streambuf that
just overloads overflow() is can only be flushed by ostream::flush().
It is not flushed by a buffer-filled condition because the buffer
grows as needed. Hence, I also don't have alot of code to check EOF.
It's a dumbing down of streambuf.

| class mostreambuf : public std::streambuf
| {
| protected:
| string sbuf;
|
| public:
|
| virtual int sync()
| {
| if( ! sbuf.empty() )
| {
| sbuf += '\0';
| SendStringToDestination( sbuf.c_str() );
| sbuf.clear();
| }
| return 0;
| }
|
| virtual int_type
| overflow( int_type c )
| {
| sbuf += c ;
| return c;
| };
|
| virtual streamsize
| xsputn( const char_type* s, streamsize n )
| {
| sbuf.append( s, s+n );
| return n;
| };
| };

Judging from your previous post, I might have given the impression
that I was considering going back to ostringstream instead of the
"string sbuf" above for the buffer within mostreambuf. Actually, I
understood your comment about the inefficiency in putting a stream
object inside a streambuf, at least in a general level. That scheme
is quite backward. It's just a hold-over from a prior implementation,
which was not based on much familiarity with stream internals or the
underlying streambuf. Fortunately, a simple string ("sbuf" above)
does the job.

From your last post, I'm also clear that the issue of relevance was
speed. Thanks for clarifying that.

Fred