Problem with user-defined IO streambuf for encoding purposes

[hsmit.home]

Hi everyone,

I'm having some difficulty with the following piece of code. I have
stripped it to it's bare minimum to demonstrate the problem at hand.

Compiler: MS Visual C++ 2005 Express Edition (similar problem arises
with 2008)

Runtime Library: All multi-threaded variants have been seen to fail
[DLL/Static] | [Debug|Release]

Purpose: define a user defined stream buffer that processes each
incoming character and translates it to an encoded value. Place the
encoded character into a local buffer for output. The most simple case
would be an encoder that translates each character to upper case. A
more complicated case would be an encoder that encodes plain-text to
base64 encoding (this is not a one-to-one character encoding, it's a 3
character to 4character encoding, this is why an internal buffer is
needed)

Problem: The code throws an "Unhandled exception at 0x00529bcc in
hsl_d.exe: 0xC0000005: Access violation reading location 0x00000000."
after the "encoderbuf::un derflow c = 51" character '3' has been read.
This basically tells me that deep down in the internals of the IO
library something is being dereferenced that is not allocated.

Question(s):
1) Does this occur with other compilers? (requires testing)
2) Is this a problem with the IO library? (unlikely I think)
3) Am I doing something stupid? (more than likely) And if so what?

References:
C++ Standard Library - A Tutorial and Reference (13.13.3 User-Defined
Stream Buffers)
Code:

#include <iostream>
#include <sstream>

class encoderbuf : public std::streambuf {

char mCharBuf[128];
int mBufLen;
int mBufPos;
public:
//--------------------------------------------------------------
/** default constructor */
encoderbuf()
: std::streambuf( )
, mBufLen(0)
, mBufPos(0)
{
}

//--------------------------------------------------------------
/** outgoing data */
virtual int_type underflow () {
int_type c = EOF;
if (mBufPos < mBufLen) {
c = mCharBuf[mBufPos++];
}
std::cout << "encoderbuf::un derflow c = " << c << std::endl;

return c;
}

//--------------------------------------------------------------
/** incoming data */
virtual int_type overflow (int_type c) {
std::cout << "encoderbuf::ov erflow c = " << c << std::endl;
//TODO: do encoding here
mCharBuf[mBufLen++] = c;
return c;
}
};

//--------------------------------------------------------------
int main (int argc, char ** argv) {
encoderbuf buf;
std::iostream iostr(&buf);
iostr << 12345 << std::endl;
std::stringstre am sstr;
iostr >sstr.rdbuf() ; // EXCEPTION AT PROCESSING CHARACTER '3'

std::string str = sstr.str();
std::cout << "main str = " << str << std::endl;
}
Output:
encoderbuf::ove rflow c = 49
encoderbuf::ove rflow c = 50
encoderbuf::ove rflow c = 51
encoderbuf::ove rflow c = 52
encoderbuf::ove rflow c = 53
encoderbuf::ove rflow c = 10
encoderbuf::und erflow c = 49
encoderbuf::und erflow c = 50
encoderbuf::und erflow c = 51
popup: Unhandled exception at 0x00529bcc in hsl_d.exe: 0xC0000005:
Access violation reading location 0x00000000.

[Jim Langston]

hs********@gmai l.com wrote:

Hi everyone,

I'm having some difficulty with the following piece of code. I have
stripped it to it's bare minimum to demonstrate the problem at hand.

Compiler: MS Visual C++ 2005 Express Edition (similar problem arises
with 2008)

Runtime Library: All multi-threaded variants have been seen to fail
[DLL/Static] | [Debug|Release]

Purpose: define a user defined stream buffer that processes each
incoming character and translates it to an encoded value. Place the
encoded character into a local buffer for output. The most simple case
would be an encoder that translates each character to upper case. A
more complicated case would be an encoder that encodes plain-text to
base64 encoding (this is not a one-to-one character encoding, it's a 3
character to 4character encoding, this is why an internal buffer is
needed)

Problem: The code throws an "Unhandled exception at 0x00529bcc in
hsl_d.exe: 0xC0000005: Access violation reading location 0x00000000."
after the "encoderbuf::un derflow c = 51" character '3' has been read.
This basically tells me that deep down in the internals of the IO
library something is being dereferenced that is not allocated.

Question(s):
1) Does this occur with other compilers? (requires testing)
2) Is this a problem with the IO library? (unlikely I think)
3) Am I doing something stupid? (more than likely) And if so what?

References:
C++ Standard Library - A Tutorial and Reference (13.13.3 User-Defined
Stream Buffers)
Code:

#include <iostream>
#include <sstream>

class encoderbuf : public std::streambuf {

char mCharBuf[128];
int mBufLen;
int mBufPos;
public:
//--------------------------------------------------------------
/** default constructor */
encoderbuf()
: std::streambuf( )
, mBufLen(0)
, mBufPos(0)
{
}

//--------------------------------------------------------------
/** outgoing data */
virtual int_type underflow () {
int_type c = EOF;
if (mBufPos < mBufLen) {
c = mCharBuf[mBufPos++];
}
std::cout << "encoderbuf::un derflow c = " << c << std::endl;

return c;
}

//--------------------------------------------------------------
/** incoming data */
virtual int_type overflow (int_type c) {
std::cout << "encoderbuf::ov erflow c = " << c << std::endl;
//TODO: do encoding here
mCharBuf[mBufLen++] = c;

Most likely you meant
mCharBuf[mBufPos++] = c;
here. mBufPos, not mBufLen.

return c;
}
};

//--------------------------------------------------------------
int main (int argc, char ** argv) {
encoderbuf buf;
std::iostream iostr(&buf);
iostr << 12345 << std::endl;
std::stringstre am sstr;
iostr >sstr.rdbuf() ; // EXCEPTION AT PROCESSING CHARACTER '3'

std::string str = sstr.str();
std::cout << "main str = " << str << std::endl;
}
Output:
encoderbuf::ove rflow c = 49
encoderbuf::ove rflow c = 50
encoderbuf::ove rflow c = 51
encoderbuf::ove rflow c = 52
encoderbuf::ove rflow c = 53
encoderbuf::ove rflow c = 10
encoderbuf::und erflow c = 49
encoderbuf::und erflow c = 50
encoderbuf::und erflow c = 51
popup: Unhandled exception at 0x00529bcc in hsl_d.exe: 0xC0000005:
Access violation reading location 0x00000000.

--
Jim Langston
ta*******@rocke tmail.com

[anon]

hs********@gmai l.com wrote:

>
Has anyone tried running this code under Linux? I don't have a Linux
machine handy at the moment, but I would be curious to know if this is
strictly a MS issue.

It crashes.

[James Kanze]

On Dec 12, 9:34 am, hsmit.h...@gmai l.com wrote:

I'm having some difficulty with the following piece of code. I have
stripped it to it's bare minimum to demonstrate the problem at hand.

Compiler: MS Visual C++ 2005 Express Edition (similar problem arises
with 2008)

Runtime Library: All multi-threaded variants have been seen to fail
[DLL/Static] | [Debug|Release]

Purpose: define a user defined stream buffer that processes each
incoming character and translates it to an encoded value. Place the
encoded character into a local buffer for output. The most simple case
would be an encoder that translates each character to upper case. A
more complicated case would be an encoder that encodes plain-text to
base64 encoding (this is not a one-to-one character encoding, it's a 3
character to 4character encoding, this is why an internal buffer is
needed)

Problem: The code throws an "Unhandled exception at 0x00529bcc in
hsl_d.exe: 0xC0000005: Access violation reading location 0x00000000."
after the "encoderbuf::un derflow c = 51" character '3' has been read.
This basically tells me that deep down in the internals of the IO
library something is being dereferenced that is not allocated.

Question(s):
1) Does this occur with other compilers? (requires testing)
2) Is this a problem with the IO library? (unlikely I think)
3) Am I doing something stupid? (more than likely) And if so what?

References:
C++ Standard Library - A Tutorial and Reference (13.13.3 User-Defined
Stream Buffers)

Code:

#include <iostream>
#include <sstream>

class encoderbuf : public std::streambuf {

char mCharBuf[128];
int mBufLen;
int mBufPos;
public:
//--------------------------------------------------------------
/** default constructor */
encoderbuf()
: std::streambuf( )
, mBufLen(0)
, mBufPos(0)
{
}

//--------------------------------------------------------------
/** outgoing data */
virtual int_type underflow () {
int_type c = EOF;
if (mBufPos < mBufLen) {
c = mCharBuf[mBufPos++];
}
std::cout << "encoderbuf::un derflow c = " << c << std::endl;
return c;
}

Attention: this function does not meet the requirements for
underflow. Underflow must return the character, leaving it in
the stream. This normally involves at least a one character
buffer. In your case, of course, you've got a longer buffer,
so you could just use that:

int_type c = EOF ;
if ( mBufPos != mBufLen ) {
setg( mCharBuf, mCharBuf + mBufPos, mCharBuf + mBufLen ) ;
c = mCharBuf[ mBufPos ] ;
mBufPos = mBufLen ;
}
return c ;

. Although the problem being addressed is somewhat different,
you'll find a short discussion concerning the requirements of
underflow and overflow in my articles on filtering streambuf's
(http://kanze.james.neuf.fr/articles-en.html).

//--------------------------------------------------------------
/** incoming data */
virtual int_type overflow (int_type c) {
std::cout << "encoderbuf::ov erflow c = " << c << std::endl;
//TODO: do encoding here
mCharBuf[mBufLen++] = c;
return c;
}
};

//--------------------------------------------------------------
int main (int argc, char ** argv) {
encoderbuf buf;
std::iostream iostr(&buf);
iostr << 12345 << std::endl;
std::stringstre am sstr;
iostr >sstr.rdbuf() ; // EXCEPTION AT PROCESSING CHARACTER '3'
std::string str = sstr.str();
std::cout << "main str = " << str << std::endl;
}

Output:
encoderbuf::ove rflow c = 49
encoderbuf::ove rflow c = 50
encoderbuf::ove rflow c = 51
encoderbuf::ove rflow c = 52
encoderbuf::ove rflow c = 53
encoderbuf::ove rflow c = 10
encoderbuf::und erflow c = 49
encoderbuf::und erflow c = 50
encoderbuf::und erflow c = 51
popup: Unhandled exception at 0x00529bcc in hsl_d.exe: 0xC0000005:
Access violation reading location 0x00000000.

I get the correct results with my version of underflow. Both
with g++ (under Linux) and VC++ (under Windows).

What's probably happening is that the implementation, having
read the character with underflow, increments the get pointer
(gptr()). Which is null, since you've never set it. The next
time around, it compares this pointer with egptr(), finds that
they aren't equal, and accesses through it directly. (This is,
at least, what happens with g++ under Solaris. At least, if the
input routing is using sgetn, this is what happens.)

The normal way of implementing input in a streambuf is to
override underflow, using a buffer of at least one character.
(The reason for this is to allow simple and direct support for
putback.)

--
James Kanze (GABI Software) email:ja******* **@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]

[hsmit.home]

Thank you James for all the useful info. Looks like I've got some
serious reading tonight.

Before I read your response I got my code to do what I wanted. It was
along the lines of what you described. It's still a work in progress.
Here is the result (comments/refinements/improvements of this code are
welcome, in fact they would very much be appreciatted).

 
#include <iostream>
#include <sstream>
 
//--------------------------------------------------------------
/**
General purpose buffered filter / encoder.
@todo: still requires testing and refinement.
*/
class encoderbuf : public std::streambuf {
 
std::stringbuf mBuf;
char mIBuf[3]; //TODO: make this dynamic
 
public:
//--------------------------------------------------------------
/** default constructor */
encoderbuf() {
mBuf.pubsetbuf(mIBuf, sizeof(mIBuf));
setg(mIBuf, mIBuf, mIBuf+sizeof(mIBuf));
}
 
private:
//--------------------------------------------------------------
/** default encoder routine - does nothing */
virtual void                  encode (int_type c) {
*gptr() = c;
gbump(1);
}
 
//--------------------------------------------------------------
/** flushes the buffered data to the final destination */
int                            flush () {
int_type c = mBuf.sbumpc();
if (c == EOF) {
return EOF;
}
while (c != EOF) {
encode(c);
if (gptr() >= egptr()) {
break;
}
c = mBuf.sbumpc();
}
setg(mIBuf, mIBuf, gptr());
 
return 0;
}
 
//--------------------------------------------------------------
/** flush the buffered data to output */
virtual int                   sync () {
return flush();
}
 
 
//--------------------------------------------------------------
/** buffer the incoming data */
virtual int_type              overflow (int_type c) {
return mBuf.sputc(c);
}
 
//--------------------------------------------------------------
/** reset the internal input buffers and flush the buffered
data to output
*/
virtual int_type              underflow () {
 
memset(mIBuf, 'x', sizeof(mIBuf));
setg(mIBuf, mIBuf, mIBuf+sizeof(mIBuf));
 
if (flush() == EOF) {
return EOF;
}
 
return ((unsigned char)(*gptr()));
}
};
 
//--------------------------------------------------------------
/**
Example encoder
*/
class uppercasebuf : public encoderbuf {
 
public:
//--------------------------------------------------------------
/** upper case encoder */
virtual void                  encode (int_type c) {
*gptr() = toupper(c);
gbump(1);
}
};
 
//--------------------------------------------------------------
int main (int argc, char ** argv) {
uppercasebuf buf;
std::iostream iostr(&buf);
iostr << "abcdef";
iostr << "123HIG";
iostr << "aabb" << std::endl;
std::string str;
iostr >str;
std::cout << "test_encoderbuf str = " << str << std::endl;
}
 
 
 

Cheers,

Hans Smit

[James Kanze]

On Dec 12, 9:55 am, "Alf P. Steinbach" <[e-mail] wrote:

* Jim Langston:

[...]

std::stringstre am sstr;
iostr >sstr.rdbuf() ; // EXCEPTION AT PROCESSING CHARACTER '3'

The call to rdbuf() looks suspicious. Since I try to avoid
the unclean iostreams as much as possible I don't know. But
at least it sort of sounds wrong.

It's the usual way of testing new, user-defined streams. It's
true that it's somewhat surprising, since it's a >operator
that doesn't parse anything, but since it's designed mainly for
debugging.

Writing your own streambuf isn't very difficult, but you do have
to respect the contract for the virtual functions you implement.
In the case of underflow(), the standard says very clearly that
unless it returns EOF, it must return "the first character of
the pending sequence, without moving the input sequence position
past it." It's possible to do so without using a buffer, but
that requires also overriding uflow() (whose default
implementation increments the pointer into the buffer), and
pbackfail() (since the implementation has to support at least
one character pushback). This has been discussed in published
articles---I know because I wrote them.

And of course, if you think that iostream is unclean, you're
free to propose something better. The original iostream, by
Jerry Schwarz, is in impressively elegant solution to the
contraints he had at that time. IMHO, the standards committee
didn't improve it, and of course, if we were designing it from
scratch today, we'd do some things differently, but I've not
seen any concrete alternatives that are better.

--
James Kanze (GABI Software) email:ja******* **@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]

[Pete Becker]

On 2007-12-12 12:10:35 -0500, hs********@gmai l.com said:

Method names such as
"showmanyc"
are a little unclear when first working with this library, i.e. I kept
seeing
"show many c" for some strange reason.

There's a footnote in the standard that addresses this: "The morphemes
of showmanyc are es-how-many-see", not "show-manic".

--
Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com) Author of "The
Standard C++ Library Extensions: a Tutorial and Reference
(www.petebecker.com/tr1book)

[hsmit.home]

On Dec 12, 7:22 pm, Pete Becker <p...@versatile coding.comwrote :

On 2007-12-12 12:10:35 -0500, hsmit.h...@gmai l.com said:

Method names such as
"showmanyc"
are a little unclear when first working with this library, i.e. I kept
seeing
"show many c" for some strange reason.

There's a footnote in the standard that addresses this: "The morphemes
of showmanyc are es-how-many-see", not "show-manic".

--
Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com) Author of "The
Standard C++ Library Extensions: a Tutorial and Reference
(www.petebecker.com/tr1book)

That footnote is in quite a few references. The fact that a footnote
had to be issued for this validates my point that some of the naming
conventions in this library are non-intuitive. Nevertheless, the power
and heritage of this library in-validates any comments I have about
it ;-)

[James Kanze]

On Dec 12, 4:33 pm, hsmit.h...@gmai l.com wrote:

Before I read your response I got my code to do what I wanted.
It was along the lines of what you described. It's still a
work in progress. Here is the result
(comments/refinements/improvements of this code are welcome,
in fact they would very much be appreciatted).

Just some quick comments:

[code]

#include <iostream>
#include <sstream>

//--------------------------------------------------------------
/**
General purpose buffered filter / encoder.
@todo: still requires testing and refinement.
*/
class encoderbuf : public std::streambuf {

std::stringbuf mBuf;
char mIBuf[3]; //TODO: make this dynamic

You can use std::vector< char >, and skip the stringbuf
entirely.

public:
//--------------------------------------------------------------
/** default constructor */
encoderbuf() {
mBuf.pubsetbuf( mIBuf, sizeof(mIBuf));

Note that the behavior of stringbuf::setb uf is implementation
defined, and that an implementation is not required to respect
it.

setg(mIBuf, mIBuf, mIBuf+sizeof(mI Buf));

And that this statement basically means that you have two
different class instances (the base class and you mBuf member)
sharing the same buffer, in an unspecified manner.

}

private:
//--------------------------------------------------------------
/** default encoder routine - does nothing */
virtual void encode (int_type c) {
*gptr() = c;
gbump(1);

Uses the buffer defined in the base class. I'd use the
std::vector, and then something like:

mBuffer.push_ba ck( c ) ;

Since you don't seem to be supporting buffering on input,
there's no reason to touch the put pointers. And you definitly
shouldn't be messing with the get pointers on output; at the
very most, you might do something like:
setg( &mBuffer[ 0 ],
&mBuffer[ 0 ] + gptr() - eback(),
&mBuffer[ 0 ] + mBuffer.size() ) ;
(but you could just as well do this in underflow()).

}

//--------------------------------------------------------------
/** flushes the buffered data to the final destination */
int flush () {
int_type c = mBuf.sbumpc();
if (c == EOF) {
return EOF;
}
while (c != EOF) {
encode(c);

Note that encode manipulates mIBuf, which you passed to mBuf.
If your code works, it is probably because mBuf is ignoring the
setbuf.

if (gptr() >= egptr()) {
break;
}
c = mBuf.sbumpc();
}
setg(mIBuf, mIBuf, gptr());

return 0;
}

//--------------------------------------------------------------
/** flush the buffered data to output */
virtual int sync () {
return flush();
}

//--------------------------------------------------------------
/** buffer the incoming data */
virtual int_type overflow (int_type c) {
return mBuf.sputc(c);
}

//--------------------------------------------------------------
/** reset the internal input buffers and flush the buffered
data to output
*/
virtual int_type underflow () {

memset(mIBuf, 'x', sizeof(mIBuf));
setg(mIBuf, mIBuf, mIBuf+sizeof(mI Buf));

if (flush() == EOF) {
return EOF;
}
return ((unsigned char)(*gptr())) ;
}
};

In fact, if I've understood at all, you're really using two
different buffers, mBuf and mIBuf, and transferring between them
in flush. As long as you are calling encode for each individual
character, and not on the complete buffer, I wouldn't bother.
Use an std::vector< char as the buffer: overflow simply calls
encode, which does a push_back on whatever it generates. (I'd
provide a protected function for this, rather than make the
std::vector directly accessible.) Underflow simply updates the
get pointers and returns *gptr() if gptr() != egptr(), e.g.:

int_type
encoderbuf::und erflow()
{
setg( &mBuffer[ 0 ],
&mBuffer[ 0 ] + gptr() - eback(),
&mBuffer[ 0 ] + mBuffer.size() ) ;
return gptr() == egptr()
? traits_type::eo f()
: traits_type::to _int_type( *gptr() ) ;
}

In this particular case, I'd also ask myself if a streambuf is
really the correct solution. On the whole, for memory to memory
translations, I'd rather go with a custom insertion_itera tor,
invoked as the destination of std::copy, e.g.:

std::copy( source.being(), source.end(),
EncodingInserte r( dest ) ) ;

(If the encoding is one to one, of course, all you need is a
function and std::transform. )

An insertion iterator is an output iterator (in other words, not
an iterator at all), which means that many of the constraints
normally present on iterators are absent. Just be aware that it
may be (and in fact will be) copied, so if it needs modifiable
internal state, you have to be careful.

--
James Kanze (GABI Software) email:ja******* **@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]

[hsmit.home]

James, you have given me so much to chew on, I am deeply indebted.

Before I read your last post, I had finally gotten a reasonably
generic encoderbuf class implemented, there are still a few kinks in
it and I will continue working on this today. I will see if I can
integrate a few of your ideas, since these "kinks" I refered to make
me feel like I am doing something terribly wrong (even though it
works). I need to investigate further before I can properly articulate
this additional problem.

Your insertion iterator idea is something I will be looking into
sometime later (more as an excercies in understanding than anything
else). I am an old newbie to the STL library. Worked with it quite a
bit 7-9 years ago, put it aside for some time, and started working
with it again 6 months ago. I am finally discovering the beautiful
intracies of this remarkable library.

I will post my "semi-final" solution when it's ready.