I need help for std::codecvt<>

[Dancefire]

Hi, everyone,

I'm trying to use std::codecvt<to do the encoding conversion. I am
using following code for encoding conversion between wchar_t string
and char string(MBCS). I am not sure am I right. The code works, but
I'm not familiar with the codecvt, and I don't know my way is the
right way to do the job. Could you help me to review the code?

This function try to convert a wide string to a MBCS in the loc's
charset. I hardcode multiple by 2 here which is not correct, but I
don't know how to get the buf length.

================================================== ================
static string to_string(const wstring& str, const locale& loc)
{
typedef codecvt<wchar_t, char, mbstate_tcodecvt_t;
const codecvt_t& cc = use_facet<codecvt_t>(loc);
mbstate_t state = mbstate_t();

int buf_size = static_cast<int>(str.length() * 2);
char* buf = new char[buf_size+1]; // FIXME: it's hardcode : "* 2"

const wchar_t* char_next;
char* byte_next;

int res = cc.out(state,
str.c_str(), str.c_str() + str.length(), char_next,
&buf[0], &buf[buf_size], byte_next);

if (res == codecvt_base::error) {
cerr << "codecvt convert fail. locale=" << loc.name() << endl;
return string();
}

*byte_next = 0;
string result(buf);
delete buf;

return result;
}
================================================== ================

The following code is trying to convert a MBCS string to a wide string
which encoded by loc's charset. I don't know this time I use
cc.length() is right or not.

================================================== ================
static wstring to_wstring(const string& str, const locale& loc)
{
typedef codecvt<wchar_t, char, mbstate_tcodecvt_t;
const codecvt_t& cc = use_facet<codecvt_t>(loc);
mbstate_t state = mbstate_t();

int buf_size = cc.length(state, str.c_str(), str.c_str() +
str.length(), str.length());
wchar_t* buf = new wchar_t[buf_size+1];

wchar_t* char_next;
const char* byte_next;

int res = cc.in(state,
str.c_str(), str.c_str() + str.length(), byte_next,
&buf[0], &buf[buf_size], char_next);

if (res == codecvt_base::error) {
cerr << "codecvt convert fail. locale=" << loc.name() << endl;
return wstring();
}

*char_next = 0;
wstring result(buf);
delete buf;

return result;
}
================================================== ================

Thanks.

[Alf P. Steinbach]

* Dancefire:

Hi, everyone,

I'm trying to use std::codecvt<to do the encoding conversion. I am
using following code for encoding conversion between wchar_t string
and char string(MBCS). I am not sure am I right. The code works, but
I'm not familiar with the codecvt, and I don't know my way is the
right way to do the job. Could you help me to review the code?

This function try to convert a wide string to a MBCS in the loc's
charset. I hardcode multiple by 2 here which is not correct, but I
don't know how to get the buf length.

Essentially you need to loop until the result isn't "partial". That's
what the "char_next" arguments are for -- continuing.

================================================== ================
static string to_string(const wstring& str, const locale& loc)

Declaring a function "static" is a bit old-fashioned.

The modern style is to use an anonymous namespace.

But, oh well, I do it too: these newfangled ways are much too verbose.

{
typedef codecvt<wchar_t, char, mbstate_tcodecvt_t;
const codecvt_t& cc = use_facet<codecvt_t>(loc);
mbstate_t state = mbstate_t();

int buf_size = static_cast<int>(str.length() * 2);
char* buf = new char[buf_size+1]; // FIXME: it's hardcode : "* 2"

An initial assumption of sizeof(wchar_t) is OK, as an estimate. Perhaps
multiply by a little fudge factor. If the first conversion call yields
a partial conversion (check the result) you can use the xxx_next
pointers to improve the estimate and increase the buffer.

For a buffer, use a std::string (use a suitable initial size).

Note: you can safely assume that a std::string's internal buffer (from
data()) is contiguous, and C++0x will require that.

>
const wchar_t* char_next;
char* byte_next;

int res = cc.out(state,
str.c_str(), str.c_str() + str.length(), char_next,
&buf[0], &buf[buf_size], byte_next);

Would possibly be better to use data() than c_str(), although as I
recall most discussion have concluded that data() in practice has no
option other than doing the same as c_str().

If you declare the result variable as codecvt_t::result instead of int a
debugger can help you see what the result value means (symbolic name).

Anyway, make it const.

>
if (res == codecvt_base::error) {
cerr << "codecvt convert fail. locale=" << loc.name() << endl;
return string();
}

Throw an exception instead of doing output or other nasty things.

*byte_next = 0;

No need to add a null-terminator.

string result(buf);

No need for this if you use a string as buffer. For the current buffer
management scheme just do

string result( buf, byte_next );

delete buf;

return result;
}

I've never used this stuff. One reason is that it's not well-defined
what result you actually get. Thus, it's useless in the first place.
Another reason is that it's complicated and suspected of being hugely
inefficient. But the main reason is that it takes (much!) less time to
just code e.g. a conversion to UTF-8 directly, assuming UTF-16 or UTF-32
for wchar_t, or use platform-specific functionality, than to figure this
out and provide reasonable wrappers and interfaces.

--
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?