Alexander Adam wrote:
Hi,
I am a bit list in encoding related stuff. Let me explain what I am
doing (yes it's C++ :)):
I am getting some input content due Expat Xml Parser. I've setup Expat
to use wchar_t.
First question is this -- what is the difference of unsigned short,
wchar_t and char?
On my compiler, they all have different sizes....
Okay, wchar_t is an built-in type of C++ and its two bytes of size
I believe that that's the case on Windows. On Linux, wchar_t is 4
bytes. You should not rely on it having any particular size.
whereas char is always one byte.
But what's the real difference when storing Text into those types i.e.
ASCII, UTF-8, UTF-16 or UTF-32 encoded text?
Afaik, UTF-8 is 2 bytes,
No. It's a variable length encoding. Look it up, e.g. on the Unicode
web site. I bet Wikipedia has a good description too.
UTF-16 is 2 bytes
No. It's a variable length encoding. For the vast majority of cases,
it will use two bytes per character, but you shouldn't rely on that.
Look it up.
and UTF-32 is up to four
bytes?
It's always exactly four bytes per character. Look it up.
Well anyway, my issue is how to correctly work with those
types. Internally I am using wchar_t for all my representations but
depending on the encoding I need to shift a current char value
bitwise, right?
Err, I'm not sure what you mean, but no I don't think that's the right
thing to do. What do you mean by "work with" these types? What are you
actually trying to do?
Okay next one -- I am storing everything of my wchar_t array into a
stream of type char,
Why?
doing so by a simple memcpy. Now how could I read
it back in? Say I have char* buffer where my wchar_t string is saved
in. I could surely do a simply memcpy(myWcharVar, buffer,
sizeof(wchar_t)) to get two bytes but this doesn't seem to be very
efficient as I'd like to read it char by char (like wchar_t nx =
buffer.next(), know what I mean?).
Beware that there are endianness issues to worry about here.
Your compiler will possibly optimise a memcpy() into efficient inline code.
But if you kept it in a whcat_t buffer, you wouldn't need to worry about
this.
And then after having read such a char, I must be able to correctly
encode it. I know the encoding whether its ASCII, UTF-8, 16 or
anything but how would I go about it *without* using any big
libraries?
Why the prohibition of libraries? POSIX systems have iconv(), which
will do it all for you. I think Windows has something similar.
If you want to write the code yourself, you should find enough
description in the definitions of the encodings.
I have done some work on strings tagged with their character sets which
I may propose for Boost at some point in the future. You'll find my
first attempt if you look for my name in the Boost list archives from
last September and October. I'm currently revising it, and my first
step has been to define char8_t, char16_t and char32_t. These types are
guaranteed to have exactly the indicated number of bits, and to be char
or wchar_t when that type is the right size. Here's the code:
template <int bits>
struct char_t {
typedef typename boost::uint_t<bits>::least type;
};
template <>
struct char_t<8*sizeof(char){
typedef char type;
};
template <>
struct char_t<8*sizeof(wchar_t){
typedef wchar_t type;
};
typedef char_t<8>::type char8_t;
typedef char_t<16>::type char16_t;
typedef char_t<32>::type char32_t;
I suggest using something like char16_t, rather than wchar_t, as the
basis for a UTF-16 string, for portability. I'm currently not sure how
this can work with string literals, though.
Regards, Phil.
