UTF-8 in char*

[Jacky Cheung]

Hi,

I am developing a vCard application which have to support UTF-8. Does the
UTF-8 in char* will crash the strlen, I mean does UTF-8 have some char which
treat as NULL character in strlen?

Jacky

[Joona I Palaste]

Jacky Cheung <ja*****@yahoo. com> scribbled the following:

Hi, I am developing a vCard application which have to support UTF-8. Does the
UTF-8 in char* will crash the strlen, I mean does UTF-8 have some char which
treat as NULL character in strlen?

AFAIK UTF-8 does not have NUL characters (most people prefer to spell it
that way to avoid confusion). UTF-8 only includes "normal" ASCII
characters and special characters with bit 7 set. You're in no more
danger of seeing NUL in UTF-8 than you are of seeing it in ASCII.
Note that vCards, by themselves, are completely off-topic here.

--
/-- Joona Palaste (pa*****@cc.hel sinki.fi) ------------- Finland --------\
\-- http://www.helsinki.fi/~palaste --------------------- rules! --------/
"He said: 'I'm not Elvis'. Who else but Elvis could have said that?"
- ALF

[Hallvard B Furuseth]

Jacky Cheung wrote:

I am developing a vCard application which have to support UTF-8. Does
the UTF-8 in char* will crash the strlen, I mean does UTF-8 have some
char which treat as NULL character in strlen?

Well, it has a null control character, but it means more or less the
same as the ASCII null character. So if you just want to handle normal
text, you can use normal C strings, and thus strlen().
BTW, if you have only written programs for ASCII before, you might note
that functions like getchar() return character values in the range of
'unsigned char' or EOF, while 'char' can be negative. So code like
char buf[] = "<UTF-8 string>";
int ch, i;
...
while ((ch = getchar()) != EOF) {
if (ch == buf[i]) ...
is wrong. (Even if you don't use UTF-8, but you may not have noticed
before.) You need to convert ch to char or buf[j] to unsigned char
before comparing the two.

--
Hallvard

[Chris Torek]

In article <news:br******* ****@imsp212.ne tvigator.com>
Jacky Cheung <ja*****@yahoo. com> writes:

I am developing a vCard application which have to support UTF-8. Does the
UTF-8 in char* will crash the strlen, I mean does UTF-8 have some char which
treat as NULL character in strlen?

UTF-8 is simply an encoding mechanism for taking larger-than-8-bit
values and storing them in 8-bit values. The details of this
mechanism are pretty much off-topic in comp.lang.c, but here we
can say that UTF-8 encoded characters will always fit in objects
of type "unsigned char", as those will have at least 8 bits.

Your actual question above cannot (quite) be answered as asked as
it appears to contain at least one false assumption, i.e., that
the presence of a '\0' character in an array of unsigned char will
"crash" strlen(). In fact, strlen() simply operates on an array
of (plain, i.e., optionally-signed at the compiler's discretion)
char, searching forward until it finds a '\0' value, then returning
the number of non-'\0'-"char"s it has skipped. Passing strlen()
the address of an array of "char" that does *not* contain a '\0'
could cause the program to crash (or indeed exhibit any behavior
at all); so I think what you really mean to ask is:

"Given some sequence of values in some wider-than-8-bit
character set (such as 16 or 32 bit Unicode), suppose I have
encoded it in 8-bit bytes using the UTF-8 scheme. Can I
(usefully) apply strlen() to the result?"

The answer to this version of the question is "maybe". In particular,
you must ensure that:

a) none of the 8-bit values is a trap representation in plain
"char" if plain "char" is signed (and the C language proper is
not terribly helpful here, but you could constrain yourself to
two's complement systems or those with wide-enough "plain" chars,
by checking that either CHAR_MAX >= 255 -- i.e., no UTF-8 value
will be negative -- or that -CHAR_MIN <= -128);

b) that the "char" array you have used to stored the encoded
values is '\0'-terminated;

c) that you did not embed any '\0' values in that array, and

d) that the resulting strlen() value meets any other criteria
you may hide beneath the word "useful".

The conditions in part (a) are met by most C systems today, so you
might simply assume them (and document that assumption somewhere).
The conditions in part (b) and (c) may, or may not, arise naturally
out of the values you are UTF-8 encoding -- this part is up to you.
Part (d) is likewise something only you can answer.
--
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) [phone]
email: forget about it http://web.torek.net/torek/index.html
Reading email is like searching for food in the garbage, thanks to spammers.

[Chris]

I remember when I did UCS2 for similar vCard application, I used the
following structure:

typedef struct ucs2_tag {
unsigned short* str_ptr;
unsigned int length;
} ucs2, *ucs2_ptr;

By doing it like this, it'll be obvious to people who need to work on your
code what you're doing and stick to memcpy() for string copying then you dun
need to worry about the NUL terminator or not, this is achieved at a small
cost of managing the memory usage for each structure that you create.

Also, you might need to write your own myStrlen() to count the number of
characters of an input string since it's length can be unpredictable.

So in your case you can declare a structure for UTF-8 as:

typedef struct utf8_tag {
unsigned char* str_ptr;
unsigned int length;
} utf8, *utf8_ptr;

The answer to your 2nd question is that NUL character is NOT EQUIVALENT to
NULL!!!
There is no such thing as "NULL character" but there exists an "NUL
character", which is the '\0' at the end of a string buffer.

Just out of curiosity, are you a mobile phone software developer?

"Jacky Cheung" <ja*****@yahoo. com> wrote in message
news:br******** ***@imsp212.net vigator.com...

Hi,

I am developing a vCard application which have to support UTF-8. Does the
UTF-8 in char* will crash the strlen, I mean does UTF-8 have some char which treat as NULL character in strlen?

Jacky

[Simon Biber]

"Chris" <ch*******@nosp am.hotmail.com> wrote:

I remember when I did UCS2 for similar vCard application, I used the
following structure:

typedef struct ucs2_tag {
unsigned short* str_ptr;
unsigned int length;
} ucs2, *ucs2_ptr;
That looks good. I would use a value of type size_t to store the
length, and as a style point I wouldn't provide a pointer typedef.
Users can declare a
ucs2 *my_ptr;
if they like.
By doing it like this, it'll be obvious to people who need to work on
your code what you're doing and stick to memcpy() for string copying
then you dun need to worry about the NUL terminator or not, this is
achieved at a small cost of managing the memory usage for each
structure that you create.
True, and useful in the case of UCS2. Or you could just use C's
wide strings if they are implemented in UCS2 on your platform.
Also, you might need to write your own myStrlen() to count the
number of characters of an input string since it's length can
be unpredictable.
That would be more a file format issue; in the case of a UTF-8
encoded text file, there should not be any embedded zero bytes
and usual I/O or string functions like fgets and strlen should
work fine.
So in your case you can declare a structure for UTF-8 as:
typedef struct utf8_tag {
unsigned char* str_ptr;
unsigned int length;
} utf8, *utf8_ptr;

This is typically not needed for UTF-8. UTF-8 has the important
property that any code value from 0 to 127 inclusive codes for
the respective ASCII character and cannot occur as part of the
multi-byte representation for a higher character. Therefore, any
zero byte occuring in the UTF-8 string is indeed a real ASCII NUL
character and therefore can be used transparently with the usual
C semantics of string termination.

--
Simon.

[grobbeltje]

Chris Torek <no****@torek.n et> wrote:

In fact, strlen() simply operates on an array
of (plain, i.e., optionally-signed at the compiler's discretion)
char,

Do you mean the compiler can basically do what it wants concerning
signing of chars? Where can I find more info on this?
I've been trying to read some iso documentation from 1999,
and how 'plain' char works is a bit hard to understand for me.

It says: "If the value of an object of type char is treated as a signed
integer when used in an expression, the value of CHAR_MIN shall be the same
as that of SCHAR_MIN and the value of CHAR_MAX shall be the same as that of
SCHAR_MAX. Otherwise, the value of CHAR_MIN shall be 0 and the value of
CHAR_MAX shall be the same as that of UCHAR_MAX." (sorry for any typo's,
they are mine).

As I read it, this would mean 'char' is always 'unsigned char' when compared
to other chars. The same document says the null character is defined as a
byte with all bits set to 0. So to my understanding a simple strlen consisting
of a for/while loop searching for a '\0' should operate on unsigned chars.

Am I wrong (again)? Does it really matter whether the chars in
this comparison are signed or not?

ps: Sorry for my bad english.
Grobbeltje (just another curious newbee).
--
Next time you think you're perfect, try walking on water.

[Kevin Goodsell]

grobbeltje wrote:

Do you mean the compiler can basically do what it wants concerning
signing of chars? Where can I find more info on this?
I've been trying to read some iso documentation from 1999,
and how 'plain' char works is a bit hard to understand for me.

It says: "If the value of an object of type char is treated as a signed
integer when used in an expression, the value of CHAR_MIN shall be the same
as that of SCHAR_MIN and the value of CHAR_MAX shall be the same as that of
SCHAR_MAX. Otherwise, the value of CHAR_MIN shall be 0 and the value of
CHAR_MAX shall be the same as that of UCHAR_MAX." (sorry for any typo's,
they are mine).

You're reading in the wrong place. The behavior of char is described
earlier, in section 6.2.5/15:

The three types char, signed char, and unsigned char are
collectively called the character types. The implementation
shall define char to have the same range, representation,
and behavior as either signed char or unsigned char.

So char behaves exactly like either signed char or unsigned char, and
the implementation must decide (and document) which. The three types are
distinct, however.
As I read it, this would mean 'char' is always 'unsigned char' when compared
to other chars.
This interpretation is wrong. I'm unsure of how you got that from the
passage you quoted.
The same document says the null character is defined as a
byte with all bits set to 0. So to my understanding a simple strlen consisting
of a for/while loop searching for a '\0' should operate on unsigned chars.
I'm not totally sure what you mean here. To the best of my knowledge,
strlen should operate correctly on any character type.

Am I wrong (again)? Does it really matter whether the chars in
this comparison are signed or not?

The 'signed-ness' of the objects involved in a comparison is certainly
important, but I'm still not sure what you are getting at.

-Kevin
--
My email address is valid, but changes periodically.
To contact me please use the address from a recent posting.

[David Resnick]

"Jacky Cheung" <ja*****@yahoo. com> wrote in message news:<br******* ****@imsp212.ne tvigator.com>.. .

Hi,

I am developing a vCard application which have to support UTF-8. Does the
UTF-8 in char* will crash the strlen, I mean does UTF-8 have some char which
treat as NULL character in strlen?

Jacky

There are no embedded NUL characters in a UTF-8 encoded string,
that is one of its primary virtues. However, you need to note
that the strlen of a UTF-8 string is greater than (unless that
string is all ASCII) the number of characters represented by
that string...

-David

[Christian Bau]

In article <br**********@n ews.tue.nl>, grobbeltje <gr*****@hotmai l.com>
wrote:

Chris Torek <no****@torek.n et> wrote:

In fact, strlen() simply operates on an array
of (plain, i.e., optionally-signed at the compiler's discretion)
char,

Do you mean the compiler can basically do what it wants concerning
signing of chars? Where can I find more info on this?
I've been trying to read some iso documentation from 1999,
and how 'plain' char works is a bit hard to understand for me.

It says: "If the value of an object of type char is treated as a signed
integer when used in an expression, the value of CHAR_MIN shall be the same
as that of SCHAR_MIN and the value of CHAR_MAX shall be the same as that of
SCHAR_MAX. Otherwise, the value of CHAR_MIN shall be 0 and the value of
CHAR_MAX shall be the same as that of UCHAR_MAX." (sorry for any typo's,
they are mine).

As I read it, this would mean 'char' is always 'unsigned char' when compared
to other chars. The same document says the null character is defined as a
byte with all bits set to 0. So to my understanding a simple strlen consisting
of a for/while loop searching for a '\0' should operate on unsigned chars.

The compiler has to make a choice between two possibilities: Either
plain "char" behaves exactly the same way as "unsigned char", or plain
"char" behaves exactly the same as "signed char". The compiler must make
its decision and then stick with it.