473,902 Members | 5,194 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

UTF-8 in char*

Hi,

I am developing a vCard application which have to support UTF-8. Does the
UTF-8 in char* will crash the strlen, I mean does UTF-8 have some char which
treat as NULL character in strlen?

Jacky
Nov 14 '05
20 15506
Christian Bau <ch***********@ cbau.freeserve. co.uk> wrote in message news:<ch******* *************** ***********@slb-newsm1.svr.pol. co.uk>...
In article <br**********@n ews.tue.nl>, grobbeltje <gr*****@hotmai l.com>
wrote:
Chris Torek <no****@torek.n et> wrote:
In fact, strlen() simply operates on an array
of (plain, i.e., optionally-signed at the compiler's discretion)
char,

Do you mean the compiler can basically do what it wants concerning
signing of chars? Where can I find more info on this?
I've been trying to read some iso documentation from 1999,
and how 'plain' char works is a bit hard to understand for me.

It says: "If the value of an object of type char is treated as a signed
integer when used in an expression, the value of CHAR_MIN shall be the same
as that of SCHAR_MIN and the value of CHAR_MAX shall be the same as that of
SCHAR_MAX. Otherwise, the value of CHAR_MIN shall be 0 and the value of
CHAR_MAX shall be the same as that of UCHAR_MAX." (sorry for any typo's,
they are mine).

As I read it, this would mean 'char' is always 'unsigned char' when compared
to other chars. The same document says the null character is defined as a
byte with all bits set to 0. So to my understanding a simple strlen consisting
of a for/while loop searching for a '\0' should operate on unsigned chars.


The compiler has to make a choice between two possibilities: Either
plain "char" behaves exactly the same way as "unsigned char", or plain
"char" behaves exactly the same as "signed char". The compiler must make
its decision and then stick with it.


Signed...Unsign ed, it doesn't really matter. Most string functions
don't care about the "real" numerical value of something, just whether
it equates to something else or not. If you're going to start testing
the range of a character, like if (c < 'a') then you have to ensure
the range doesn't cross a sign change. Just ensure all lvalues are
either signed or unsigned, casting them if necessary.

Of course the UFT-8 aspect complicates things. Strings could have the
same number of characters encoded in them but be different byte
lengths, so characters need to be decoded into a larger data type
before comparing. However, this has already been noted in this
thread.

---
Jared Dykstra
http://www.bork.org/~jared
Nov 14 '05 #11
[on plain "char" in general, and strlen() specifically]
grobbeltje wrote:
Do you mean the compiler can basically do what it wants concerning
sign[edness] of chars?

(The answer is "sort of", as Kevin Goodsell explained in text I snipped.)

Now suppose we have a UTF-8 encoding in an array "a" defined
as:

unsigned char a[UTF8_MAX_LEN];

where a[len] == '\0' for some len in [0..UTF8_MAX_LEN ).

In article <news:%x******* **********@news read2.news.pas. earthlink.net>
Kevin Goodsell <us************ *********@never box.com> writes, in part:... To the best of my knowledge,
strlen should operate correctly on any character type.


The concern here is for plain char that is the same as "signed
char" on a machine where CHAR_MIN is -127 rather than -128. The
UTF-8 encoding potentially uses all 256 possible values in the
range [0..255], which will certainly fit into a C array of "unsigned
char", because UCHAR_MAX must be at least 255. But what if UCHAR_MAX
is indeed 255, and some array element a[i] (where i < len) is set
to either 255 (for ones' complement systems) or 128 (for
sign-and-magnitude)? Then a[i], interpreted as a plain (and thus
signed) char, will be what C99 calls an "object representation" of
the value negative zero (which exists in ones' complement and
signed-magnitude, but not in the much more common two's complement
that most C systems use). This may be a "trap representation" ,
and it may be the case that strlen(a) traps, instead of returning
some number.

On any ordinary two's complement system today, you will find that
CHAR_MIN is either -128 (char being signed) or 255 (char being
unsigned), so that strlen(a) will indeed find that a[len]=='\0'
byte even if a[i]==128 or a[i]==255. And if CHAR_MIN exceeds
255 (e.g., on DSP C compilers with CHAR_BIT being 16 or more),
then all a[i] values, for i in [0..len), are valid "char" values
as well. Thus, strlen(a) will "work right" on all these systems.
But, this being comp.lang.c, we must worry about systems that
do not have these properties, too. :-)
--
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) +1 801 277 2603
email: forget about it http://web.torek.net/torek/index.html
Reading email is like searching for food in the garbage, thanks to spammers.
Nov 14 '05 #12
Chris Torek <no****@torek.n et> writes:
On any ordinary two's complement system today, you will find that
CHAR_MIN is either -128 (char being signed) or 255 (char being
unsigned), so that strlen(a) will indeed find that a[len]=='\0'
byte even if a[i]==128 or a[i]==255. And if CHAR_MIN exceeds
255 (e.g., on DSP C compilers with CHAR_BIT being 16 or more),


Are you getting a little sleepy this late at night, Chris? There
is no possible way that CHAR_MIN can be 255, and certainly no way
that CHAR_MIN can exceed 255.
--
"I hope, some day, to learn to read.
It seems to be even harder than writing."
--Richard Heathfield
Nov 14 '05 #13
>Chris Torek <no****@torek.n et> writes:
On any ordinary two's complement system today, you will find that
CHAR_MIN is either -128 (char being signed) or 255 (char being
unsigned), so that strlen(a) will indeed find that a[len]=='\0'
byte even if a[i]==128 or a[i]==255. And if CHAR_MIN exceeds
255 (e.g., on DSP C compilers with CHAR_BIT being 16 or more),

In article <news:87******* *****@pfaff.sta nford.edu>,
Ben Pfaff <bl*@cs.stanfor d.edu> wrote:Are you getting a little sleepy this late at night, Chris? There
is no possible way that CHAR_MIN can be 255, and certainly no way
that CHAR_MIN can exceed 255.


Er, right. CHAR_M{IN,AX} -- just two little letters... :-)

Make that:

On ordinary two's complement systems today, you will find that
either CHAR_MIN is -128 (char being signed), or CHAR_MAX is 255
(char being unsigned), so that strlen(a) will indeed find the
a[len]=='\0' byte even if a[i]==128 (-128 if signed) or 255
(-1 if signed). And if CHAR_MAX exceeds 255 ...
--
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) +1 801 277 2603
email: forget about it http://web.torek.net/torek/index.html
Reading email is like searching for food in the garbage, thanks to spammers.
Nov 14 '05 #14
"Chris" <ch*******@nosp am.hotmail.com> wrote in message news:<br******* ***@news8.svr.p ol.co.uk>...
"Jacky Cheung" <ja*****@yahoo. com> wrote in message
news:br******** ***@imsp212.net vigator.com...

I am developing a vCard application which have to support UTF-8. Does the
UTF-8 in char* will crash the strlen, I mean does UTF-8 have some char
which treat as NULL character in strlen?


The answer to your 2nd question is that NUL character is NOT EQUIVALENT to
NULL!!!
There is no such thing as "NULL character" but there exists an "NUL
character", which is the '\0' at the end of a string buffer.


Just for pedantry's sake: ASCII gives the character with value 0
the name "NUL" and the description "the null character".
Nov 14 '05 #15
in comp.lang.c i read:
The answer to your 2nd question is that NUL character is NOT EQUIVALENT to
NULL!!!
There is no such thing as "NULL character" but there exists an "NUL
character", which is the '\0' at the end of a string buffer.


NUL is used exactly once in a footnote -- footnotes are not normative.

the standard defines the term `null character' (below). it doesn't present
the word null in all upper-case, so there certainly is room for confusion
with the macro, and it's good that this be clarified, i.e., the intention
of this correction is well meant but it is slightly flawed. there is a
null character, which will compare equal with NULL, but they are not the
same, in that NULL may be of type void* rather than int.

| A byte with all bits set to 0, called the null character, shall exist in
| the basic execution character set; it is used to terminate a character
| string.

--
a signature
Nov 14 '05 #16
In article <ba************ **************@ posting.google. com>,
dy******@hotmai l.com (Jared Dykstra) wrote:
Of course the UFT-8 aspect complicates things. Strings could have the
same number of characters encoded in them but be different byte
lengths, so characters need to be decoded into a larger data type
before comparing. However, this has already been noted in this
thread.


The beauty of UTF-8 is that there is no need to do this.

If two sequences of characters A and B are encoded in UTF-8, then the
_bytes_ of the encoding of A will match a subsequence of bytes in the
encoding of B if and only if the characters of A match a subsequence of
the characters in B.
Nov 14 '05 #17

"J. J. Farrell" <jj*@bcs.org.uk > wrote in message
news:5c******** *************** ***@posting.goo gle.com...
"Chris" <ch*******@nosp am.hotmail.com> wrote in message

news:<br******* ***@news8.svr.p ol.co.uk>...
"Jacky Cheung" <ja*****@yahoo. com> wrote in message
news:br******** ***@imsp212.net vigator.com...

I am developing a vCard application which have to support UTF-8. Does the UTF-8 in char* will crash the strlen, I mean does UTF-8 have some char
which treat as NULL character in strlen?


The answer to your 2nd question is that NUL character is NOT EQUIVALENT to NULL!!!
There is no such thing as "NULL character" but there exists an "NUL
character", which is the '\0' at the end of a string buffer.


Just for pedantry's sake: ASCII gives the character with value 0
the name "NUL" and the description "the null character".


Well, if I'm going to be pedantic, what has ASCII got to do with
anything? What's relevant is that C defines the "null character"
as a character with value 0. "NUL" is the name of that character
in both the ASCII and EBCDIC character sets (and those based on
them, and perhaps a few others as well) but they aren't relevant
to C. Chris's correction would have been better as

! There is no such thing as "NULL character" but there exists a
! "null character", which is the '\0' at the end of a string buffer.
Nov 14 '05 #18
Christian Bau <ch***********@ cbau.freeserve. co.uk> wrote in message news:<ch******* *************** ***********@slb-newsm1.svr.pol. co.uk>...
In article <ba************ **************@ posting.google. com>,
dy******@hotmai l.com (Jared Dykstra) wrote:
Of course the UFT-8 aspect complicates things. Strings could have the
same number of characters encoded in them but be different byte
lengths, so characters need to be decoded into a larger data type
before comparing. However, this has already been noted in this
thread.


The beauty of UTF-8 is that there is no need to do this.

If two sequences of characters A and B are encoded in UTF-8, then the
_bytes_ of the encoding of A will match a subsequence of bytes in the
encoding of B if and only if the characters of A match a subsequence of
the characters in B.


True. An encoding scheme is less useful if it doesn't encode the same
data the same way twice.

The original post wasn't clear if all data was encoded this way or
just some of it. Obviousally if the two strings are encoded
differently, conversion to a common encoding scheme is required for
any useful byte-wise comparison. If not, compare bytes.

----
Jared Dykstra
http://www.bork.org/~jared
Nov 14 '05 #19
In article <ba************ **************@ posting.google. com>,
dy******@hotmai l.com (Jared Dykstra) wrote:
Christian Bau <ch***********@ cbau.freeserve. co.uk> wrote in message
news:<ch******* *************** ***********@slb-newsm1.svr.pol. co.uk>...
In article <ba************ **************@ posting.google. com>,
dy******@hotmai l.com (Jared Dykstra) wrote:
Of course the UFT-8 aspect complicates things. Strings could have the
same number of characters encoded in them but be different byte
lengths, so characters need to be decoded into a larger data type
before comparing. However, this has already been noted in this
thread.
The beauty of UTF-8 is that there is no need to do this.

If two sequences of characters A and B are encoded in UTF-8, then the
_bytes_ of the encoding of A will match a subsequence of bytes in the
encoding of B if and only if the characters of A match a subsequence of
the characters in B.


True. An encoding scheme is less useful if it doesn't encode the same
data the same way twice.


That is not what I said. UTF8 is better than that. Consider the case
where x, y, and a are single characters that are all three encoded to
two bytes each. It would be possible that the last byte of x matches the
first byte of a, and the first byte of y matches the second byte of a,
so the encoding of xy has the encoding of a as a substring. This is not
the case with UTF8.
The original post wasn't clear if all data was encoded this way or
just some of it. Obviousally if the two strings are encoded
differently, conversion to a common encoding scheme is required for
any useful byte-wise comparison. If not, compare bytes.

Nov 14 '05 #20

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

9
4200
by: lawrence | last post by:
Someone on www.php.net suggested using a seems_utf8() method to test text for UTF-8 character encoding but didn't specify how to write such a method. Can anyone suggest a test that might work? Something that maybe gives 90% confidence that a given block of text is or is not UTF-8 encoded?
4
6428
by: Alban Hertroys | last post by:
Another python/psycopg question, for which the solution is probably quite simple; I just don't know where to look. I have a query that inserts data originating from an utf-8 encoded XML file. And guess what, it contains utf-8 encoded characters... Now my problem is that psycopg will only accept queries of type str, so how do I get my utf-8 encoded data into the DB? I can't do query.encode('ascii'), that would be similar to: >>> x =...
12
8244
by: Mike Dee | last post by:
A very very basic UTF-8 question that's driving me nuts: If I have this in the beginning of my Python script in Linux: #!/usr/bin/env python # -*- coding: UTF-8 -*- should I - or should I not - be able to use non-ASCII characters in strings and in Tk GUI button labels and GUI window titles and in raw_input data without Python returning wrong case in manipulated
38
5766
by: Haines Brown | last post by:
I'm having trouble finding the character entity for the French abbreviation for "number" (capital N followed by a small supercript o, period). My references are not listing it. Where would I find an answer to this question (don't find it in the W3C_char_entities document). -- Haines Brown brownh@hartford-hwp.com
6
18781
by: jmgonet | last post by:
Hello everybody, I'm having troubles loading a Xml string encoded in UTF-8. If I try this code: ------------------------------ XmlDocument doc=new XmlDocument(); String s="<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"yes\"?><a>Schönbühl</a>"; doc.LoadXml(s); doc.Save("d:\\temp\\test.xml");
6
13898
by: archana | last post by:
Hi all, can someone tell me difference between unicode and utf 8 or utf 18 and which one is supporting more character set. whic i should use to support character ucs-2. I want to use ucs-2 character in streamreader and streamwriter. How unicode and utf chacters are stored.
7
12172
by: Jimmy Shaw | last post by:
Hi everybody, Is there any SIMPLE way to convert from UTF-16 to UTF-32? I may be mixed up, but is it possible that all UTF-16 "code points" that are 16 bits long appear just the same in UTF-32, but with zero padding and hence no real conversion is necessary? If I am completely wrong and some intricate conversion operation needs to take place, can anyone give me some primer on the subject?
1
7303
by: sheldon.regular | last post by:
I am new to unicode so please bear with my stupidity. I am doing the following in a Python IDE called Wing with Python 23. äöü äöü '\xc3\xa4\xc3\xb6\xc3\xbc' u'\xe4\xf6\xfc' u'\xe4\xf6\xfc' äöü
4
2358
by: shreshth.luthra | last post by:
Hi All, I am having a GUI which accepts a Unicode string and searches a given set of xml files for that string. Now, i have 2 XML files both of them saved in UTF-8 format, having characters of different language. Although both of them are having UTF-8 as BoM, but only first file is having UTF-8 defined in XML declration at the top of the XML file as
23
5051
by: Allan Ebdrup | last post by:
I hava an ajax web application where i hvae problems with UTF-8 encoding oc chineese chars. My Ajax webapplication runs in a HTML page that is UTF-8 Encoded. I copy and paste some chineese chars from another HTML page viewed in IE7, that is also UTF-8 encoded (search for "china" on google.com). I paste the chineese chars into a content editable div. My Ajax webservice compiles an XML where the data from the content editable div is...
0
9997
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9845
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10872
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10981
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9675
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
7205
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
6085
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
4307
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
3323
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.