UTF-8 locales

[saroj.yadav]

As I understand it (correct me, if I am wrong) Unicode came into
picture so that a document containing multiple language characters can
be supported like somebody can write a document comparing Korean and
Chinese in French language.

Now, I am looking at all UNIX platforms and seems like all Unix (AIX,
HP, Solaris) platforms support Unicode by supporting language/region
specific UTF-8 locales like fr_FR.UTF-8, ja_JP.UTF-8, ko_KR.UTF-8 etc.

Now in order to use UTF-8 for Japanese, I have to set locale to
ja_JP.UTF-8. To use UTF-8 for Korean, I have to set locale to
ko_KR.UTF-8.

With this approach it's not possible to mix multiple language
characters. Doesn't this defeat the whole purpose of Unicode ?
Am I missing something ?

Thanks in advance for any insight you can provide.

[Christian Bau]

In article <11************ **********@z14g 2000cwz.googleg roups.com>,
sa*********@gma il.com wrote:

As I understand it (correct me, if I am wrong) Unicode came into
picture so that a document containing multiple language characters can
be supported like somebody can write a document comparing Korean and
Chinese in French language.

Now, I am looking at all UNIX platforms and seems like all Unix (AIX,
HP, Solaris) platforms support Unicode by supporting language/region
specific UTF-8 locales like fr_FR.UTF-8, ja_JP.UTF-8, ko_KR.UTF-8 etc.

Now in order to use UTF-8 for Japanese, I have to set locale to
ja_JP.UTF-8. To use UTF-8 for Korean, I have to set locale to
ko_KR.UTF-8.

With this approach it's not possible to mix multiple language
characters. Doesn't this defeat the whole purpose of Unicode ?
Am I missing something ?

A locale describes much more than just the characters.

There are for example the rules for formatting a number. Threethousand
twohundred and six hundredths is 3,200.06 in one locale and 3.200,06 in
another and 3'200.06 in a third locale.

There are rules how to display monetary values. There are rules how to
sort text. There are rules how to display dates. They all will be
different between locales.

[Spacen Jasset]

sa*********@gma il.com wrote:

As I understand it (correct me, if I am wrong) Unicode came into
picture so that a document containing multiple language characters can
be supported like somebody can write a document comparing Korean and
Chinese in French language.

Now, I am looking at all UNIX platforms and seems like all Unix (AIX,
HP, Solaris) platforms support Unicode by supporting language/region
specific UTF-8 locales like fr_FR.UTF-8, ja_JP.UTF-8, ko_KR.UTF-8 etc.

Now in order to use UTF-8 for Japanese, I have to set locale to
ja_JP.UTF-8. To use UTF-8 for Korean, I have to set locale to
ko_KR.UTF-8.

With this approach it's not possible to mix multiple language
characters. Doesn't this defeat the whole purpose of Unicode ?
Am I missing something ?

Thanks in advance for any insight you can provide.

I am not certain how this works on Unix platforms. However I suspect
that the locale has an effect on local conventions as the previous
poster mentioned. I.e date formats and what not.

You still should be able to display any language no matter what locale
you have set, i.e in ja_JP.UTF-8 you should still be able to display
French an Chinese and so on. Try it. ( You must have fonts capable of
mapping those characters of course, if it doesn't work try changing font
- look on the web for names of the fonts that support the glyphs that
are needed )

[CBFalconer]

sa*********@gma il.com wrote:

As I understand it (correct me, if I am wrong) Unicode came into
picture so that a document containing multiple language characters
can be supported like somebody can write a document comparing
Korean and Chinese in French language.

Now, I am looking at all UNIX platforms and seems like all Unix
(AIX, HP, Solaris) platforms support Unicode by supporting
language/region specific UTF-8 locales like fr_FR.UTF-8,
ja_JP.UTF-8, ko_KR.UTF-8 etc.

.... snip ...

This is off-topic in c.l.c. Try a newsgroup with unix, posix, aix,
solaris, etc. in its name. c.l.c deals with the actual portable
language as defined in the C standard. As soon as you have to
specify a system or compiler, you become off-topic.

--
"I conclude that there are two ways of constructing a software
design: One way is to make it so simple that there are obviously
no deficiencies and the other way is to make it so complicated
that there are no obvious deficiencies." -- C. A. R. Hoare

[Kevin Bracey]

In message <42************ ***@yahoo.com>
CBFalconer <cb********@yah oo.com> wrote:

sa*********@gma il.com wrote:

Now, I am looking at all UNIX platforms and seems like all Unix
(AIX, HP, Solaris) platforms support Unicode by supporting
language/region specific UTF-8 locales like fr_FR.UTF-8,
ja_JP.UTF-8, ko_KR.UTF-8 etc.

... snip ...

This is off-topic in c.l.c. Try a newsgroup with unix, posix, aix,
solaris, etc. in its name. c.l.c deals with the actual portable
language as defined in the C standard. As soon as you have to
specify a system or compiler, you become off-topic.

Which is of course why the whole locale system and wchar_t are effectively
useless for portable programming. Everything about them is
implementation-defined, so you can't even talk about them in comp.*.c :(

Is there anywhere where one could discuss "best practice" or standardisation
for these facilities?

--
Kevin Bracey, Principal Software Engineer
Tematic Ltd Tel: [phone]
182-190 Newmarket Road Fax: [phone]
Cambridge, CB5 8HE, United Kingdom WWW: http://www.tematic.com/

[Michael Mair]

Kevin Bracey wrote:

In message <42************ ***@yahoo.com>
CBFalconer <cb********@yah oo.com> wrote:

sa*********@g mail.com wrote:

Now, I am looking at all UNIX platforms and seems like all Unix
(AIX, HP, Solaris) platforms support Unicode by supporting
language/region specific UTF-8 locales like fr_FR.UTF-8,
ja_JP.UTF-8, ko_KR.UTF-8 etc.

... snip ...

This is off-topic in c.l.c. Try a newsgroup with unix, posix, aix,
solaris, etc. in its name. c.l.c deals with the actual portable
language as defined in the C standard. As soon as you have to
specify a system or compiler, you become off-topic.

Which is of course why the whole locale system and wchar_t are effectively
useless for portable programming. Everything about them is
implementation-defined, so you can't even talk about them in comp.*.c :(

Is there anywhere where one could discuss "best practice" or standardisation
for these facilities?

Ack.
I really would be interested in that, too.
Cheers
Michael
--
E-Mail: Mine is an /at/ gmx /dot/ de address.

[CBFalconer]

Michael Mair wrote:

Kevin Bracey wrote:

CBFalconer <cb********@yah oo.com> wrote:

sa*********@gma il.com wrote:

Now, I am looking at all UNIX platforms and seems like all Unix
(AIX, HP, Solaris) platforms support Unicode by supporting
language/region specific UTF-8 locales like fr_FR.UTF-8,
ja_JP.UTF-8, ko_KR.UTF-8 etc.

... snip ...

This is off-topic in c.l.c. Try a newsgroup with unix, posix, aix,
solaris, etc. in its name. c.l.c deals with the actual portable
language as defined in the C standard. As soon as you have to
specify a system or compiler, you become off-topic.

Which is of course why the whole locale system and wchar_t are
effectively useless for portable programming. Everything about
them is implementation-defined, so you can't even talk about them
in comp.*.c :(

Is there anywhere where one could discuss "best practice" or
standardisation for these facilities?

Ack.
I really would be interested in that, too.

I would consider comp.programmin g quite suitable for this sort of
general internationaliz ation discussion.

--
Chuck F (cb********@yah oo.com) (cb********@wor ldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home .att.net> USE worldnet address!

[websnarf]

Kevin Bracey wrote:

CBFalconer <cb********@yah oo.com> wrote:

sa*********@gma il.com wrote:

Now, I am looking at all UNIX platforms and seems like all Unix
(AIX, HP, Solaris) platforms support Unicode by supporting
language/region specific UTF-8 locales like fr_FR.UTF-8,
ja_JP.UTF-8, ko_KR.UTF-8 etc.
... snip ...

This is off-topic in c.l.c. Try a newsgroup with unix, posix, aix,
solaris, etc. in its name. c.l.c deals with the actual portable
language as defined in the C standard. As soon as you have to
specify a system or compiler, you become off-topic.

Which is of course why the whole locale system and wchar_t are
effectively useless for portable programming. Everything about
them is implementation-defined, so you can't even talk about
them in comp.*.c :(

Yeah, amazing he doesn't see the irony of his own statement. The
wchar_t types and functions are only portable in the very narrow scope
of "I can compile that source code to do something" sense. But since
the definition of wchar_t is so open ended, system dependent, and in
fact *NOT* universal in any real sense, there's no way you can actually
use it for any truly portable applications.

And there's no way to somehow "augment" the specification, or use these
types/function in some special way to *make* them portable. There are
no ifdef tricks or anything like that -- you're just screwed. The
whole wchar_t idea literally has no relevant functional content.

C's whole misconception of what it means to be portable, is in fact
what *prevents* it from being portable where it counts, on issues like
localization.
Is there anywhere where one could discuss "best practice" or
standardisation for these facilities?

Well more importantly, portability in general, really can't be
discussed in comp.lang.c. I lurk on comp.programmin g, so you can
always throw questions about portability there.

---
Paul Hsieh
http://www.pobox.com/~qed/
http://bstring.sf.net/