Internationalization - Unicode - Format used for command line arguments?

[Sune]

Hi!

For example:

1)
I want to open a file in a Chinese locale and print it.

2)
The program takes the file name as a command line argument.

Chinese characters are multibyte but I don't know what particular
encoding is used in the argv vector of main.

Once I get an answer to this I expect that the ICU package can help me
sort out the rest.

Any comments are appreciated.

BRs
/Sune

[Simon Biber]

Sune wrote:

Hi!

For example:

1)
I want to open a file in a Chinese locale and print it.

2)
The program takes the file name as a command line argument.

Chinese characters are multibyte but I don't know what particular
encoding is used in the argv vector of main.

The way locales and filenames work depends on your platform. For
example, it is different on Windows and Linux.

If you are using the standard form of main:
int main(int argc, char **argv)
{
}
then the argv array probably contains strings in the current locale's
encoding. If the system is actually set to a Chinese locale, this is
likely to be one of GBK, GB2312, BIG5 or UTF-8. On Windows, it will be
GBK for Simplified Chinese, or BIG5 for Traditional Chinese. On Linux,
it may be UTF-8 if your system is quite recent, otherwise will probably
be either GB2312 or Big5.

In addition, filenames may be stored in different encodings.

Windows typically uses Unicode (UTF-16) to store filenames, though you
need platform specific functions to actually open a file using a Unicode
filename. If the filename contains characters that are not in the system
locale's encoding, then you cannot use fopen to open the file.

Linux uses multibyte strings as filenames, and you can always use fopen
to open them. These days they are often UTF-8, but may also be in a
legacy encoding such as GB2312 or BIG5.

Getting to the point: there is unfortunately no way to tell what
encoding is being used, from the point of view of portable standard C.
You may have to insert some conditional code for each platform.

--
Simon.

[Sune]

Hi,

thanks for being clear and to the point. My conclusion is that I will
have to rely on ICU (IBM's International Components for Unicode) in the
sense that: locales and character sets that ICU can detect and turn
into Unicode (if necessary) will be the locales I support. Going
further doesn't seem feasible. Is this considered bad style/laziness?
Is there any other way?

You seem to be well informed on the subject, so could you please
provide a link where I can find all locales and their supported
characters? What I want to find out is if the so-called invariants are
part of the character set of all locales, or if I will mess things up
if I try to add a file extension such as '.xml' to a processed file?
E.g. a succeeding fopen will fail (as you pointed out earlier) if the
locale is Chinese.

Thanks again
/Sune

[Simon Biber]

Sune wrote:

Hi,

thanks for being clear and to the point. My conclusion is that I will
have to rely on ICU (IBM's International Components for Unicode) in the
sense that: locales and character sets that ICU can detect and turn
into Unicode (if necessary) will be the locales I support. Going
further doesn't seem feasible. Is this considered bad style/laziness?
Is there any other way?

You seem to be well informed on the subject, so could you please
provide a link where I can find all locales and their supported
characters? What I want to find out is if the so-called invariants are
part of the character set of all locales, or if I will mess things up
if I try to add a file extension such as '.xml' to a processed file?
E.g. a succeeding fopen will fail (as you pointed out earlier) if the
locale is Chinese.

Character encodings can be divided into those which are based around
ASCII, where ASCII text is valid in that encoding, and those which do
not allow plain ASCII text to exist unchanged.

ASCII-based encodings:

ISO-8859-1 has 1 byte per character
UTF-8 has 1, 2, 3, or 4 bytes per character
Big5 has 1 or 2 bytes per character
GB2312 has 1 or 2 bytes per character
GBK has 1 or 2 bytes per character
GB18030 has 1, 2 or 4 bytes per character

Non-ASCII-based encodings:

UTF-16 has 2 or 4 bytes per character
UTF-32 has 4 bytes per character

You can usually use a C string (zero-terminated array of char) to store
any of the ASCII-based encodings. The problem with UTF-16 and UTF-32 is
that they have embedded zero bytes, so you could instead use an array of
uint16_t or uint32_t respectively.

wchar_t is 2 bytes on most C implementations on Windows, and is used to
contain strings in UTF-16. wchar_t is 4 bytes on most C implementations
on Linux, and is used to contain strings in UTF-32.

For one of the ASCII-based encodings, you can append ".xml" to the
string using strcat, and not break the encoding.

For one of the non-ASCII based encodings, you need to convert your
".xml" into the correct Unicode format before appending it to the string.

UTF-16 LE: (uint16_t *)(char[]) {'.',0, 'x',0, 'm',0, 'l',0, 0,0}

UTF-16 BE: (uint16_t *)(char[]) { 0,'.', 0,'x', 0,'m', 0,'l', 0,0}

UTF-32 LE: (uint32_t *)(char[]) { 0,0,0,'.', 0,0,0,'x', 0,0,0,'m',
0,0,0,'l', 0,0,0,0 }

UTF-32 BE: (uint32_t *)(char[]) {'.',0,0,0, 'x',0,0,0, 'm',0,0,0,
'l',0,0,0, 0,0,0,0 }

On most systems, L".xml" is equivalent to one of the four above. Which
one it is equivalent to depends on the choices made by your platform's C
compiler and library implementors.

--
Simon.

[Sune]

Simon,

this made me get the part of the picture I was missing. It also made my
program a lot smaller.

Thanks for your patience
/Sune

[Mark McIntyre]

On 17 Dec 2005 07:31:58 -0800, in comp.lang.c , "Sune"
<su**********@hotmail.com> wrote:

(a contextless answer).

Sune, can you please take note of this:

--
Please quote enough of the previous message for context. To do so from
Google, click "show options" and use the Reply shown in the expanded
header.

----== Posted via Newsfeeds.Com - Unlimited-Unrestricted-Secure Usenet News==----
http://www.newsfeeds.com The #1 Newsgroup Service in the World! 120,000+ Newsgroups
----= East and West-Coast Server Farms - Total Privacy via Encryption =----

[Simon Biber]

Simon Biber wrote:
[...]

For one of the non-ASCII based encodings, you need to convert your
".xml" into the correct Unicode format before appending it to the string.

UTF-16 LE: (uint16_t *)(char[]) {'.',0, 'x',0, 'm',0, 'l',0, 0,0}

UTF-16 BE: (uint16_t *)(char[]) { 0,'.', 0,'x', 0,'m', 0,'l', 0,0}

UTF-32 LE: (uint32_t *)(char[]) { 0,0,0,'.', 0,0,0,'x', 0,0,0,'m',
0,0,0,'l', 0,0,0,0 }

UTF-32 BE: (uint32_t *)(char[]) {'.',0,0,0, 'x',0,0,0, 'm',0,0,0,
'l',0,0,0, 0,0,0,0 }

Replying to myself here.

I got UTF-32 LE and BE mixed around. {0,0,0,'.'} is actually big-endian
UTF-32, and {'.',0,0,0} is little-endian UTF-32.

Also remember that these shortcuts assume that your character constants
are in ASCII (or ISO-8859-1), and will not work on an EBCDIC machine.

--
Simon.

[SM Ryan]

Simon Biber <ne**@ralmin.cc> wrote:

# Also remember that these shortcuts assume that your character constants
# are in ASCII (or ISO-8859-1), and will not work on an EBCDIC machine.

Any poor wretch using an EBCDIC machine has enough problems
without Unicode.

--
SM Ryan http://www.rawbw.com/~wyrmwif/
Death is the worry of the living. The dead, like myself,
only worry about decay and necrophiliacs.