473,324 Members | 2,196 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,324 software developers and data experts.

ansi c compiler character encoding

Hi!

Is it determined that the C standard compiler always encode characters
with the same character excoding? If for example the functions Foo and
Bar are compiled by different compilers, is it unambiguous how to
interpret the character string in Bar?

Does string.h expect a specific string format?

void Foo(void)
{
char myTextString[11] = "stuvxyzåäö";
Bar(myTextString);
}

void Bar(char* inp)
{
What character set to expect?
}
Aug 18 '08 #1
12 3869
Andreas Lundgren wrote:
Hi!

Is it determined that the C standard compiler always encode characters
with the same character excoding?
No.
Aug 18 '08 #2
Andreas Lundgren <d9****@efd.lth.sewrites:
Is it determined that the C standard compiler always encode characters
with the same character excoding? If for example the functions Foo and
Bar are compiled by different compilers, is it unambiguous how to
interpret the character string in Bar?

Does string.h expect a specific string format?

void Foo(void)
{
char myTextString[11] = "stuvxyzåäö";
Bar(myTextString);
}

void Bar(char* inp)
{
What character set to expect?
}
No.

But if the two compilers are being used on the same system, it's very
likely that they'll use the same encoding. Since you're calling one
function from the other, presumably you're using the compilers on the
same system and linking the resulting code into a single executable or
equivalent.

Typically a given operating system will impose representations for
certain things. Though this is outside the scope of the C standard,
it's in the best interest of compiler writers to make their generate
code work and play well with that of other compilers. (For example, a
C compiler for Linux that generates code that's incompatible with code
generated by gcc wouldn't be very useful.)

This goes far beyond character set issues and includes things like
integer and floating-point type representations and function calling
conventions.

Your later followup suggests that you're concerned about some
real-world situation, presumably on some specific system. You should
ask in a newsgroup that deals with that system.

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
Nokia
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"
Aug 18 '08 #3
Keith Thompson <ks***@mib.orgwrote:
Andreas Lundgren <d9****@efd.lth.sewrites:
Is it determined that the C standard compiler always encode characters
with the same character excoding? If for example the functions Foo and
Bar are compiled by different compilers, is it unambiguous how to
interpret the character string in Bar?

Does string.h expect a specific string format?

void Foo(void)
{
char myTextString[11] = "stuvxyzåäö";
Bar(myTextString);
}

void Bar(char* inp)
{
What character set to expect?
}
No.
But if the two compilers are being used on the same system, it's very
likely that they'll use the same encoding. Since you're calling one
function from the other, presumably you're using the compilers on the
same system and linking the resulting code into a single executable or
equivalent.
Is it actually a question about the compiler at all? As far as
I can see the compiler will happily create a string literal with
whatever there is in the string, not caring a bit about the en-
coding of the string. I guess the problem is much more one of
how the source files are generated and the expectations of the
output medium.

Consider the case of using one editor for the first file, set
to output files in e.g. one of the different (and incompatible)
russian extended ASCII code pages, and the second file genera-
ted with another editor, set to output in a different encoding.
Even if you use the same compiler this should lead to trouble.
And if then the terminal that receives the output of the pro-
gram is set to a third encoding it becomes a complete mess;-)

Regards, Jens
--
\ Jens Thoms Toerring ___ jt@toerring.de
\__________________________ http://toerring.de
Aug 18 '08 #4
Jens Thoms Toerring wrote:
Keith Thompson <ks***@mib.orgwrote:
>[...]
But if the two compilers are being used on the same system, it's very
likely that they'll use the same encoding. Since you're calling one
function from the other, presumably you're using the compilers on the
same system and linking the resulting code into a single executable or
equivalent.

Is it actually a question about the compiler at all? As far as
I can see the compiler will happily create a string literal with
whatever there is in the string, not caring a bit about the en-
coding of the string. I guess the problem is much more one of
how the source files are generated and the expectations of the
output medium.
A crucial point here is that the encoding of characters in the
C source files need have nothing to do with the encoding of
characters in the execution environment. The compiler generates
execution-encoded strings from source-encoded string literals, and
the transformation is not necessarily the identity mapping. For
example, consider a compiler that reads ASCII-encoded source and
produces a program for an EBCDIC environment: An X in a source
literal would go into the compiler as the value 88, but produce a
character with the value 231 in the executed program.

The fact that the source-to-execution mapping might not be
a simple copy is surprising, but it really shouldn't be. There are
plenty of other non-copy steps in the manufacture of an execution
string from a source literal: Escapes (hex, octal, and symbolic)
are translated, adjacent literals are spliced, the quotation marks
vanish, a trailing zero appears out of thin air -- in light of all
the other things that happen to a source character on its way into
the executable program, why should we imagine that the encoding of
an 'X' would be immune to change?

--
Er*********@sun.com
Aug 18 '08 #5
On Aug 18, 7:48 am, Andreas Lundgren <d99...@efd.lth.sewrote:
Hi!

Is it determined that the C standard compiler always encode characters
with the same character excoding? If for example the functions Foo and
Bar are compiled by different compilers, is it unambiguous how to
interpret the character string in Bar?
No, it does not depends on the compiler...
>
Does string.h expect a specific string format?

void Foo(void)
{
char myTextString[11] = "stuvxyzåäö";
Here, instead of char, try with wchar_t and
related functions if you are using unicode
for your messages and your .c files
Bar(myTextString);

}

void Bar(char* inp)
{
What character set to expect?
Thats depends on the user environment, but if the
user environments is using unicode, you can expect no
more than an array of bytes, other case is with
wchar_t and related functions...
>
}
Regards,
DMW
Aug 18 '08 #6
Daniel Molina Wegener wrote:
On Aug 18, 7:48 am, Andreas Lundgren <d99...@efd.lth.sewrote:
Hi!

Is it determined that the C standard compiler always encode characters
with the same character excoding? If for example the functions Foo and
Bar are compiled by different compilers, is it unambiguous how to
interpret the character string in Bar?

No, it does not depends on the compiler...

Does string.h expect a specific string format?

void Foo(void)
{
char myTextString[11] = "stuvxyz���";

Here, instead of char, try with wchar_t and
related functions if you are using unicode
for your messages and your .c files
Whether or not wchar_t has anything to do with unicode depends upon
the compiler; the standard makes no such requirement. When it does,
the way in which you can take advantage of that fact depends upon the
compiler as well.
Aug 18 '08 #7
Daniel Molina Wegener wrote, On 18/08/08 18:29:
On Aug 18, 7:48 am, Andreas Lundgren <d99...@efd.lth.sewrote:
>Hi!

Is it determined that the C standard compiler always encode characters
with the same character excoding? If for example the functions Foo and
Bar are compiled by different compilers, is it unambiguous how to
interpret the character string in Bar?

No, it does not depends on the compiler...
You are wrong. See the replies others posted before you for details.
>Does string.h expect a specific string format?

void Foo(void)
{
char myTextString[11] = "stuvxyzåäö";

Here, instead of char, try with wchar_t and
related functions if you are using unicode
for your messages and your .c files
> Bar(myTextString);

}

void Bar(char* inp)
{
What character set to expect?

Thats depends on the user environment,
Wrong. It depends on what the function is written to expect and
(assuming the function expects a simple C string, which is likely) on
the encoding the implementation expects.

Actually, the expected encodings for standard C library functions which
handle strings and characters can be changed at run-time using the
setlocale() function, so it could also depend on what the program has
done before calling this function.
but if the
user environments is using unicode, you can expect no
more than an array of bytes,
Not necessarily.
other case is with
wchar_t and related functions...
For a start, an array of wchar_t is not simply an array of bytes.
>}
--
Flash Gordon
Aug 18 '08 #8
Many inputs and some disagreement.

A simple example may be the letter Ö that in ASCII is represented by
the number 153, but in ISO-8859-1 and Unicode is represented by the
number 214.

From what I have read out, I have to specify to customers that a
specific method has an input of a city name _coded with ISO-8859-1_ in
a char pointer. Elsewhise 'Göthenborg' stores in ISO-8859-1 encoding
will not match a search for 'Göthenborg' provided in ASCII format.

Best Regards,
Andreas Lundgren
Aug 20 '08 #9
In article <d8**********************************@26g2000hsk.g ooglegroups.com>,
Andreas Lundgren <d9****@efd.lth.sewrote:
>A simple example may be the letter Ö that in ASCII is represented by
the number 153
That's not ASCII. It's a Microsoft extension of ASCII called "code
page 437". ASCII has only 128 characters.

-- Richard
--
Please remember to mention me / in tapes you leave behind.
Aug 20 '08 #10
Andreas Lundgren wrote:
Many inputs and some disagreement.

A simple example may be the letter Ö that in ASCII is represented by
the number 153,
There is no letter 'Ö' in ASCII, and the maximum value that an ASCII
character can have is 0177 = 127. 153 is out of range.
but in ISO-8859-1 and Unicode is represented by the
number 214.

From what I have read out, I have to specify to customers that a
specific method has an input of a city name _coded with ISO-8859-1_ in
a char pointer. Elsewhise 'Göthenborg' stores in ISO-8859-1 encoding
will not match a search for 'Göthenborg' provided in ASCII format.
If you have two different encodings for the same glyph, for example 214
in ISO-8869-1 and 153 in god-knows-what-but-not-ASCII, then they cannot
compare to be equal. The computer sees values, not the shapes of glyphs
on your output device.
Aug 20 '08 #11
Andreas Lundgren <d9****@efd.lth.sewrites:
Many inputs and some disagreement.

A simple example may be the letter Ö that in ASCII is represented by
the number 153,
Not ASCII, but your point remains valid.
but in ISO-8859-1 and Unicode is represented by the
number 214.
There is another big issue hidden in that phrase. That character is
indeed 214 in Unicode, but there are at least two well-known ways to
represent the U+214 in C (as a wide char and as some multi-byte
encoded string -- UTF-8 being the most commonly used in Europe and the
US).
From what I have read out, I have to specify to customers that a
specific method has an input of a city name _coded with ISO-8859-1_ in
a char pointer. Elsewhise 'Göthenborg' stores in ISO-8859-1 encoding
will not match a search for 'Göthenborg' provided in ASCII format.
You can either mandate one uniform encoding for everything or you can
allow the user to specify the encoding and convert to some suitably
all-embracing internally. You *could* tied the encoding to the string
and convert only as and when you need to but that will be a
maintenance nightmare.

--
Ben.
Aug 20 '08 #12
In article <d8**********************************@26g2000hsk.g ooglegroups.comAndreas Lundgren <d9****@efd.lth.sewrites:
A simple example may be the letter =D6 that in ASCII is represented by
the number 153, but in ISO-8859-1 and Unicode is represented by the
number 214.
That letter is not represented in ASCII. ASCII contains the code points
0 to 127, no more.
--
dik t. winter, cwi, kruislaan 413, 1098 sj amsterdam, nederland, +31205924131
home: bovenover 215, 1025 jn amsterdam, nederland; http://www.cwi.nl/~dik/
Aug 21 '08 #13

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: aa | last post by:
Is it OK to include an ANSI file into a UTF-8 file?
20
by: Petter Reinholdtsen | last post by:
Is the code fragment 'char a = ("a");' valid ANSI C? The problematic part is '("a")'. I am sure 'char a = "a";' is valid ANSI C, but I am more unsure if it is allowed to place () around the...
2
by: Ziver MALHASOGLU | last post by:
Hi, I produce a text file using my windows application written with c#. -- System.Text.Encoding encOutput=null; encOutput=System.Text.Encoding.UTF8; StreamWriter sw=new...
4
by: Nick | last post by:
Hi, I am trying to output a string of chinese characters as a text file. When I open a file for writing from VB, the file is automatically set to UTF-8 encoding (can tell by opening the file...
11
by: LucaJonny | last post by:
Hi, I've got a problem using StreamReader in VB.NET. I try to read a txt file that contains extended characters and theese are removed from the line that is being read. I've read a lot of...
10
by: Mark Rae | last post by:
Hi, I'm in the process if converting the data out of an old DOS-based SunAccounts system (don't ask!) into SQL Server. The data has been sent to me as a collection of hundreds of SunAccounts...
2
by: gizmo | last post by:
Hi, Here's a little hack I put together to try to get to the bottom of a problem I'm having with trying to base64 encode a hash value. The hash value contains character codes 135 and 130...
0
NeoPa
by: NeoPa | last post by:
ANSI-89 v ANSI-92 Before we get into all the various types of pattern matching that can be used, there are two ANSI standards used for the main types of wildcard matching (matching zero or more...
1
by: Tejas | last post by:
Hi, I am using ldap_get_values() call to get the user attributes from LDAP. This call is returning the user attributes in UTF-8 encoding and its a PCHAR*. For normal English characters this...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.