A question about reading an UTF-8 text file

[Claude Yih]

Hi, everyone. The other day I wrote a piece of code to see how the
UTF-8 text file was saved in the memory. Although the code worked
seemingly well, I still cannot understand a problem about the output
format.

The code is like this (I ignored the #include and #define here) :
int main(void)
{
FILE* fp = NULL;
char buffer[MAX_SIZE];
int byteRead = 0;
char* cp = NULL;

memset(buffer, '\0', sizeof(buffer)) ;

if ((fp = fopen(FILE_NAME , "r")) != NULL)
{
byteRead = fread(buffer, sizeof(char), MAX_SIZE, fp);
if (byteRead != 0)
{
cp = (char *)buffer;
while ((*cp) != '\0')
{
printf("%#X\t", *cp++);
}
}
else
{
}
fclose(fp);
}
else
{
perror("open");
}
return 0;
}

The UTF-8 text file it reads in contains five Japanese characters
("ã�‚ã�„ã�†ã�ˆã �Š") and four ascii character ("abcd"). And the output
is as below:

0XFFFFFFEF 0XFFFFFFBB 0XFFFFFFBF 0XFFFFFFE3
0XFFFFFF81
0XFFFFFF82 0XFFFFFFE3 0XFFFFFF81 0XFFFFFF84
0XFFFFFFE3
0XFFFFFF81 0XFFFFFF86 0XFFFFFFE3 0XFFFFFF81
0XFFFFFF88
0XFFFFFFE3 0XFFFFFF81 0XFFFFFF8A 0X61 0X62 0X63
0X64

Well，The result is fine, but I don't understand how do those "FFFFFF"
come out? In my opinion, printf("%#X\t", *cp++) will only output the
content of one byte, but the output seems like four bytes. I don't know
why. Can anybody tell my how does it happen? Thanx~~~

[Barry Schwarz]

On 17 Mar 2006 21:58:23 -0800, "Claude Yih" <wi******@gmail .com>
wrote:

Hi, everyone. The other day I wrote a piece of code to see how the
UTF-8 text file was saved in the memory. Although the code worked
seemingly well, I still cannot understand a problem about the output
format.

The code is like this (I ignored the #include and #define here) :
int main(void)
{
FILE* fp = NULL;
char buffer[MAX_SIZE];
int byteRead = 0;
char* cp = NULL;

memset(buffer, '\0', sizeof(buffer)) ;

if ((fp = fopen(FILE_NAME , "r")) != NULL)
{
byteRead = fread(buffer, sizeof(char), MAX_SIZE, fp);
if (byteRead != 0)
{
cp = (char *)buffer;
The cast is superfluous.
while ((*cp) != '\0')
{
printf("%#X\t", *cp++);
%x requires an unsigned int. On your system, char is apparently
signed. Try changing cp to an unsigned char* and buffer to an
unsigned char array.
}
}
else
{
}
If you don't have any else action, you don't need the else at all.
fclose(fp);
}
else
{
perror("open");
}
return 0;
}

The UTF-8 text file it reads in contains five Japanese characters
("?????") and four ascii character ("abcd"). And the output
is as below:

0XFFFFFFEF 0XFFFFFFBB 0XFFFFFFBF 0XFFFFFFE3
0XFFFFFF81
0XFFFFFF82 0XFFFFFFE3 0XFFFFFF81 0XFFFFFF84
0XFFFFFFE3
0XFFFFFF81 0XFFFFFF86 0XFFFFFFE3 0XFFFFFF81
0XFFFFFF88
0XFFFFFFE3 0XFFFFFF81 0XFFFFFF8A 0X61 0X62 0X63
0X64

Well?The result is fine, but I don't understand how do those "FFFFFF"
come out? In my opinion, printf("%#X\t", *cp++) will only output the
content of one byte, but the output seems like four bytes. I don't know
why. Can anybody tell my how does it happen? Thanx~~~

Since printf is a variadic function, all integer arguments smaller
than int (unsigned int) are promoted to int (or unsigned int). Think
about why 0x81 has the Fs but 0x61 doesn't.

Similarly, all floats would be promoted to double.
Remove del for email

[Keith Thompson]

"Claude Yih" <wi******@gmail .com> writes:

Hi, everyone. The other day I wrote a piece of code to see how the
UTF-8 text file was saved in the memory. Although the code worked
seemingly well, I still cannot understand a problem about the output
format.

The code is like this (I ignored the #include and #define here) :
int main(void)
{

[snip]

Why would you want to ignore the #include and #define directives?
Your program is invalid without them. If you're going to post a
program, post the whole thing (unless it's too big to post).

--
Keith Thompson (The_Other_Keit h) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.

[Martin Ambuhl]

Claude Yih wrote:

Hi, everyone. The other day I wrote a piece of code to see how the
UTF-8 text file was saved in the memory. Although the code worked
seemingly well, I still cannot understand a problem about the output
format.

The code is like this (I ignored the #include and #define here) :
I won't ignore #includes and #defines. It is expected that code posted
here be compilable. The remainder of Claude Yih's question follows this
note.
See what the following does on your implementation. Note the 'unsigned'.

#include <stdio.h>
#define FILE_NAME "./japchar"

int main(void)
{
FILE *fp = NULL;
unsigned char buffer[BUFSIZ], *cp;

if ((fp = fopen(FILE_NAME , "rb"))) {
if (fread(buffer, 1, BUFSIZ, fp))
for (cp = buffer; *cp; cp++)
printf("%#X\t", *cp);
putchar('\n');
fclose(fp);
}
else
perror("open");

return 0;
}

[output for my implementation with ./japchar containing the characters
����� + EOL characters]

0XE3 0X81 0X82 0XE3 0X81 0X84 0XE3 0X81 0X86
0XE3 0X81 0X88 0XE3 0X81 0X8A 0XD 0XA



int main(void)
{
FILE* fp = NULL;
char buffer[MAX_SIZE];
int byteRead = 0;
char* cp = NULL;

memset(buffer, '\0', sizeof(buffer)) ;

if ((fp = fopen(FILE_NAME , "r")) != NULL)
{
byteRead = fread(buffer, sizeof(char), MAX_SIZE, fp);
if (byteRead != 0)
{
cp = (char *)buffer;
while ((*cp) != '\0')
{
printf("%#X\t", *cp++);
}
}
else
{
}
fclose(fp);
}
else
{
perror("open");
}
return 0;
}

The UTF-8 text file it reads in contains five Japanese characters
("ã�‚ã�„ã�†ã�ˆã �Š") and four ascii character ("abcd"). And the output
is as below:

0XFFFFFFEF 0XFFFFFFBB 0XFFFFFFBF 0XFFFFFFE3
0XFFFFFF81
0XFFFFFF82 0XFFFFFFE3 0XFFFFFF81 0XFFFFFF84
0XFFFFFFE3
0XFFFFFF81 0XFFFFFF86 0XFFFFFFE3 0XFFFFFF81
0XFFFFFF88
0XFFFFFFE3 0XFFFFFF81 0XFFFFFF8A 0X61 0X62 0X63
0X64

Well，The result is fine, but I don't understand how do those "FFFFFF"
come out? In my opinion, printf("%#X\t", *cp++) will only output the
content of one byte, but the output seems like four bytes. I don't know
why. Can anybody tell my how does it happen? Thanx~~~

[Claude Yih]

Yeh, referring to the <Computer Systems - A Programmer's Perspective>,
I knew why I should change the char into unsigned char here. Thanx,
buddies :)

P.S: Next time, I won't ignore #include and #define any more :) Thanks
for reminding me of that.

[Vladimir S. Oka]

Claude Yih opined:

Yeh, referring to the <Computer Systems - A Programmer's
Perspective>, I knew why I should change the char into
unsigned char here. Thanx, buddies :)
But we don't, as you didn't post any context. Read:

<http://cfaj.freeshell. org/google/>
<http://clc-wiki.net/wiki/Introduction_to _comp.lang.c>
P.S: Next time, I won't ignore #include and #define any more
:) Thanks for reminding me of that.

Also, don't forget to follow good advice at the above URLs.

--
BR, Vladimir

For adult education nothing beats children.

[Claude Yih]

<http://cfaj.freeshell. org/google/>
<http://clc-wiki.net/wiki/Introduction_to _comp.lang.c>
Also, don't forget to follow good advice at the above URLs.

I'd admit those URLs helped a lot. I'll pay attention to that when I
post another message next time.

[Vladimir S. Oka]

Claude Yih opined:

<http://cfaj.freeshell. org/google/>
<http://clc-wiki.net/wiki/Introduction_to _comp.lang.c>

Also, don't forget to follow good advice at the above URLs.

I'd admit those URLs helped a lot. I'll pay attention to that when I
post another message next time.

Thank you!

--
BR, Vladimir

Your business will go through a period of considerable expansion.

[CBFalconer]

Claude Yih wrote:

<http://cfaj.freeshell. org/google/>
<http://clc-wiki.net/wiki/Introduction_to _comp.lang.c>

Also, don't forget to follow good advice at the above URLs.

I'd admit those URLs helped a lot. I'll pay attention to that
when I post another message next time.

Good. Here are some more useful links.

--
Some informative links:
news:news.annou nce.newusers
http://www.geocities.com/nnqweb/
http://www.catb.org/~esr/faqs/smart-questions.html
http://www.caliburn.nl/topposting.html
http://www.netmeister.org/news/learn2quote.html