Umlaut characters in Unicode

[Jürgen Kahrs]

Hello,

do you think that this file is a proper Unicode file?

http://belnet.dl.sourceforge.net/sou...t-example3.xml

<?xml version="1.0" encoding="UTF-8"?>
...
<resource id="1" name="Andreas Plüschke" function="10" contacts=""/>

I am asking because of the ü Umlaut character.
I am guessing that the author used an ISO-8859-1
environment but forgot to change the encoding
declaration from UTF-8 to ISO-8859-1.

[Martin Honnen]

Jürgen Kahrs wrote:

do you think that this file is a proper Unicode file?

http://belnet.dl.sourceforge.net/sou...t-example3.xml
<?xml version="1.0" encoding="UTF-8"?>
...
<resource id="1" name="Andreas Plüschke" function="10" contacts=""/>

I am asking because of the ü Umlaut character.

Why is an umlaut a problem? Unicode certainly contains/allows umlaut
characters.
--

Martin Honnen
http://JavaScript.FAQTs.com/

[Jürgen Kahrs]

Martin Honnen wrote:

Why is an umlaut a problem? Unicode certainly contains/allows umlaut
characters.

Umlaut is not a problem for Unicode.
Umlaut is a problem if you write a text
with an editor in ISO-8859-1 mode and
watch the text with an editor in UTF-8
mode.

For example, while writing this posting,
I use ISO-8859-1 mode and this is an u-Umlaut: ü
Now, switch your news reader to UTF-8 and you
will find that the character does not look like
an u-umlaut anymore.

[Steve W. Jackson]

In article <2v*************@uni-berlin.de>,
Jürgen Kahrs <Ju*********************@vr-web.de> wrote:

:Martin Honnen wrote:
:
:> Why is an umlaut a problem? Unicode certainly contains/allows umlaut
:> characters.
:
:Umlaut is not a problem for Unicode.
:Umlaut is a problem if you write a text
:with an editor in ISO-8859-1 mode and
:watch the text with an editor in UTF-8
:mode.
:
:For example, while writing this posting,
:I use ISO-8859-1 mode and this is an u-Umlaut: ü
:Now, switch your news reader to UTF-8 and you
:will find that the character does not look like
:an u-umlaut anymore.

That's precisely the problem we've encountered with our application,
which stores its data in UTF-8 encoded XML documents.

We maintain everything internally in our Java application as part of a
DOM, and it's saved to an external file on request. But we failed to
force the byte stream written to the file to be encoded to UTF-8, so it
used the default ISO-8859-1 on our American systems. When the next
attempt was made to read the file (only if such characters appeared),
errors occurred because there were non-UTF-8 characters present.

The solution we found was to serialize the DOM with UTF-8 encoding
specified (which we were already doing) and then also specify UTF-8
encoding on the output file stream when writing. When this was done,
opening such an XML file in an editor clearly showed something that did
not resemble the letter with umlaut, or accent, or other special feature.

= Steve =
--
Steve W. Jackson
Montgomery, Alabama

[Jürgen Kahrs]

Steve W. Jackson wrote:

We maintain everything internally in our Java application as part of a
DOM, and it's saved to an external file on request. But we failed to
force the byte stream written to the file to be encoded to UTF-8, so it
used the default ISO-8859-1 on our American systems. When the next
attempt was made to read the file (only if such characters appeared),
errors occurred because there were non-UTF-8 characters present.

Yes, this is the situation I was thinking of.
Now, with your unpleasant experience in mind,
would you say that the following document was
also encoded in an inadequate way ?

http://belnet.dl.sourceforge.net/sou...t-example3.xml

As I said in my original posting, I am guessing
that the author used an ISO-8859-1 environment
(just like you) but forgot to change the encoding
declaration from UTF-8 to ISO-8859-1.

Thanks for answering !

[Steve W. Jackson]

In article <2v*************@uni-berlin.de>,
Jürgen Kahrs <Ju*********************@vr-web.de> wrote:

:Steve W. Jackson wrote:
:
:> We maintain everything internally in our Java application as part of a
:> DOM, and it's saved to an external file on request. But we failed to
:> force the byte stream written to the file to be encoded to UTF-8, so it
:> used the default ISO-8859-1 on our American systems. When the next
:> attempt was made to read the file (only if such characters appeared),
:> errors occurred because there were non-UTF-8 characters present.
:
:Yes, this is the situation I was thinking of.
:Now, with your unpleasant experience in mind,
:would you say that the following document was
:also encoded in an inadequate way ?
:
: http://belnet.dl.sourceforge.net/sou...tproject-examp
: le3.xml
:
:As I said in my original posting, I am guessing
:that the author used an ISO-8859-1 environment
:(just like you) but forgot to change the encoding
:declaration from UTF-8 to ISO-8859-1.
:
:Thanks for answering !

It looks to me as if it's not encoded properly, based on the visual
appearance of the <resource> element near the end.

Just to make clear what I said earlier, the problem we encountered did
not stem from using an ISO-8859-1 encoding in the XML itself. All of
our files already included <?xml version="1.0" encoding="UTF-8"?> at the
top when serialized, since we told the XML serializer to use UTF-8.

Instead, we also write the file using Java's OutputStreamWriter, in
which we specify the stream being written (in this case, Java's
FileOutputStream class designating the file) and the encoding to use
when writing the stream. Only if *both* of these things were done would
non-ASCII characters get correctly written and then parse without error
next time around. We got a separate report of this same problem from a
German user who used a directory name containing an umlaut-o (as in ö)
and from a French user with an accented e (as in é).

= Steve =
--
Steve W. Jackson
Montgomery, Alabama

[Richard Tobin]

In article <2v*************@uni-berlin.de>,
Jürgen Kahrs <Ju*********************@vr-web.de> wrote:

do you think that this file is a proper Unicode file?

http://belnet.dl.sourceforge.net/sou...t-example3.xml

The file at that URL appears to be well-formed, and contains a
correctly encoded UTF-8 u-with-umlaut. I don't see any problem with it.

Putting a UTF-8 declaration on a file that is really Latin-1 (and which
contains non-ascii characters) will almost always result in a detectable
error because the result will almost always be an illegal UTF-8 byte
sequence. An XML parser should detect the error.

-- Richard

[Alan J. Flavell]

On Fri, 12 Nov 2004, Richard Tobin wrote:

Putting a UTF-8 declaration on a file that is really Latin-1 (and which
contains non-ascii characters) will almost always result in a detectable
error
Indeed...
because the result will almost always be an illegal UTF-8 byte
sequence. An XML parser should detect the error.

In fact, anything which is supposed to handle utf-8 should give up at
that point, if only for security reasons. XML is a higher layer in
the protocol layer-cake: I'm not sure that it really should be allowed
to have any say in these lower-level problems. That way lie dragons,
from a security analysis point of view.

[Martin Honnen]

Jürgen Kahrs wrote:

Now, with your unpleasant experience in mind,
would you say that the following document was
also encoded in an inadequate way ?

http://belnet.dl.sourceforge.net/sou...t-example3.xml
As I said in my original posting, I am guessing
that the author used an ISO-8859-1 environment
(just like you) but forgot to change the encoding
declaration from UTF-8 to ISO-8859-1.

I have no problems viewing that file with Netscape 7 or IE 6, I don't
see anything displayed incorrectly that suggests the encoding has not
been declared correctly.
--

Martin Honnen
http://JavaScript.FAQTs.com/

[Jürgen Kahrs]

Richard Tobin wrote:

Putting a UTF-8 declaration on a file that is really Latin-1 (and which
contains non-ascii characters) will almost always result in a detectable
error because the result will almost always be an illegal UTF-8 byte
I should have looked into the hexdump immediately:

00002250 20 6e 61 6d 65 3d 22 41 6e 64 72 65 61 73 20 50 | name="Andreas P|
00002260 6c c3 bc 73 63 68 6b 65 22 20 66 75 6e 63 74 69 |l..schke" functi|

C3BC in UTF-8 converts to position 0FC as described here:

http://www.pemberley.com/janeinfo/latin1.html#utf8

And 0FC is really the position of the ü as described
on page 2 of this one:

http://www.unicode.org/charts/PDF/U0080.pdf

This mixture of bitwise encoding and character sets
is a pain if you work with it rarely.
sequence. An XML parser should detect the error.

The problem was that I did not trust my parser.
I think I should put the Unicode 4.0 book onto my book shelf.

Thanks to all who answered.

[Richard Tobin]

In article <2v*************@uni-berlin.de>,
Jürgen Kahrs <Ju*********************@vr-web.de> wrote:

I think I should put the Unicode 4.0 book onto my book shelf.

You might find this page useful:

http://www.cogsci.ed.ac.uk/~richard/utf-8.html

-- Richard

[Jürgen Kahrs]

Richard Tobin wrote:

You might find this page useful:

http://www.cogsci.ed.ac.uk/~richard/utf-8.html

Yes, this helps a lot.
I have appended this link to my bookmarks.