UTF-8 & Unicode

[EU citizen]

Do web pages have to be created in unicode in order to use UTF-8 encoding?
If so, can anyone name a free application which I can use under Windows 98
to create web pages?

[Richard Tobin]

In article <Pt************ ***@newsfe1-win.ntli.net>,
EU citizen <no*******@form e.com> wrote:

Through experimentation with the W3C HTML vakidator, I've worked
out that iso-8859-1will work for Notepad files with standard english text
plus acute accented vowels.
Beware that Microsoft uses some proprietary encodings that are ISO-8859-1
for characters A0-FF, but use the C1 controls (81-9F) for other purposes.
If you don't use any of those (and the Euro symbol is quite likely one
of them) you should be OK.
The need for the XML encoding statement to match the original file format
was not mentioned in any of the (many) articles I've read on XM:/XHTML over
the last *four* years.

In most circumstances UTF-8 is the default encoding for XML if there
is no encoding declaration. In theory for text/* served by HTTP,
8859-1 is (or was - they may have changed it) the default. But if you
stick to ascii, it won't matter. And remember that you *can* stick to
ASCII and use character references (such as &#xa3;) or entity
references (if you declare them in your DTD) for all non-ascii
characters.

-- Richard

[Andreas Prilop]

On Wed, 2 Feb 2005, EU citizen wrote:

X-Newsreader: Microsoft Outlook Express 6.00.2800.1437

XML documents can contain foreign characters like Norwegian ???, or French
???.

You need to set up your newsreader^W Outlook Express correctly
in order to transmit special, non-ASCII characters:

Tools > Options > Send
Mail Sending Format > Plain Text Settings > Message format MIME
News Sending Format > Plain Text Settings > Message format MIME
Encode text using: None

--
Top-posting.
What's the most irritating thing on Usenet?

[Alan J. Flavell]

On Wed, 2 Feb 2005, Richard Tobin wrote:

In most circumstances UTF-8 is the default encoding for XML if there
is no encoding declaration.
It's a bit more complicated than that.

http://www.w3.org/TR/2000/REC-xml-20001006#charencoding

The /default/ is to look for a BOM - failing which, utf-8 is assumed.

On the other hand, it seems you've caught me out with the next bit:
In theory for text/* served by HTTP,
8859-1 is (or was - they may have changed it) the default.
HTTP hasn't changed. RFC2616 section 3.7.1, last paragraph. Thanks!

So I suppose /that/ was the explanation for the W3C validator not
failing the cited page from w3schools. Thanks.
But if you stick to ascii, it won't matter.

True - although that's hardly a very efficient way to write, say,
Cyrillic, or Arabic, or Japanese.

[Lachlan Hunt]

EU citizen wrote:

My original question asked for suggestions about suitable applications, and
yet no one has named one.

If you cared to take the time to read the guide to unicode I linked to
earlier, you would have found editors mentioned in part 2. Within it, I
mentioned two windows editors that support Unicode: SuperEdi [1] and
Macromedia Dreamweaver. A simple search for "Unicode Editor" also
reveals many other editors that may be capable of doing the job.

[1] http://www.wolosoft.com/en/superedi/

--
Lachlan Hunt
http://lachy.id.au/
http://GetFirefox.com/ Rediscover the Web
http://SpreadFirefox.com/ Igniting the Web

[Lachlan Hunt]

EU citizen wrote:

"Richard Tobin" <ri*****@cogsci .ed.ac.uk> wrote in message
news:ct******** **@pc-news.cogsci.ed. ac.uk...

<?xml version="1.0" encoding="whate ver-the-notepad-encoding-is"?>
Based on what I know now, I agree. I always assumed that Notepad, being a
simple text editor, saved files in Ascii format.

By default, Notepad saves files as Windows-1252. The characters from 0
to 127 (0x7F) are identical to US-ASCII, ISO-8859-1, UTF-8 and many
other character sets that make use of the same subset. Thus, any file
saved using Windows-1252 that only makes use of those characters is
compatible with all those other encodings.

The characters from 160 (0xA0) to 255 (0xFF) match those contained in
ISO-8859-1. Thus, any file saved using Windows-1252 that only makes use
of the aforementioned US-ASCII subset and that range of characters is
compatible with ISO-8859-1.

The characters from 128 (0x80) to 159 (0x9F), however, do not match
those in any other encoding, making any Windows-1252 file using these
characters incompatible with any other encoding. For XML, this must be
declared appropriately in the XML declaration. The characters in this
range contain the infamous "smart quotes" (Left and Right, single and
double quotation marks: ‘ ’ “ �) that cause so many problems for the
uneducated. Use of this range while declaring ISO-8859-1, UTF-8 or any
other encoding, will cause errors because they are control characters in
the character repertoires used by those encodings.
Nothing in Notepad's Help, Windows' Help or Microsoft's website says anything
about the formt used by Notepad.
It is actually mentioned in a few places on the web, though it's not
easy to find. Microsoft tend to incorrectly refer to it as ANSI, even
though it is not.

Through experimentation with the W3C HTML vakidator, I've worked out that
iso-8859-1will work for Notepad files with standard english text plus acute
accented vowels.

That's because Windows-1252 is compatible with ISO-8859-1 when that
subset is used.

Windows 95/98 Notepad files must be saved with an encoding attribute.

This is mysterious. What does it mean? That Notepad won't save
them without one? Or that you have to add one to make it work
in the web browser?

I can't make head or tail of it.

It actually means that version of Notepad will only save as
Windows-1252, so it needs to be declared in the XML declaration. That
is because an XML parser will assume UTF-8 without it and that
assumption is acceptable only when the US-ASCII subset is used.

--
Lachlan Hunt
http://lachy.id.au/
http://GetFirefox.com/ Rediscover the Web
http://SpreadFirefox.com/ Igniting the Web

[Alan J. Flavell]

On Wed, 2 Feb 2005, EU citizen wrote:

The need for the XML encoding statement to match the original file
format was not mentioned in any of the (many) articles I've read on
XM:/XHTML over the last *four* years.

The XML coding has to comply with the relevant bit of the XML
specification. Whether you read it "over the last four years" or not.
http://www.w3.org/TR/REC-xml/#charencoding

Talking about the "original file format" could be misleading, bearing
in mind that some HTTP servers are set up to transcode the
internally-stored file format into one that's more appropriate for use
on the web. For XML-based markups, that may call for appropriate
rewriting of the document's XML encoding specification. And if you're
using XHTML/1.0 Appendix C then the transcoded document would need to
confirm to its constraints too.

[Philippe Poulard]

Alan J. Flavell wrote:

On Wed, 2 Feb 2005, EU citizen wrote:

The need for the XML encoding statement to match the original file
format was not mentioned in any of the (many) articles I've read on
XM:/XHTML over the last *four* years.

The XML coding has to comply with the relevant bit of the XML
specification. Whether you read it "over the last four years" or not.
http://www.w3.org/TR/REC-xml/#charencoding

Talking about the "original file format" could be misleading, bearing
in mind that some HTTP servers are set up to transcode the
internally-stored file format into one that's more appropriate for use
on the web. For XML-based markups, that may call for appropriate
rewriting of the document's XML encoding specification. And if you're
using XHTML/1.0 Appendix C then the transcoded document would need to
confirm to its constraints too.

RFC3023 talk about XML media types

i retain that text/xml (and text/and-others-related-to-xml) should be
avoid on behalf of application/xml (and
application/and-others-related-to-xml)

Here we get utf-8:
Content-type: text/xml; charset="utf-8"
<?xml version="1.0" encoding="utf-8"?>

!?!?! Here we get US-ACII, despite the encoding specified:
Content-type: text/xml
<?xml version="1.0" encoding="utf-8"?>

Here we get utf-16:
Content-type: application/xml; charset="utf-16"
{BOM}<?xml version="1.0" encoding="utf-16"?>

Here we get the right encoding-known-by-your-parser:
Content-type: application/xml
<?xml version="1.0" encoding="encod ing-known-by-your-parser"?>

--
Cordialement,

///
(. .)
-----ooO--(_)--Ooo-----
| Philippe Poulard |
-----------------------

[Henri Sivonen]

In article <Pi************ *************** ***@ppepc56.ph. gla.ac.uk>,
"Alan J. Flavell" <fl*****@ph.gla .ac.uk> wrote:

The /conclusions/ are fine, in their way:

* Use an editor that supports encoding.
* Make sure you know what encoding it uses.
* Use the same encoding attribute in your XML documents.

That is not a safe conclusion. XML processors are only required to
support UTF-8 and UTF-16. Support for any other encoding is an XML
processor-specific extra feature. It follows that using any encoding
other than UTF-8 or UTF-16 is unsafe. If communication fails, because
someone sent an XML document in an encoding other than UTF-8 or UTF-16,
the sender is to blame.

This simplifies to a rule of thumb:
When producing XML, always use UTF-8 (and Unicode Normalization Form C).
Those who absolutely insist on using UTF-16 can use UTF-16 instead of
UTF-8.

--
Henri Sivonen
hs******@iki.fi
http://iki.fi/hsivonen/
Mozilla Web Author FAQ: http://mozilla.org/docs/web-developer/faq.html

[Philippe Poulard]

Henri Sivonen wrote:

In article <Pi************ *************** ***@ppepc56.ph. gla.ac.uk>,
"Alan J. Flavell" <fl*****@ph.gla .ac.uk> wrote:

The /conclusions/ are fine, in their way:

* Use an editor that supports encoding.
* Make sure you know what encoding it uses.
* Use the same encoding attribute in your XML documents.

That is not a safe conclusion. XML processors are only required to
support UTF-8 and UTF-16. Support for any other encoding is an XML
processor-specific extra feature. It follows that using any encoding
other than UTF-8 or UTF-16 is unsafe. If communication fails, because
someone sent an XML document in an encoding other than UTF-8 or UTF-16,
the sender is to blame.

This simplifies to a rule of thumb:
When producing XML, always use UTF-8 (and Unicode Normalization Form C).
Those who absolutely insist on using UTF-16 can use UTF-16 instead of
UTF-8.

this is theory

is there anybody who knows a parser that doesn't handle iso-8859-1
corresctly ? i don't think so; otherwise, you should change, and
communication became safe :)

--
Cordialement,

///
(. .)
-----ooO--(_)--Ooo-----
| Philippe Poulard |
-----------------------

[Alan J. Flavell]

On Fri, 4 Feb 2005, Henri Sivonen wrote:

"Alan J. Flavell" <fl*****@ph.gla .ac.uk> wrote:

The /conclusions/ are fine, in their way:

* Use an editor that supports encoding.
* Make sure you know what encoding it uses.
* Use the same encoding attribute in your XML documents.
That is not a safe conclusion.

I guess that was one of the penalties of responding to a cross-posted
article.
XML processors are only required to
support UTF-8 and UTF-16. Support for any other encoding is an XML
processor-specific extra feature.
But that's OK, since any plausible encoding produced by the editor can
be transformed by rote into utf-8 prior to subsequent XML processing
(that's the XML relevance). And pretty much any plausible encoding
produced by an editor that's meant for WWW use, is going to be
supported by the available web browsers (that's the c.i.w.a.h
relevance).
It follows that using any encoding other than UTF-8 or UTF-16 is
unsafe.

I take your point, but again: as long as the document is correctly
labelled, it can be transformed by rote into utf-8, it needs no
special heuristics, nor does it run risks of being damaged in the
process.

all the best