[poll] which charset

On Sat, 12 Mar 2005, nose wrote:

A little poll for the week-end:
Oh dear. Is this a technical question or a popularity contest?
Which charset do you use for your website(s):
On the whole, I follow my own advice
http://ppewww.ph.gla.ac.uk/~flavell/charset/checklist

Following scenario 1 or 2 when possible; 5 when appropriate; and
scenario 6 or 7 when necessary.
What charset is better now, say, for European languages like french,
german, spanish...

iso-8859-1 is more compact. If you don't care about Netscape 4 any
more, you can also include occasional non-Latin characters when
necessary by &#number; notation. Don't use iso-8859-15. utf-8 isn't
wrong by any means, but it's less compact if the majority of the
content is Latin-1.

There are plenty of ways of converting "by rote" from one
representation to another. For example Mozilla Composer can save the
document with a different encoding, and will adapt to and from
&-notations as needed. For another example there's free recode. So
you can author documents in whichever encoding you find convenient,
and then serve them out in whichever encoding you think best meets
your readers' needs.

Jul 23 '05 #4

In article <Pi******************************@ppepc56.ph.gla.a c.uk>,
"Alan J. Flavell" <fl*****@ph.gla.ac.uk> wrote:

What charset is better now, say, for European languages like french,
german, spanish...

iso-8859-1 is more compact.

However, UTF-8 is easier to deal with if there are forms (assuming you
have a library that does Unicode normalization). With ISO-8859-1 user
can enter characters that aren't representable as ISO-8859-1.

I use UTF-8 in NFC on output and accept UTF-8 as input and normalize it
to NFC up front.

--
Henri Sivonen
hs******@iki.fi
http://hsivonen.iki.fi/
Mozilla Web Author FAQ: http://mozilla.org/docs/web-developer/faq.html

Jul 23 '05 #5

On Sat, 12 Mar 2005, Henri Sivonen wrote:

In article <Pi******************************@ppepc56.ph.gla.a c.uk>,
"Alan J. Flavell" <fl*****@ph.gla.ac.uk> wrote:
iso-8859-1 is more compact.
However, UTF-8 is easier to deal with if there are forms

Good point. (I have a page on that issue too ;-)
http://ppewww.ph.gla.ac.uk/~flavell/...form-i18n.html
(assuming you have a library that does Unicode normalization). With
ISO-8859-1 user can enter characters that aren't representable as
ISO-8859-1.

Indeed, and with some bizarre consequences.

So yes, if i18n forms input is to be expected, then one of the options
which get sent out as utf-8 certainly have some advantages (scenarios
6 and 7 on my checklist page).

But here too, Netscape 4 will have to be left behind.

Jul 23 '05 #6

In article <Pi******************************@ppepc56.ph.gla.a c.uk>,
"Alan J. Flavell" <fl*****@ph.gla.ac.uk> wrote:

ISO-8859-1.

Indeed, and with some bizarre consequences.

So yes, if i18n forms input is to be expected, then one of the options
which get sent out as utf-8 certainly have some advantages (scenarios
6 and 7 on my checklist page).

But here too, Netscape 4 will have to be left behind.

But Netscape 4.x users can still use the ISO-8859-1 repertoire with
UTF-8 forms, so I do not see Netscape 4.x as a reason for using
ISO-8859-1 over UTF-8.

--
Henri Sivonen
hs******@iki.fi
http://hsivonen.iki.fi/
Mozilla Web Author FAQ: http://mozilla.org/docs/web-developer/faq.html

Jul 23 '05 #7

On Sat, 12 Mar 2005, Henri Sivonen wrote:

But here too, Netscape 4 will have to be left behind.

But Netscape 4.x users can still use the ISO-8859-1 repertoire with
UTF-8 forms, so I do not see Netscape 4.x as a reason for using
ISO-8859-1 over UTF-8.

Aha, I see your point. I had written:

| It's true that Latin-1 characters can be typed-in (or pasted in from
| other windows that are using iso-8859-1 or windows-1252 coding), but
| that isn't particularly useful, after all, because if you only wanted
| Latin-1, you wouldn't be likely to choose utf-8 coding.

which is true as far as it goes, but your interpretation is also
valid, in the terms you put it. Thanks for the insight!

Jul 23 '05 #8

LÄÊ»ie Techie

On Sat, 12 Mar 2005 00:36:29 +0100, nose wrote:

Hi,
A little poll for the week-end:
Which charset do you use for your website(s): utf-8?
iso-8859-1 (...)?
other?
just don't care?
What charset is better now, say, for European languages like french,
german, spanish...

I use UTF-8 when dealing with Hawaiian or if I'm expecting form input from
multiple languages.

La'ie Techie

Jul 23 '05 #9

Alan J. Flavell wrote:

Don't use iso-8859-15. utf-8 isn't
wrong by any means, but it's less compact if the majority of the
content is Latin-1.

Why don't use Iso Latin-9 ? Nowdays we can't anymore keep iso Latin-1
(euro sign of course, not very easy to deal with it as an entity on a
whole website). And iso latin-9 as the same advantage over utf-8 than
iso latin-1 : it's more compact. And, by the way, not everyone needs
Unicode as lots of website only contains texts written in latin script
languages. So I keep wondering : why don't use ISO Latin-9 ?

Jul 23 '05 #10

Tim

On Mon, 14 Mar 2005 10:49:22 +0100,
Pierre Goiffon <pg******@invalid.fr> posted:

And iso latin-9 as the same advantage over utf-8 than
iso latin-1 : it's more compact.

Only if you happen to use the multi-byte characters...

For instance, the text of this message uses the same number of bytes
whether I sent it as US-ASCII, ISO-8859-1, or UTF-8.

--
If you insist on e-mailing me, use the reply-to address (it's real but
temporary). But please reply to the group, like you're supposed to.

This message was sent without a virus, please delete some files yourself.

Jul 23 '05 #11

Jukka K. Korpela

Pierre Goiffon <pg******@invalid.fr> wrote:

Why don't use Iso Latin-9 ?
Because it has virtually no practical benefits over ISO Latin 1 but the
serious drawback that it is less widely supported. And when your
attempts to specify character encoding fail (they shouldn't, but they
often do), browsers will in practice imply ISO Latin 1 (or windows-1252
to be honest), and there you go.
Nowdays we can't anymore keep iso
Latin-1 (euro sign of course, not very easy to deal with it as an
entity on a whole website).
The euro sign is a banality in more than two ways, and it is almost
always better and more stylish to use a word ("euro", "euroa", "euros"
or whatever the language and context require). In price lists, you
might wish to use the euro sign, but it can't be a serious problem to
make your price list generator or converted produce a character
reference (or an entity reference).
And, by the way,
not everyone needs Unicode as lots of website only contains texts
written in latin script languages.
There are more characters in Latin script languages than most people
know. And neither ISO Latin-1 nor ISO Latin-9 will let you use directly
more than a handful of them.
So I keep wondering : why don't use ISO Latin-9 ?

Because it was unnecessary from the beginning, and in Web usage (where
you can use character references or utf-8, according to your
consideration of the situation) even more so. It is close enough to ISO
Latin-1 to be almost useless _and_ to be too easily confuseable with
it.

--
Yucca, http://www.cs.tut.fi/~jkorpela/
Pages about Web authoring: http://www.cs.tut.fi/~jkorpela/www.html

Jul 23 '05 #12

Jukka K. Korpela wrote:

Why don't use Iso Latin-9 ?

Because it has virtually no practical benefits over ISO Latin 1 but the
serious drawback that it is less widely supported.

Thanks for the answer. Just one remaining question : can you give more
details about this lack of support ?

Jul 23 '05 #13

In article <42***********************@news.free.fr>,
Pierre Goiffon <pg******@invalid.fr> wrote:

Alan J. Flavell wrote:
Don't use iso-8859-15. utf-8 isn't
wrong by any means, but it's less compact if the majority of the
content is Latin-1.
Why don't use Iso Latin-9 ?

The proliferation of encodings is a problem. ISO-8859-15 adds to the
problem instead of solving it. The proliferation of parochial encodings
that address a few pet characters at time makes no sense considering
that UTF-8 exists.

ISO-8859-15 is an anachronism. There is software that does not support
it but supports UTF-8.

ISO-8859-15 is even more problematic with form submissions than
ISO-8859-1. UTF-8 solves the form problems.
Nowdays we can't anymore keep iso Latin-1
(euro sign of course, not very easy to deal with it as an entity on a
whole website).
So use UTF-8. ISO-8859-15 is it just fire fighting!
And iso latin-9 as the same advantage over utf-8 than
iso latin-1 : it's more compact.
My attitude is that those who don't use mod_gzip should not complain
about the non-compactness of UTF-8. Besides, the differences become
negligible relative to other data once you have images on the pages.
And, by the way, not everyone needs
Unicode as lots of website only contains texts written in latin script
languages.

Bah. It is a fallacy that Europeans/Americans/Japanese do not need
Unicode. We're all better off without the parochial legacy encodings.

For those who read Finnish my considered harmful rant is at
http://hsivonen.iki.fi/iso-8859-15/

--
Henri Sivonen
hs******@iki.fi
http://hsivonen.iki.fi/
Mozilla Web Author FAQ: http://mozilla.org/docs/web-developer/faq.html

Jul 23 '05 #14

On Mon, 14 Mar 2005, Pierre Goiffon wrote:

Alan J. Flavell wrote:
Don't use iso-8859-15. utf-8 isn't wrong by any means, but it's less
compact if the majority of the content is Latin-1.
Why don't use Iso Latin-9 ?

I was under the impression that the notes in my checklist gave some
explanation for why one should not use iso-8859-15 encoding in HTML.

http://ppewww.ph.gla.ac.uk/~flavell/...checklist#Lat9

Latin-9 defines a *character repertoire*. Certainly you may use the
Latin-9 *repertoire* - it is a proper subset of the Unicode character
repertoire of HTML4, after all.

What I am saying is that there is no point, in HTML, in encoding
that repertoire in iso-8859-15.
Nowdays we can't anymore keep iso Latin-1
Right, but that doesn't stop you from using other acceptable character
*encodings* such as iso-8859-1 or utf-8 or us-ascii for your HTML,
depending on circumstances. iso-8859-15 is a solution for a problem
which does not exist in HTML.
(euro sign of course, not very easy to deal with it as an entity on
a whole website).

€ is always a valid representation, no matter what the encoding.
Or you can follow one of the other approaches indicated by Jukka.

Jul 23 '05 #15

On Mon, 14 Mar 2005, Henri Sivonen wrote:

The proliferation of encodings is a problem.
HTML pages will continue to be served out in the traditional 8-bit
encodings for quite some years yet, and I don't see that necessarily
as a problem (modulo the problem of getting servers to put the correct
charset= parameter on their HTTP headers). But I can agree there
seems to be no benefit in adding yet more 8-bit special cases to the
mix.
ISO-8859-15 is an anachronism. There is software that does not support
it but supports UTF-8.
Right.
Bah. It is a fallacy that Europeans/Americans/Japanese do not need
Unicode.
It's unfortunately only too true (as I know from many previous usenet
discussions) that many USA users have browser installations that are
incapable of displaying a decent character repertoire. (They seem to
have thought that installing the optinal multinational support would
get them hauled up by Homeland Security, for un-American Activities.)
We're all better off without the parochial legacy encodings.

We will be living with them for years yet; but there's nothing to be
gained by adding more of them, and - as far as HTML is concerned - the
cutoff is somewhere before iso-8859-15, IMHO.

If we were discussing plain text formats then I might have a different
view. But that's OT here.

Jul 23 '05 #16

On Mon, 14 Mar 2005, Pierre Goiffon wrote:

Why don't use Iso Latin-9 ?

ISO-8859-15 is okay for text/*plain* but pointless for text/*html*
because UTF-8 or € are more widely supported. In text/plain
you cannot use "escape sequences" such as € .

--
Mars, unlike Earth, has no atmosphere.
The Chicago manual of style, 15th ed., p. 362

Jul 23 '05 #17

On Mon, 14 Mar 2005, Jukka K. Korpela wrote:

Why don't use Iso Latin-9 ?

Because it has virtually no practical benefits over ISO Latin 1 [...]

I think it has - but we need to distinguish carefully between text/plain
and text/html. ISO-8859-15 may be okay for text/plain, but it is
certainly pointless for text/html.

--
Mars, unlike Earth, has no atmosphere.
The Chicago manual of style, 15th ed., p. 362

Jul 23 '05 #18

Andreas Prilop wrote:

ISO-8859-15 may be okay for text/plain, but it is
certainly pointless for text/html.

I can't figure exactly why ? Is it because, as said by Jukka, in a HTML
source the euro sign could be replaced by an entity ?

Jul 23 '05 #19

On Mon, 14 Mar 2005, Pierre Goiffon wrote:

ISO-8859-15 may be okay for text/plain, but it is
certainly pointless for text/html.

I can't figure exactly why ? Is it because, as said by Jukka, in a HTML
source the euro sign could be replaced by an entity ?

ISO-8859-15 came years^W decades too late. It would have been a
good idea if this character set had been adopted as standard Western
8-bit character set back in the middle ages. However, when ISO-8859-15
was finally published and adopted as MIME charset, ISO-8859-1,
Windows-1252, and UTF-8 were already on their way and more or less
supported by various software.

Today, ISO-8859-15 adds nothing for text/html. It's almost as useless
as Romanian ISO-8859-16.

(The situation may be different for text/plain - but this group
discusses mainly text/html.)

Jul 23 '05 #20

Andreas Prilop wrote:
(...)

in the middle ages

Very funny expression :)

Well, OK, that's approximatly what Jukka wrote.
I would be very very interested in learning what differences between
text/html and text/plain ? Maybe by private mail ?

Jul 23 '05 #21

Andreas Prilop wrote:
(...)

in the middle ages

Very funny expression :)

Well, OK, that's approximatly what Jukka wrote.
I would be very very interested in learning what differences between
text/html and text/plain ? Maybe by private mail if it's off topic here ?

Jul 23 '05 #22

On Mon, 14 Mar 2005, Pierre Goiffon wrote:

I would be very very interested in learning what differences between
text/html and text/plain ?

You can write € and œ in text/html to represent
the euro sign and the "oe" ligature, for example.
If you want to write the euro sign and the "oe" ligature in
text/plain, you can't do with ISO-8859-1.

Jul 23 '05 #23

On Mon, 14 Mar 2005, Pierre Goiffon wrote:

Andreas Prilop wrote:
ISO-8859-15 may be okay for text/plain, but it is
certainly pointless for text/html.
I can't figure exactly why ?

Because, as has already been said (in different words) several times
on this thread: in HTML it offers NO extra functionality that was not
*already* better supported *before* browser support for iso-8859-15
became available.

Consequently, it is always (on average) less compatible to use
iso-8859-15. The difference in compatibility will perhaps become less
important with time, but there will always be some other way to do it,
which offers some advantages and fewer disadvantages than iso-8859-15.

Basically, -15 came much too late for it to be of any real relevance
to HTML. Yes, there /is/ support for it; but my Windows browser also
supports (to take just one example) Mac Cyrillic, however I certainly
wouldn't recommend anybody to serve-out WWW documents coded in Mac
Cyrillic just because some browsers support it - no matter how much
more convenient it might be to /author/ in that encoding.
Is it because, as said by Jukka, in a HTML source
the euro sign could be replaced by an entity ?

iso-8859-15 (or rather, the Latin-9 repertoire) isn't solely about
the euro character!! As I say: in HTML, all of the things which were
introduced in Latin-9 were already better-supported across browsers by
/existing/ means, by the time that support for iso-8859-15 was being
introduced. iso-8859-15 is a solution for a problem that HTML did not
have.

Jul 23 '05 #24

nose

Thanks for all your contributions.
As for me, I understand that utf-8 is the best option... at least in theory.

In practice, it seems that for west-european languages the majority of
websites are still using iso-8859-1, and I wonder if this will evolve very
soon. By experience, moving from one charset [I realise that I'm using this
term loosely] to the other is not so easy, and we can't expect the average
webdesigner to grasp all the options of recode.

Jul 23 '05 #25

nose

Pierre Goiffon wrote:
....

And iso latin-9 as the same advantage over utf-8 than
iso latin-1 : it's more compact.

The gain in compactness is so small that you can't really count that as an
argument in favor of iso-8859-1 or -15 over utf-8. Take a newspaper in
french, german or spanish: how many accented characters? 1 in 20? If you
count the markup I think you generally end up in a 3-5% increase in size
with utf-8 compared to latin-1.

The only iso-8859-* charsets for which there is a significant gain are those
for greek, russian, hebrew..., and even there, counting the markup and the
images the loss of compactness is negligible in most situations.

Jul 23 '05 #26

In article <Pine.GSO.4.44.0503141856360.19988-100000@s5b004>,
Andreas Prilop <nh******@rrzn-user.uni-hannover.de> wrote:

If you want to write the euro sign and the "oe" ligature in
text/plain, you can't do with ISO-8859-1.

But you can with UTF-8.

--
Henri Sivonen
hs******@iki.fi
http://hsivonen.iki.fi/
Mozilla Web Author FAQ: http://mozilla.org/docs/web-developer/faq.html

Jul 23 '05 #27

In article <Pi*******************************@ppepc56.ph.gla. ac.uk>,
"Alan J. Flavell" <fl*****@ph.gla.ac.uk> wrote:

iso-8859-15 is a solution for a problem that HTML did not
have.

And the problem was getting some political ISO-rubber stamped non-MS
8-bit band-aid for the euro sign. (A reliable source told me the
French/Finnish stuff came in the process and was not the original
motivation.)

--
Henri Sivonen
hs******@iki.fi
http://hsivonen.iki.fi/
Mozilla Web Author FAQ: http://mozilla.org/docs/web-developer/faq.html

Jul 23 '05 #28

Alan J. Flavell wrote:

in HTML it offers NO extra functionality that was not
*already* better supported *before* browser support for iso-8859-15
became available.

Yes, it's clear for me now thanks.

Anyway I can see a good reason to use iso latin-9 : if all the contents
are stored encoded in latin-9, and you need to use it for a website AND
to generate emails, and got nothing to make conversions to utf-8 for
example. Indeed it seems to me UTF-8 support in email agents is very poor ?

Jul 23 '05 #29

Alan J. Flavell wrote:

Why don't use Iso Latin-9 ?

I was under the impression that the notes in my checklist gave some
explanation for why one should not use iso-8859-15 encoding in HTML.

http://ppewww.ph.gla.ac.uk/~flavell/...checklist#Lat9

Yes it is all clearly written - I just missed this particular point
reading your page, witch is a very good source of informations for me.

Thanks all for your informations
And sorry I did ask a question in
<news://news.free.fr:119/42***********************@news.free.fr> that
was laready answered in you message Alan, or in
<news://news.free.fr:119/Pine.GSO.4.44.0503141602120.19764-100000@s5b004>
or <news://news.free.fr:119/42***********************@news.skynet.be> :
I just read all these messages this morning, it seems my news server get
the contributions of this newsgroup in a very strange order ?

Jul 23 '05 #30

On Tue, 15 Mar 2005, Henri Sivonen wrote:

In article <Pi*******************************@ppepc56.ph.gla. ac.uk>,
"Alan J. Flavell" <fl*****@ph.gla.ac.uk> wrote:
iso-8859-15 is a solution for a problem that HTML did not
have.

And the problem was getting some political ISO-rubber stamped non-MS
8-bit band-aid for the euro sign. (A reliable source told me the
French/Finnish stuff came in the process and was not the original
motivation.)

Well, I had fairly close contact at the time, in the "SHARE Europe"
IBM user group, with a couple of the people involved in the ISO
Latin-9/ iso-8859-15 development process, and I can say that *they*
did not emphasise the importance of the euro character.

Of course, in any development process of this kind, each of the
participants can be expected to have their own private agenda. One of
the Francophone commentators several times repeated the claim that the
iso-8859-1 specification had only taken out the oe-ligature because
the decision had been taken while the French delegate went out for a
pee, and -15 was their way to finally get this back again. But
another said this story was untrue, and I really can't tell you for
sure which one was right.

Jul 23 '05 #31

In article <42***********************@news.free.fr>,
Pierre Goiffon <pg******@invalid.fr> wrote:

Anyway I can see a good reason to use iso latin-9 : if all the contents
are stored encoded in latin-9, and you need to use it for a website AND
to generate emails, and got nothing to make conversions to utf-8 for
example. Indeed it seems to me UTF-8 support in email agents is very poor ?

UTF-8 is supported in virtually every contemporary dedicated GUI email
client. The notable problematic email client is the official pine
without the iconv patch. Then there are broken gateways as in FirstClass.

FWIW, my email client (properly IMO) refuses to send ISO-8859-15 encoded
mail and pragmatically selects from US-ASCII, ISO-8859-1, Windows-1252,
ISO-2022-JP and UTF-8. The pragmatic choices for mail with the euro
character are Windows-1252 and UTF-8.

--
Henri Sivonen
hs******@iki.fi
http://hsivonen.iki.fi/
Mozilla Web Author FAQ: http://mozilla.org/docs/web-developer/faq.html

Jul 23 '05 #32

Henri Sivonen wrote:

UTF-8 is supported in virtually every contemporary dedicated GUI email
client.

Yes you're right, but almost all the webmails I know don't support
correctly emails encoded in UTF-8.

Jul 23 '05 #33