More elegant UTF-8 encoder

[Bjoern Hoehrmann]

Hi,

For a free software project, I had to write a routine that, given a
Unicode scalar value U+0000 - U+10FFFF, returns an integer that holds
the UTF-8 encoded form of it, for example, U+00F6 becomes 0x0000C3B6.
I came up with the following. I am looking for a more elegant solution,
that is, roughly, faster, shorter, more readable, ... while producing
the same ouput for the cited range.

unsigned int
utf8toint(unsig ned int c) {
unsigned int len, res, i;

if (c < 0x80) return c;

len = c < 0x800 ? 1 : c < 0x10000 ? 2 : 3;

/* this could be replaced with a array lookup */
res = (2 << len) - 1 << (7 - len) << len * 8;

for (i = len; i 0; --i, c >>= 6)
res |= ((c & 0x3f) | 0x80) << (len - i) * 8;

/* while unusual, the desired result is an int */
return res | c << len * 8;
}

Any ideas? Thanks,
--
Björn Höhrmann · mailto:bj****@h oehrmann.de · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: [phone] · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

[Richard Tobin]

In article <11************ **********@e9g2 000prf.googlegr oups.com>,
<we******@gmail .comwrote:

>is a big win. If you include western European languages, it will
still get about 90% of characters.

>Tell that to the Greeks, French or Russians.

Russian is not western European. French has accented characters, but
less than 10% (yes, I checked some examples). Overall, I believe that
about 90% of characters in western European languages will be from
the Unicode range below 0x80.

-- Richard
--
"Considerat ion shall be given to the need for as many as 32 characters
in some alphabets" - X3.4, 1963.

[Dik T. Winter]

In article <f5***********@ pc-news.cogsci.ed. ac.ukri*****@cogsci. ed.ac.uk (Richard Tobin) writes:

In article <11************ **********@e9g2 000prf.googlegr oups.com>,
<we******@gmail .comwrote:

is a big win. If you include western European languages, it will
still get about 90% of characters.

Tell that to the Greeks, French or Russians.

Russian is not western European.

How do you define western European?

Russian is not western European. French has accented characters, but
less than 10% (yes, I checked some examples). Overall, I believe that
about 90% of characters in western European languages will be from
the Unicode range below 0x80.

That may be the case. But there is only one of the western European
languages that will fit completely in that range, and I do not know
whether it is indeed 10% accented characters in all other languages
(and there are more than you think). I would think that figure is
exceeded in Frisian, one of the official languages of the Netherlands.
--
dik t. winter, cwi, kruislaan 413, 1098 sj amsterdam, nederland, [phone]
home: bovenover 215, 1025 jn amsterdam, nederland; http://www.cwi.nl/~dik/

[Joachim Schmitz]

"Dik T. Winter" <Di********@cwi .nlschrieb im Newsbeitrag
news:JJ******** @cwi.nl...

In article <f5***********@ pc-news.cogsci.ed. ac.ukri*****@cogsci. ed.ac.uk
(Richard Tobin) writes:

In article <11************ **********@e9g2 000prf.googlegr oups.com>,
<we******@gmail .comwrote:

>is a big win. If you include western European languages, it will
>still get about 90% of characters.

>Tell that to the Greeks, French or Russians.

Russian is not western European.

How do you define western European?

Does it matter? Russia is eastern Europe and even goes into Asia, there's no
european country that is more eastern, so it can't be western, can it?

Eastern Europe used to be devided from western Europe by the Iron Curtain.
Even now this still is the line to draw with the minor exception of eastern
Germany, maybe 8-)

Also Russia uses a completly different character set. The Greek too. While
the French only have a couple of accented characters, the Dutch having one
extra character (ij), etc, in addition to the Latin alphabeth

Bye, Jojo

[Richard Bos]

"Dik T. Winter" <Di********@cwi .nlwrote:

In article <f5***********@ pc-news.cogsci.ed. ac.ukri*****@cogsci. ed.ac.uk (Richard Tobin) writes:

In article <11************ **********@e9g2 000prf.googlegr oups.com>,
<we******@gmail .comwrote:
>

is a big win. If you include western European languages, it will
still get about 90% of characters.

>

Tell that to the Greeks, French or Russians.

>
Russian is not western European.

How do you define western European?

From the West of Europe, obviously. Russia is about as far East as you
can go while still being in Europe.

Russian is not western European. French has accented characters, but
less than 10% (yes, I checked some examples). Overall, I believe that
about 90% of characters in western European languages will be from
the Unicode range below 0x80.

That may be the case. But there is only one of the western European
languages that will fit completely in that range,

Two, if you count dead languages. Since English is completely identical
to Latin in all other regards (hence the ban on split infinitives), this
is but proper.

and I do not know whether it is indeed 10% accented characters in all
other languages (and there are more than you think). I would think
that figure is exceeded in Frisian, one of the official languages of
the Netherlands.

Is not! It's a speech defect. But no, you'd be surprised how few accents
there are in a typical Frisian text. Odd ones, such as a circonflexe on
the 'y', but not that many. Too bloody many unaccented 'y's and 'j's
inserted any which where, but not that many accents.
All this, and nobody has yet mentioned that categorical statements about
what kind of code is more time-efficient are usually the sign of a very
poor programmer? Measure, people, measure! And don't be surprised to
find a difference of only 1% either way.

Richard

[Dik T. Winter]

In article <f5**********@o nline.de"Joachi m Schmitz" <jo**@schmitz-digital.dewrite s:

"Dik T. Winter" <Di********@cwi .nlschrieb im Newsbeitrag
news:JJ******** @cwi.nl...

In article <f5***********@ pc-news.cogsci.ed. ac.ukri*****@cogsci. ed.ac.uk
(Richard Tobin) writes:

In article <11************ **********@e9g2 000prf.googlegr oups.com>,
<we******@gmail .comwrote:
>
is a big win. If you include western European languages, it will
still get about 90% of characters.
>
Tell that to the Greeks, French or Russians.
>
Russian is not western European.

How do you define western European?

Does it matter? Russia is eastern Europe and even goes into Asia, there's no
european country that is more eastern, so it can't be western, can it?

But there are many people who would not call German western European either.
Rather central European.

Eastern Europe used to be devided from western Europe by the Iron Curtain.
Even now this still is the line to draw with the minor exception of eastern
Germany, maybe 8-)

Ah. But because Greece and in fact also Yugoslavia were not behind the
Iron Curtain they belong to Western Europe? And Cyrillic is also used
in some of the former Yugoslavian Republics (it was actually invented
in Croatia, but not used there). And we have now also Bulgaria in the
EU, using a Cyrillic script. It will not be long before the banknotes
are adapted to include that script.

Also Russia uses a completly different character set. The Greek too. While
the French only have a couple of accented characters, the Dutch having one
extra character (ij), etc, in addition to the Latin alphabeth

Take care. Linguists do not agree that "ij" is an extra character in
Dutch, it is a bit controversial. And you are missing the accented letters
that are used in Dutch quite a lot (diaeresis amongst others, with a function
different from the Umlaut in German, our neighbouring country is called
België in Dutch).
--
dik t. winter, cwi, kruislaan 413, 1098 sj amsterdam, nederland, [phone]
home: bovenover 215, 1025 jn amsterdam, nederland; http://www.cwi.nl/~dik/

[Dik T. Winter]

In article <46************ ****@news.xs4al l.nlrl*@hoekstra-uitgeverij.nl (Richard Bos) writes:

"Dik T. Winter" <Di********@cwi .nlwrote:

In article <f5***********@ pc-news.cogsci.ed. ac.ukri*****@cogsci. ed.ac.uk (Richard Tobin) writes:

....

Russian is not western European.

How do you define western European?

From the West of Europe, obviously. Russia is about as far East as you
can go while still being in Europe.

But that script is used more west than the most western part of (former)
Russia.

and I do not know whether it is indeed 10% accented characters in all
other languages (and there are more than you think). I would think
that figure is exceeded in Frisian, one of the official languages of
the Netherlands.

Is not! It's a speech defect.

Perhaps, but in that case it is an official speech defect. And there are
two more such in the Netherlands (called regional languages).

All this, and nobody has yet mentioned that categorical statements about
what kind of code is more time-efficient are usually the sign of a very
poor programmer? Measure, people, measure! And don't be surprised to
find a difference of only 1% either way.

Right. And: make it correct first, bother about optimisation only when
time is a problem.
--
dik t. winter, cwi, kruislaan 413, 1098 sj amsterdam, nederland, [phone]
home: bovenover 215, 1025 jn amsterdam, nederland; http://www.cwi.nl/~dik/