UTF-8 to HTML conversion

[Martin Trautmann]

Hi all,

is there any kind of 'hiconv' or other (unix-like) conversion tool that
would convert UTF-8 to HTML (ISO-Latin-1 and Unicode)?

The database output is UTF-8 or UTF-16 only - Thus almost every character
starts with ^@.

I've seen e.g.
http://aktuell.de.selfhtml.org/artik...64/utf8.htm#a5 as
JavaScript decoder - but maybe there's a recommended little helper that
could do:

- get rid of UTF-8 declarations where Latin is good enough
- convert others to most widely used HTML

such as Bulgarian and Russian charsets to &#10..;
.... or even ISO-Latin-1 8bits to HTML
ä -> ä
.... and maybe
EUR -> €

Thanks,
Martin

[Martin Trautmann]

On Wed, 02 Mar 2005 23:21:35 +1100, Lachlan Hunt wrote:

Stripping away one byte from a UTF-16 character does not necessarily
produce the equivalent UTF-8 character. That would only apply to the
US-ASCII subset of Unicode, for which the high-order octet is set to 0
and the low-order octet matches the US-ASCII character, but does not
apply to any other character.
True - but that's exactly the range where I wanted ASCII/Latin instead
of UTF-16.
However any program that performed such a conversion like that is
extremely broken. Read the Unicode spec and implement it properly, or
use some existing libraries that have already been written correctly for
the job, just don't implement it the way you suggested above.

That's why I was looking for a conversion tool. recode -d u6..h $file
is exactly what I was looking for.

Thanks,
Martin

[Martin Trautmann]

On 2 Mar 2005 04:30:56 -0800, ph*******@treef ic.com wrote:

$ recode utf8..html

ah, this would be an option. However, I'll have to exclude the real
<, > ", &

Then you need -d:

$ recode -d utf8..html
<h1>¿Dondé?</h1>
<h1>&iquest;Don d&eacute;?</h1>

Thanks - that's perfect. I would not have recognized the -d option
"convert only diacritics or alike for HTML/LaTeX"
as the best option.

Martin

[Martin Trautmann]

On Wed, 02 Mar 2005 23:31:15 +1100, Lachlan Hunt wrote:

What about for Unicode characters that aren't included within the
subsets with named character entity references in HTML4? Does it
produce numeric character references instead?

Could you send me a sample?

recode does not produce the named characters first or at all - a named
output from my html editor is e.g.

Greek &Theta;&alpha;& nu;&alpha;&tau; &eta;&phi;ό&rho ;&omicron;&sigm af; &Beta;&omicron; &eta;&theta;ό&s igmaf;

where recode does produce
Greek Θανατηφόρος Β&#95
9;ηθός

- Martin

[phil_gg04]

>> $ recode -d utf8..html

Thanks - that's perfect. I would not have recognized the -d option
"convert only diacritics or alike for HTML/LaTeX"
as the best option.

No, not from that description! Try this for the better documentation
(near the end):

$ info recode HTML

--Phil.

[phil_gg04]

>> $ recode utf8..html

What about for Unicode characters that aren't included within the
subsets with named character entity references in HTML4? Does it
produce numeric character references instead?

Yes. Again, whether you see this depends on what happens to this
message between me and you. Appologies to any thai readers, I've just
copied and pasted this at random.

$ recode utf8..html
สื่à¸*ฮ ่à¸*งà¸�ง รายงา นว่าน ายตุง
ชี หว่า
สื่อฮ่องกงรายงา นว่านายตุง
ชี หว่า

Quote from the documentation:

Codes not having a mnemonic entity are output by `recode' using the
`&#NNN;' notation, where NNN is a decimal representation of the UCS
code value. When there is an entity name for a character, it is always
preferred over a numeric character reference. ASCII printable
characters are always generated directly. So is the newline.
See for example:

http://www.delorie.com/gnu/docs/recode/recode_49.html

Phil.

[Martin Trautmann]

On 2 Mar 2005 05:50:50 -0800, ph*******@treef ic.com wrote:

When there is an entity name for a character, it is always
preferred over a numeric character reference.

Oops - you're right. It was my HTML editor which created numbers
instead of names, while recode used the name.

thanks,
Martin

[Alan J. Flavell]

On Wed, 2 Mar 2005, Lachlan Hunt wrote:

Martin Trautmann wrote:

it's not really UTF-8, but UTF-16. I don't have a major problem to make
UTF-8 from UTF-16 - just stripping the other byte.

That would - at best - make iso-8859-1, rather than utf-8. But
horribly wrong if the stripped byte wasn't zero.
Stripping away one byte from a UTF-16 character does not necessarily
produce the equivalent UTF-8 character. That would only apply to
the US-ASCII subset
Indeed. us-ascii is indistinguishab le from iso-8859-1 or utf-8 under
those restricted circumstances.

It you want a single additional step that does this code conversion,
then I'd be looking at recode (as others have also said). But it
would be better to look at existing process and identify a point at
which characters can be transparently converted as part of the
process. XML processors can typically do this with appropriate
parameters, for example, if an XML process is already involved.

Database access interfaces may be able to recode the data too, and so
on.
However any program that performed such a conversion like that is
extremely broken. Read the Unicode spec and implement it properly,
or use some existing libraries that have already been written
correctly for the job, just don't implement it the way you suggested
above.

Oh, quite.

Btw, for manual processes, Mozilla Composer seems to do quite a
reasonable job when you "save and change character encoding". Back in
Netscape (<=4) days we used to call it "Netscape Composter", but it's
much better in its Mozilla embodiment, AFAICS.

[Martin Trautmann]

On Wed, 2 Mar 2005 14:14:42 +0000, Alan J. Flavell wrote:

On Wed, 2 Mar 2005, Lachlan Hunt wrote:

Martin Trautmann wrote:

it's not really UTF-8, but UTF-16. I don't have a major problem to make
UTF-8 from UTF-16 - just stripping the other byte.

That would - at best - make iso-8859-1, rather than utf-8. But
horribly wrong if the stripped byte wasn't zero.

I was assuming a zero byte - whatever it is that is shown in vim as But it
would be better to look at existing process and identify a point at
which characters can be transparently converted as part of the
process. XML processors can typically do this with appropriate
parameters, for example, if an XML process is already involved.
We are talking about a 'stupid' data export to a text file which happens
to be UTF-16.
Btw, for manual processes, Mozilla Composer seems to do quite a
reasonable job when you "save and change character encoding". Back in
Netscape (<=4) days we used to call it "Netscape Composter", but it's
much better in its Mozilla embodiment, AFAICS.

Indeed - and that was my manual process first and was my reference for
comparison of the results.

I was surprised that it did not attempt to 'improve' anything that I
built myself (such as repairing "<a name=x>x</a>" -> "<a name=x></a>x")
It did fix e.g. </i><i> or <i></i> - and it did not create new
paragraphs on its own where the composter always exchanged 'random'
amounts of <p> and <br>.

Even better: it created validator approved HTML ;-)
However, I did not want to use Mozilla as my required UTF-16 convertor.

- Martin

[Shmuel (Seymour J.) Metz]

In <11************ **********@f14g 2000cwb.googleg roups.com>, on
03/01/2005
at 11:07 AM, ph*******@treef ic.com said:

Anyway, if you really want to convert Unicode to latin1 + html
character entities, I believe that GNU recode can do what you want:

What about bidi text?

--
Shmuel (Seymour J.) Metz, SysProg and JOAT <http://patriot.net/~shmuel>

Unsolicited bulk E-mail subject to legal action. I reserve the
right to publicly post or ridicule any abusive E-mail. Reply to
domain Patriot dot net user shmuel+news to contact me. Do not
reply to sp******@librar y.lspace.org

[phil_gg04]

>>Anyway, if you really want to convert Unicode to latin1 + html

character entities, I believe that GNU recode can do what you want:

What about bidi text?

Good question. I believe that it generates one numeric character
entity for each unicode character without special consideration for
this. I think that this should just work, if the browser does the
right thing, shouldn't it?

--Phil.