utf - string translation

[hg]

Hi,

I'm bringing over a thread that's going on on f.c.l.python.

The point was to get rid of french accents from words.

We noticed that len('à') != len('a') and I found the hack below to fix
the "problem" ... yet I do not understand - especially since 'à' is
included in the extended ASCII table, and thus can be stored in one byte.

Any clue ?

hg

# -*- coding: utf-8 -*-
import string

def convert(mot):
print len(mot)
print mot[0]
print '%x' % ord(mot[1])
table =
string.maketran s('àâäéèêëîïôöù üû','\x00a\x00a \x00a\x00e\x00e \x00e\x00e\x00i \x00i\x00o\x00o \x00u\x00u\x00u ')

return mot.translate(t able).replace(' \x00','')
c = 'àbôö a '
print convert(c)

[John Machin]

Fredrik Lundh wrote:

John Machin wrote:

3. ... and to check for missing maps. The OP may be working only with
French text, and may not care about Icelandic and German letters, but
other readers who stumble on this (and miss past thread(s) on this
topic) may like something done with \xde (capital thorn), \xfe (small
thorn) and \xdf (sharp s aka Eszett).

I did post links to code that does this to this thread, several days ago...

Ah yes, I missed that -- and your posting doesn't advertise that the
code fixed the "one character should be mapped to two" cases :-)

This code
(http://effbot.python-hosting.com/fil...xt/unaccent.py)
looks generally very good, but I'm left wondering why "AE" and "OE" in
the table, not "Ae and "Oe":
[snip]
0xc6: u"AE", # LATIN CAPITAL LETTER AE <<<=== ??
0xd0: u"D", # LATIN CAPITAL LETTER ETH
0xd8: u"OE", # LATIN CAPITAL LETTER O WITH STROKE <<<=== ??
0xde: u"Th", # LATIN CAPITAL LETTER THORN
[snip]

Another point: there are many non-latin1 characters that could be
mapped to ASCII. For example:
u"\u0141ukaszie wicz".translate (unaccented_map ())
doesn't work unless an entry is added to the no-decomposition table:
0x0141: u"L", # LATIN CAPITAL LETTER L WITH STROKE

It looks like generating extra entries like that could be done, with
the aid of unicodedata.nam e():

LATIN CAPITAL LETTER X WITH blahblah -"X"
LATIN SMALL LETTER X WITH blahblah -"X".lower()

This would require a fair bit of care -- obviously there are special
cases like LATIN CAPITAL LETTER O WITH STROKE. Eyeballing by regional
experts is probably required.

Cheers,
John

[Fredrik Lundh]

John Machin wrote:

Another point: there are many non-latin1 characters that could be
mapped to ASCII. For example:
u"\u0141ukaszie wicz".translate (unaccented_map ())
doesn't work unless an entry is added to the no-decomposition table:
0x0141: u"L", # LATIN CAPITAL LETTER L WITH STROKE

It looks like generating extra entries like that could be done, with
the aid of unicodedata.nam e():

LATIN CAPITAL LETTER X WITH blahblah -"X"
LATIN SMALL LETTER X WITH blahblah -"X".lower()

This would require a fair bit of care -- obviously there are special
cases like LATIN CAPITAL LETTER O WITH STROKE. Eyeballing by regional
experts is probably required.

see the comments over at

http://effbot.org/zone/unicode-convert.htm

for an extended table, eyeballed by a regional expert (and since he
makes the same point about OE vs Oe as you do, I'll probably have to
change the code ;-)

</F>

[John Machin]

Fredrik Lundh wrote:

John Machin wrote:

Another point: there are many non-latin1 characters that could be
mapped to ASCII. For example:
u"\u0141ukaszie wicz".translate (unaccented_map ())
doesn't work unless an entry is added to the no-decomposition table:
0x0141: u"L", # LATIN CAPITAL LETTER L WITH STROKE

It looks like generating extra entries like that could be done, with
the aid of unicodedata.nam e():

LATIN CAPITAL LETTER X WITH blahblah -"X"
LATIN SMALL LETTER X WITH blahblah -"X".lower()

This would require a fair bit of care -- obviously there are special
cases like LATIN CAPITAL LETTER O WITH STROKE. Eyeballing by regional
experts is probably required.

see the comments over at

http://effbot.org/zone/unicode-convert.htm

Don't rush me, I was getting to that next :-)

>
for an extended table, eyeballed by a regional expert (and since he
makes the same point about OE vs Oe as you do, I'll probably have to
change the code ;-)

Slightly extended. My point is that there is a large number of LATIN
(CAPITAL|SMALL) LETTER X WITH twiddly-bits that don't have a
decomposition; the table entries could be generated automatically

As well as regional experts, Google can be handy: googling for Thord,
Thordh, Thordsson and Thordhsson and noting the number of hits for each
tends to indicate that you and I are right about the treatment of
"eth"; Marcin's "dh" might better indicate how it's pronounced, but "d"
is AFAICT the standard transcription.

Cheers,
John