utf - string translation

[hg]

Hi,

I'm bringing over a thread that's going on on f.c.l.python.

The point was to get rid of french accents from words.

We noticed that len('à') != len('a') and I found the hack below to fix
the "problem" ... yet I do not understand - especially since 'à' is
included in the extended ASCII table, and thus can be stored in one byte.

Any clue ?

hg

# -*- coding: utf-8 -*-
import string

def convert(mot):
print len(mot)
print mot[0]
print '%x' % ord(mot[1])
table =
string.maketran s('àâäéèêëîïôöù üû','\x00a\x00a \x00a\x00e\x00e \x00e\x00e\x00i \x00i\x00o\x00o \x00u\x00u\x00u ')

return mot.translate(t able).replace(' \x00','')
c = 'àbôö a '
print convert(c)

[John Machin]

Dan wrote:

Thank you for your answers.

In fact, I'm getting start with Python.

That was a good decision. Welcome!

>
I was looking for transform a text through elementary cryptographic
processes (Vigenère).

So why do you want to strip off accents? The history of communication
has several examples of significant difference in meaning caused by
minute differences in punctuation or accents including one of which you
may have heard: a will that could be read (in part) as either "a chacun
d'eux million francs" or "a chacun deux million francs" with the
remainder to a 3rd party.

The initial text is in a file, and my system is under UTF-8 by default
(Ubuntu)

Your system being "under UTF-8" does give you some clue, I suppose. Do
find the time to locate some data with accents and do print(repr(data ))
as I suggested, to *verify* what you've got.

Don't guess. Different underlying representations can look the same
when rendered on your screen. Don't rely on what sysadmins tell you.
Peculiar things can happen, e.g.

me: How is your data encoded?
them: XYZese [a language]
me: I'll try again; Are you using encoding A or encoding B?
them: We've heard A mentioned; what's an encoding anyway?
[snip long explanation plus investigation of what locales [plural] had
been used when configuring their workstations and servers]
them: OK, so there's more than one way of representing XYZese on a
computer. That might explain why the government regulatory authority
for our industry is very sad [to put it mildly] about not being able to
read our monthly filings!!!

Cheers,
John

[David H Wild]

In article <11************ **********@k70g 2000cwa.googleg roups.com>,
John Machin <sj******@lexic on.netwrote:

So why do you want to strip off accents? The history of communication
has several examples of significant difference in meaning caused by
minute differences in punctuation or accents including one of which you
may have heard: a will that could be read (in part) as either "a chacun
d'eux million francs" or "a chacun deux million francs" with the
remainder to a 3rd party.

The difference there, though, is a punctuation character, not an accent.

--
David Wild using RISC OS on broadband

[John Machin]

David H Wild wrote:

In article <11************ **********@k70g 2000cwa.googleg roups.com>,
John Machin <sj******@lexic on.netwrote:

So why do you want to strip off accents? The history of communication
has several examples of significant difference in meaning caused by
minute differences in punctuation or accents including one of which you
may have heard: a will that could be read (in part) as either "a chacun
d'eux million francs" or "a chacun deux million francs" with the
remainder to a 3rd party.

The difference there, though, is a punctuation character, not an accent.

I did say "difference s in punctuation or accents". Yes, the only
example I could recall OTTOMH was a difference in punctuation --
according to legend, a fly-spot IIRC :-)

[Klaas]

David H Wild wrote:

In article <11************ **********@k70g 2000cwa.googleg roups.com>,
John Machin <sj******@lexic on.netwrote:

So why do you want to strip off accents? The history of communication
has several examples of significant difference in meaning caused by
minute differences in punctuation or accents including one of which you
may have heard: a will that could be read (in part) as either "a chacun
d'eux million francs" or "a chacun deux million francs" with the
remainder to a 3rd party.

The difference there, though, is a punctuation character, not an accent.

It's not too hard to imagine an accentual difference, eg:

Le soldat protège avec le fusil --the soldier protects with the gun
Le soldat protégé avec le fusil --the soldier who is protected by
the gun (perhaps a cannon)

Contrived example, I realize, but there are scads of such instances.
(Caveat: my french is also very rusty).

-Mike

[Fredrik Lundh]

Klaas wrote:

It's not too hard to imagine an accentual difference, eg:

especially in languages where certain combinations really are distinct
letters, not just letters with accents or silly marks.

I have a Swedish children's book somewhere, in which some characters are
harassed by a big ugly monster who carries a sign around his neck that
says "Monster".

the protagonist ends up adding two dots to that sign, turning it into
"Mönster" (meaning "model", in the "model citizen" sense), and all ends
well.

just imagine that story in reverse.

</F>

[Eric Brunel]

On Wed, 22 Nov 2006 22:59:01 +0100, John Machin <sj******@lexic on.net>
wrote:
[snip]

So why do you want to strip off accents? The history of communication
has several examples of significant difference in meaning caused by
minute differences in punctuation or accents including one of which you
may have heard: a will that could be read (in part) as either "a chacun
d'eux million francs" or "a chacun deux million francs" with the
remainder to a 3rd party.

It may not be to store or even use the actual text. I stumbled on a
problem like this some time ago: I had some code building an index for a
document and wanted the entries starting with "e", "é", "è" or "ê" to be
in the same section...
--
python -c "print ''.join([chr(154 - ord(c)) for c in
'U(17zX(%,5.zmz 5(17l8(%,5.Z*(9 3-965$l7+-'])"

[Dan]

On 22 nov, 22:59, "John Machin" <sjmac...@lexic on.netwrote:

processes (Vigenère)

So why do you want to strip off accents? The history of communication
has several examples of significant difference in meaning caused by
minute differences in punctuation or accents including one of which you
may have heard: a will that could be read (in part) as either "a chacun
d'eux million francs" or "a chacun deux million francs" with the
remainder to a 3rd party.

of course.
My purpose is not doing something realistic on a cryptographic view.
It's for learning rudiments of programming.
In fact, coding characters is a kind of cryptography I mean, sometimes,
when friends can't read an email because of the characters used...

I wanted to strip off accents because I use the frequences of the
charactacters. If I only have 26 char, it's more easy to analyse (the
text can be shorter for example)

[Frederic Rentsch]

Dan wrote:

On 22 nov, 22:59, "John Machin" <sjmac...@lexic on.netwrote:

>>processes (Vigenère)

So why do you want to strip off accents? The history of communication
has several examples of significant difference in meaning caused by
minute differences in punctuation or accents including one of which you
may have heard: a will that could be read (in part) as either "a chacun
d'eux million francs" or "a chacun deux million francs" with the
remainder to a 3rd party.

of course.
My purpose is not doing something realistic on a cryptographic view.
It's for learning rudiments of programming.
In fact, coding characters is a kind of cryptography I mean, sometimes,
when friends can't read an email because of the characters used...

I wanted to strip off accents because I use the frequences of the
charactacters. If I only have 26 char, it's more easy to analyse (the
text can be shorter for example)

Try this:

from_characters =
'\xc0\xc1\xc2\x c3\xc4\xc5\xc6\ xc7\xc8\xc9\xca \xcb\xcc\xcd\xc e\xcf\xd0\xd1\x d2\xd3\xd4\xd5\ xd6\xd8\xd9\xda \xdb\xdc\xdd\xe 0\xe1\xe2\xe3\x e4\xe5\xe6\xec\ xed\xee\xef\xf0 \xf1\xf2\xf3\xf 4\xf5\xf6\xf8\x f9\xfa\xfb\xfc\ xfd\xff\xe7\xe8 \xe9\xea\xeb'
to_characters =
'AAAAAAACEEEEII IIDNOOOOOOUUUUY aaaaaaaiiiionoo oooouuuuyyceeee '
translation_tab le = string.maketran s (from_character s, to_characters)
translated_stri ng = string.translat e (original_strin g, translation_tab le)
Frederic

[John Machin]

Frederic Rentsch wrote:

Try this:

from_characters =
'\xc0\xc1\xc2\x c3\xc4\xc5\xc6\ xc7\xc8\xc9\xca \xcb\xcc\xcd\xc e\xcf\xd0\xd1\x d2\xd3\xd4\xd5\ xd6\xd8\xd9\xda \xdb\xdc\xdd\xe 0\xe1\xe2\xe3\x e4\xe5\xe6\xec\ xed\xee\xef\xf0 \xf1\xf2\xf3\xf 4\xf5\xf6\xf8\x f9\xfa\xfb\xfc\ xfd\xff\xe7\xe8 \xe9\xea\xeb'
to_characters =
'AAAAAAACEEEEII IIDNOOOOOOUUUUY aaaaaaaiiiionoo oooouuuuyyceeee '
translation_tab le = string.maketran s (from_character s, to_characters)
translated_stri ng = string.translat e (original_strin g, translation_tab le)

A few observations on the above:

1. This assumes that "original_strin g" is a str object, and the text is
encoded in latin1 or similar (e.g. cp1252).

2. Presentation of the map could be improved greatly, along the lines
of:

import pprint
import unicodedata
fromc = \
[snip]
toc = 'AAAAAAACEEEEII IIDNOOOOOOUUUUY aaaaaaaiiiionoo oooouuuuyyceeee '
assert len(fromc) == len(toc)
tups = list(zip(unicod e(fromc, 'latin1'), toc))
tups.sort()
tupsu = [(x[1], x[0], unicodedata.nam e(x[0], '** no name **')) for x in
tups]
pprint.pprint(t upsu)

which produces:

[('A', u'\xc0', 'LATIN CAPITAL LETTER A WITH GRAVE'),
('A', u'\xc1', 'LATIN CAPITAL LETTER A WITH ACUTE'),
[snip]
('D', u'\xd0', 'LATIN CAPITAL LETTER ETH'),
[snip]
('Y', u'\xdd', 'LATIN CAPITAL LETTER Y WITH ACUTE'),
('a', u'\xe0', 'LATIN SMALL LETTER A WITH GRAVE'),
[snip]
('o', u'\xf0', 'LATIN SMALL LETTER ETH'),
[snip]
('y', u'\xfd', 'LATIN SMALL LETTER Y WITH ACUTE'),
('y', u'\xff', 'LATIN SMALL LETTER Y WITH DIAERESIS')]

This makes it a lot easier to see what is going on, and check for
weirdness, like the inconsistent treatment of \xd0 and \xf0.

3. ... and to check for missing maps. The OP may be working only with
French text, and may not care about Icelandic and German letters, but
other readers who stumble on this (and miss past thread(s) on this
topic) may like something done with \xde (capital thorn), \xfe (small
thorn) and \xdf (sharp s aka Eszett).

Cheers,
John

[Fredrik Lundh]

John Machin wrote:

3. ... and to check for missing maps. The OP may be working only with
French text, and may not care about Icelandic and German letters, but
other readers who stumble on this (and miss past thread(s) on this
topic) may like something done with \xde (capital thorn), \xfe (small
thorn) and \xdf (sharp s aka Eszett).

I did post links to code that does this to this thread, several days ago...

</F>