utf - string translation

[hg]

Hi,

I'm bringing over a thread that's going on on f.c.l.python.

The point was to get rid of french accents from words.

We noticed that len('à') != len('a') and I found the hack below to fix
the "problem" ... yet I do not understand - especially since 'à' is
included in the extended ASCII table, and thus can be stored in one byte.

Any clue ?

hg

# -*- coding: utf-8 -*-
import string

def convert(mot):
print len(mot)
print mot[0]
print '%x' % ord(mot[1])
table =
string.maketran s('àâäéèêëîïôöù üû','\x00a\x00a \x00a\x00e\x00e \x00e\x00e\x00i \x00i\x00o\x00o \x00u\x00u\x00u ')

return mot.translate(t able).replace(' \x00','')
c = 'àbôö a '
print convert(c)

[Fredrik Lundh]

hg wrote:

We noticed that len('à') != len('a')

sounds odd.

>>len('à') == len('a')

True

are you perhaps using an UTF-8 editor?

to keep your sanity, no matter what editor you're using, I recommend
adding a coding directive to the source file, and using *only* Unicode
string literals for non-ASCII text.

or in other words, put this at the top of your file (where "utf-8" is
whatever your editor/system is using):

# -*- coding: utf-8 -*-

and use

u'<text>'

for all non-ASCII literals.

</F>

[hg]

Fredrik Lundh wrote:

hg wrote:

>We noticed that len('à') != len('a')

sounds odd.

>>>len('à') == len('a')

True

are you perhaps using an UTF-8 editor?

to keep your sanity, no matter what editor you're using, I recommend
adding a coding directive to the source file, and using *only* Unicode
string literals for non-ASCII text.

or in other words, put this at the top of your file (where "utf-8" is
whatever your editor/system is using):

# -*- coding: utf-8 -*-

and use

u'<text>'

for all non-ASCII literals.

</F>

Hi,

The problem is that:

# -*- coding: utf-8 -*-
import string
print len('a')
print len('à')

returns 1 then 2

and string.maketran s(str1, str2) requires that len(str1) == len(str2)

hg

[hg]

hg wrote:

Fredrik Lundh wrote:

>hg wrote:

>>We noticed that len('à') != len('a')

sounds odd.

>>>>len('à') == len('a')

True

are you perhaps using an UTF-8 editor?

to keep your sanity, no matter what editor you're using, I recommend
adding a coding directive to the source file, and using *only* Unicode
string literals for non-ASCII text.

or in other words, put this at the top of your file (where "utf-8" is
whatever your editor/system is using):

# -*- coding: utf-8 -*-

and use

u'<text>'

for all non-ASCII literals.

</F>

Hi,

The problem is that:

# -*- coding: utf-8 -*-
import string
print len('a')
print len('à')

returns 1 then 2

and string.maketran s(str1, str2) requires that len(str1) == len(str2)

hg

PS: I'm running this under Idle

[Duncan Booth]

hg <[e-mail] rote:

>or in other words, put this at the top of your file (where "utf-8" is
whatever your editor/system is using):

# -*- coding: utf-8 -*-

and use

u'<text>'

for all non-ASCII literals.

</F>

Hi,

The problem is that:

# -*- coding: utf-8 -*-
import string
print len('a')
print len('à')

returns 1 then 2

And if you do what was suggested and write:

# -*- coding: utf-8 -*-
import string
print len(u'a')
print len(u'à')

then you get:

1
1

[hg]

Duncan Booth wrote:

hg <[e-mail] rote:

>>or in other words, put this at the top of your file (where "utf-8" is
whatever your editor/system is using):

# -*- coding: utf-8 -*-

and use

u'<text>'

for all non-ASCII literals.

</F>

Hi,

The problem is that:

# -*- coding: utf-8 -*-
import string
print len('a')
print len('à')

returns 1 then 2

And if you do what was suggested and write:

# -*- coding: utf-8 -*-
import string
print len(u'a')
print len(u'à')

then you get:

1
1

OK,

How would you handle the string.maketran s then ?

hg

[Fredrik Lundh]

hg wrote:

How would you handle the string.maketran s then ?

maketrans works on bytes, not characters. what makes you think that you
can use maketrans if you haven't gotten the slightest idea what's in the
string?

if you want to get rid of accents in a Unicode string, you can do the
approaches described here

http://www.peterbe.com/plog/unicode-to-ascii

or here

http://effbot.org/zone/unicode-convert.htm

which both works on any Unicode string.

</F>

[hg]

Fredrik Lundh wrote:

hg wrote:

>How would you handle the string.maketran s then ?

maketrans works on bytes, not characters. what makes you think that you
can use maketrans if you haven't gotten the slightest idea what's in the
string?

if you want to get rid of accents in a Unicode string, you can do the
approaches described here

http://www.peterbe.com/plog/unicode-to-ascii

or here

http://effbot.org/zone/unicode-convert.htm

which both works on any Unicode string.

</F>

Thanks

[John Machin]

hg wrote:

Duncan Booth wrote:

hg <[e-mail] rote:

>or in other words, put this at the top of your file (where "utf-8" is
whatever your editor/system is using):

# -*- coding: utf-8 -*-

and use

u'<text>'

for all non-ASCII literals.

</F>

Hi,

The problem is that:

# -*- coding: utf-8 -*-
import string
print len('a')
print len('à')

returns 1 then 2

And if you do what was suggested and write:

# -*- coding: utf-8 -*-
import string
print len(u'a')
print len(u'à')

then you get:

1
1

Some general comments:

1. There has been at least one thread on the subject of ripping accents
off Latin1 characters in the last 3 or 4 months. Try Google.

2. About your earlier problem, when len(thing1) != len(thing2):
In that and similar situations, it can be *very* useful to use this
technique:
print repr(thing1), type(thing1)
print repr(thing2), type(thing2)
Go back now and try it out!

OK,

How would you handle the string.maketran s then ?

I suggest that you first read the documentation on the str and unicode
"translate" methods.
You can obtain this quickly at the interactive prompt by doing
help(''.transla te)
and
help(u''.transl ate)
respectively.

Next steps:

Is your *real* data (not the examples you were hard-coding earlier)
encoded (latin1, utf8) in str objects or is it in unicode objects?
After reading previous posts my head is spinning & I'm not going to
guess; you determine it yourself.

[pseudocode -- blend of Pythonic & Knuthian styles]
if latin1: (A) you can use string.maketran s and str.translate
immediately.

elif unicode: (B) either (1) encode to latin1; goto (A) or (2) use
unicode.transla te with do-it-yourself mapping

elif utf8: decode to unicode; goto (B)

else: ???

HTH,
John

[Dan]

Thank you for your answers.

In fact, I'm getting start with Python.

I was looking for transform a text through elementary cryptographic
processes (Vigenère).
The initial text is in a file, and my system is under UTF-8 by default
(Ubuntu)