Re: convert unicode characters to visibly similar ascii characters

[Laszlo Nagy]

Peter Bulychev wrote:

Hello.

I want to convert unicode character into ascii one.
The method ".encode('ASCII') " can convert only those unicode
characters, which fit into 0..128 range.

But there are still lots of characters beyond this range, which can be
manually converted to some visibly similar ascii characters. For
instance, there are several quotation marks in unicode, which can be
converted into ascii quotation mark.

Please be more specific. There is no general solution. Unicode can
handle latin, cyrilic (russian), chinese, japanese and arabic characters
in the same string. There are thousands of possible non-ascii characters
and many of them are not similar to any ascii character.

If you only want this to work for a subset, please define that subset.

Laszlo

[Jim]

Peter Bulychev wrote:

I want to convert unicode character into ascii one.

You have to make some arbitrary choices of what to translate. Based
on some materials on effbot's site, and a recipe, I made
ftp://alan.smcvt.edu/hefferon/unicode2ascii.py
which has at least some of what you are looking for.
$ grep HYPHEN unicode2ascii.py
u'\N{SOFT HYPHEN}':u'-',
u'\N{HYPHEN}':u'-',
u'\N{NON-BREAKING HYPHEN}':u'-',
u'\N{SOFT HYPHEN}': '-',
No doubt I have some terrible gaffes and some things missing.
Corrections appreciated.

Jim

[Jim]

Peter Bulychev wrote:

I want to convert unicode character into ascii one.

You have to make some arbitrary choices of what to translate. Based
on some materials on effbot's site, and a recipe, I made
ftp://alan.smcvt.edu/hefferon/unicode2ascii.py
which has at least some of what you are looking for.
$ grep HYPHEN unicode2ascii.py
u'\N{SOFT HYPHEN}':u'-',
u'\N{HYPHEN}':u'-',
u'\N{NON-BREAKING HYPHEN}':u'-',
u'\N{SOFT HYPHEN}': '-',
No doubt I have some terrible gaffes and some things missing.
Corrections appreciated.

Jim

[John Machin]

On Jul 2, 9:55 am, Jim <[e-mail]:

Peter Bulychev wrote:

I want to convert unicode character into ascii one.

You have to make some arbitrary choices of what to translate. Based
on some materials on effbot's site, and a recipe, I made
ftp://alan.smcvt.edu/hefferon/unicode2ascii.py
which has at least some of what you are looking for.
$ grep HYPHEN unicode2ascii.py
u'\N{SOFT HYPHEN}':u'-',
u'\N{HYPHEN}':u'-',
u'\N{NON-BREAKING HYPHEN}':u'-',
u'\N{SOFT HYPHEN}': '-',
No doubt I have some terrible gaffes and some things missing.
Corrections appreciated.

Comments on the above grep output:
1. You have SOFT HYPHEN twice, mapping it to u'-' and '-'
2. The idea of a soft hyphen is as a hint to a hyphenator about where
to insert a hyphen if one is necessary and the hyphenator is suspected
of acting cluelessly without the hint. IMHO, asciification should
substitute u'', not u'-'.
3. Read PEP 8. s/:/: /

Cheers,
John

[Jim]

On Jul 1, 8:29 pm, John Machin <[e-mail]:

On Jul 2, 9:55 am, Jim <[e-mail]:

Comments on the above grep output:
1. You have SOFT HYPHEN twice, mapping it to u'-' and '-'

Hmph. I'll correct that. Thanks.

2. The idea of a soft hyphen is as a hint to a hyphenator about where
to insert a hyphen if one is necessary and the hyphenator is suspected
of acting cluelessly without the hint. IMHO, asciification should
substitute u'', not u'-'.

Thanks also here. I'll think about it.

3. Read PEP 8. s/:/: /

I don't like the spacing in 8, personally.

Thanks,
Jim

[Jim]

On Jul 1, 8:29 pm, John Machin <[e-mail]:

On Jul 2, 9:55 am, Jim <[e-mail]:

Comments on the above grep output:
1. You have SOFT HYPHEN twice, mapping it to u'-' and '-'

Hmph. I'll correct that. Thanks.

2. The idea of a soft hyphen is as a hint to a hyphenator about where
to insert a hyphen if one is necessary and the hyphenator is suspected
of acting cluelessly without the hint. IMHO, asciification should
substitute u'', not u'-'.

Thanks also here. I'll think about it.

3. Read PEP 8. s/:/: /

I don't like the spacing in 8, personally.

Thanks,
Jim

[Ben Finney]

Jim <ji**********@gmail.comwrites:

I don't like the spacing in [PEP 8], personally.

Nevertheless, your Python code will be much less effort to read by
others (and yourself in future) if it is written in conformance with
PEP 8.

Writing all your Python code to conform with that standard is the
simplest step you can take to ensure that your code won't cause other
Python programmers undue reading effort.

--
\ “There's no excuse to be bored. Sad, yes. Angry, yes. |
`\ Depressed, yes. Crazy, yes. But there's no excuse for boredom, |
_o__) ever.� —Viggo Mortensen |
Ben Finney

[Jim]

On Jul 1, 8:42 pm, Jim <[e-mail]:

On Jul 1, 8:29 pm, John Machin <[e-mail]:

Comments on the above grep output:
1. You have SOFT HYPHEN twice, mapping it to u'-' and '-'

Hmph. I'll correct that. Thanks.

Well, maybe not. I forgot that I got the by-hand conversions from
three different sources and that's why that character appears in two
different places. (I thought that listing all cases for each source
was less confusing. Arguable, for sure.)

2. The idea of a soft hyphen is as a hint to a hyphenator about where

to insert a hyphen if one is necessary and the hyphenator is suspected
of acting cluelessly without the hint. IMHO, asciification should
substitute u'', not u'-'.

Thanks also here. I'll think about it.

Googling "soft hyphen" showed me that the question is not perfectly
clear-- some people seem to have very elaborate opinions on the
topic-- but I've gone with your suggestion. Thank you.

Again, I'd appreciate additional corrections. Not do I only speak
ASCII :-( but I admit to entering the data while watching a basketball
game, so no doubt there are some real blunders.

Thanks,
Jim