I found at least one case where decombining and recombining a unicode
character does not result in the same character (see at end).
I have no extensive knowledge about Unicode, yet I believe that this
must be a problem of the Unicode 3.2 specification and not Python's.
However, I haven't found out how the decomp_data (in unicodedata_db. h)
is built, and neither did I find much more info about the specifics of
Unicode 3.2. I thought about posting here; anyone more knowing could
give it a look.
If we find out that it's a problem with Python, I'll open a bug report
(and volunteer work).
*** Example *** import unicodedata as ud def report(utext):
for uchar in utext:
print ord(uchar), ud.name(uchar)
u1=u'\N{greek small letter alpha with oxia}' report(u1)
8049 GREEK SMALL LETTER ALPHA WITH OXIA u2=ud.normalize ('NFD', u1) report(u2)
945 GREEK SMALL LETTER ALPHA
769 COMBINING ACUTE ACCENT u3=ud.normalize ('NFC', u2) report(u3)
940 GREEK SMALL LETTER ALPHA WITH TONOS
*** End of Example ***
I can understand this confusion; if, as I have found, there is no
COMBINING GREEK TONOS or COMBINING TONOS ACCENT in the Unicode table,
decombining, one has to use the 'oxeia' (acute) accent...
--
TZOTZIOY, I speak England very best,
"Tssss!" --Brad Pitt as Achilles in unprecedented Ancient Greek 3 4137
Christos TZOTZIOY Georgiou wrote: I have no extensive knowledge about Unicode, yet I believe that this must be a problem of the Unicode 3.2 specification and not Python's.
Without checking the details: very well possible. Could this be
an instance of python.org/sf/1054943 ?
Regards,
Martin
Christos TZOTZIOY Georgiou wrote: I found at least one case where decombining and recombining a unicode character does not result in the same character (see at end).
I have no extensive knowledge about Unicode, yet I believe that this must be a problem of the Unicode 3.2 specification and not Python's.
I've been spending some time lately writing a normalizer (in PHP of all
things -- yeesh!), and yes Unicode is a scary world. :) Although it may
seem counterintuitiv e, it is in fact perfectly legitimate for a
character not to be its own canonical composition. u1=u'\N{gre ek small letter alpha with oxia}' report(u1 ) 8049 GREEK SMALL LETTER ALPHA WITH OXIA
This character is a "singleton decomposition". It decomposes into GREEK
SMALL LETTER ALPHA WITH TONOS, which further decomposes into GREEK SMALL
LETTER ALPHA and a COMBINING ACUTE ACCENT.
It is by definition not normalized, so when you normalize it to form C
it will turn into GREEK SMALL LETTER ALPHA WITH TONOS; there is no way
to get "back" to the original character in a normalized string. For some
more info see: http://www.unicode.org/unicode/repor...ion_List_Table u2=ud.norma lize('NFD', u1) report(u2 ) 945 GREEK SMALL LETTER ALPHA 769 COMBINING ACUTE ACCENT u3=ud.norma lize('NFC', u2) report(u3 )
940 GREEK SMALL LETTER ALPHA WITH TONOS
You should get this same result directly for ud.normalize('N FC', u1).
Converting directly to NFC should always give the same result as
converting to NFD and then NFC. Either will give you back the string you
started with if and only if it's already normalized to form C.
-- brion vibber (brion @ pobox.com)
On Mon, 08 Nov 2004 17:40:47 -0800, rumours say that Brion Vibber
<br***@pobox.co m> might have written: I've been spending some time lately writing a normalizer (in PHP of all things -- yeesh!), and yes Unicode is a scary world. :)
.... http://www.unicode.org/unicode/repor...ion_List_Table
Thanks for the pointer, very informative, explaining why the observed
behaviour is well inside the definition of Unicode. Thanks go to Martin
also for taking a look at this.
--
TZOTZIOY, I speak England very best,
"Tssss!" --Brad Pitt as Achilles in unprecedented Ancient Greek This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: David Opstad |
last post by:
Hi, all! I'm relatively new to Python, but have definitely fallen in
love with it. It reminds me of Mesa (old Xerox development language) and
LISP a bit.
Anyway, on to the question. Now that Unicode 4.0 has been released (just
got my copy today), any guesses on how long before the unicodedata
module will be updated to include all the new names? How do things like
that work, anyway; is there somebody whose task it is to update that, or...
|
by: Scott Chapman |
last post by:
There seems to be an inconsistency here:
Python 2.3.2 (#1, Oct 3 2003, 19:04:58)
on linux2
>>> 1 == True
True
>>> 3 == True
False
>>> if 1: print "true"
....
|
by: Ken Beesley |
last post by:
Newbie question: on unicodedata.name
If I do
import unicodedata
unicodedata.name(u"a")
or
unicodedata.name(u"\u0061")
|
by: Xah Lee |
last post by:
python has this nice unicodedata module that deals with unicode nicely.
#-*- coding: utf-8 -*-
# python
from unicodedata import *
# each unicode char has a unique name.
# one can use the “lookup” func to find it
|
by: Szabolcs Nagy |
last post by:
the unicodedata manual sais:
"
name( unichr)
Returns the name assigned to the Unicode character unichr as a
string. If no name is defined, default is returned, or, if not given,
ValueError is raised.
"
what is the difference between "no name defined" and "not given"?
eg. '\n' why gives a ValueError?
| |
by: kollatjorva |
last post by:
Hi all
I'm trying to get a value from an xml node 'Publisher' use the value as
a name of an .css class. This works fine until I get a value from the
Publisher node with white space in it.
I've been trying to use normalize-space function on this but I can't
make this work
here is what I'm trying to do
|
by: James Abley |
last post by:
Hi,
I'm looking into implementing this module for Jython, and I'm trying
to understand the contracts promised by the various methods. Please
bear in mind that means I'm probably targeting the CPython
implementation as of 2.3, although I would obviously be quite happy if
my implementation doesn't need too much extra to fit the 2.5
functionality!
|
by: =?iso-8859-1?B?TWF0dGlhcyBCcuRuZHN0cvZt?= |
last post by:
Hello!
I'm trying to find what package I should use if I want to:
1. Create 3d vectors.
2. Normalize those vectors.
3. Create a 3x3 rotation matrix from a unit 3-d vector and an angle in
radians.
4. Perform matrix multiplication.
|
by: James Abley |
last post by:
Hi,
I'm trying to understand how CPython implements unicodedata, with a view to
providing an implementation for Jython. This is a background, low priority
thing for me, since I last posted to this list about it in February!
Python 2.5.1 (r251:54863, May 2 2007, 16:56:35)
on linux2
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, well explore What is ONU, What Is Router, ONU & Routers main usage, and What is the difference between ONU and Router. Lets take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
| |
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development projectplanning, coding, testing, and deploymentwithout human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupr who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |