A bug for unicode strings in Python 2.4?

[Thomas Moore]

Hi:

Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.

u=u'\u9019\u662f\u4e2d\u6587\u5b57\u4e32'
u.split() [u'\u9019\u662f\u4e2d\u6587\u5b57\u4e32']

I think u should get split.

--Frank

[Neil Hodgson]

Thomas Moore:

u=u'\u9019\u662f\u4e2d\u6587\u5b57\u4e32'
u.split()

[u'\u9019\u662f\u4e2d\u6587\u5b57\u4e32']
I think u should get split.

Where do you think "這是ä¸*æ–‡å*—串" should be split and why?

Neil

[Thomas Moore]

> Thomas Moore:

>u=u'\u9019\u662f\u4e2d\u6587\u5b57\u4e32'
>u.split()

[u'\u9019\u662f\u4e2d\u6587\u5b57\u4e32']
I think u should get split.

Where do you think "這是ä¸*æ–‡å*—串" should be split and why?

Isn't a unicode string character by character?

-Frank

[Fredrik Lundh]

Thomas Moore wrote:

Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.

u=u'\u9019\u662f\u4e2d\u6587\u5b57\u4e32'
u.split() [u'\u9019\u662f\u4e2d\u6587\u5b57\u4e32']
I think u should get split.

why? split splits on whitespace (basically unicode category Zs), and
there are no whitespace symbols in there:

u=u'\u9019\u662f\u4e2d\u6587\u5b57\u4e32'
[c.isspace() for c in u]

[False, False, False, False, False, False]

there's no universal "split on words in all languages" function in the
standard python library. You may be able to roll your own using the
information in http://www.unicode.org/reports/tr29/ plus functions
in the unicodedata module (which currently doesn't include the
BreakTest tables; patches are welcome). Or maybe google can
help you find an existing implementation.

</F>

[Thomas Moore]

Hi:

Thanks. I'll write my own split().

Frank

[Szabolcs Nagy]

> Thanks. I'll write my own split().

do you want to split character by character?
then use
list(u'\u9019\u662f\u4e2d\u6587\u5b57\u4e32')