Python 2.5 and Unicode on MacOS

[Michael J. Fromberger]

I'm not sure whether this is a bug, or a feature that I do not fully
understand. I would therefore appreciate some advice, if you have any,
on the following problem:

I have just installed Python 2.5 (release) under MacOS 10.4.7 ("Tiger").
Unfortunately, the unit test "test_unicodefile.py" (in Lib/tests/)
exhibits a warning when invoked, as shown here:

[Lib/test] % python2.5 test_unicode_file.py
test_directories (__main__.TestUnicodeFiles) ... ok
test_unicode_file.py:103: UnicodeWarning: Unicode equal comparison
failed to convert both arguments to Unicode - interpreting them as being
unequal
filename1==filename2
test_equivalent_files (__main__.TestUnicodeFiles) ... ok
test_single_files (__main__.TestUnicodeFiles) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.252s

As you can see, the unit tests appear to pass, but I suspect there is a
hidden problem, as a subsequent build of the latest version of
BeautifulSoup now fails its unit tests, citing the same warning. I
can't tell whether this is a problem with the way BeautifulSoup is doing
things, or whether there is a more general problem with Unicode support
in the MacOS build of Python 2.5.

For reference, I have done a framework build of Python, and it seems to
work fine for everything else I have tried. Do you have any idea what
might be causing this trouble? I'd be grateful for your insights.

Cheers,
-M

--
Michael J. Fromberger | Lecturer, Dept. of Computer Science
http://www.dartmouth.edu/~sting/ | Dartmouth College, Hanover, NH, USA

[Peter Otten]

Michael J. Fromberger wrote:

I'm not sure whether this is a bug, or a feature that I do not fully
understand. I would therefore appreciate some advice, if you have any,
on the following problem:

I have just installed Python 2.5 (release) under MacOS 10.4.7 ("Tiger").
Unfortunately, the unit test "test_unicodefile.py" (in Lib/tests/)
exhibits a warning when invoked, as shown here:

[Lib/test] % python2.5 test_unicode_file.py
test_directories (__main__.TestUnicodeFiles) ... ok
test_unicode_file.py:103: UnicodeWarning: Unicode equal comparison
failed to convert both arguments to Unicode - interpreting them as being
unequal
filename1==filename2
test_equivalent_files (__main__.TestUnicodeFiles) ... ok
test_single_files (__main__.TestUnicodeFiles) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.252s

As you can see, the unit tests appear to pass, but I suspect there is a
hidden problem, as a subsequent build of the latest version of
BeautifulSoup now fails its unit tests, citing the same warning. I
can't tell whether this is a problem with the way BeautifulSoup is doing
things, or whether there is a more general problem with Unicode support
in the MacOS build of Python 2.5.

For reference, I have done a framework build of Python, and it seems to
work fine for everything else I have tried. Do you have any idea what
might be causing this trouble? I'd be grateful for your insights.

This Problem is not Mac-specific. Should u"ö" == "ö" evaluate to True, or
False, or should it raise an exception?

I would favor always raising an exception, but the current behavior is the
result of an extensive discussion on python-dev (that I did not follow
completely) -- google for "unicode hell" and "dicts are broken".

My personal conclusion is not to mix unicode and non-ascii str. I hope I'm
safe then...

Peter