Is it possible to consume UTF8 XML documents using xml.dom.pulldom?

Simon Willison

I'm having a horrible time trying to get xml.dom.pulldom to consume a
UTF8 encoded XML file. Here's what I've tried so far:

>>xml_utf8 = """<?xml version="1.0" encoding="UTF-8" ?>

<msg>Simon\xe2\x80\x99s XML nightmare</msg>
"""

>>from xml.dom import pulldom
parser = pulldom.parseString(xml_utf8)
parser.next()

('START_DOCUMENT', <xml.dom.minidom.Document instance at 0x6f06c0>)

>>parser.next()

('START_ELEMENT', <DOM Element: msg at 0x6f0710>)

>>parser.next()

....
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in
position 21: ordinal not in range(128)

xml.dom.minidom can handle the string just fine:

>>from xml.dom import minidom
dom = minidom.parseString(xml_utf8)
dom.toxml()

u'<?xml version="1.0" ?><msg>Simon\u2019s XML nightmare</msg>'

If I pass a unicode string to pulldom instead of a utf8 encoded
bytestring it still breaks:

>>xml_unicode = u'<?xml version="1.0" ?><msg>Simon\u2019s XML nightmare</msg>'
parser = pulldom.parseString(xml_unicode)

....
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/
xml/dom/pulldom.py in parseString(string, parser)
346
347 bufsize = len(string)
--348 buf = StringIO(string)
349 if not parser:
350 parser = xml.sax.make_parser()
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in
position 32: ordinal not in range(128)

Is it possible to consume utf8 or unicode using xml.dom.pulldom or
should I try something else?

Thanks,

Simon Willison

Jul 30 '08 #1

Simon Willison

Follow up question: what's the best way of incrementally consuming XML
in Python that's character encoding aware? I have a very large file to
consume but I'd rather not have to fall back to the raw SAX API.

Jul 30 '08 #2

Paul Boddie

On 30 Jul, 16:32, Simon Willison <si...@simonwillison.netwrote:

I'm having a horrible time trying to get xml.dom.pulldom to consume a
UTF8 encoded XML file. Here's what I've tried so far:

>xml_utf8 = """<?xml version="1.0" encoding="UTF-8" ?>

<msg>Simon\xe2\x80\x99s XML nightmare</msg>
""">>from xml.dom import pulldom

>parser = pulldom.parseString(xml_utf8)
parser.next()

('START_DOCUMENT', <xml.dom.minidom.Document instance at 0x6f06c0>)>>parser.next()

('START_ELEMENT', <DOM Element: msg at 0x6f0710>)>>parser.next()

...
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in
position 21: ordinal not in range(128)

I can't reproduce this on Python 2.3.6 or 2.4.4 on RHEL 4. Instead, I
get the usual...

('CHARACTERS', <DOM Text node "Simon\u2019s XM...">)

And I can get the content of the text node as a proper Unicode object.

[...]

Is it possible to consume utf8 or unicode using xml.dom.pulldom or
should I try something else?

Yes, it is possible, at least in Python 2.3.6 and 2.4.4 configured
with --enable-unicode=ucs4 (which is what Red Hat does and expects).

Paul

P.S. You shouldn't try and pass Unicode to the parser, since XML
parsing in its entirety deals with byte sequences and character
encodings, although I suppose that there's some kind of character-
based (ie. Unicode value-based) parsing method defined somewhere by
some committee or other.

Jul 30 '08 #3

Simon Willison

On Jul 30, 4:43*pm, Paul Boddie <p...@boddie.org.ukwrote:

I can't reproduce this on Python 2.3.6 or 2.4.4 on RHEL 4. Instead, I
get the usual...

('CHARACTERS', <DOM Text node "Simon\u2019s XM...">)

I'm using Python 2.5.1 on OS X Leopard:

$ python
Python 2.5.1 (r251:54863, Feb 4 2008, 21:48:13)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin

I just tried it out on Python 2.4.2 on an Ubuntu machine and it worked
fine! I guess this must be an OS X Python bug. How absolutely
infuriating.

Thanks,

Simon

Jul 30 '08 #4

Simon Willison

On Jul 30, 4:59*pm, Simon Willison <si...@simonwillison.netwrote:

I just tried it out on Python 2.4.2 on an Ubuntu machine and it worked
fine! I guess this must be an OS X Python bug. How absolutely
infuriating.

Some very useful people in #python on Freenode pointed out that my bug
occurs because I'm trying to display things interactively in the
console. Saving to a variable instead fixes the problem.

Thanks for your help,

Simon

Jul 30 '08 #5

Paul Boddie

On 30 Jul, 18:17, Simon Willison <si...@simonwillison.netwrote:

>
Some very useful people in #python on Freenode pointed out that my bug
occurs because I'm trying to display things interactively in the
console. Saving to a variable instead fixes the problem.

What's strange about that is how the object is represented when
displayed:

('CHARACTERS', <DOM Text node "Simon\u2019s XM...">)

Here, there's no attempt made to encode \u2019 as an ASCII byte
sequence. Does the OS X version of Python do anything special with
string representations?

Paul

Jul 30 '08 #6

Peter Otten

Paul Boddie wrote:

On 30 Jul, 18:17, Simon Willison <si...@simonwillison.netwrote:

>>
Some very useful people in #python on Freenode pointed out that my bug
occurs because I'm trying to display things interactively in the
console. Saving to a variable instead fixes the problem.

What's strange about that is how the object is represented when
displayed:

('CHARACTERS', <DOM Text node "Simon\u2019s XM...">)

Here, there's no attempt made to encode \u2019 as an ASCII byte
sequence. Does the OS X version of Python do anything special with
string representations?

I'm on Kubuntu 7.10 and see the same error as Simon. The problem is in the
minidom.CharacterData class which has the following method

def __repr__(self):
data = self.data
if len(data) 10:
dotdotdot = "..."
else:
dotdotdot = ""
return "<DOM %s node \"%s%s\">" % (
self.__class__.__name__, data[0:10], dotdotdot)

The data attribute is a unicode instance...

Peter

Jul 30 '08 #7

Stefan Behnel

Simon Willison wrote:

Follow up question: what's the best way of incrementally consuming XML
in Python that's character encoding aware?

iterparse(), as implemented in (c)ElementTree and lxml. Note that ElementTree
and cElementTree are part of Python 2.5, in the xml.etree package.

I have a very large file to
consume but I'd rather not have to fall back to the raw SAX API.

Large is fairly relative. Both cElementTree and lxml are pretty memory
friendly, even when parsing into an in-memory tree.

Stefan

Jul 30 '08 #8

Paul Boddie

On 30 Jul, 19:23, Peter Otten <__pete...@web.dewrote:

>
I'm on Kubuntu 7.10 and see the same error as Simon. The problem is in the
minidom.CharacterData class which has the following method

* * def __repr__(self):
* * * * data = self.data
* * * * if len(data) 10:
* * * * * * dotdotdot = "..."
* * * * else:
* * * * * * dotdotdot = ""
* * * * return "<DOM %s node \"%s%s\">" % (
* * * * * * self.__class__.__name__, data[0:10], dotdotdot)

The data attribute is a unicode instance...

Who wants to be first to submit a patch? ;-)

Paul

Jul 30 '08 #9

Peter Otten

Paul Boddie wrote:

Who wants to be first to submit a patch? ;-)

And where? The sourceforge page says

"PyXML is no longer maintained."

Peter

Jul 30 '08 #10

Paul Boddie

On 30 Jul, 20:15, Peter Otten <__pete...@web.dewrote:

Paul Boddie wrote:

Who wants to be first to submit a patch? ;-)

And where? The sourceforge page says

"PyXML is no longer maintained."

The minidom code is in the standard library:

http://svn.python.org/view/python/trunk/Lib/xml/dom/

Paul

Jul 31 '08 #11