472,331 Members | 2,001 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,331 software developers and data experts.

Is it possible to consume UTF8 XML documents using xml.dom.pulldom?

I'm having a horrible time trying to get xml.dom.pulldom to consume a
UTF8 encoded XML file. Here's what I've tried so far:
>>xml_utf8 = """<?xml version="1.0" encoding="UTF-8" ?>
<msg>Simon\xe2\x80\x99s XML nightmare</msg>
"""
>>from xml.dom import pulldom
parser = pulldom.parseString(xml_utf8)
parser.next()
('START_DOCUMENT', <xml.dom.minidom.Document instance at 0x6f06c0>)
>>parser.next()
('START_ELEMENT', <DOM Element: msg at 0x6f0710>)
>>parser.next()
....
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in
position 21: ordinal not in range(128)

xml.dom.minidom can handle the string just fine:
>>from xml.dom import minidom
dom = minidom.parseString(xml_utf8)
dom.toxml()
u'<?xml version="1.0" ?><msg>Simon\u2019s XML nightmare</msg>'

If I pass a unicode string to pulldom instead of a utf8 encoded
bytestring it still breaks:
>>xml_unicode = u'<?xml version="1.0" ?><msg>Simon\u2019s XML nightmare</msg>'
parser = pulldom.parseString(xml_unicode)
....
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/
xml/dom/pulldom.py in parseString(string, parser)
346
347 bufsize = len(string)
--348 buf = StringIO(string)
349 if not parser:
350 parser = xml.sax.make_parser()
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in
position 32: ordinal not in range(128)

Is it possible to consume utf8 or unicode using xml.dom.pulldom or
should I try something else?

Thanks,

Simon Willison
Jul 30 '08 #1
10 2134
Follow up question: what's the best way of incrementally consuming XML
in Python that's character encoding aware? I have a very large file to
consume but I'd rather not have to fall back to the raw SAX API.
Jul 30 '08 #2
On 30 Jul, 16:32, Simon Willison <si...@simonwillison.netwrote:
I'm having a horrible time trying to get xml.dom.pulldom to consume a
UTF8 encoded XML file. Here's what I've tried so far:
>xml_utf8 = """<?xml version="1.0" encoding="UTF-8" ?>

<msg>Simon\xe2\x80\x99s XML nightmare</msg>
""">>from xml.dom import pulldom
>parser = pulldom.parseString(xml_utf8)
parser.next()

('START_DOCUMENT', <xml.dom.minidom.Document instance at 0x6f06c0>)>>parser.next()

('START_ELEMENT', <DOM Element: msg at 0x6f0710>)>>parser.next()

...
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in
position 21: ordinal not in range(128)
I can't reproduce this on Python 2.3.6 or 2.4.4 on RHEL 4. Instead, I
get the usual...

('CHARACTERS', <DOM Text node "Simon\u2019s XM...">)

And I can get the content of the text node as a proper Unicode object.

[...]
Is it possible to consume utf8 or unicode using xml.dom.pulldom or
should I try something else?
Yes, it is possible, at least in Python 2.3.6 and 2.4.4 configured
with --enable-unicode=ucs4 (which is what Red Hat does and expects).

Paul

P.S. You shouldn't try and pass Unicode to the parser, since XML
parsing in its entirety deals with byte sequences and character
encodings, although I suppose that there's some kind of character-
based (ie. Unicode value-based) parsing method defined somewhere by
some committee or other.
Jul 30 '08 #3
On Jul 30, 4:43*pm, Paul Boddie <p...@boddie.org.ukwrote:
I can't reproduce this on Python 2.3.6 or 2.4.4 on RHEL 4. Instead, I
get the usual...

('CHARACTERS', <DOM Text node "Simon\u2019s XM...">)
I'm using Python 2.5.1 on OS X Leopard:

$ python
Python 2.5.1 (r251:54863, Feb 4 2008, 21:48:13)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin

I just tried it out on Python 2.4.2 on an Ubuntu machine and it worked
fine! I guess this must be an OS X Python bug. How absolutely
infuriating.

Thanks,

Simon
Jul 30 '08 #4
On Jul 30, 4:59*pm, Simon Willison <si...@simonwillison.netwrote:
I just tried it out on Python 2.4.2 on an Ubuntu machine and it worked
fine! I guess this must be an OS X Python bug. How absolutely
infuriating.
Some very useful people in #python on Freenode pointed out that my bug
occurs because I'm trying to display things interactively in the
console. Saving to a variable instead fixes the problem.

Thanks for your help,

Simon
Jul 30 '08 #5
On 30 Jul, 18:17, Simon Willison <si...@simonwillison.netwrote:
>
Some very useful people in #python on Freenode pointed out that my bug
occurs because I'm trying to display things interactively in the
console. Saving to a variable instead fixes the problem.
What's strange about that is how the object is represented when
displayed:

('CHARACTERS', <DOM Text node "Simon\u2019s XM...">)

Here, there's no attempt made to encode \u2019 as an ASCII byte
sequence. Does the OS X version of Python do anything special with
string representations?

Paul
Jul 30 '08 #6
Paul Boddie wrote:
On 30 Jul, 18:17, Simon Willison <si...@simonwillison.netwrote:
>>
Some very useful people in #python on Freenode pointed out that my bug
occurs because I'm trying to display things interactively in the
console. Saving to a variable instead fixes the problem.

What's strange about that is how the object is represented when
displayed:

('CHARACTERS', <DOM Text node "Simon\u2019s XM...">)

Here, there's no attempt made to encode \u2019 as an ASCII byte
sequence. Does the OS X version of Python do anything special with
string representations?
I'm on Kubuntu 7.10 and see the same error as Simon. The problem is in the
minidom.CharacterData class which has the following method

def __repr__(self):
data = self.data
if len(data) 10:
dotdotdot = "..."
else:
dotdotdot = ""
return "<DOM %s node \"%s%s\">" % (
self.__class__.__name__, data[0:10], dotdotdot)

The data attribute is a unicode instance...

Peter
Jul 30 '08 #7
Simon Willison wrote:
Follow up question: what's the best way of incrementally consuming XML
in Python that's character encoding aware?
iterparse(), as implemented in (c)ElementTree and lxml. Note that ElementTree
and cElementTree are part of Python 2.5, in the xml.etree package.

I have a very large file to
consume but I'd rather not have to fall back to the raw SAX API.
Large is fairly relative. Both cElementTree and lxml are pretty memory
friendly, even when parsing into an in-memory tree.

Stefan
Jul 30 '08 #8
On 30 Jul, 19:23, Peter Otten <__pete...@web.dewrote:
>
I'm on Kubuntu 7.10 and see the same error as Simon. The problem is in the
minidom.CharacterData class which has the following method

* * def __repr__(self):
* * * * data = self.data
* * * * if len(data) 10:
* * * * * * dotdotdot = "..."
* * * * else:
* * * * * * dotdotdot = ""
* * * * return "<DOM %s node \"%s%s\">" % (
* * * * * * self.__class__.__name__, data[0:10], dotdotdot)

The data attribute is a unicode instance...
Who wants to be first to submit a patch? ;-)

Paul
Jul 30 '08 #9
Paul Boddie wrote:
Who wants to be first to submit a patch? ;-)
And where? The sourceforge page says

"PyXML is no longer maintained."

Peter
Jul 30 '08 #10
On 30 Jul, 20:15, Peter Otten <__pete...@web.dewrote:
Paul Boddie wrote:
Who wants to be first to submit a patch? ;-)

And where? The sourceforge page says

"PyXML is no longer maintained."
The minidom code is in the standard library:

http://svn.python.org/view/python/trunk/Lib/xml/dom/

Paul
Jul 31 '08 #11

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Joshua Beall | last post by:
Hi All, I have been using the SAX library in PHP to parse XHTML documents, and one thing I have noted is that the <!DOCTYPE> line is ignored. ...
4
by: David Pinto | last post by:
I'm trying to use either the minidom or pulldom to find table tags in html web pages. I've tried parsing two web pages that show up fine in my...
6
by: Spamtrap | last post by:
I only work in Perl occasionaly, and have been searching for a solution for a conversion, and everything I found seems much too complex. All I...
8
by: jog | last post by:
Hi, I want to get text out of some nodes of a huge xml file (1,5 GB). The architecture of the xml file is something like this <parent> <page>...
1
by: Srini | last post by:
Hi, I am working on a project and a portion of which involves receiving xml files on internet, extract values to build a string and pass that...
7
by: EmeraldShield | last post by:
We have an application that uses UTF8 everywhere to load / save / process documents. One of our clients is having a problem with BIG Encoded files...
0
by: sun | last post by:
I have a large xml file parsed by pulldom. I did some editing on some node,change attributes and remove some child node, how do I save it back to...
0
by: Tim Golden | last post by:
Lawrence, Anna K (US SSA) wrote: From where I'm sitting, I can't see enough to help. The crucial thing seems to be in this phrase: "When I try to...
0
by: susan_ali | last post by:
I'm using xml.dom.pulldom to parse through an XML file. I use expandNode() to scrutinize certain blocks of it that I'm interested in. Once I...
0
by: concettolabs | last post by:
In today's business world, businesses are increasingly turning to PowerApps to develop custom business applications. PowerApps is a powerful tool...
0
by: teenabhardwaj | last post by:
How would one discover a valid source for learning news, comfort, and help for engineering designs? Covering through piles of books takes a lot of...
0
by: Kemmylinns12 | last post by:
Blockchain technology has emerged as a transformative force in the business world, offering unprecedented opportunities for innovation and...
0
by: CD Tom | last post by:
This only shows up in access runtime. When a user select a report from my report menu when they close the report they get a menu I've called Add-ins...
0
by: Naresh1 | last post by:
What is WebLogic Admin Training? WebLogic Admin Training is a specialized program designed to equip individuals with the skills and knowledge...
0
jalbright99669
by: jalbright99669 | last post by:
Am having a bit of a time with URL Rewrite. I need to incorporate http to https redirect with a reverse proxy. I have the URL Rewrite rules made...
0
by: antdb | last post by:
Ⅰ. Advantage of AntDB: hyper-convergence + streaming processing engine In the overall architecture, a new "hyper-convergence" concept was...
0
by: Matthew3360 | last post by:
Hi there. I have been struggling to find out how to use a variable as my location in my header redirect function. Here is my code. ...
0
by: AndyPSV | last post by:
HOW CAN I CREATE AN AI with an .executable file that would suck all files in the folder and on my computerHOW CAN I CREATE AN AI with an .executable...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.