473,544 Members | 1,420 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Is it possible to consume UTF8 XML documents using xml.dom.pulldom ?

I'm having a horrible time trying to get xml.dom.pulldom to consume a
UTF8 encoded XML file. Here's what I've tried so far:
>>xml_utf8 = """<?xml version="1.0" encoding="UTF-8" ?>
<msg>Simon\xe2\ x80\x99s XML nightmare</msg>
"""
>>from xml.dom import pulldom
parser = pulldom.parseSt ring(xml_utf8)
parser.next ()
('START_DOCUMEN T', <xml.dom.minido m.Document instance at 0x6f06c0>)
>>parser.next ()
('START_ELEMENT ', <DOM Element: msg at 0x6f0710>)
>>parser.next ()
....
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\u2019' in
position 21: ordinal not in range(128)

xml.dom.minidom can handle the string just fine:
>>from xml.dom import minidom
dom = minidom.parseSt ring(xml_utf8)
dom.toxml()
u'<?xml version="1.0" ?><msg>Simon\u2 019s XML nightmare</msg>'

If I pass a unicode string to pulldom instead of a utf8 encoded
bytestring it still breaks:
>>xml_unicode = u'<?xml version="1.0" ?><msg>Simon\u2 019s XML nightmare</msg>'
parser = pulldom.parseSt ring(xml_unicod e)
....
/System/Library/Frameworks/Python.framewor k/Versions/2.5/lib/python2.5/
xml/dom/pulldom.py in parseString(str ing, parser)
346
347 bufsize = len(string)
--348 buf = StringIO(string )
349 if not parser:
350 parser = xml.sax.make_pa rser()
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\u2019' in
position 32: ordinal not in range(128)

Is it possible to consume utf8 or unicode using xml.dom.pulldom or
should I try something else?

Thanks,

Simon Willison
Jul 30 '08 #1
10 2206
Follow up question: what's the best way of incrementally consuming XML
in Python that's character encoding aware? I have a very large file to
consume but I'd rather not have to fall back to the raw SAX API.
Jul 30 '08 #2
On 30 Jul, 16:32, Simon Willison <si...@simonwil lison.netwrote:
I'm having a horrible time trying to get xml.dom.pulldom to consume a
UTF8 encoded XML file. Here's what I've tried so far:
>xml_utf8 = """<?xml version="1.0" encoding="UTF-8" ?>

<msg>Simon\xe2\ x80\x99s XML nightmare</msg>
""">>from xml.dom import pulldom
>parser = pulldom.parseSt ring(xml_utf8)
parser.next( )

('START_DOCUMEN T', <xml.dom.minido m.Document instance at 0x6f06c0>)>>par ser.next()

('START_ELEMENT ', <DOM Element: msg at 0x6f0710>)>>par ser.next()

...
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\u2019' in
position 21: ordinal not in range(128)
I can't reproduce this on Python 2.3.6 or 2.4.4 on RHEL 4. Instead, I
get the usual...

('CHARACTERS', <DOM Text node "Simon\u201 9s XM...">)

And I can get the content of the text node as a proper Unicode object.

[...]
Is it possible to consume utf8 or unicode using xml.dom.pulldom or
should I try something else?
Yes, it is possible, at least in Python 2.3.6 and 2.4.4 configured
with --enable-unicode=ucs4 (which is what Red Hat does and expects).

Paul

P.S. You shouldn't try and pass Unicode to the parser, since XML
parsing in its entirety deals with byte sequences and character
encodings, although I suppose that there's some kind of character-
based (ie. Unicode value-based) parsing method defined somewhere by
some committee or other.
Jul 30 '08 #3
On Jul 30, 4:43*pm, Paul Boddie <p...@boddie.or g.ukwrote:
I can't reproduce this on Python 2.3.6 or 2.4.4 on RHEL 4. Instead, I
get the usual...

('CHARACTERS', <DOM Text node "Simon\u201 9s XM...">)
I'm using Python 2.5.1 on OS X Leopard:

$ python
Python 2.5.1 (r251:54863, Feb 4 2008, 21:48:13)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin

I just tried it out on Python 2.4.2 on an Ubuntu machine and it worked
fine! I guess this must be an OS X Python bug. How absolutely
infuriating.

Thanks,

Simon
Jul 30 '08 #4
On Jul 30, 4:59*pm, Simon Willison <si...@simonwil lison.netwrote:
I just tried it out on Python 2.4.2 on an Ubuntu machine and it worked
fine! I guess this must be an OS X Python bug. How absolutely
infuriating.
Some very useful people in #python on Freenode pointed out that my bug
occurs because I'm trying to display things interactively in the
console. Saving to a variable instead fixes the problem.

Thanks for your help,

Simon
Jul 30 '08 #5
On 30 Jul, 18:17, Simon Willison <si...@simonwil lison.netwrote:
>
Some very useful people in #python on Freenode pointed out that my bug
occurs because I'm trying to display things interactively in the
console. Saving to a variable instead fixes the problem.
What's strange about that is how the object is represented when
displayed:

('CHARACTERS', <DOM Text node "Simon\u201 9s XM...">)

Here, there's no attempt made to encode \u2019 as an ASCII byte
sequence. Does the OS X version of Python do anything special with
string representations ?

Paul
Jul 30 '08 #6
Paul Boddie wrote:
On 30 Jul, 18:17, Simon Willison <si...@simonwil lison.netwrote:
>>
Some very useful people in #python on Freenode pointed out that my bug
occurs because I'm trying to display things interactively in the
console. Saving to a variable instead fixes the problem.

What's strange about that is how the object is represented when
displayed:

('CHARACTERS', <DOM Text node "Simon\u201 9s XM...">)

Here, there's no attempt made to encode \u2019 as an ASCII byte
sequence. Does the OS X version of Python do anything special with
string representations ?
I'm on Kubuntu 7.10 and see the same error as Simon. The problem is in the
minidom.Charact erData class which has the following method

def __repr__(self):
data = self.data
if len(data) 10:
dotdotdot = "..."
else:
dotdotdot = ""
return "<DOM %s node \"%s%s\">" % (
self.__class__. __name__, data[0:10], dotdotdot)

The data attribute is a unicode instance...

Peter
Jul 30 '08 #7
Simon Willison wrote:
Follow up question: what's the best way of incrementally consuming XML
in Python that's character encoding aware?
iterparse(), as implemented in (c)ElementTree and lxml. Note that ElementTree
and cElementTree are part of Python 2.5, in the xml.etree package.

I have a very large file to
consume but I'd rather not have to fall back to the raw SAX API.
Large is fairly relative. Both cElementTree and lxml are pretty memory
friendly, even when parsing into an in-memory tree.

Stefan
Jul 30 '08 #8
On 30 Jul, 19:23, Peter Otten <__pete...@web. dewrote:
>
I'm on Kubuntu 7.10 and see the same error as Simon. The problem is in the
minidom.Charact erData class which has the following method

* * def __repr__(self):
* * * * data = self.data
* * * * if len(data) 10:
* * * * * * dotdotdot = "..."
* * * * else:
* * * * * * dotdotdot = ""
* * * * return "<DOM %s node \"%s%s\">" % (
* * * * * * self.__class__. __name__, data[0:10], dotdotdot)

The data attribute is a unicode instance...
Who wants to be first to submit a patch? ;-)

Paul
Jul 30 '08 #9
Paul Boddie wrote:
Who wants to be first to submit a patch? ;-)
And where? The sourceforge page says

"PyXML is no longer maintained."

Peter
Jul 30 '08 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
1887
by: Joshua Beall | last post by:
Hi All, I have been using the SAX library in PHP to parse XHTML documents, and one thing I have noted is that the <!DOCTYPE> line is ignored. I am wondering is there any way to get the <!DOCTYPE> using the SAX functions in PHP? I am looking over the manual, but nothing is jumping out at me... reference:...
4
1708
by: David Pinto | last post by:
I'm trying to use either the minidom or pulldom to find table tags in html web pages. I've tried parsing two web pages that show up fine in my browser, but I get errors when I call minidom.parse, or try to get events with pulldom. Is there a parser that is as forgiving as web browsers?
6
18313
by: Spamtrap | last post by:
I only work in Perl occasionaly, and have been searching for a solution for a conversion, and everything I found seems much too complex. All I need to do is take a simple text file and copy it, however some specific lines are in fact in UTF8 as printed garbagy characters and they need to be converted to Unicode, so that the new text file...
8
4276
by: jog | last post by:
Hi, I want to get text out of some nodes of a huge xml file (1,5 GB). The architecture of the xml file is something like this <parent> <page> <title>bla</title> <id></id> <revision> <id></id> <text>blablabla</text>
1
2326
by: Srini | last post by:
Hi, I am working on a project and a portion of which involves receiving xml files on internet, extract values to build a string and pass that string to legacy system. I am planning on using XMLReader to read passed xml file and extract values that need to be mapped to string buffer. However, in this method I am not able to get the full...
7
4451
by: EmeraldShield | last post by:
We have an application that uses UTF8 everywhere to load / save / process documents. One of our clients is having a problem with BIG Encoded files being trashed after running through our app. Indeed I have verified that if I go to a website in Taiwan and save the file in BIG5 and then just load / save the file with a UTF8 text reader /...
0
999
by: sun | last post by:
I have a large xml file parsed by pulldom. I did some editing on some node,change attributes and remove some child node, how do I save it back to this xml file or write a new xml file? The save method in minidom does not work for pulldom. Thanks
0
991
by: Tim Golden | last post by:
Lawrence, Anna K (US SSA) wrote: From where I'm sitting, I can't see enough to help. The crucial thing seems to be in this phrase: "When I try to take a string that is coming out of the database and export it to a Word document, no amount of encoding/decoding will make the Word doc display properly". It presumably doesn't matter what web...
0
792
by: susan_ali | last post by:
I'm using xml.dom.pulldom to parse through an XML file. I use expandNode() to scrutinize certain blocks of it that I'm interested in. Once I find a block of XML in the input file that I'm interested in, I need to add my own block <MyTag>.....</MyTagto the pulldom tree I'm building in memory. The documentation on PullDom is worse than...
0
7424
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7607
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
7772
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7376
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
7709
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
3415
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
1
1841
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
988
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
661
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.