I'm having a horrible time trying to get xml.dom.pulldom to consume a
UTF8 encoded XML file. Here's what I've tried so far:
>>xml_utf8 = """<?xml version="1.0" encoding="UTF-8" ?>
<msg>Simon\xe2\ x80\x99s XML nightmare</msg>
"""
>>from xml.dom import pulldom parser = pulldom.parseSt ring(xml_utf8) parser.next ()
('START_DOCUMEN T', <xml.dom.minido m.Document instance at 0x6f06c0>)
>>parser.next ()
('START_ELEMENT ', <DOM Element: msg at 0x6f0710>)
>>parser.next ()
....
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\u2019' in
position 21: ordinal not in range(128)
xml.dom.minidom can handle the string just fine:
>>from xml.dom import minidom dom = minidom.parseSt ring(xml_utf8) dom.toxml()
u'<?xml version="1.0" ?><msg>Simon\u2 019s XML nightmare</msg>'
If I pass a unicode string to pulldom instead of a utf8 encoded
bytestring it still breaks:
>>xml_unicode = u'<?xml version="1.0" ?><msg>Simon\u2 019s XML nightmare</msg>' parser = pulldom.parseSt ring(xml_unicod e)
....
/System/Library/Frameworks/Python.framewor k/Versions/2.5/lib/python2.5/
xml/dom/pulldom.py in parseString(str ing, parser)
346
347 bufsize = len(string)
--348 buf = StringIO(string )
349 if not parser:
350 parser = xml.sax.make_pa rser()
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\u2019' in
position 32: ordinal not in range(128)
Is it possible to consume utf8 or unicode using xml.dom.pulldom or
should I try something else?
Thanks,
Simon Willison 10 2226
Follow up question: what's the best way of incrementally consuming XML
in Python that's character encoding aware? I have a very large file to
consume but I'd rather not have to fall back to the raw SAX API.
On 30 Jul, 16:32, Simon Willison <si...@simonwil lison.netwrote:
I'm having a horrible time trying to get xml.dom.pulldom to consume a
UTF8 encoded XML file. Here's what I've tried so far:
>xml_utf8 = """<?xml version="1.0" encoding="UTF-8" ?>
<msg>Simon\xe2\ x80\x99s XML nightmare</msg>
""">>from xml.dom import pulldom
>parser = pulldom.parseSt ring(xml_utf8) parser.next( )
('START_DOCUMEN T', <xml.dom.minido m.Document instance at 0x6f06c0>)>>par ser.next()
('START_ELEMENT ', <DOM Element: msg at 0x6f0710>)>>par ser.next()
...
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\u2019' in
position 21: ordinal not in range(128)
I can't reproduce this on Python 2.3.6 or 2.4.4 on RHEL 4. Instead, I
get the usual...
('CHARACTERS', <DOM Text node "Simon\u201 9s XM...">)
And I can get the content of the text node as a proper Unicode object.
[...]
Is it possible to consume utf8 or unicode using xml.dom.pulldom or
should I try something else?
Yes, it is possible, at least in Python 2.3.6 and 2.4.4 configured
with --enable-unicode=ucs4 (which is what Red Hat does and expects).
Paul
P.S. You shouldn't try and pass Unicode to the parser, since XML
parsing in its entirety deals with byte sequences and character
encodings, although I suppose that there's some kind of character-
based (ie. Unicode value-based) parsing method defined somewhere by
some committee or other.
On Jul 30, 4:43*pm, Paul Boddie <p...@boddie.or g.ukwrote:
I can't reproduce this on Python 2.3.6 or 2.4.4 on RHEL 4. Instead, I
get the usual...
('CHARACTERS', <DOM Text node "Simon\u201 9s XM...">)
I'm using Python 2.5.1 on OS X Leopard:
$ python
Python 2.5.1 (r251:54863, Feb 4 2008, 21:48:13)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
I just tried it out on Python 2.4.2 on an Ubuntu machine and it worked
fine! I guess this must be an OS X Python bug. How absolutely
infuriating.
Thanks,
Simon
On Jul 30, 4:59*pm, Simon Willison <si...@simonwil lison.netwrote:
I just tried it out on Python 2.4.2 on an Ubuntu machine and it worked
fine! I guess this must be an OS X Python bug. How absolutely
infuriating.
Some very useful people in #python on Freenode pointed out that my bug
occurs because I'm trying to display things interactively in the
console. Saving to a variable instead fixes the problem.
Thanks for your help,
Simon
On 30 Jul, 18:17, Simon Willison <si...@simonwil lison.netwrote:
>
Some very useful people in #python on Freenode pointed out that my bug
occurs because I'm trying to display things interactively in the
console. Saving to a variable instead fixes the problem.
What's strange about that is how the object is represented when
displayed:
('CHARACTERS', <DOM Text node "Simon\u201 9s XM...">)
Here, there's no attempt made to encode \u2019 as an ASCII byte
sequence. Does the OS X version of Python do anything special with
string representations ?
Paul
Paul Boddie wrote:
On 30 Jul, 18:17, Simon Willison <si...@simonwil lison.netwrote:
>> Some very useful people in #python on Freenode pointed out that my bug occurs because I'm trying to display things interactively in the console. Saving to a variable instead fixes the problem.
What's strange about that is how the object is represented when
displayed:
('CHARACTERS', <DOM Text node "Simon\u201 9s XM...">)
Here, there's no attempt made to encode \u2019 as an ASCII byte
sequence. Does the OS X version of Python do anything special with
string representations ?
I'm on Kubuntu 7.10 and see the same error as Simon. The problem is in the
minidom.Charact erData class which has the following method
def __repr__(self):
data = self.data
if len(data) 10:
dotdotdot = "..."
else:
dotdotdot = ""
return "<DOM %s node \"%s%s\">" % (
self.__class__. __name__, data[0:10], dotdotdot)
The data attribute is a unicode instance...
Peter
Simon Willison wrote:
Follow up question: what's the best way of incrementally consuming XML
in Python that's character encoding aware?
iterparse(), as implemented in (c)ElementTree and lxml. Note that ElementTree
and cElementTree are part of Python 2.5, in the xml.etree package.
I have a very large file to
consume but I'd rather not have to fall back to the raw SAX API.
Large is fairly relative. Both cElementTree and lxml are pretty memory
friendly, even when parsing into an in-memory tree.
Stefan
On 30 Jul, 19:23, Peter Otten <__pete...@web. dewrote:
>
I'm on Kubuntu 7.10 and see the same error as Simon. The problem is in the
minidom.Charact erData class which has the following method
* * def __repr__(self):
* * * * data = self.data
* * * * if len(data) 10:
* * * * * * dotdotdot = "..."
* * * * else:
* * * * * * dotdotdot = ""
* * * * return "<DOM %s node \"%s%s\">" % (
* * * * * * self.__class__. __name__, data[0:10], dotdotdot)
The data attribute is a unicode instance...
Who wants to be first to submit a patch? ;-)
Paul
Paul Boddie wrote:
Who wants to be first to submit a patch? ;-)
And where? The sourceforge page says
"PyXML is no longer maintained."
Peter This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Joshua Beall |
last post by:
Hi All,
I have been using the SAX library in PHP to parse XHTML documents, and one
thing I have noted is that the <!DOCTYPE> line is ignored.
I am wondering is there any way to get the <!DOCTYPE> using the SAX
functions in PHP? I am looking over the manual, but nothing is jumping out
at me...
reference: http://us4.php.net/manual/en/ref.xml.php
|
by: David Pinto |
last post by:
I'm trying to use either the minidom or pulldom to find table tags in
html web pages. I've tried parsing two web pages that show up fine in
my browser, but I get errors when I call minidom.parse, or try to get
events with pulldom. Is there a parser that is as forgiving as web
browsers?
|
by: Spamtrap |
last post by:
I only work in Perl occasionaly, and have been searching for a
solution for a conversion, and everything I found seems much too
complex.
All I need to do is take a simple text file and copy it, however some specific lines are in
fact in UTF8 as printed garbagy characters and they need to be
converted to Unicode, so that the new text file can be imported into a
desktop program and into some Word documents.
For the moment I would be...
|
by: jog |
last post by:
Hi,
I want to get text out of some nodes of a huge xml file (1,5 GB). The
architecture of the xml file is something like this
<parent>
<page>
<title>bla</title>
<id></id>
<revision>
<id></id>
<text>blablabla</text>
|
by: Srini |
last post by:
Hi,
I am working on a project and a portion of which involves
receiving xml files on internet, extract values to build a
string and pass that string to legacy system.
I am planning on using XMLReader to read passed xml file
and extract values that need to be mapped to string
buffer. However, in this method I am not able to get the
full path (ie., path information from root to this node)
| |
by: EmeraldShield |
last post by:
We have an application that uses UTF8 everywhere to load / save / process
documents.
One of our clients is having a problem with BIG Encoded files being trashed
after running through our app.
Indeed I have verified that if I go to a website in Taiwan and save the file
in BIG5 and then just load / save the file with a UTF8 text reader / write
some bytes are modified.
How can I correct this? It was my understanding the UTF8 was...
|
by: sun |
last post by:
I have a large xml file parsed by pulldom. I did some editing on some
node,change attributes and remove some child node, how do I save it back
to this xml file or write a new xml file? The save method in minidom
does not work for pulldom.
Thanks
|
by: Tim Golden |
last post by:
Lawrence, Anna K (US SSA) wrote:
From where I'm sitting, I can't see enough to help. The crucial
thing seems to be in this phrase: "When I try to take a string that
is coming out of the database and export it to a Word document,
no amount of encoding/decoding will make the Word doc display properly".
It presumably doesn't matter what web framework etc. you're using:
you either get a unicode object or an encoded string from the database...
|
by: susan_ali |
last post by:
I'm using xml.dom.pulldom to parse through an XML file. I use
expandNode() to scrutinize certain blocks of it that I'm interested
in.
Once I find a block of XML in the input file that I'm interested in, I
need to add my own block <MyTag>.....</MyTagto the pulldom tree I'm
building in memory.
The documentation on PullDom is worse than atrocious. It is simply non-
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
| |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |