473,561 Members | 3,118 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

iterparse and unicode

It seems xml.etree.cElem entTree.iterpar se() is not unicode aware:
>>from StringIO import StringIO
from xml.etree.cElem entTree import iterparse
s = u'<name>\u03a0\ u03b1\u03bd\u03 b1\u03b3\u03b9\ u03ce\u03c4\u03 b7\u03c2</name>'
for event,elem in iterparse(Strin gIO(s)):
.... print elem.text
....
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeEr ror: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?

George
Aug 20 '08 #1
13 5043
On Aug 21, 8:36 am, George Sakkis <george.sak...@ gmail.comwrote:
It seems xml.etree.cElem entTree.iterpar se() is not unicode aware:
>from StringIO import StringIO
from xml.etree.cElem entTree import iterparse
s = u'<name>\u03a0\ u03b1\u03bd\u03 b1\u03b3\u03b9\ u03ce\u03c4\u03 b7\u03c2</name>'
for event,elem in iterparse(Strin gIO(s)):

... print elem.text
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeEr ror: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?
Hi George,
I'm no XML guru by any means but as far as I understand it, you would
need to encode your text into UTF-8, and prepend something like '<?xml
version="1.0" encoding="UTF-8" standalone="yes "?>' to it. This appears
to be the way XML is, rather than an ElementTree problem.

E.g.
>>from StringIO import StringIO
from xml.etree.cElem entTree import iterparse
s = u'<wrapper><nam e>\u03a0\u03b1 </name><digits>01 234567</digits></wrapper>'
>>h = '<?xml version="1.0" encoding="UTF-8" standalone="yes "?>'
xml = h + s.encode('utf8' )
for event,elem in iterparse(Strin gIO(xml)):
.... print elem.tag, repr(elem.text)
....
name u'\u03a0\u03b1'
digits '01234567'
wrapper None
>>>
HTH,
John
Aug 20 '08 #2
On Wed, 2008-08-20 at 15:36 -0700, George Sakkis wrote:
It seems xml.etree.cElem entTree.iterpar se() is not unicode aware:
>from StringIO import StringIO
from xml.etree.cElem entTree import iterparse
s = u'<name>\u03a0\ u03b1\u03bd\u03 b1\u03b3\u03b9\ u03ce\u03c4\u03 b7\u03c2</name>'
for event,elem in iterparse(Strin gIO(s)):
... print elem.text
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeEr ror: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?

George
--
http://mail.python.org/mailman/listinfo/python-list
As iterparse expects an actual file as input, using a unicode string is
problematic. If you want to use iterparse, the simplest way would be to
encode your string before inserting it into the StringIO object, as so:

>>for event,elem in iterparse(Strin gIO(s.encode('U TF8')):
.... print elem.text
....

If you encode using UTF-8, you don't need to worry about the <?xml header
bit as suggested previously, as it's the default for XML.

If you're using unicode extensively, you should consider using lxml,
which implements the same interface as ElementTree, but handles unicode
better (though it also doesn't run your example above without first
encoding the string):
http://codespeak.net/lxml/parsing.ht...nicode-strings

You may also find the target parser interface to be more accepting of
unicode than iterparse, though it requires a different parsing interface:
http://codespeak.net/lxml/parsing.ht...rser-interface

--
John Krukoff <jk******@ltgc. com>
Land Title Guarantee Company

Aug 21 '08 #3
Thank you both for the suggestions. I made a few more experiments to
understand how iterparse behaves with respect to three dimensions:

a. Is the encoding declared in the header (if there is one) ?
b. Is the text ascii-encodable (i.e. within range(128)) ?
c. Does the passed file object's read() method return str or unicode
(e.g. codecs.open(f,e ncoding='utf8') ) ?

Feel free to correct me if I misinterpreted what is really happening.

As John Krukoff mentioned, omitting the encoding is equivalent to
encoding="utf-8" for all other combinations. This leaves (b) and (c).

If a text node is ascii-encodable, iterparse() returns it as a byte
string, regardless of the declared encoding and the input file's
read() return type.

(c) becomes relevant only if a text node is not ascii-encodable. In
this case iterparse() returns unicode if the underlying file's read()
returns bytes in an encoding that matches (or at least is compatible
with) the declared encoding in the header (or the implied utf8).
Passing a file object whose read() returns unicode characters
implicitly encodes them to ascii, which raises a UnicodeEncodeEr ror
since the text node is not ascii-encodable.

It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.

George
Aug 21 '08 #4
George Sakkis wrote:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeEr ror: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?
iterparse parses XML documents. XML documents are streams of encoded
characters, not streams of Unicode characters.

</F>

Aug 21 '08 #5
George Sakkis wrote:
Thank you both for the suggestions. I made a few more experiments to
understand how iterparse behaves with respect to three dimensions:
Spending time researching undefined behaviour is pretty pointless. ET
parsers expect byte streams, because that's what XML files are. If you
pass it anything else, an ET implementation may attempt to convert that
thing to a byte string, run the game "rogue", or do something else that
it finds appropriate.
It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.
If you don't care about memory and execution performance, there are
plenty of toolkits that guarantee that you always get Unicode strings.

</F>

Aug 21 '08 #6
On Aug 21, 1:48*am, Fredrik Lundh <fred...@python ware.comwrote:
George Sakkis wrote:
It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from *BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.

If you don't care about memory and execution performance, there are
plenty of toolkits that guarantee that you always get Unicode strings.
As long as they are documented, both approaches are fine for different
cases. Currently the only reference I found about unicode in
ElementTree is "All strings can either be Unicode strings, or 8-bit
strings containing US-ASCII only." [1], which is rather ambiguous; at
least I read it as "all strings are Unicode or all strings are 8-bit
strings", not a potentially mix of both in the same tree.

Regards,
George

[1] http://effbot.org/zone/element.htm
Aug 21 '08 #7
George Sakkis wrote:
On Aug 21, 1:48 am, Fredrik Lundh <fred...@python ware.comwrote:
>George Sakkis wrote:
>>It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.
If you don't care about memory and execution performance, there are
plenty of toolkits that guarantee that you always get Unicode strings.

As long as they are documented, both approaches are fine for different
cases. Currently the only reference I found about unicode in
ElementTree is "All strings can either be Unicode strings, or 8-bit
strings containing US-ASCII only." [1], which is rather ambiguous
It's not ambiguous in Py2.x, where ASCII byte strings and unicode strings are
compatible. No need to feel "uneasy". :)

Stefan
Aug 24 '08 #8
George Sakkis wrote:
It seems xml.etree.cElem entTree.iterpar se() is not unicode aware:
>>>from StringIO import StringIO
from xml.etree.cElem entTree import iterparse
s = u'<name>\u03a0\ u03b1\u03bd\u03 b1\u03b3\u03b9\ u03ce\u03c4\u03 b7\u03c2</name>'
for event,elem in iterparse(Strin gIO(s)):
... print elem.text
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeEr ror: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?
If you want to parse XML from Python unicode strings, you can use lxml.etree.
The XML specification allows transport protocols and other sources to provide
external encoding information. lxml supports the Python unicode type as a
transport and reads the internal byte sequence of the unicode string.

To be clear, this does not mean that the parsing happens at the unicode
character level. Parsing XML is about parsing bytes, not characters.

Stefan
Aug 24 '08 #9
On Aug 24, 1:12 am, Stefan Behnel <stefan...@behn el.dewrote:
George Sakkis wrote:
On Aug 21, 1:48 am, Fredrik Lundh <fred...@python ware.comwrote:
George Sakkis wrote:
It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.
If you don't care about memory and execution performance, there are
plenty of toolkits that guarantee that you always get Unicode strings.
As long as they are documented, both approaches are fine for different
cases. Currently the only reference I found about unicode in
ElementTree is "All strings can either be Unicode strings, or 8-bit
strings containing US-ASCII only." [1], which is rather ambiguous

It's not ambiguous in Py2.x, where ASCII byte strings and unicode strings are
compatible. No need to feel "uneasy". :)
It depends on what you mean by "compatible "; e.g. you can't safely do
[s.decode('utf8' ) for s in strings] if you have byte strings mixed
with unicode.

George
Aug 25 '08 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
17600
by: Michael Weir | last post by:
I'm sure this is a very simple thing to do, once you know how to do it, but I am having no fun at all trying to write utf-8 strings to a unicode file. Does anyone have a couple of lines of code that - opens a file appropriately for output - writes to this file Thanks very much. Michael Weir
8
5255
by: Bill Eldridge | last post by:
I'm trying to grab a document off the Web and toss it into a MySQL database, but I keep running into the various encoding problems with Unicode (that aren't a problem for me with GB2312, BIG 5, etc.) What I'd like is something as simple as: CREATE TABLE junk (junklet VARCHAR(2500) CHARACTER SET UTF8)); import MySQLdb, re,urllib
8
3644
by: Francis Girard | last post by:
Hi, For the first time in my programmer life, I have to take care of character encoding. I have a question about the BOM marks. If I understand well, into the UTF-8 unicode binary representation, some systems add at the beginning of the file a BOM mark (Windows?), some don't. (Linux?). Therefore, the exact same text encoded in the same...
48
4593
by: Zenobia | last post by:
Recently I was editing a document in GoLive 6. I like GoLive because it has some nice features such as: * rewrite source code * check syntax * global search & replace (through several files at once) * regular expression search & replace. Normally my documents are encoded with the ISO setting. Recently I was writing an XHTML...
4
6049
by: webdev | last post by:
lo all, some of the questions i'll ask below have most certainly been discussed already, i just hope someone's kind enough to answer them again to help me out.. so i started a python 2.3 script that grabs some web pages from the web, regex parse the data and stores it localy to xml file for further use.. at first i had no problem using...
4
4647
by: paul.sherwood | last post by:
Hi Im trying to parse a large(150MB) xml file in order to extract specific required records. import sys from elementtree.ElementTree import ElementTree root = ElementTree(file=big.xml')
2
2613
by: Neil Schemenauer | last post by:
python-dev@python.org.] The PEP has been rewritten based on a suggestion by Guido to change str() rather than adding a new built-in function. Based on my testing, I believe the idea is feasible. It would be helpful if people could test the patched Python with their own applications and report any incompatibilities. PEP: 349
24
9018
by: ChaosKCW | last post by:
Hi I am reading from an oracle database using cx_Oracle. I am writing to a SQLite database using apsw. The oracle database is returning utf-8 characters for euopean item names, ie special charcaters from an ASCII perspective. I get the following error: > SQLiteCur.execute(sql, row)
2
3761
by: Stuart McGraw | last post by:
I have a broad (~200K nodes) but shallow xml file I want to parse with Elementtree. There are too many nodes to read into memory simultaneously so I use iterparse() to process each node sequentially. Now I find i need to get and save the input file line number of each node. Googling turned up a way to do it by subclassing...
0
7558
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7851
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8072
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7605
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
6195
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5462
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5177
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3591
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
1171
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.