469,658 Members | 1,803 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,658 developers. It's quick & easy.

iterparse and unicode

It seems xml.etree.cElementTree.iterparse() is not unicode aware:
>>from StringIO import StringIO
from xml.etree.cElementTree import iterparse
s = u'<name>\u03a0\u03b1\u03bd\u03b1\u03b3\u03b9\u03ce \u03c4\u03b7\u03c2</name>'
for event,elem in iterparse(StringIO(s)):
.... print elem.text
....
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeError: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?

George
Aug 20 '08 #1
13 4597
On Aug 21, 8:36 am, George Sakkis <george.sak...@gmail.comwrote:
It seems xml.etree.cElementTree.iterparse() is not unicode aware:
>from StringIO import StringIO
from xml.etree.cElementTree import iterparse
s = u'<name>\u03a0\u03b1\u03bd\u03b1\u03b3\u03b9\u03ce \u03c4\u03b7\u03c2</name>'
for event,elem in iterparse(StringIO(s)):

... print elem.text
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeError: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?
Hi George,
I'm no XML guru by any means but as far as I understand it, you would
need to encode your text into UTF-8, and prepend something like '<?xml
version="1.0" encoding="UTF-8" standalone="yes"?>' to it. This appears
to be the way XML is, rather than an ElementTree problem.

E.g.
>>from StringIO import StringIO
from xml.etree.cElementTree import iterparse
s = u'<wrapper><name>\u03a0\u03b1</name><digits>01234567</digits></wrapper>'
>>h = '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
xml = h + s.encode('utf8')
for event,elem in iterparse(StringIO(xml)):
.... print elem.tag, repr(elem.text)
....
name u'\u03a0\u03b1'
digits '01234567'
wrapper None
>>>
HTH,
John
Aug 20 '08 #2
On Wed, 2008-08-20 at 15:36 -0700, George Sakkis wrote:
It seems xml.etree.cElementTree.iterparse() is not unicode aware:
>from StringIO import StringIO
from xml.etree.cElementTree import iterparse
s = u'<name>\u03a0\u03b1\u03bd\u03b1\u03b3\u03b9\u03ce \u03c4\u03b7\u03c2</name>'
for event,elem in iterparse(StringIO(s)):
... print elem.text
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeError: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?

George
--
http://mail.python.org/mailman/listinfo/python-list
As iterparse expects an actual file as input, using a unicode string is
problematic. If you want to use iterparse, the simplest way would be to
encode your string before inserting it into the StringIO object, as so:

>>for event,elem in iterparse(StringIO(s.encode('UTF8')):
.... print elem.text
....

If you encode using UTF-8, you don't need to worry about the <?xml header
bit as suggested previously, as it's the default for XML.

If you're using unicode extensively, you should consider using lxml,
which implements the same interface as ElementTree, but handles unicode
better (though it also doesn't run your example above without first
encoding the string):
http://codespeak.net/lxml/parsing.ht...nicode-strings

You may also find the target parser interface to be more accepting of
unicode than iterparse, though it requires a different parsing interface:
http://codespeak.net/lxml/parsing.ht...rser-interface

--
John Krukoff <jk******@ltgc.com>
Land Title Guarantee Company

Aug 21 '08 #3
Thank you both for the suggestions. I made a few more experiments to
understand how iterparse behaves with respect to three dimensions:

a. Is the encoding declared in the header (if there is one) ?
b. Is the text ascii-encodable (i.e. within range(128)) ?
c. Does the passed file object's read() method return str or unicode
(e.g. codecs.open(f,encoding='utf8')) ?

Feel free to correct me if I misinterpreted what is really happening.

As John Krukoff mentioned, omitting the encoding is equivalent to
encoding="utf-8" for all other combinations. This leaves (b) and (c).

If a text node is ascii-encodable, iterparse() returns it as a byte
string, regardless of the declared encoding and the input file's
read() return type.

(c) becomes relevant only if a text node is not ascii-encodable. In
this case iterparse() returns unicode if the underlying file's read()
returns bytes in an encoding that matches (or at least is compatible
with) the declared encoding in the header (or the implied utf8).
Passing a file object whose read() returns unicode characters
implicitly encodes them to ascii, which raises a UnicodeEncodeError
since the text node is not ascii-encodable.

It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.

George
Aug 21 '08 #4
George Sakkis wrote:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeError: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?
iterparse parses XML documents. XML documents are streams of encoded
characters, not streams of Unicode characters.

</F>

Aug 21 '08 #5
George Sakkis wrote:
Thank you both for the suggestions. I made a few more experiments to
understand how iterparse behaves with respect to three dimensions:
Spending time researching undefined behaviour is pretty pointless. ET
parsers expect byte streams, because that's what XML files are. If you
pass it anything else, an ET implementation may attempt to convert that
thing to a byte string, run the game "rogue", or do something else that
it finds appropriate.
It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.
If you don't care about memory and execution performance, there are
plenty of toolkits that guarantee that you always get Unicode strings.

</F>

Aug 21 '08 #6
On Aug 21, 1:48*am, Fredrik Lundh <fred...@pythonware.comwrote:
George Sakkis wrote:
It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from *BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.

If you don't care about memory and execution performance, there are
plenty of toolkits that guarantee that you always get Unicode strings.
As long as they are documented, both approaches are fine for different
cases. Currently the only reference I found about unicode in
ElementTree is "All strings can either be Unicode strings, or 8-bit
strings containing US-ASCII only." [1], which is rather ambiguous; at
least I read it as "all strings are Unicode or all strings are 8-bit
strings", not a potentially mix of both in the same tree.

Regards,
George

[1] http://effbot.org/zone/element.htm
Aug 21 '08 #7
George Sakkis wrote:
On Aug 21, 1:48 am, Fredrik Lundh <fred...@pythonware.comwrote:
>George Sakkis wrote:
>>It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.
If you don't care about memory and execution performance, there are
plenty of toolkits that guarantee that you always get Unicode strings.

As long as they are documented, both approaches are fine for different
cases. Currently the only reference I found about unicode in
ElementTree is "All strings can either be Unicode strings, or 8-bit
strings containing US-ASCII only." [1], which is rather ambiguous
It's not ambiguous in Py2.x, where ASCII byte strings and unicode strings are
compatible. No need to feel "uneasy". :)

Stefan
Aug 24 '08 #8
George Sakkis wrote:
It seems xml.etree.cElementTree.iterparse() is not unicode aware:
>>>from StringIO import StringIO
from xml.etree.cElementTree import iterparse
s = u'<name>\u03a0\u03b1\u03bd\u03b1\u03b3\u03b9\u03ce \u03c4\u03b7\u03c2</name>'
for event,elem in iterparse(StringIO(s)):
... print elem.text
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeError: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?
If you want to parse XML from Python unicode strings, you can use lxml.etree.
The XML specification allows transport protocols and other sources to provide
external encoding information. lxml supports the Python unicode type as a
transport and reads the internal byte sequence of the unicode string.

To be clear, this does not mean that the parsing happens at the unicode
character level. Parsing XML is about parsing bytes, not characters.

Stefan
Aug 24 '08 #9
On Aug 24, 1:12 am, Stefan Behnel <stefan...@behnel.dewrote:
George Sakkis wrote:
On Aug 21, 1:48 am, Fredrik Lundh <fred...@pythonware.comwrote:
George Sakkis wrote:
It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.
If you don't care about memory and execution performance, there are
plenty of toolkits that guarantee that you always get Unicode strings.
As long as they are documented, both approaches are fine for different
cases. Currently the only reference I found about unicode in
ElementTree is "All strings can either be Unicode strings, or 8-bit
strings containing US-ASCII only." [1], which is rather ambiguous

It's not ambiguous in Py2.x, where ASCII byte strings and unicode strings are
compatible. No need to feel "uneasy". :)
It depends on what you mean by "compatible"; e.g. you can't safely do
[s.decode('utf8') for s in strings] if you have byte strings mixed
with unicode.

George
Aug 25 '08 #10
George Sakkis wrote:
It depends on what you mean by "compatible"; e.g. you can't safely do
[s.decode('utf8') for s in strings] if you have byte strings mixed
with unicode.
why would you want to decode strings given to you by a library that
returns decoded strings?

if you meant to write "encode", you can indeed safely do
[s.encode('utf8') for s in strings] as long as all strings are returned
by an ET implementation.

</F>

Aug 25 '08 #11
On Aug 25, 4:45*pm, Fredrik Lundh <fred...@pythonware.comwrote:
George Sakkis wrote:
It depends on what you mean by "compatible"; e.g. you can't safely do
[s.decode('utf8') for s in strings] if you have byte strings mixed
with unicode.

why would you want to decode strings given to you by a library that
returns decoded strings?

if you meant to write "encode", you can indeed safely do
[s.encode('utf8') for s in strings] as long as all strings are returned
by an ET implementation.
I was replying to the general assertion that "in 2.x ASCII byte
strings and unicode strings are compatible", not specifically about
the strings returned by ET.

George
Aug 27 '08 #12
George Sakkis wrote:
>if you meant to write "encode", you can indeed safely do
[s.encode('utf8') for s in strings] as long as all strings are returned
by an ET implementation.

I was replying to the general assertion that "in 2.x ASCII byte
strings and unicode strings are compatible", not specifically about
the strings returned by ET.
that assertion was made in the context of ET. having to unilaterially
change the topic to "win" an argument is pretty lame.

and if you really meant to write "decode", you picked a rather stupid
example to support your complaint about ET not returning Unicode -- your
example does work fine for byte strings (whether they contain pure ASCII
or not), but doesn't work at all for arbitrary Unicode strings, because
decoding things that are already decoded makes very little sense (which
explains why that method was removed in 3.0).
>>"hello".decode("utf-8")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute 'decode'

are you sure you understand the distinction between Unicode strings and
encoded strings?

</F>

Aug 27 '08 #13
On Aug 27, 5:42 am, Fredrik Lundh <fred...@pythonware.comwrote:
George Sakkis wrote:
if you meant to write "encode", you can indeed safely do
[s.encode('utf8') for s in strings] as long as all strings are returned
by an ET implementation.
I was replying to the general assertion that "in 2.x ASCII byte
strings and unicode strings are compatible", not specifically about
the strings returned by ET.

that assertion was made in the context of ET. having to unilaterially
change the topic to "win" an argument is pretty lame.
I took Stefan's comment as a general statement, not in the context of
ET. Feeling the need to keep "defending" ET at this point is, to
borrow your words, pretty lame.
and if you really meant to write "decode", you picked a rather stupid
example to support your complaint about ET not returning Unicode -- your
example does work fine for byte strings (whether they contain pure ASCII
or not), but doesn't work at all for arbitrary Unicode strings, because
decoding things that are already decoded makes very little sense (which
explains why that method was removed in 3.0).
The thing is, a user might be happily using ET and call "decode" on
the returned byte strings as long as the files happen to be all ASCII,
without knowing that ET may also return Unicode. Then after some weeks/
months/years a non-ASCII file comes in and the program breaks. As far
as I am concerned, it's a documentation issue, nothing more.

George
Aug 27 '08 #14

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

3 posts views Thread by Michael Weir | last post: by
8 posts views Thread by Bill Eldridge | last post: by
8 posts views Thread by Francis Girard | last post: by
4 posts views Thread by webdev | last post: by
4 posts views Thread by paul.sherwood | last post: by
2 posts views Thread by Neil Schemenauer | last post: by
24 posts views Thread by ChaosKCW | last post: by
2 posts views Thread by Stuart McGraw | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.