iterparse and unicode

[George Sakkis]

It seems xml.etree.cElementTree.iterparse() is not unicode aware:

>>from StringIO import StringIO
from xml.etree.cElementTree import iterparse
s = u'<name>\u03a0\u03b1\u03bd\u03b1\u03b3\u03b9\u03ce \u03c4\u03b7\u03c2</name>'
for event,elem in iterparse(StringIO(s)):

.... print elem.text
....
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeError: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?

George

[John Machin]

On Aug 21, 8:36 am, George Sakkis <[e-mail]:

It seems xml.etree.cElementTree.iterparse() is not unicode aware:

>from StringIO import StringIO
from xml.etree.cElementTree import iterparse
s = u'<name>\u03a0\u03b1\u03bd\u03b1\u03b3\u03b9\u03ce \u03c4\u03b7\u03c2</name>'
for event,elem in iterparse(StringIO(s)):

... print elem.text
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeError: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?

Hi George,
I'm no XML guru by any means but as far as I understand it, you would
need to encode your text into UTF-8, and prepend something like '<?xml
version="1.0" encoding="UTF-8" standalone="yes"?>' to it. This appears
to be the way XML is, rather than an ElementTree problem.

E.g.

>>from StringIO import StringIO
from xml.etree.cElementTree import iterparse
s = u'<wrapper><name>\u03a0\u03b1</name><digits>01234567</digits></wrapper>'

>>h = '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
xml = h + s.encode('utf8')
for event,elem in iterparse(StringIO(xml)):

.... print elem.tag, repr(elem.text)
....
name u'\u03a0\u03b1'
digits '01234567'
wrapper None

>>>

HTH,
John

[John Krukoff]

On Wed, 2008-08-20 at 15:36 -0700, George Sakkis wrote:

It seems xml.etree.cElementTree.iterparse() is not unicode aware:

>from StringIO import StringIO
from xml.etree.cElementTree import iterparse
s = u'<name>\u03a0\u03b1\u03bd\u03b1\u03b3\u03b9\u03ce \u03c4\u03b7\u03c2</name>'
for event,elem in iterparse(StringIO(s)):

... print elem.text
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeError: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?

George
--
http://mail.python.org/mailman/listinfo/python-list

As iterparse expects an actual file as input, using a unicode string is
problematic. If you want to use iterparse, the simplest way would be to
encode your string before inserting it into the StringIO object, as so:

>>for event,elem in iterparse(StringIO(s.encode('UTF8')):
.... print elem.text
....

If you encode using UTF-8, you don't need to worry about the <?xml header
bit as suggested previously, as it's the default for XML.

If you're using unicode extensively, you should consider using lxml,
which implements the same interface as ElementTree, but handles unicode
better (though it also doesn't run your example above without first
encoding the string):
http://codespeak.net/lxml/parsing.ht...nicode-strings

You may also find the target parser interface to be more accepting of
unicode than iterparse, though it requires a different parsing interface:
http://codespeak.net/lxml/parsing.ht...rser-interface

--
John Krukoff <jk******@ltgc.com>
Land Title Guarantee Company

[George Sakkis]

Thank you both for the suggestions. I made a few more experiments to
understand how iterparse behaves with respect to three dimensions:

a. Is the encoding declared in the header (if there is one) ?
b. Is the text ascii-encodable (i.e. within range(128)) ?
c. Does the passed file object's read() method return str or unicode
(e.g. codecs.open(f,encoding='utf8')) ?

Feel free to correct me if I misinterpreted what is really happening.

As John Krukoff mentioned, omitting the encoding is equivalent to
encoding="utf-8" for all other combinations. This leaves (b) and (c).

If a text node is ascii-encodable, iterparse() returns it as a byte
string, regardless of the declared encoding and the input file's
read() return type.

(c) becomes relevant only if a text node is not ascii-encodable. In
this case iterparse() returns unicode if the underlying file's read()
returns bytes in an encoding that matches (or at least is compatible
with) the declared encoding in the header (or the implied utf8).
Passing a file object whose read() returns unicode characters
implicitly encodes them to ascii, which raises a UnicodeEncodeError
since the text node is not ascii-encodable.

It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.

George

[Fredrik Lundh]

George Sakkis wrote:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeError: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?

iterparse parses XML documents. XML documents are streams of encoded
characters, not streams of Unicode characters.

</F>

[Fredrik Lundh]

George Sakkis wrote:

Thank you both for the suggestions. I made a few more experiments to
understand how iterparse behaves with respect to three dimensions:

Spending time researching undefined behaviour is pretty pointless. ET
parsers expect byte streams, because that's what XML files are. If you
pass it anything else, an ET implementation may attempt to convert that
thing to a byte string, run the game "rogue", or do something else that
it finds appropriate.

It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.

If you don't care about memory and execution performance, there are
plenty of toolkits that guarantee that you always get Unicode strings.

</F>

[George Sakkis]

On Aug 21, 1:48*am, Fredrik Lundh <[e-mail]:

George Sakkis wrote:

It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from *BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.

If you don't care about memory and execution performance, there are
plenty of toolkits that guarantee that you always get Unicode strings.

As long as they are documented, both approaches are fine for different
cases. Currently the only reference I found about unicode in
ElementTree is "All strings can either be Unicode strings, or 8-bit
strings containing US-ASCII only." [1], which is rather ambiguous; at
least I read it as "all strings are Unicode or all strings are 8-bit
strings", not a potentially mix of both in the same tree.

Regards,
George

[1] http://effbot.org/zone/element.htm

[Stefan Behnel]

George Sakkis wrote:

On Aug 21, 1:48 am, Fredrik Lundh <[e-mail]:

>George Sakkis wrote:

>>It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.

If you don't care about memory and execution performance, there are
plenty of toolkits that guarantee that you always get Unicode strings.

As long as they are documented, both approaches are fine for different
cases. Currently the only reference I found about unicode in
ElementTree is "All strings can either be Unicode strings, or 8-bit
strings containing US-ASCII only." [1], which is rather ambiguous

It's not ambiguous in Py2.x, where ASCII byte strings and unicode strings are
compatible. No need to feel "uneasy". :)

Stefan

[Stefan Behnel]

George Sakkis wrote:

It seems xml.etree.cElementTree.iterparse() is not unicode aware:

>>>from StringIO import StringIO
from xml.etree.cElementTree import iterparse
s = u'<name>\u03a0\u03b1\u03bd\u03b1\u03b3\u03b9\u03ce \u03c4\u03b7\u03c2</name>'
for event,elem in iterparse(StringIO(s)):

... print elem.text
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 64, in __iter__
UnicodeEncodeError: 'ascii' codec can't encode characters in position
6-15: ordinal not in range(128)

Am I using it incorrectly or it doesn't currently support unicode ?

If you want to parse XML from Python unicode strings, you can use lxml.etree.
The XML specification allows transport protocols and other sources to provide
external encoding information. lxml supports the Python unicode type as a
transport and reads the internal byte sequence of the unicode string.

To be clear, this does not mean that the parsing happens at the unicode
character level. Parsing XML is about parsing bytes, not characters.

Stefan

[George Sakkis]

On Aug 24, 1:12 am, Stefan Behnel <[e-mail]:

George Sakkis wrote:

On Aug 21, 1:48 am, Fredrik Lundh <[e-mail]:

George Sakkis wrote:
It's interesting that the element text attributes after a successful
parse do not necessarily have the same type, i.e. all be str or all
unicode. I ported some text extraction code from BeautifulSoup (which
handles all text as unicode) and I was surprized to find out that in
xml.etree the returned text's type is not fixed, even within the same
file. Although it's not a bug, having a mixed collection of byte and
unicode strings from the same source makes me somewhat uneasy.
If you don't care about memory and execution performance, there are
plenty of toolkits that guarantee that you always get Unicode strings.

As long as they are documented, both approaches are fine for different
cases. Currently the only reference I found about unicode in
ElementTree is "All strings can either be Unicode strings, or 8-bit
strings containing US-ASCII only." [1], which is rather ambiguous

It's not ambiguous in Py2.x, where ASCII byte strings and unicode strings are
compatible. No need to feel "uneasy". :)

It depends on what you mean by "compatible"; e.g. you can't safely do
[s.decode('utf8') for s in strings] if you have byte strings mixed
with unicode.

George

[Fredrik Lundh]

George Sakkis wrote:

It depends on what you mean by "compatible"; e.g. you can't safely do
[s.decode('utf8') for s in strings] if you have byte strings mixed
with unicode.

why would you want to decode strings given to you by a library that
returns decoded strings?

if you meant to write "encode", you can indeed safely do
[s.encode('utf8') for s in strings] as long as all strings are returned
by an ET implementation.

</F>

[George Sakkis]

On Aug 25, 4:45*pm, Fredrik Lundh <[e-mail]:

George Sakkis wrote:

It depends on what you mean by "compatible"; e.g. you can't safely do
[s.decode('utf8') for s in strings] if you have byte strings mixed
with unicode.

why would you want to decode strings given to you by a library that
returns decoded strings?

if you meant to write "encode", you can indeed safely do
[s.encode('utf8') for s in strings] as long as all strings are returned
by an ET implementation.

I was replying to the general assertion that "in 2.x ASCII byte
strings and unicode strings are compatible", not specifically about
the strings returned by ET.

George

[Fredrik Lundh]

George Sakkis wrote:

>if you meant to write "encode", you can indeed safely do
[s.encode('utf8') for s in strings] as long as all strings are returned
by an ET implementation.

I was replying to the general assertion that "in 2.x ASCII byte
strings and unicode strings are compatible", not specifically about
the strings returned by ET.

that assertion was made in the context of ET. having to unilaterially
change the topic to "win" an argument is pretty lame.

and if you really meant to write "decode", you picked a rather stupid
example to support your complaint about ET not returning Unicode -- your
example does work fine for byte strings (whether they contain pure ASCII
or not), but doesn't work at all for arbitrary Unicode strings, because
decoding things that are already decoded makes very little sense (which
explains why that method was removed in 3.0).

>>"hello".decode("utf-8")

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute 'decode'

are you sure you understand the distinction between Unicode strings and
encoded strings?

</F>

[George Sakkis]

On Aug 27, 5:42 am, Fredrik Lundh <[e-mail]:

George Sakkis wrote:

if you meant to write "encode", you can indeed safely do
[s.encode('utf8') for s in strings] as long as all strings are returned
by an ET implementation.

I was replying to the general assertion that "in 2.x ASCII byte
strings and unicode strings are compatible", not specifically about
the strings returned by ET.

that assertion was made in the context of ET. having to unilaterially
change the topic to "win" an argument is pretty lame.

I took Stefan's comment as a general statement, not in the context of
ET. Feeling the need to keep "defending" ET at this point is, to
borrow your words, pretty lame.

and if you really meant to write "decode", you picked a rather stupid
example to support your complaint about ET not returning Unicode -- your
example does work fine for byte strings (whether they contain pure ASCII
or not), but doesn't work at all for arbitrary Unicode strings, because
decoding things that are already decoded makes very little sense (which
explains why that method was removed in 3.0).

The thing is, a user might be happily using ET and call "decode" on
the returned byte strings as long as the files happen to be all ASCII,
without knowing that ET may also return Unicode. Then after some weeks/
months/years a non-ASCII file comes in and the program breaks. As far
as I am concerned, it's a documentation issue, nothing more.

George