the tostring and XML methods in ElementTree

[mirandacascade]

O/S: Windows XP Home
Vsn of Python: 2.4

Copy/paste of interactive window is immediately below; the
text/questions toward the bottom of this post will refer to the content
of the copy/paste

from elementtree import ElementTree
beforeRoot = ElementTree.Element('beforeRoot')
beforeCtag = ElementTree.SubElement(beforeRoot, 'C')
beforeCtag.text = 'I\x92m confused'
type(beforeCtag.text) <type 'str'> print beforeCtag.text I'm confused resultToStr = ElementTree.tostring(beforeRoot)
resultToStr '<beforeRoot><C>I’m confused</C></beforeRoot>' afterRoot = ElementTree.XML(resultToStr)
afterCtag = afterRoot[0]
type(afterCtag.text) <type 'unicode'> print afterCtag.text I?m confused
I wanted to see what would happen if one used the results of a tostring
method as input into the XML method. What I observed is this:
a) beforeCtag.text is of type <type 'str'>
b) beforeCtag.text when printed displays: I'm confused
c) afterCtag.text is of type <type 'unicode'>
d) afterCtag.text when printed displays: I?m confused

Question 1: assuming the following:
a) beforeCtag.text gets assigned a value of 'I\x92m confused'
b) afterRoot is built using the XML() method where the input to the
XML() method is the results of a tostring() method from beforeRoot
Are there any settings/arguments that could have been modified that
would have resulted in afterCtag.text being of type <type 'str'> and
afterCtag.text when printed displays:
I'm confused

?

Another snippet from interactive window
resultToStr2 = ElementTree.tostring(beforeRoot, encoding="utf-8")
resultToStr2 '<beforeRoot><C>I’m confused</C></beforeRoot>' if resultToStr == resultToStr2: .... print 'equal'
....
equal
If I'm reading the signature of the tostring method in ElementTree.py
correctly, it looks like encoding gets assigned a value of None if the
tostring method gets called without a 2nd argument. In the specific
examples above, the result of the tostring method was the same when an
encoding of utf-8 was specified as it was when no encoding was
specified.

Question 2: Does the fact that resultToStr is equal to resultToStr2
mean that an encoding of utf-8 is the defacto default when no encoding
is passed as an argument to the tostring method, or does it only mean
that in this particular example, they happened to be the same?

Another snippet
fileHandle = open('c:/output1.text', 'w')
fileHandle.write(beforeCtag.text)
fileHandle.write(afterCtag.text) Traceback (most recent call last):
File "<interactive input>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\x92' in
position 1: ordinal not in range(128) encodedCtagtext = afterCtag.text.encode("utf-8")
type(encodedCtagtext) <type 'str'> encodedCtagtext 'I\xc2\x92m confused' print encodedCtagtext IÂ'm confused fileHandle.write(encodedCtagtext)
ord(encodedCtagtext[1]) 194

In this snippet, I am trying to discern what can be written to a file
without raising an exception. The variable beforeCtag.text can be
written, but an exception is raised when an attempt is made to write
the unicode variable afterCtag.text to the file. The statement

encodedCtagtext = afterCtag.text.encode("utf-8")

was a shot-in-the-dark attempt to transform afterCtag.text to something
that can be written to a file without raising an exception. What I
observed is that:
a) encodedCtagtext can be written to a file without raising an
exception
b) the second character in encodedCtagtext has an ordinal value of 194

Questions 3 and 4:
3) would it be possible to construct a statement of the form

newResult = afterCtag.text.encode(?? some argument ??)

where newResult was the same as beforeCtag.text? If so, what should
the argument be to the encode method?

4) what is the second character in encodedCtagtext (the character with
an ordinal value of 194)?

[Serge Orlov]

mi************@yahoo.com wrote:

Question 1: assuming the following:
a) beforeCtag.text gets assigned a value of 'I\x92m confused'
b) afterRoot is built using the XML() method where the input to the
XML() method is the results of a tostring() method from beforeRoot
Are there any settings/arguments that could have been modified that
would have resulted in afterCtag.text being of type <type 'str'> and
afterCtag.text when printed displays:
I'm confused

?
str type (also known as byte string) is only suitable for ascii text.
chr(0x92) is outside of ascii so you should use unicode strings or
you\x92ll be confused :)

print u"I\u2019m not confused" I'm not confused

Question 2: Does the fact that resultToStr is equal to resultToStr2
mean that an encoding of utf-8 is the defacto default when no encoding
is passed as an argument to the tostring method, or does it only mean
that in this particular example, they happened to be the same?

No. Dejure default encoding is ascii, defacto people try to change it,
but it's not a good idea. I'm not sure how you got the strings to be
the same, but it's definately host-specific result, when I repeat your
interactive session I get different resultToStr at this point:
afterRoot = ElementTree.XML(resultToStr)
resultToStr '<beforeRoot><C>I’m confused</C></beforeRoot>'

3) would it be possible to construct a statement of the form

newResult = afterCtag.text.encode(?? some argument ??)

where newResult was the same as beforeCtag.text? If so, what should
the argument be to the encode method?
Dealing with unicode doesn't require you to pollute your code with
encode methods, just open the file using codecs module and then write
unicode strings directly:

import codecs
fileHandle = codecs.open('c:/output1.text', 'w',"utf-8")
fileHandle.write(u"I\u2019m not confused, because I'm using unicode")
4) what is the second character in encodedCtagtext (the character with
an ordinal value of 194)?

That is byte with value 194, it's not a character. It is part of
unicode code point U+0092 when it is encoded in utf-8
'\xc2\x92'.decode("utf-8") u'\x92'

This code point actually has no name, so you shouldn't produce it:
import unicodedata
unicodedata.name('\xc2\x92'.decode("utf-8"))

Traceback (most recent call last):
File "<pyshell#40>", line 1, in -toplevel-
unicodedata.name('\xc2\x92'.decode("utf-8"))
ValueError: no such name

[Serge Orlov]

mi************@yahoo.com wrote:

O/S: Windows XP Home
Vsn of Python: 2.4

[snip fighting with unicode character U+2019 (RIGHT SINGLE QUOTATION
MARK) ]

I don't know what console you use but if it is IDLE you'll get confused
even more because it is buggy and improperly handles that character:

print repr(u'I'm confused') u'I\x92m confused'

I'm using Lightning Compiler
<http://cheeseshop.python.org/pypi/Lightning%20Compiler> to run
snippets of code and in the editor tab it handles that character fine:
print repr(u'I'm confused')

u'I\u2019m confused'

But in the console tab it produces the same buggy result :) It looks
like handling unicode is like rocket science :)

[Fredrik Lundh]

mi************@yahoo.com wrote:

I wanted to see what would happen if one used the results of a tostring
method as input into the XML method. What I observed is this:
a) beforeCtag.text is of type <type 'str'>
b) beforeCtag.text when printed displays: I'm confused
c) afterCtag.text is of type <type 'unicode'>
d) afterCtag.text when printed displays: I?m confused

the XML file format isn't a Python string serialization format, it's an XML infoset
serialization format.

as stated in the documentation, ET always uses Unicode strings for text that
contain non-ASCII characters. for text that *only* contains ASCII, it may use
either Unicode strings or 8-bit strings, depending on the implementation.

the behaviour if you're passing in non-ASCII text as 8-bit strings is undefined
(which means that you shouldn't do that; it's not portable).

to learn more about Unicode in Python, google for "python unicode".

</F>

[George Sakkis]

Fredrik Lundh wrote:

mi************@yahoo.com wrote:

I wanted to see what would happen if one used the results of a tostring
method as input into the XML method. What I observed is this:
a) beforeCtag.text is of type <type 'str'>
b) beforeCtag.text when printed displays: I'm confused
c) afterCtag.text is of type <type 'unicode'>
d) afterCtag.text when printed displays: I?m confused

the XML file format isn't a Python string serialization format, it's an XML infoset
serialization format.

as stated in the documentation, ET always uses Unicode strings for text that
contain non-ASCII characters. for text that *only* contains ASCII, it may use
either Unicode strings or 8-bit strings, depending on the implementation.

the behaviour if you're passing in non-ASCII text as 8-bit strings is undefined
(which means that you shouldn't do that; it's not portable).

I was about to post a similar question when I found this thread.
Fredrik, can you explain why this is not portable ? I'm currently using
(a variation of) the workaround below instead of ET.tostring and it
works fine for me:

def tostring(element, encoding=None):
text = element.text
if text:
if not isinstance(text, basestring):
text2 = str(text)
elif isinstance(text, str) and encoding:
text2 = text.decode(encoding)
element.text = text2
s = ET.tostring(element, encoding)
element.text = text
return s
Why isn't this the standard behaviour ?

Thanks,
George

[Stefan Behnel]

George Sakkis wrote:

Fredrik Lundh wrote:

mi************@yahoo.com wrote:

I wanted to see what would happen if one used the results of a tostring
method as input into the XML method. What I observed is this:
a) beforeCtag.text is of type <type 'str'>
b) beforeCtag.text when printed displays: I'm confused
c) afterCtag.text is of type <type 'unicode'>
d) afterCtag.text when printed displays: I?m confused the XML file format isn't a Python string serialization format, it's an XML infoset
serialization format.

as stated in the documentation, ET always uses Unicode strings for text that
contain non-ASCII characters. for text that *only* contains ASCII, it may use
either Unicode strings or 8-bit strings, depending on the implementation.

the behaviour if you're passing in non-ASCII text as 8-bit strings is undefined
(which means that you shouldn't do that; it's not portable).

I was about to post a similar question when I found this thread.
Fredrik, can you explain why this is not portable ?

Because there is no such things as a default encoding for 8-bit strings.

I'm currently using
(a variation of) the workaround below instead of ET.tostring and it
works fine for me:

def tostring(element, encoding=None):
text = element.text
if text:
if not isinstance(text, basestring):
text2 = str(text)
elif isinstance(text, str) and encoding:
text2 = text.decode(encoding)
element.text = text2
s = ET.tostring(element, encoding)
element.text = text
return s
Why isn't this the standard behaviour ?

Because it wouldn't work. What if you wanted to serialize a different encoding
than that of the strings you put into the .text fields? How is ET supposed to
know what encoding your strings have? And how should it know that you didn't
happily mix various different byte encodings in your strings?

Use unicode, that works *and* is portable.

Stefan

[George Sakkis]

Stefan Behnel wrote:

George Sakkis wrote:

Fredrik Lundh wrote:

mi************@yahoo.com wrote:

I wanted to see what would happen if one used the results of a tostring
method as input into the XML method. What I observed is this:
a) beforeCtag.text is of type <type 'str'>
b) beforeCtag.text when printed displays: I'm confused
c) afterCtag.text is of type <type 'unicode'>
d) afterCtag.text when printed displays: I?m confused
the XML file format isn't a Python string serialization format, it's an XML infoset
serialization format.

as stated in the documentation, ET always uses Unicode strings for text that
contain non-ASCII characters. for text that *only* contains ASCII, it may use
either Unicode strings or 8-bit strings, depending on the implementation.

the behaviour if you're passing in non-ASCII text as 8-bit strings is undefined
(which means that you shouldn't do that; it's not portable).
I was about to post a similar question when I found this thread.
Fredrik, can you explain why this is not portable ?

Because there is no such things as a default encoding for 8-bit strings.

I'm currently using
(a variation of) the workaround below instead of ET.tostring and it
works fine for me:

def tostring(element, encoding=None):
text = element.text
if text:
if not isinstance(text, basestring):
text2 = str(text)
elif isinstance(text, str) and encoding:
text2 = text.decode(encoding)
element.text = text2
s = ET.tostring(element, encoding)
element.text = text
return s
Why isn't this the standard behaviour ?

Because it wouldn't work. What if you wanted to serialize a different encoding
than that of the strings you put into the .text fields? How is ET supposed to
know what encoding your strings have? And how should it know that you didn't
happily mix various different byte encodings in your strings?

If you're mixing different encodings, no tool can help you clean up the
mess, you're on your own. This is very different though from having
nice utf-8 strings everywhere, asking ET.tostring explicitly to print
them in utf-8 and getting back garbage. Isn't the most reasonable
assumption that the input's encoding is the same with the output, or
does this fall under the "refuse the temptation to guess" motto ? If
this is the case, ET could at least accept an optional input encoding
parameter and convert everything to unicode internally.
Use unicode, that works *and* is portable.
*and* it's not supported by all the 3rd party packages, databases,
middleware, etc. you have to or want to use.
Stefan

George

[Serge Orlov]

George Sakkis wrote:

I'm currently using
(a variation of) the workaround below instead of ET.tostring and it
works fine for me:

def tostring(element, encoding=None):
text = element.text
if text:
if not isinstance(text, basestring):
text2 = str(text)
elif isinstance(text, str) and encoding:
text2 = text.decode(encoding)
element.text = text2
s = ET.tostring(element, encoding)
element.text = text
return s
Why isn't this the standard behaviour ?

Because it wouldn't work. What if you wanted to serialize a different encoding
than that of the strings you put into the .text fields? How is ET supposed to
know what encoding your strings have? And how should it know that you didn't
happily mix various different byte encodings in your strings?

If you're mixing different encodings, no tool can help you clean up the
mess, you're on your own. This is very different though from having
nice utf-8 strings everywhere, asking ET.tostring explicitly to print
them in utf-8 and getting back garbage. Isn't the most reasonable
assumption that the input's encoding is the same with the output, or
does this fall under the "refuse the temptation to guess" motto ? If
this is the case, ET could at least accept an optional input encoding
parameter and convert everything to unicode internally.

This is an optimization. Basically you're delaying decoding. First of
all have you measured the impact on your program if you delay decoding?
I'm sure for many programs it doesn't matter, so what you're proposing
will just pollute their source code with optimization they don't need.
That doesn't mean it's a bad idea in general. I'd prefer it implemented
in python core with minimal impact on such programs, decoding delayed
until you try to access individual characters. The code below can be
implemented without actual decoding:

utf8_text_file.write("abc".decode("utf-8") + " def".decode("utf-8"))

But this example will require decoding done during split method:

a = ("abc".decode("utf-8") + " def".decode("utf-8")).split()

Use unicode, that works *and* is portable.

*and* it's not supported by all the 3rd party packages, databases,
middleware, etc. you have to or want to use.

You can always call .encode method. Granted that could be a waste of
CPU and memory, but it works.