xml parsing escape characters

[Luis P. Mendes]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

I only know a little bit of xml and I'm trying to parse a xml document
in order to save its elements in a file (dictionaries inside a list).

When I access a url from python 2.3.3 running in Linux with the
following lines:
resposta = urllib.urlopen( url)
xmldoc = minidom.parse(r esposta)
resposta.close( )

I get the following result:

<?xml version="1.0" encoding="utf-8"?>
<string xmlns="http://www......">&lt; DataSet&gt;
~ &lt;Order&gt ;
~ &lt;Customer&gt ;439&lt;/Customer&gt;
(... others ...)
~ &lt;/Order&gt;
&lt;/DataSet&gt;</string>
_______________ _______________ _______________ _______________ _

In the lines below, I try to get all the child nodes from string, first
by counting them, and then ignoring the /n ones:

stringNode = xmldoc.childNod es[0]
print stringNode.toxm l()
dataSetNode = stringNode.chil dNodes[0]
numNos = len(dataSetNode .childNodes)
todosNos={}
for no in range(numNos):
todosNos[no] = dataSetNode.chi ldNodes[no].toxml()
posicaoXml = [no for no in todosNos.keys() if len(todosNos[no])>4]
print posicaoXml

(I'm almost sure there's a simpler way to do this...)
_______________ _______________ _______________ _______________ _

I don't get any elements. But, if I access the same url via a browser,
the result in the browser window is something like:

<string xmlns="http://www......">
~ <DataSet>
~ <Order>
~ <Customer>439 </Customer>
(... others ...)
~ </Order>
~ </DataSet>
</string>

and the lines I posted work as intended.

I already browsed the web, I know it's about the escape characters, but
I didn't find a simple solution for this.

I tried to use LL2XML.py and unescape function with a simple replace
text = text.replace("& lt;", "<")
but I had to convert the xml document to string and then I could not (or
don't know) how to convert it back to xml object.

How can I solve this? Please, explain it having in mind that I'm just
beggining with Xml and I'm not very experienced in Python, too.
Luis
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFB7rzKHn4 UHCY8rB8RAhnlAK CYA6t0gd8rRDhIv Z5sdmNJlEPSeQCg teB3
XUtZ0JoHeTavBOC Yi6YYnNo=
=VORM
-----END PGP SIGNATURE-----

[Martin v. Löwis]

Luis P. Mendes wrote:

I get the following result:

<?xml version="1.0" encoding="utf-8"?>
<string xmlns="http://www......">&lt; DataSet&gt;
~ &lt;Order&gt ;
Most likely, this result is correct, and your document
really does contain

&lt;Order&gt ;

I don't get any elements. But, if I access the same url via a browser,
the result in the browser window is something like:

<string xmlns="http://www......">
~ <DataSet>
Most likely, your browser is incorrect (or atleast confusing), and
renders &lt; as "<", even though this is not markup.
I already browsed the web, I know it's about the escape characters, but
I didn't find a simple solution for this.

Not sure what "this" is. AFAICT, everything works correctly.

Regards,
Martin

[Luis P. Mendes]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

this is the xml document:

<?xml version="1.0" encoding="utf-8"?>
<string xmlns="http://www......">&lt; DataSet&gt;
~ &lt;Order&gt ;
~ &lt;Customer&gt ;439&lt;/Customer&gt;
(... others ...)
~ &lt;/Order&gt;
&lt;/DataSet&gt;</string>

When I do:

print xmldoc.toxml()

it prints:
<?xml version="1.0" ?>
<string xmlns="http://www...">&lt;Dat aSet&gt;
~ &lt;Order&gt ;
~ &lt;Customer&gt ;439&lt;/Customer&gt;

~ &lt;/Order&gt;
&lt;/DataSet&gt;</string>

_______________ _______________ _______________ _____________
with: stringNode = xmldoc.childNod es[0]
print stringNode.toxm l()
I get:
<string xmlns="http://www.......">&lt ;DataSet&gt;
~ &lt;Order&gt ;
~ &lt;Customer&gt ;439&lt;/Customer&gt;

~ &lt;/Order&gt;
&lt;/DataSet&gt;</string>
_______________ _______________ _______________ _______________ __________

with: DataSetNode = stringNode.chil dNodes[0]
print DataSetNode.tox ml()

I get:

&lt;DataSet& gt;
~ &lt;Order&gt ;
~ &lt;Customer&gt ;439&lt;/Customer&gt;

~ &lt;/Order&gt;
&lt;/DataSet&gt;
_______________ _______________ _______________ _______________ ___-

so far so good, but when I issue the command:

print DataSetNode.chi ldNodes[0]

I get:
IndexError: tuple index out of range

Why the error, and why does it return a tuple?
Why doesn't it return:
&lt;Order&gt ;
&lt;Customer&gt ;439&lt;/Customer&gt;

&lt;/Order&gt;
??
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFB76y3Hn4 UHCY8rB8RAvQsAK CFD/hps8ybQli8HAs3i SCvRjwqjACfS/12
5gctpB91S5cy299 e/TVLGQk=
=XR2a
-----END PGP SIGNATURE-----

[Kent Johnson]

Luis P. Mendes wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

this is the xml document:

<?xml version="1.0" encoding="utf-8"?>
<string xmlns="http://www......">&lt; DataSet&gt;
~ &lt;Order&gt ;
~ &lt;Customer&gt ;439&lt;/Customer&gt;
(... others ...)
~ &lt;/Order&gt;
&lt;/DataSet&gt;</string>

This is an XML document containing a single tag, <string>, whose content is text containing
entity-escaped XML.

This is *not* an XML document containing tags <DataSet>, <Order>, <Customer>, etc.

All the behaviour you are seeing is a consequence of this. You need to unescape the contents of the
<string> tag to be able to treat it as structured XML.

Kent

[Irmen de Jong]

Kent Johnson wrote:
[...]

This is an XML document containing a single tag, <string>, whose content
is text containing entity-escaped XML.

This is *not* an XML document containing tags <DataSet>, <Order>,
<Customer>, etc.

All the behaviour you are seeing is a consequence of this. You need to
unescape the contents of the <string> tag to be able to treat it as
structured XML.

The unescaping is usually done for you by the xml parser that you use.

--Irmen

[Kent Johnson]

Irmen de Jong wrote:

Kent Johnson wrote:
[...]

This is an XML document containing a single tag, <string>, whose
content is text containing entity-escaped XML.

This is *not* an XML document containing tags <DataSet>, <Order>,
<Customer>, etc.

All the behaviour you are seeing is a consequence of this. You need to
unescape the contents of the <string> tag to be able to treat it as
structured XML.

The unescaping is usually done for you by the xml parser that you use.

Yes, so if your XML contains for example
<stuff>&lt;no t a tag&gt;</stuff>

and you parse this and ask for the *text* content of the <stuff> tag, you will get the string
"<not a tag>"

but it's still *not* a tag. If you try to get child elements of the <stuff> element there will be none.

This is exactly the confusion the OP has.

--Irmen

[Martin v. Löwis]

Luis P. Mendes wrote:

with: DataSetNode = stringNode.chil dNodes[0]
print DataSetNode.tox ml()

I get:

&lt;DataSet& gt;
~ &lt;Order&gt ;
~ &lt;Customer&gt ;439&lt;/Customer&gt;

~ &lt;/Order&gt;
&lt;/DataSet&gt;
_______________ _______________ _______________ _______________ ___-

so far so good, but when I issue the command:

print DataSetNode.chi ldNodes[0]

I get:
IndexError: tuple index out of range

Why the error, and why does it return a tuple?

The DataSetNode has no children, because it is not
an Element node, but a Text node. In XML, an element
is denoted by

<DataSet>...</DataSet>

and *not* by

&lt;DataSet&gt; ...&lt;/DataSet&gt;

The latter is just a single string, represented
in XML as a Text node. It does not give you any
hierarchy whatsoever.

As a text node does not have any children, its
childNode members is a empty tuple; accessing
that tuple gives you an IndexError.

Regards,
Martin

[Martin v. Löwis]

Irmen de Jong wrote:

The unescaping is usually done for you by the xml parser that you use.

Usually, but not in this case. If you have a text that looks like
XML, and you want to put it into an XML element, the XML file uses
&lt; and &gt;. The XML parser unescapes that as < and >. However, it
does not then consider the < and > as markup, and it shouldn't.

Regards,
Martin

[Irmen de Jong]

Martin v. Löwis wrote:

Irmen de Jong wrote:

The unescaping is usually done for you by the xml parser that you use.

Usually, but not in this case. If you have a text that looks like
XML, and you want to put it into an XML element, the XML file uses
&lt; and &gt;. The XML parser unescapes that as < and >. However, it
does not then consider the < and > as markup, and it shouldn't.

That's also what I said?

The unescaping of the XML entities in the contents of the OP's
<string> element is done for you by the parser,
so you will get a text node with the <,>,&,whateve r in there.
The OP probably wants to feed that to a new xml parser instance
to process it as markup.
Or perhaps the way the original XML document is constructed is
flawed.

--Irmen

[Martin v. Löwis]

Irmen de Jong wrote:

Usually, but not in this case. If you have a text that looks like
XML, and you want to put it into an XML element, the XML file uses
&lt; and &gt;. The XML parser unescapes that as < and >. However, it
does not then consider the < and > as markup, and it shouldn't.

That's also what I said?

You said it in response to

All the behaviour you are seeing is a consequence of this. You need
to unescape the contents of the <string> tag to be able to treat it
as structured XML.

In that context, I interpreted
The unescaping is usually done for you by the xml parser that you
use.

as "The parser should have done what you want; if the parser didn't,
that is is bug in the parser".
The OP probably wants to feed that to a new xml parser instance
to process it as markup.
Or perhaps the way the original XML document is constructed is
flawed.

Either of these, indeed - probably the latter.

Regards,
Martin