By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
438,216 Members | 1,001 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 438,216 IT Pros & Developers. It's quick & easy.

XML Tree Discovery (script, tool, __?)

P: n/a
Hi all,

Finally diving into XML programmatically. Does anyone have a best
practice recommendation for programmatically discovering the structure
of an arbitrary XML document via Python?

It seems like it is a common wheel I'd be re-inventing.

Thanks and cheers
EP

Oct 24 '05 #1
Share this Question
Share on Google+
14 Replies


P: n/a
er***********@gmail.com wrote:
Finally diving into XML programmatically. Does anyone have a best
practice recommendation for programmatically discovering the structure
of an arbitrary XML document via Python?

It seems like it is a common wheel I'd be re-inventing.


It is. Look up XML DOM.

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
An ounce of hypocrisy is worth a pound of ambition.
-- Michael Korda
Oct 24 '05 #2

P: n/a
"inally diving into XML programmatically. Does anyone have a best
practice recommendation for programmatically discovering the structure
of an arbitrary XML document via Python?"

You can do this with DOM or SAX, or any of the many more friendly XML
processing libraries out there. You might want to be more specific.
What sort of output do you want from this discovery?

--
Uche Ogbuji Fourthought, Inc.
http://uche.ogbuji.net http://fourthought.com
http://copia.ogbuji.net http://4Suite.org
Articles: http://uche.ogbuji.net/tech/publications/

Oct 24 '05 #3

P: n/a
The output I was contemplating was a DOM "DNA" - that is the DOM
without the instances of the elements or their data, a bare tree, a
prototype tree based on what is in the document (rather than what is
legal to include in the document).

Just enough data that for an arbitrary element I would know:

1) whether the element was in a document
2) where to find it (the chain of parents)
As I mentioned, I'm just starting to think about the subject, so maybe
the best practice is something else like loading the full DOM into
memory. I'm still in the make one to throw away one mode.

EP

Oct 24 '05 #4

P: n/a
er***********@gmail.com wrote:
The output I was contemplating was a DOM "DNA" - that is the DOM
without the instances of the elements or their data, a bare tree, a
prototype tree based on what is in the document (rather than what is
legal to include in the document).

Just enough data that for an arbitrary element I would know:

1) whether the element was in a document
2) where to find it (the chain of parents)


how to you identify the elements ? do they have unique tags, unique identifiers,
or some other "globally unique" identifier (attributes or contents) ?

</F>

Oct 25 '05 #5

P: n/a
just namespace + tag

Oct 25 '05 #6

P: n/a
er***********@gmail.com wrote:
just namespace + tag


here's an ElementTree-based example:

# http://effbot.org/tag/elementtree
import elementtree.ElementTree as ET

FILE = "example.xml"

path = ()
path_map = {}

for event, elem in ET.iterparse(FILE, events=("start", "end")):
if event == "start":
path = path + (elem.tag,)
else:
path_map.setdefault(path, []).append(elem)
path = path[:-1]
elem.clear() # won't need the contents any more

for path in path_map:
print "/".join(path), len(path_map[path])

given this document:

<document>
<chapter>
<title>chapter 1</title>
</chapter>
<chapter>
<title>chapter 2</title>
</chapter>
</document>

the above script prints

document 1
document/chapter 2
document/chapter/title 2

the script uses universal names for tags that live in a namespace. for
information on how to decipher such names, see:

http://www.effbot.org/zone/element.htm#xml-namespaces

hope this helps!

</F>

Oct 26 '05 #7

P: n/a
"""
The output I was contemplating was a DOM "DNA" - that is the DOM
without the instances of the elements or their data, a bare tree, a
prototype tree based on what is in the document (rather than what is
legal to include in the document).

Just enough data that for an arbitrary element I would know:

1) whether the element was in a document
2) where to find it (the chain of parents)
"""

This is easy to do in SAX. For some hints, see page 2 of my article:

http://www.xml.com/pub/a/2004/11/24/py-xml.html

--
Uche Ogbuji Fourthought, Inc.
http://uche.ogbuji.net http://fourthought.com
http://copia.ogbuji.net http://4Suite.org
Articles: http://uche.ogbuji.net/tech/publications/

Oct 26 '05 #8

P: n/a
All I can add to this is:

- don't use SAX unless your document is huge
- don't use DOM unless someone is putting a gun to your head

There's a good selection of nice and simple XML processing libraries in
python. You could start with ElementTree.

Oct 26 '05 #9

P: n/a
Istvan Albert wrote:
All I can add to this is:

- don't use SAX unless your document is huge
- don't use DOM unless someone is putting a gun to your head


+1 QOTW
Oct 26 '05 #10

P: n/a
> - don't use SAX unless your document is huge
- don't use DOM unless someone is putting a gun to your head


What I say is: use what works for you. I think SAX would be fine for
this task, but, hey, I personally would use Amara (
http://uche.ogbuji.net/tech/4suite/amara/ ), of course. The following
does the trick:

import sets
import amara
from amara import binderytools

#element_skeleton_rule suppresses char data from the resulting binding
#tree. If you have a large document and only care about element/attr
#structure and not text, this saves a lot of memory
rules = [binderytools.element_skeleton_rule()]
#XML can be a file path, URI, string, or even an open-file-like object
doc = amara.parse(XML, rules=rules)
elems = {}
for e in doc.xml_xpath('//*'):
paths = elems.setdefault((e.namespaceURI, e.localName), sets.Set())
path = u'/'.join([n.nodeName for n in e.xml_xpath(u'ancestor::*')])
paths.add(u'/' + path)

#Pretty-print output
for name in elems:
print name, '\n\t\t\t', '\n\t\t\t'.join(elems[name])
--
Uche Ogbuji Fourthought, Inc.
http://uche.ogbuji.net http://fourthought.com
http://copia.ogbuji.net http://4Suite.org
Articles: http://uche.ogbuji.net/tech/publications/

Oct 28 '05 #11

P: n/a
"er***********@gmail.com" wrote:
Hi all,

Finally diving into XML programmatically. Does anyone have a best
practice recommendation for programmatically discovering the structure
of an arbitrary XML document via Python?

It seems like it is a common wheel I'd be re-inventing.

Thanks and cheers

I was looking for something similar (XML to DTD inference) but I didn't
find anything related in python. Trang
(http://www.thaiopensource.com/relaxn...#introduction),
on the other hand seems impressive after a few non-trivial tests. It
would be neat to have it ported in python, at least the inference part.

George

Oct 28 '05 #12

P: n/a
"""
I was looking for something similar (XML to DTD inference) but I didn't
find anything related in python. Trang
(http://www.thaiopensource.com/relaxn...#introduction),
on the other hand seems impressive after a few non-trivial tests. It
would be neat to have it ported in python, at least the inference part.

"""

If you're OK with RELAX NG rather than DTD as the schema output
(probably a good idea if you're using namespaces), consider
Examplotron, which I've used on many such production tasks.

http://www-128.ibm.com/developerwork...ary/x-xmptron/

It's XSLT rather than Python, but the good news is that XSLT is easy to
invoke from Python using tools such as 4Suite.

http://uche.ogbuji.net/tech/akara/no...01/python-xslt

--
Uche Ogbuji Fourthought, Inc.
http://uche.ogbuji.net http://fourthought.com
http://copia.ogbuji.net http://4Suite.org
Articles: http://uche.ogbuji.net/tech/publications/

Oct 28 '05 #13

P: n/a
<uc*********@gmail.com> wrote:
"""
I was looking for something similar (XML to DTD inference) but I didn't
find anything related in python. Trang
(http://www.thaiopensource.com/relaxn...#introduction),
on the other hand seems impressive after a few non-trivial tests. It
would be neat to have it ported in python, at least the inference part.

"""

If you're OK with RELAX NG rather than DTD as the schema output
(probably a good idea if you're using namespaces), consider
Examplotron, which I've used on many such production tasks.

http://www-128.ibm.com/developerwork...ary/x-xmptron/

It's XSLT rather than Python, but the good news is that XSLT is easy to
invoke from Python using tools such as 4Suite.

http://uche.ogbuji.net/tech/akara/no...01/python-xslt


Neat, though non-trivial XSLT makes my head spin. Just for kicks, I
rewrote in python Michael Kay's DTDGenerator
(http://saxon.sourceforge.net/dtdgen.html), though as the original it
has several limitations on the accuracy of the inferred DTD.

George

Oct 29 '05 #14

P: n/a
"Neat, though non-trivial XSLT makes my head spin."

Well, you don't have to know XSLT at all to use the Examplotron
transform, although I can understand wanting to understand and hack
what you're using.

"Just for kicks, I
rewrote in python Michael Kay's DTDGenerator
(http://saxon.sourceforge.net/dtdgen.html), though as the original it
has several limitations on the accuracy of the inferred DTD. "

Ah. Cool. Got a link?

--
Uche Ogbuji Fourthought, Inc.
http://uche.ogbuji.net http://fourthought.com
http://copia.ogbuji.net http://4Suite.org
Articles: http://uche.ogbuji.net/tech/publications/

Oct 30 '05 #15

This discussion thread is closed

Replies have been disabled for this discussion.