[jog]
I want to get text out of some nodes of a huge xml file (1,5 GB). The
architecture of the xml file is something like this
[snip]
I want to combine the text out of page:title and page:revision:t ext
for every single page element. One by one I want to index these
combined texts (so for each page one index)
What is the most efficient API for that?:
SAX ( I don´t thonk so)
SAX is perfect for the job. See code below.
DOM
If your XML file is 1.5G, you'll need *lots* of RAM and virtual memory
to load it into a DOM.
or pulldom?
Not sure how pulldom does it's pull "optimizations" , but I think it
still builds an in-memory object structure for your document, which will
still take buckets of memory for such a big document. I could be wrong
though.
Or should I just use Xpath somehow.
Using xpath normally requires building a (D)OM, which will consume
*lots* of memory for your document, regardless of how efficient the OM is.
Best to use SAX and XPATH-style expressions.
You can get a limited subset of xpath using a SAX handler and a stack.
Your problem is particularly well suited to that kind of solution. Code
that does a basic job of this for your specific problem is given below.
Note that there are a number of caveats with this code
1. characterdata handlers may get called multiple times for a single xml
text() node. This is permitted in the SAX spec, and is basically a
consequence of using buffered IO to read the contents of the xml file,
e.g. the start of a text node is at the end of the last buffer read, and
the rest of the text node is at the beginning of the next buffer.
2. This code assumes that your "revision/text" nodes do not contain
mixed content, i.e. a mixture of elements and text, e.g.
"<revision><tex t>This is a piece of <b>revision</b>
text</text></revision>. The below code will fail to extract all
character data in that case.
#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
import xml.sax
class Page:
def append(self, field_name, new_value):
old_value = ""
if hasattr(self, field_name):
old_value = getattr(self, field_name)
setattr(self, field_name, "%s%s" % (old_value, new_value))
class page_matcher(xm l.sax.handler.C ontentHandler):
def __init__(self, page_handler=No ne):
xml.sax.handler .ContentHandler .__init__(self)
self.page_handl er = page_handler
self.stack = []
def check_stack(sel f):
stack_expr = "/" + "/".join(self.sta ck)
if '/parent/page' == stack_expr:
self.page = Page()
elif '/parent/page/title/text()' == stack_expr:
self.page.appen d('title', self.chardata)
elif '/parent/page/revision/id/text()' == stack_expr:
self.page.appen d('revision_id' , self.chardata)
elif '/parent/page/revision/text/text()' == stack_expr:
self.page.appen d('revision_tex t', self.chardata)
else:
pass
def startElement(se lf, elemname, attrs):
self.stack.appe nd(elemname)
self.check_stac k()
def endElement(self , elemname):
if elemname == 'page' and self.page_handl er:
self.page_handl er(self.page)
self.page = None
self.stack.pop( )
def characters(self , data):
self.chardata = data
self.stack.appe nd('text()')
self.check_stac k()
self.stack.pop( )
testdoc = """
<parent>
<page>
<title>Page number 1</title>
<id>p1</id>
<revision>
<id>r1</id>
<text>revisio n one</text>
</revision>
</page>
<page>
<title>Page number 2</title>
<id>p2</id>
<revision>
<id>r2</id>
<text>revisio n two</text>
</revision>
</page>
</parent>
"""
def page_handler(ne w_page):
print "New page"
print "title\t\t% s" % new_page.title
print "revision_id\t% s" % new_page.revisi on_id
print "revision_text\ t%s" % new_page.revisi on_text
print
if __name__ == "__main__":
parser = xml.sax.make_pa rser()
parser.setConte ntHandler(page_ matcher(page_ha ndler))
parser.setFeatu re(xml.sax.hand ler.feature_nam espaces, 0)
parser.feed(tes tdoc)
#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
HTH,
--
alan kennedy
------------------------------------------------------
email alan:
http://xhaus.com/contact/alan 
