huge XML files, XSLT memory problems, Java & SAX...

[Jeff Calico]

I have 2 XML data files that I want to extract data from simultaneously

and transform with XSLT to generate a report. The first file is huge
and when XSLT builds the DOM tree in memory, it runs out of space.

I only need a few branches of elements from the original XML, so I am
seeking a recomended way of building a DOM for XSLT of only the
elements
that I need. I'm writing a Java application that invokes Xalan, and
reading up on SAX parsers this afternoon... I'm sure this is a common
problem, and as such, there is probably a clean and easy way to do it,
but I haven't found that one yet...

thanks,
Jeff

[Henry S. Thompson]

Don't write Java, use a pipeline -- Markup pipeline demo site [1]
includes a pipeline to split a large document into small chunks for
validation, similar approach would work for.

ht

[1] http://www.markup.co.uk/showcase/
--
Henry S. Thompson, Markup Technology Ltd.
4 Buccleuch Place, Edinburgh EH8 9LW, SCOTLAND -- [phone]
Fax: [phone], e-mail: ht@markuptechno logy.com
URL: http://www.markup.co.uk/
[mail really from me _always_ has this .sig -- mail without it is forged spam]

[Joe Kesselman]

> Lemme see if I've got anything on tap that's simple enough to be a good

pedagogical illustration...

I don't have anything really good on hand, but look for examples of use
of org.xml.sax.hel pers.XMLFilterI mpl.

That starts out as a no-op filter which just passes everything through.
What you'd need to do is add enough logic to recognize which portions of
the document you're interested in, and pass those (and only those) along
to the next stage of processing. Plus, probably, the document element
(or a synthesized document element) so it's well-formed XML. Handling
namespaces properly complicates this somewhat but not horribly.

Note that the next stage has to be aware that it's seeing a filtered
view of the document; if you've passed along only some subtrees, search
patterns that look at the context they appeared in may of course not
work as expected. For example, if you're prefiltering before running a
stylesheet, some of the XPaths in that stylesheet may have to be rewritten.

As I say, I haven't had much trouble running recent versions of Xalan on
large documents... but this kind of explicit prefiltering may save you
some cycles and storage, at the cost of requiring more cycles of
developer time to create and maintain it.

--
() ASCII Ribbon Campaign | Joe Kesselman
/\ Stamp out HTML e-mail! | System architexture and kinetic poetry

[Jeff Calico]

Thanks, Joe. I shall take a look at org.xml.sax.hel pers.XMLFilterI mpl.

--Jeff