Parsing Large XML files

[doug]

How can I parse a large XML file that is to large for memory? I am
currently using php 5.0.3 and the libxml parser, I would like to read
it incrementally from a file, but the parser gets the entire contents
from as String?

[Derek Fountain]

doug wrote:

How can I parse a large XML file that is to large for memory? I am
currently using php 5.0.3 and the libxml parser, I would like to read
it incrementally from a file, but the parser gets the entire contents
from as String?

The standard XML solution to this problem is to use a SAX parser instead of
a DOM one. However, there doesn't seem to be a SAX parser in the PHP XML
library. One solution appears to be:

http://www.engageinteractive.com/mam...tent&task=view
&id=3628&Itemid=10159

Google might help find others. Or maybe use an external SAX based tool to
boil the XML down to something a bit smaller that you can manipulate from
PHP?

--
The email address used to post is a spam pit. Contact me at
http://www.derekfountain.org : <a
href="http://www.derekfountain.org/">Derek Fountain</a>

[Chung Leong]

"doug" <dd*****@shortbus.net> wrote in message
news:11**********************@g14g2000cwa.googlegr oups.com...

How can I parse a large XML file that is to large for memory? I am
currently using php 5.0.3 and the libxml parser, I would like to read
it incrementally from a file, but the parser gets the entire contents
from as String?

Use the expat functions instead.

http://us4.php.net/manual/en/ref.xml.php

[doug]

How do I use this SAX parser to read XML directly from a file?

Can you show me an instantiation example?

[doug]

So in the example below, every line is parsed individually with no
contents stored in memory but the current line?
<?php
$file = "data.xml";
$depth = array();

function startElement($parser, $name, $attrs)
{
global $depth;
for ($i = 0; $i < $depth[$parser]; $i++) {
echo " ";
}
echo "$name\n";
$depth[$parser]++;
}

function endElement($parser, $name)
{
global $depth;
$depth[$parser]--;
}

$xml_parser = xml_parser_create();
xml_set_element_handler($xml_parser, "startElement", "endElement");
if (!($fp = fopen($file, "r"))) {
die("could not open XML input");
}

while ($data = fread($fp, 4096)) {
if (!xml_parse($xml_parser, $data, feof($fp))) {
die(sprintf("XML error: %s at line %d",
xml_error_string(xml_get_error_code($xml_parser)),
xml_get_current_line_number($xml_parser)));
}
}
xml_parser_free($xml_parser);
?>

[Derek Fountain]

doug wrote:

How do I use this SAX parser to read XML directly from a file?
Erm, read the docs that come with it! Sounds like you need to read up on SAX
first.
Can you show me an instantiation example?

No, I've never used it. You're actually going to have to do some work here
yourself!

If it were me, and the application allowed it, I'd be writing a separate
utility to do the XML grunt work. PHP, for all it's strengths, isn't ideal
for heavy duty XML parsing.

--
The email address used to post is a spam pit. Contact me at
http://www.derekfountain.org : <a
href="http://www.derekfountain.org/">Derek Fountain</a>

[Chung Leong]

"doug" <dd*****@shortbus.net> wrote in message
news:11**********************@l41g2000cwc.googlegr oups.com...

So in the example below, every line is parsed individually with no
contents stored in memory but the current line?

The expat parser reuses the same buffers in calls to the handlers. If PHP
garbage collection works correctly, then the XML data should not take up
memory if your handlers don't save it.