XML::Simple in perl?

[Dan]

Using XML::Simple in perl is
extreemly slow to parse big
XML files (can be up to 250M,
taking ~1h).

How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?

Should I consider using (learning)
Expat.c for increased performance?

How long would parsing a 250M XML
file take with Expat?

Thanks for any suggestions you can give,
Dan.

[Janek Schleicher]

yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100:

Using XML::Simple in perl is
extreemly slow to parse big
XML files (can be up to 250M,
taking ~1h).
XML::Simple is not the only module on CPAN.
There are also
XML::Smart
XML::Parser
XML::LibXML
....
How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?
Note that SAX is basical an API for XML triggering starting and ending
tags and similar. That's different to the previous mentioned modules, as
they also generate a huge tree view to the XML document.

But SAX can reduce the memory wastage and gain in time advantage.
In Perl you can use e.g. XML::SAX for it.
Should I consider using (learning)
Expat.c for increased performance?

XML::Parser and XML::SAX::Expat are e.g. based on Expat, so you might use
the Perl interfaces if you want.

Greetings,
Janek

[Dan]

Janek Schleicher wrote:

yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100:

Using XML::Simple in perl is
extreemly slow to parse big
XML files (can be up to 250M,
taking ~1h).

XML::Simple is not the only module on CPAN.
There are also
XML::Smart
XML::Parser
XML::LibXML
...

Which ones have the same 'front end' as
XML simple? I would rather not change
code if I don't have to.

Which one uses least memory on big files?

How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?

Note that SAX is basical an API for XML triggering starting and ending
tags and similar. That's different to the previous mentioned modules, as
they also generate a huge tree view to the XML document.

But SAX can reduce the memory wastage and gain in time advantage.
In Perl you can use e.g. XML::SAX for it.

Should I consider using (learning)
Expat.c for increased performance?

XML::Parser and XML::SAX::Expat are e.g. based on Expat, so you might use
the Perl interfaces if you want.

I think I will end up doing this...

Is it any slower?

My requirements are extreemly simple, I just need
to print tab delimited lines to a file, so I was
thinking I could take this opportunity to try
to learn c....

Cheers,
Dan.

Greetings,
Janek

[Tad McClellan]

Dan <dm*@mrc-dunn.cam.ac.uk> wrote:

Janek Schleicher wrote:

yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100:

extreemly slow to parse big

There are also
XML::Smart
XML::Parser
XML::LibXML
...
Which ones have the same 'front end' as
XML simple?

None of them.

The "simple" front end is why the module is called Simple.

I would rather not change
code if I don't have to.

Too late. :-(

Which one uses least memory on big files?

None of those ones.

How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?

But SAX can reduce the memory wastage and gain in time advantage.

That one.

There is also a mailing list specifically for doing XML processing
using Perl:

http://listserv.ActiveState.com/mail...tinfo/perl-xml
--
Tad McClellan SGML consulting
ta***@augustmail.com Perl programming
Fort Worth, Texas

[cp]

In article <3F**************@mrc-dunn.cam.ac.uk>, Dan
<dm*@mrc-dunn.cam.ac.uk> wrote:

Using XML::Simple in perl is
extreemly slow to parse big
XML files (can be up to 250M,
taking ~1h).
XML::Simple is limited. The author points it out in the docs. It was
designed for a fairly specific purpose, parsing small configuration
files written in XML. It has since been expanded on, but it's core
remains, well, simple.
How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?

The docs for XML::Simple Version 2.05 suggest that you can (as of
version 1.08) use a SAX parser with XML::Simple, the benefits of which
are:

Applications written to the SAX API can extract data
from huge XML documents without the memory overheads
of a DOM or tree API.

So you might read the docs, use a SAX parser, and see some speed
benefit. You might not.

--
cp

[Dan]

I am now using XML::Parser,
which is working nicely, apart
from the occasional weird behaviour,
in some cases characters go missing,
(i.e. I get a 1 instead of 19).

The error is persistant, i.e. not a random
caracter, but the same character each time
goes missing.

It is really confusing.

Also my Char event gets called 3 times
per tag, even though there are no new
lines anywhere in teh tag text.

Little frustrating problems....

Dan.

cp wrote:

In article <3F**************@mrc-dunn.cam.ac.uk>, Dan
<dm*@mrc-dunn.cam.ac.uk> wrote:

Using XML::Simple in perl is
extreemly slow to parse big
XML files (can be up to 250M,
taking ~1h).

XML::Simple is limited. The author points it out in the docs. It was
designed for a fairly specific purpose, parsing small configuration
files written in XML. It has since been expanded on, but it's core
remains, well, simple.

How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?

The docs for XML::Simple Version 2.05 suggest that you can (as of
version 1.08) use a SAX parser with XML::Simple, the benefits of which
are:

Applications written to the SAX API can extract data
from huge XML documents without the memory overheads
of a DOM or tree API.

So you might read the docs, use a SAX parser, and see some speed
benefit. You might not.

[Michel Rodriguez]

Tad McClellan wrote:

Dan <dm*@mrc-dunn.cam.ac.uk> wrote:

Janek Schleicher wrote:

yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100: [XML::Simple is] extreemly slow to parse big [files] There are also
XML::Smart
XML::Parser
XML::LibXML
...

Which ones have the same 'front end' as XML simple?

None of them.

The "simple" front end is why the module is called Simple.

As a matter of fact XML::Smart has the same 'front end', and XML::Twig has a
method names 'simplify', which generates the same data structure as
XML::Simple, on a tree or on a sub-tree.

One possible cause for the problem might be that XML::Simple could be using
XML::SAX::PurePerl as its parser, which is very slow. This depends on your
installation. If you have XML::LibXML or XML::Parser installed you can set
the $XML::Simple::PREFERRED_PARSER variable to tell it to use an other
parser, see the docs.

Which one uses least memory on big files?

You might want to have a look at XML::Twig, which is specially designed for
big files (but I might be slightly biased ;--)

__
Michel Rodriguez
Perl &amp; XML
http://xmltwig.com

[Tad McClellan]

Dan <dm*@mrc-dunn.cam.ac.uk> wrote:

I am now using XML::Parser,
which is working nicely, apart
from the occasional weird behaviour,
in some cases characters go missing, The error is persistant, i.e. not a random
caracter, but the same character each time
goes missing.

If you show us a short and complete program that we can run that
illustrates your problem, then we can surely help you solve
your problem.

It is really confusing.

Can't help with unseen code...

Also my Char event gets called 3 times
per tag,

That is "normal".

You'll get the PCDATA in dribs and drabs, so you need to keep collecting
it until you reach the end of the containing element.

even though there are no new
lines anywhere in teh tag text.

The concept of "lines" is not present in XML.

Remove every newline from an XML document, and it is still
an XML document.

[snip upside-down quoting]

--
Tad McClellan SGML consulting
ta***@augustmail.com Perl programming
Fort Worth, Texas

[Dan]

Ta, the problem is fixed now,
I forgot to unset my global $currentTag
in the &endTag event handler, leading to
the 'dribs and drabs' below, which actually
belonged to outer tags (I was mistakenly
giving them to $currentTag).

With this bug gone I can now safely

$data{$currentTag} .= $text;

where I had been

$data{$currentTag} = $text if !$data{$currentTag};

Hence my occasional missing characters.

Thanks very much for all the kind help
and advice,

Regards,

Dan.

DIY GENOME...
perl -e '@A=qw(A T C G); for(1..10**6){print $A[rand(@A)]}' > \
myGenome.txt

Tad McClellan wrote:

Dan <dm*@mrc-dunn.cam.ac.uk> wrote:

I am now using XML::Parser,
which is working nicely, apart
from the occasional weird behaviour,
in some cases characters go missing,

The error is persistant, i.e. not a random
caracter, but the same character each time
goes missing.

If you show us a short and complete program that we can run that
illustrates your problem, then we can surely help you solve
your problem.

It is really confusing.

Can't help with unseen code...

Also my Char event gets called 3 times
per tag,

That is "normal".

You'll get the PCDATA in dribs and drabs, so you need to keep collecting
it until you reach the end of the containing element.

even though there are no new
lines anywhere in teh tag text.

The concept of "lines" is not present in XML.

Remove every newline from an XML document, and it is still
an XML document.

[snip upside-down quoting]