473,372 Members | 1,039 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,372 software developers and data experts.

XML::Simple in perl?

Dan
Using XML::Simple in perl is
extreemly slow to parse big
XML files (can be up to 250M,
taking ~1h).

How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?

Should I consider using (learning)
Expat.c for increased performance?

How long would parsing a 250M XML
file take with Expat?

Thanks for any suggestions you can give,
Dan.

Jul 20 '05 #1
8 6462
yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100:
Using XML::Simple in perl is
extreemly slow to parse big
XML files (can be up to 250M,
taking ~1h).
XML::Simple is not the only module on CPAN.
There are also
XML::Smart
XML::Parser
XML::LibXML
....
How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?
Note that SAX is basical an API for XML triggering starting and ending
tags and similar. That's different to the previous mentioned modules, as
they also generate a huge tree view to the XML document.

But SAX can reduce the memory wastage and gain in time advantage.
In Perl you can use e.g. XML::SAX for it.
Should I consider using (learning)
Expat.c for increased performance?


XML::Parser and XML::SAX::Expat are e.g. based on Expat, so you might use
the Perl interfaces if you want.

Greetings,
Janek
Jul 20 '05 #2
Dan


Janek Schleicher wrote:
yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100:

Using XML::Simple in perl is
extreemly slow to parse big
XML files (can be up to 250M,
taking ~1h).

XML::Simple is not the only module on CPAN.
There are also
XML::Smart
XML::Parser
XML::LibXML
...


Which ones have the same 'front end' as
XML simple? I would rather not change
code if I don't have to.

Which one uses least memory on big files?

How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?

Note that SAX is basical an API for XML triggering starting and ending
tags and similar. That's different to the previous mentioned modules, as
they also generate a huge tree view to the XML document.

But SAX can reduce the memory wastage and gain in time advantage.
In Perl you can use e.g. XML::SAX for it.

Should I consider using (learning)
Expat.c for increased performance?

XML::Parser and XML::SAX::Expat are e.g. based on Expat, so you might use
the Perl interfaces if you want.


I think I will end up doing this...

Is it any slower?

My requirements are extreemly simple, I just need
to print tab delimited lines to a file, so I was
thinking I could take this opportunity to try
to learn c....

Cheers,
Dan.

Greetings,
Janek


Jul 20 '05 #3
Dan <dm*@mrc-dunn.cam.ac.uk> wrote:
Janek Schleicher wrote:
yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100:
extreemly slow to parse big
There are also
XML::Smart
XML::Parser
XML::LibXML
...
Which ones have the same 'front end' as
XML simple?

None of them.

The "simple" front end is why the module is called Simple.

I would rather not change
code if I don't have to.

Too late. :-(

Which one uses least memory on big files?

None of those ones.

How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?
But SAX can reduce the memory wastage and gain in time advantage.

That one.

There is also a mailing list specifically for doing XML processing
using Perl:

http://listserv.ActiveState.com/mail...tinfo/perl-xml
--
Tad McClellan SGML consulting
ta***@augustmail.com Perl programming
Fort Worth, Texas
Jul 20 '05 #4
cp
In article <3F**************@mrc-dunn.cam.ac.uk>, Dan
<dm*@mrc-dunn.cam.ac.uk> wrote:
Using XML::Simple in perl is
extreemly slow to parse big
XML files (can be up to 250M,
taking ~1h).
XML::Simple is limited. The author points it out in the docs. It was
designed for a fairly specific purpose, parsing small configuration
files written in XML. It has since been expanded on, but it's core
remains, well, simple.
How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?


The docs for XML::Simple Version 2.05 suggest that you can (as of
version 1.08) use a SAX parser with XML::Simple, the benefits of which
are:

Applications written to the SAX API can extract data
from huge XML documents without the memory overheads
of a DOM or tree API.

So you might read the docs, use a SAX parser, and see some speed
benefit. You might not.

--
cp
Jul 20 '05 #5
Dan
I am now using XML::Parser,
which is working nicely, apart
from the occasional weird behaviour,
in some cases characters go missing,
(i.e. I get a 1 instead of 19).

The error is persistant, i.e. not a random
caracter, but the same character each time
goes missing.

It is really confusing.

Also my Char event gets called 3 times
per tag, even though there are no new
lines anywhere in teh tag text.

Little frustrating problems....

Dan.

cp wrote:
In article <3F**************@mrc-dunn.cam.ac.uk>, Dan
<dm*@mrc-dunn.cam.ac.uk> wrote:

Using XML::Simple in perl is
extreemly slow to parse big
XML files (can be up to 250M,
taking ~1h).

XML::Simple is limited. The author points it out in the docs. It was
designed for a fairly specific purpose, parsing small configuration
files written in XML. It has since been expanded on, but it's core
remains, well, simple.

How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?

The docs for XML::Simple Version 2.05 suggest that you can (as of
version 1.08) use a SAX parser with XML::Simple, the benefits of which
are:

Applications written to the SAX API can extract data
from huge XML documents without the memory overheads
of a DOM or tree API.

So you might read the docs, use a SAX parser, and see some speed
benefit. You might not.


Jul 20 '05 #6
Tad McClellan wrote:
Dan <dm*@mrc-dunn.cam.ac.uk> wrote:
Janek Schleicher wrote:
yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100: [XML::Simple is] extreemly slow to parse big [files] There are also
XML::Smart
XML::Parser
XML::LibXML
...


Which ones have the same 'front end' as XML simple?


None of them.

The "simple" front end is why the module is called Simple.


As a matter of fact XML::Smart has the same 'front end', and XML::Twig has a
method names 'simplify', which generates the same data structure as
XML::Simple, on a tree or on a sub-tree.

One possible cause for the problem might be that XML::Simple could be using
XML::SAX::PurePerl as its parser, which is very slow. This depends on your
installation. If you have XML::LibXML or XML::Parser installed you can set
the $XML::Simple::PREFERRED_PARSER variable to tell it to use an other
parser, see the docs.
Which one uses least memory on big files?


You might want to have a look at XML::Twig, which is specially designed for
big files (but I might be slightly biased ;--)

__
Michel Rodriguez
Perl &amp; XML
http://xmltwig.com
Jul 20 '05 #7
Dan <dm*@mrc-dunn.cam.ac.uk> wrote:
I am now using XML::Parser,
which is working nicely, apart
from the occasional weird behaviour,
in some cases characters go missing, The error is persistant, i.e. not a random
caracter, but the same character each time
goes missing.

If you show us a short and complete program that we can run that
illustrates your problem, then we can surely help you solve
your problem.

It is really confusing.

Can't help with unseen code...

Also my Char event gets called 3 times
per tag,

That is "normal".

You'll get the PCDATA in dribs and drabs, so you need to keep collecting
it until you reach the end of the containing element.

even though there are no new
lines anywhere in teh tag text.

The concept of "lines" is not present in XML.

Remove every newline from an XML document, and it is still
an XML document.

[snip upside-down quoting]

--
Tad McClellan SGML consulting
ta***@augustmail.com Perl programming
Fort Worth, Texas
Jul 20 '05 #8
Dan
Ta, the problem is fixed now,
I forgot to unset my global $currentTag
in the &endTag event handler, leading to
the 'dribs and drabs' below, which actually
belonged to outer tags (I was mistakenly
giving them to $currentTag).

With this bug gone I can now safely

$data{$currentTag} .= $text;

where I had been

$data{$currentTag} = $text if !$data{$currentTag};

Hence my occasional missing characters.

Thanks very much for all the kind help
and advice,

Regards,

Dan.

DIY GENOME...
perl -e '@A=qw(A T C G); for(1..10**6){print $A[rand(@A)]}' > \
myGenome.txt

Tad McClellan wrote:
Dan <dm*@mrc-dunn.cam.ac.uk> wrote:
I am now using XML::Parser,
which is working nicely, apart
from the occasional weird behaviour,
in some cases characters go missing,


The error is persistant, i.e. not a random
caracter, but the same character each time
goes missing.


If you show us a short and complete program that we can run that
illustrates your problem, then we can surely help you solve
your problem.
It is really confusing.


Can't help with unseen code...
Also my Char event gets called 3 times
per tag,


That is "normal".

You'll get the PCDATA in dribs and drabs, so you need to keep collecting
it until you reach the end of the containing element.
even though there are no new
lines anywhere in teh tag text.


The concept of "lines" is not present in XML.

Remove every newline from an XML document, and it is still
an XML document.

[snip upside-down quoting]


Jul 20 '05 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

13
by: Paulo Pinto | last post by:
Hi, does anyone know of a Python package that is able to load XML like the XML::Simple Perl package does? For those that don't know it, this package maps the XML file to a dictionary.
0
by: Randy | last post by:
Is there a dotnet class that formats XML simple types. I'm making a xmldocument which has a timestamp element (among others). The format is: yyyy-MM-ddThh:mm:ss (which looks like...
6
by: Lindy | last post by:
I'm using VB .Net and am brand new to XML. I need to create an XML file with the following lines: <?xml version="1.0" encoding="UTF-8" ?> - <HC_DATA...
1
by: Miguel Manso | last post by:
Hi there, I'm a Perl programmer trying to get into Python. I've been reading some documentation and I've choosed Python has being the "next step" to give. Can you point me out to Python...
1
by: jack | last post by:
Hi all, I am working on perl..and am using XML::Simple to parse a xml document. I've been trying to retrieve character data from tags whose occurance is recursive.. The scenario can be better...
0
by: Marv | last post by:
Is it possible to print the path of all leaf nodes of an XML using XML::Simple This is the kind of output text that i'm trying to print (not the leaf node values but the path to reach them) c:\>...
0
by: JohnLucas | last post by:
Hi all, I have just started working with the XML::Simple module to parse an XML file. I'm trying to pull some values from the file that I need in another program. The problem is that the XML...
4
by: Steven M. O'Neill | last post by:
I have an xml structure like this: <Meta name="fieldAttributes"> <MetaString name="name">SUB_PHONE</MetaString> <MetaString name="value">999999999</MetaString> </Meta> <Meta...
5
by: 0xception | last post by:
Hi, I'm attempting to create a perl script that will modify a series of RRD databases (a couple hundred of them). in order to do this the RRD database can be exported to XML modified and then...
0
by: ryjfgjl | last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.