Using XML::Simple in perl is
extreemly slow to parse big
XML files (can be up to 250M,
taking ~1h).
How can I increase my performance /
reduce my memory usage?
Is SAX the way forward?
Should I consider using (learning)
Expat.c for increased performance?
How long would parsing a 250M XML
file take with Expat?
Thanks for any suggestions you can give,
Dan. 8 6506
yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100: Using XML::Simple in perl is extreemly slow to parse big XML files (can be up to 250M, taking ~1h).
XML::Simple is not the only module on CPAN.
There are also
XML::Smart
XML::Parser
XML::LibXML
....
How can I increase my performance / reduce my memory usage?
Is SAX the way forward?
Note that SAX is basical an API for XML triggering starting and ending
tags and similar. That's different to the previous mentioned modules, as
they also generate a huge tree view to the XML document.
But SAX can reduce the memory wastage and gain in time advantage.
In Perl you can use e.g. XML::SAX for it.
Should I consider using (learning) Expat.c for increased performance?
XML::Parser and XML::SAX::Expat are e.g. based on Expat, so you might use
the Perl interfaces if you want.
Greetings,
Janek
Janek Schleicher wrote: yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100:
Using XML::Simple in perl is extreemly slow to parse big XML files (can be up to 250M, taking ~1h).
XML::Simple is not the only module on CPAN. There are also XML::Smart XML::Parser XML::LibXML ...
Which ones have the same 'front end' as
XML simple? I would rather not change
code if I don't have to.
Which one uses least memory on big files?
How can I increase my performance / reduce my memory usage?
Is SAX the way forward?
Note that SAX is basical an API for XML triggering starting and ending tags and similar. That's different to the previous mentioned modules, as they also generate a huge tree view to the XML document.
But SAX can reduce the memory wastage and gain in time advantage. In Perl you can use e.g. XML::SAX for it.
Should I consider using (learning) Expat.c for increased performance?
XML::Parser and XML::SAX::Expat are e.g. based on Expat, so you might use the Perl interfaces if you want.
I think I will end up doing this...
Is it any slower?
My requirements are extreemly simple, I just need
to print tab delimited lines to a file, so I was
thinking I could take this opportunity to try
to learn c....
Cheers,
Dan.
Greetings, Janek
Dan <dm*@mrc-dunn.cam.ac.uk> wrote: Janek Schleicher wrote: yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100:
extreemly slow to parse big
There are also XML::Smart XML::Parser XML::LibXML ... Which ones have the same 'front end' as XML simple?
None of them.
The "simple" front end is why the module is called Simple.
I would rather not change code if I don't have to.
Too late. :-(
Which one uses least memory on big files?
None of those ones.
How can I increase my performance / reduce my memory usage?
Is SAX the way forward?
But SAX can reduce the memory wastage and gain in time advantage.
That one.
There is also a mailing list specifically for doing XML processing
using Perl: http://listserv.ActiveState.com/mail...tinfo/perl-xml
--
Tad McClellan SGML consulting ta***@augustmai l.com Perl programming
Fort Worth, Texas
In article <3F************ **@mrc-dunn.cam.ac.uk> , Dan
<dm*@mrc-dunn.cam.ac.uk> wrote: Using XML::Simple in perl is extreemly slow to parse big XML files (can be up to 250M, taking ~1h).
XML::Simple is limited. The author points it out in the docs. It was
designed for a fairly specific purpose, parsing small configuration
files written in XML. It has since been expanded on, but it's core
remains, well, simple.
How can I increase my performance / reduce my memory usage?
Is SAX the way forward?
The docs for XML::Simple Version 2.05 suggest that you can (as of
version 1.08) use a SAX parser with XML::Simple, the benefits of which
are:
Applications written to the SAX API can extract data
from huge XML documents without the memory overheads
of a DOM or tree API.
So you might read the docs, use a SAX parser, and see some speed
benefit. You might not.
--
cp
I am now using XML::Parser,
which is working nicely, apart
from the occasional weird behaviour,
in some cases characters go missing,
(i.e. I get a 1 instead of 19).
The error is persistant, i.e. not a random
caracter, but the same character each time
goes missing.
It is really confusing.
Also my Char event gets called 3 times
per tag, even though there are no new
lines anywhere in teh tag text.
Little frustrating problems....
Dan.
cp wrote: In article <3F************ **@mrc-dunn.cam.ac.uk> , Dan <dm*@mrc-dunn.cam.ac.uk> wrote:
Using XML::Simple in perl is extreemly slow to parse big XML files (can be up to 250M, taking ~1h).
XML::Simple is limited. The author points it out in the docs. It was designed for a fairly specific purpose, parsing small configuration files written in XML. It has since been expanded on, but it's core remains, well, simple.
How can I increase my performance / reduce my memory usage?
Is SAX the way forward?
The docs for XML::Simple Version 2.05 suggest that you can (as of version 1.08) use a SAX parser with XML::Simple, the benefits of which are:
Applications written to the SAX API can extract data from huge XML documents without the memory overheads of a DOM or tree API.
So you might read the docs, use a SAX parser, and see some speed benefit. You might not.
Tad McClellan wrote: Dan <dm*@mrc-dunn.cam.ac.uk> wrote: Janek Schleicher wrote: yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100: [XML::Simple is] extreemly slow to parse big [files] There are also XML::Smart XML::Parser XML::LibXML ...
Which ones have the same 'front end' as XML simple?
None of them.
The "simple" front end is why the module is called Simple.
As a matter of fact XML::Smart has the same 'front end', and XML::Twig has a
method names 'simplify', which generates the same data structure as
XML::Simple, on a tree or on a sub-tree.
One possible cause for the problem might be that XML::Simple could be using
XML::SAX::PureP erl as its parser, which is very slow. This depends on your
installation. If you have XML::LibXML or XML::Parser installed you can set
the $XML::Simple::P REFERRED_PARSER variable to tell it to use an other
parser, see the docs. Which one uses least memory on big files?
You might want to have a look at XML::Twig, which is specially designed for
big files (but I might be slightly biased ;--)
__
Michel Rodriguez
Perl & XML http://xmltwig.com
Dan <dm*@mrc-dunn.cam.ac.uk> wrote: I am now using XML::Parser, which is working nicely, apart from the occasional weird behaviour, in some cases characters go missing,
The error is persistant, i.e. not a random caracter, but the same character each time goes missing.
If you show us a short and complete program that we can run that
illustrates your problem, then we can surely help you solve
your problem.
It is really confusing.
Can't help with unseen code...
Also my Char event gets called 3 times per tag,
That is "normal".
You'll get the PCDATA in dribs and drabs, so you need to keep collecting
it until you reach the end of the containing element.
even though there are no new lines anywhere in teh tag text.
The concept of "lines" is not present in XML.
Remove every newline from an XML document, and it is still
an XML document.
[snip upside-down quoting]
--
Tad McClellan SGML consulting ta***@augustmai l.com Perl programming
Fort Worth, Texas
Ta, the problem is fixed now,
I forgot to unset my global $currentTag
in the &endTag event handler, leading to
the 'dribs and drabs' below, which actually
belonged to outer tags (I was mistakenly
giving them to $currentTag).
With this bug gone I can now safely
$data{$currentT ag} .= $text;
where I had been
$data{$currentT ag} = $text if !$data{$current Tag};
Hence my occasional missing characters.
Thanks very much for all the kind help
and advice,
Regards,
Dan.
DIY GENOME...
perl -e '@A=qw(A T C G); for(1..10**6){p rint $A[rand(@A)]}' > \
myGenome.txt
Tad McClellan wrote: Dan <dm*@mrc-dunn.cam.ac.uk> wrote:
I am now using XML::Parser, which is working nicely, apart from the occasional weird behaviour, in some cases characters go missing,
The error is persistant, i.e. not a random caracter, but the same character each time goes missing. If you show us a short and complete program that we can run that illustrates your problem, then we can surely help you solve your problem. It is really confusing. Can't help with unseen code... Also my Char event gets called 3 times per tag, That is "normal".
You'll get the PCDATA in dribs and drabs, so you need to keep collecting it until you reach the end of the containing element. even though there are no new lines anywhere in teh tag text. The concept of "lines" is not present in XML.
Remove every newline from an XML document, and it is still an XML document. [snip upside-down quoting] This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Paulo Pinto |
last post by:
Hi,
does anyone know of a Python package that
is able to load XML like the XML::Simple
Perl package does?
For those that don't know it, this package
maps the XML file to a dictionary.
|
by: Randy |
last post by:
Is there a dotnet class that formats XML simple types. I'm making a
xmldocument which has a timestamp element (among others). The format is:
yyyy-MM-ddThh:mm:ss (which looks like 2003-09-28T07:55:06).
The schema looks like:
<xsd:simpleType name="TimestampType">
<xsd:annotation>
<xsd:documentation>Base typefor a date and time stamp</xsd:documentation>
</xsd:annotation>
|
by: Lindy |
last post by:
I'm using VB .Net and am brand new to XML.
I need to create an XML file with the following lines:
<?xml version="1.0" encoding="UTF-8" ?>
- <HC_DATA xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://.../asc.xsd">
other stuff...
</HC_DATA>
|
by: Miguel Manso |
last post by:
Hi there,
I'm a Perl programmer trying to get into Python. I've been reading some
documentation and I've choosed Python has being the "next step" to give.
Can you point me out to Python solutions for:
1) Perl's Data::Dumper
It dumps any perl variable to the stdout in a "readable" way.
|
by: jack |
last post by:
Hi all,
I am working on perl..and am using XML::Simple to parse a xml document.
I've been trying to retrieve character data from tags whose occurance
is recursive.. The scenario can be better explained as below:
For a structure like,
<employees>
<employee>
| |
by: Marv |
last post by:
Is it possible to print the path of all leaf nodes of an XML using XML::Simple
This is the kind of output text that i'm trying to print (not the leaf node values but the path to reach them)
c:\> perl ListLeafs.pl config.xml
{RootNode}->{ChildNode1}->{LeafNode1}
{RootNode}->{ChildNode1}->{LeafNode2}
{RootNode}->{ChildNode1}->{LeafNode3}
{RootNode}->{ChildNode2}->{LeafNode1}
{RootNode}->{ChildNode2}->{LeafNode2}
|
by: JohnLucas |
last post by:
Hi all,
I have just started working with the XML::Simple module to parse an XML file.
I'm trying to pull some values from the file that I need in another program.
The problem is that the XML file is "nested" and I can't get the correct values out of the file.
This is part of my xml file:
<?xml version="1.0" encoding="ISO-8859-1"?>
<Root>
|
by: Steven M. O'Neill |
last post by:
I have an xml structure like this:
<Meta name="fieldAttributes">
<MetaString name="name">SUB_PHONE</MetaString>
<MetaString name="value">999999999</MetaString>
</Meta>
<Meta name="fieldAttributes">
<MetaString name="name">SUB_NAME</MetaString>
<MetaString name="value">XML PLUS</MetaString>
|
by: 0xception |
last post by:
Hi,
I'm attempting to create a perl script that will modify a series of RRD databases (a couple hundred of them). in order to do this the RRD database can be exported to XML modified and then restored from XML.
My issue is when using XML::Simple to read in and edit the xml structure and then output it back out the section of XML that was
previously:
<cdp_prep>
<ds>
...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |