473,320 Members | 2,024 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

How to search HUGE XML with DOM?

a relation database has admiring search efficiency when the database is
very big (several thousands or tens of thousands of records). But my
current project is based on XML, for its tree-like data structure has
much more flexibility; and DOM, which could be manipulated just like a
tree. However, how to establish such a XML data base for search when it
contains 10,000 records (One record usually contain 10~30 tags) or
more?

My search needs:
1. Search and return all the record (an element) with specific id.
2. Search and return all the record whose child nodes has a specific id
or attribute.

the xml.dom.minidom object is too slow when parsing such a big XML file
to a DOM object. while pulldom should spend quite a long time going
through the whole database file. How to enhance the searching speed?
Are there existing solution or algorithm? Thank you for your
suggetion...

Mar 31 '06 #1
8 1676
> the xml.dom.minidom object is too slow when parsing such a big XML file
to a DOM object. while pulldom should spend quite a long time going
through the whole database file. How to enhance the searching speed?
Are there existing solution or algorithm? Thank you for your
suggetion...


I've told you that before, and I tell you again: RDBMS is the way to go.
There might be XML-parsers that work faster - I suppose cElementTree can
gain you some speed - but ultimately the problems are inherent in the
representation as DOM: no type-information, no indices, no nothing. Just a
huge pile of nodes in memory.

So all searches are linear in the number of nodes. Of course you might be
able to create indices yourself, even devise a clever scheme to make using
them as declarative as possible. But that would in the end mean nothing but
re-creating RDBMS technology - why do that, if it's already there?

Maybe there are frameworks out there that support you in this, but the very
nature of XML makes that for sure a more tedious task than just defining a
simple SQL-Schema. If I'd have to search for some XML-tools that go beyond
DOM, I'd go for uche ogbuji's 4suite as a starter and work my way down from
there - maybe AMARA is what you need?

Now having said that: I'm not a SQL-bigot. Just use the right tool for the
job.

Regards,

Diez
Mar 31 '06 #2
Sullivan WxPyQtKinter wrote:
a relation database has admiring search efficiency when the database is
very big (several thousands or tens of thousands of records). But my
current project is based on XML, for its tree-like data structure has
much more flexibility; and DOM, which could be manipulated just like a
tree. However, how to establish such a XML data base for search when it
contains 10,000 records (One record usually contain 10~30 tags) or
more?

My search needs:
1. Search and return all the record (an element) with specific id.
2. Search and return all the record whose child nodes has a specific id
or attribute.

the xml.dom.minidom object is too slow when parsing such a big XML file
to a DOM object. while pulldom should spend quite a long time going
through the whole database file. How to enhance the searching speed?
Are there existing solution or algorithm? Thank you for your
suggetion...


- have a look at cElementTree ?
- store your XML as persistant objects in a ZODB instance, then use ZODB
catalog for queries ?
- index relevant data in a DB (RDBMS, Berkeley, whatever...) ?
- have a look at 4suite (http://4suite.org/index.xhtml) ?

My 2 cents...
--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom.gro'.split('@')])"
Mar 31 '06 #3
Diez B. Roggisch wrote:
the xml.dom.minidom object is too slow when parsing such a big XML file
to a DOM object. while pulldom should spend quite a long time going
through the whole database file. How to enhance the searching speed?
Are there existing solution or algorithm? Thank you for your
suggetion...
I've told you that before, and I tell you again: RDBMS is the way to go.


We've lost some context from the original post that may be relevant
here, but if populating what the original questioner calls "the
database" is an infrequent operation, then an RDBMS probably is the way
to go, in general. On the other hand, if a lot of parsing has to happen
in order to perform a search, such parsing would probably incur a lot
of overhead from SQL inserts that wouldn't be particularly desirable.
There might be XML-parsers that work faster - I suppose cElementTree can
gain you some speed - but ultimately the problems are inherent in the
representation as DOM: no type-information, no indices, no nothing. Just a
huge pile of nodes in memory.
Well, I would hope that W3C DOM operations like getElementById would be
supported by some index in the implementation: that would make some of
the searches mentioned by the questioner fairly rapid, given enough
memory.
So all searches are linear in the number of nodes. Of course you might be
able to create indices yourself, even devise a clever scheme to make using
them as declarative as possible. But that would in the end mean nothing but
re-creating RDBMS technology - why do that, if it's already there?


I agree that careful usage of RDBMS technology would solve the general
problems of searching large amounts of data, but the stated queries
should involve indexes and be fairly quick.

Paul

Mar 31 '06 #4
Mind, that XML documents are not more flexible than RDBMS.

You can represent any XML document in a RDBMS. You cannot represent any
RDBMS in an XML document. RDBMS are (strictly spoken) relations and XML
documents are trees. Relations are superior to trees, at least
mathematically speaking.

Once you have set up your system in a practicable way (e.G. not needing
to create a new table via SQL Queries for a new type of node, which
would be a pain) SQL is far superior to XML.

Anyway, cElementTree seems to be the best way to go for you now. Its
performance is untopped by any other python xml library, as far as I
know.

Mar 31 '06 #5

On 31-Mar-06, at 11:17 AM, bayerj wrote:
Mind, that XML documents are not more flexible than RDBMS.

You can represent any XML document in a RDBMS. You cannot represent
any
RDBMS in an XML document. RDBMS are (strictly spoken) relations and
XML
documents are trees. Relations are superior to trees, at least
mathematically speaking.

Once you have set up your system in a practicable way (e.G. not
needing
to create a new table via SQL Queries for a new type of node, which
would be a pain) SQL is far superior to XML.

Anyway, cElementTree seems to be the best way to go for you now. Its
performance is untopped by any other python xml library, as far as I
know.

--
http://mail.python.org/mailman/listinfo/python-list


If I may hijack this thread for a bit, I'd like to dig deeper into
this issue :)

Currently my simulation program produces an XML log file with events
represented as nodes.
Often those files grow to multiple GB size. I like this setup because
the format is open
and easily parse-able with a variety of tools. So I have a bunch I
scripts that can analyze
different aspects of the simulation.

I have not much clue about databases, except that they exist,
somewhat complex, and often
use proprietary formats for efficiency. So any points on whether RDBM-
based setup
would be better would be greatly appreciated.

Even trivial aspects, such as whether to produce RDBM during the
simulation, or convert the complete XML log file into one, are not
entirely clear to me. I gather that RDBM would be much better suited
for analysis, but what about portability ? Is database file a
separate entity that may be passed around?

Apologies if this seems like a selfish question, perhaps consider it
a full disclosure, different set-ups/examples would be appreciated as
well.

--
Cheers, Ivan

Mar 31 '06 #6
Perhaps what you have said is correct. But XML is more direct for
programmers and readers in my view point.

bayerj 写道:
Mind, that XML documents are not more flexible than RDBMS.

You can represent any XML document in a RDBMS. You cannot represent any
RDBMS in an XML document. RDBMS are (strictly spoken) relations and XML
documents are trees. Relations are superior to trees, at least
mathematically speaking.

Once you have set up your system in a practicable way (e.G. not needing
to create a new table via SQL Queries for a new type of node, which
would be a pain) SQL is far superior to XML.

Anyway, cElementTree seems to be the best way to go for you now. Its
performance is untopped by any other python xml library, as far as I
know.


Apr 1 '06 #7
Ivan Vinogradov wrote:
I have not much clue about databases, except that they exist, somewhat
complex, and often use proprietary formats for efficiency.
Prorietary storage format, but a standardized API...
So any points on whether RDBM-based setup
would be better would be greatly appreciated.
The typical use case for RDBMS is that you have a number
of record types (classes/relations/tables) with a regular
structure, and all data fits into these structures. When
you want to search for something, you know exactly in what
field of what table to look (but not which item of course).
You also typically have multiple users who need to be able
to update the same database simultaneously without getting
in each others way.
Even trivial aspects, such as whether to produce RDBM during the
simulation, or convert the complete XML log file into one, are not
entirely clear to me.
Most databases as suited at writing data in fairly small chunks,
although it's typically much faster to write 100 items in a
transaction, than to write 100 transactions with one item each.
I gather that RDBM would be much better suited for
analysis, but what about portability ? Is database file a separate
entity that may be passed around?


Who says that a database needs to reside in a file? Most
databases reside on disk, but it might well be in raw
partitions.

In general, you should see the database as a persistent
representation of data in a system. It's not a transport
mechanism.
Apr 7 '06 #8
Sullivan WxPyQtKinter wrote:
My search needs:
1. Search and return all the record (an element) with specific id.
2. Search and return all the record whose child nodes has a specific id
or attribute.


Try lxml, which is based on the libxml2 library. The current SVN version has
support for xml:id through the XMLDTDID function. It simply returns an XML
tree and an ID dictionary.

http://codespeak.net/lxml

Stefan
Apr 25 '06 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: leegold2 | last post by:
Commonly done, eg. you enter a word in a search engine and when a hit-page comes up the search word(s) are highlighted. I'm doing a fulltext search that works well but I've tried a few "packaged...
0
by: j | last post by:
Hi, Anyone out there with binary search tree experience. Working on a project due tomorrow and really stuck. We need a function that splits a binary tree into a bigger one and smaller one(for a...
6
by: richard.pasco | last post by:
Hey all I am trying to write a script that allows users to search through a database of names. But rather than give a search string and then return all those that match, I would like it to...
0
by: Huihong | last post by:
Please check out our source code search engine here, http://www.codase.com e.g., search main method,...
3
by: jdworley | last post by:
Howdy! The search is a 1 GB+ file from a crash to be exact - had an e-mail correspondence (window) hopefully in it so I saved the file on reboot before windoze kicked in and want to open and...
8
by: Huihong | last post by:
Please check out our newly released source code search engine here, http://www.codase.com e.g., search socket method call,...
6
by: DC | last post by:
Hi, I am programming a search catalogue with 200000 items (and growing). I am currently using the SQL Server 2000 fulltext engine for this task but it does not fit the requirements anymore. ...
0
by: Huihong | last post by:
Please check out our source code search engine here, http://www.codase.com e.g., search the "main" method,...
9
by: Rick | last post by:
I have a large list of objects where each object has a unique (non-overlapping) date range. The list is sorted chronologically. What is the most efficient way to search this list for a single...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, youll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shllpp 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.