a relation database has admiring search efficiency when the database is
very big (several thousands or tens of thousands of records). But my
current project is based on XML, for its tree-like data structure has
much more flexibility; and DOM, which could be manipulated just like a
tree. However, how to establish such a XML data base for search when it
contains 10,000 records (One record usually contain 10~30 tags) or
more?
My search needs:
1. Search and return all the record (an element) with specific id.
2. Search and return all the record whose child nodes has a specific id
or attribute.
the xml.dom.minidom object is too slow when parsing such a big XML file
to a DOM object. while pulldom should spend quite a long time going
through the whole database file. How to enhance the searching speed?
Are there existing solution or algorithm? Thank you for your
suggetion... 8 1676
> the xml.dom.minidom object is too slow when parsing such a big XML file to a DOM object. while pulldom should spend quite a long time going through the whole database file. How to enhance the searching speed? Are there existing solution or algorithm? Thank you for your suggetion...
I've told you that before, and I tell you again: RDBMS is the way to go.
There might be XML-parsers that work faster - I suppose cElementTree can
gain you some speed - but ultimately the problems are inherent in the
representation as DOM: no type-information, no indices, no nothing. Just a
huge pile of nodes in memory.
So all searches are linear in the number of nodes. Of course you might be
able to create indices yourself, even devise a clever scheme to make using
them as declarative as possible. But that would in the end mean nothing but
re-creating RDBMS technology - why do that, if it's already there?
Maybe there are frameworks out there that support you in this, but the very
nature of XML makes that for sure a more tedious task than just defining a
simple SQL-Schema. If I'd have to search for some XML-tools that go beyond
DOM, I'd go for uche ogbuji's 4suite as a starter and work my way down from
there - maybe AMARA is what you need?
Now having said that: I'm not a SQL-bigot. Just use the right tool for the
job.
Regards,
Diez
Sullivan WxPyQtKinter wrote: a relation database has admiring search efficiency when the database is very big (several thousands or tens of thousands of records). But my current project is based on XML, for its tree-like data structure has much more flexibility; and DOM, which could be manipulated just like a tree. However, how to establish such a XML data base for search when it contains 10,000 records (One record usually contain 10~30 tags) or more?
My search needs: 1. Search and return all the record (an element) with specific id. 2. Search and return all the record whose child nodes has a specific id or attribute.
the xml.dom.minidom object is too slow when parsing such a big XML file to a DOM object. while pulldom should spend quite a long time going through the whole database file. How to enhance the searching speed? Are there existing solution or algorithm? Thank you for your suggetion...
- have a look at cElementTree ?
- store your XML as persistant objects in a ZODB instance, then use ZODB
catalog for queries ?
- index relevant data in a DB (RDBMS, Berkeley, whatever...) ?
- have a look at 4suite ( http://4suite.org/index.xhtml) ?
My 2 cents...
--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom.gro'.split('@')])"
Diez B. Roggisch wrote: the xml.dom.minidom object is too slow when parsing such a big XML file to a DOM object. while pulldom should spend quite a long time going through the whole database file. How to enhance the searching speed? Are there existing solution or algorithm? Thank you for your suggetion... I've told you that before, and I tell you again: RDBMS is the way to go.
We've lost some context from the original post that may be relevant
here, but if populating what the original questioner calls "the
database" is an infrequent operation, then an RDBMS probably is the way
to go, in general. On the other hand, if a lot of parsing has to happen
in order to perform a search, such parsing would probably incur a lot
of overhead from SQL inserts that wouldn't be particularly desirable.
There might be XML-parsers that work faster - I suppose cElementTree can gain you some speed - but ultimately the problems are inherent in the representation as DOM: no type-information, no indices, no nothing. Just a huge pile of nodes in memory.
Well, I would hope that W3C DOM operations like getElementById would be
supported by some index in the implementation: that would make some of
the searches mentioned by the questioner fairly rapid, given enough
memory.
So all searches are linear in the number of nodes. Of course you might be able to create indices yourself, even devise a clever scheme to make using them as declarative as possible. But that would in the end mean nothing but re-creating RDBMS technology - why do that, if it's already there?
I agree that careful usage of RDBMS technology would solve the general
problems of searching large amounts of data, but the stated queries
should involve indexes and be fairly quick.
Paul
Mind, that XML documents are not more flexible than RDBMS.
You can represent any XML document in a RDBMS. You cannot represent any
RDBMS in an XML document. RDBMS are (strictly spoken) relations and XML
documents are trees. Relations are superior to trees, at least
mathematically speaking.
Once you have set up your system in a practicable way (e.G. not needing
to create a new table via SQL Queries for a new type of node, which
would be a pain) SQL is far superior to XML.
Anyway, cElementTree seems to be the best way to go for you now. Its
performance is untopped by any other python xml library, as far as I
know.
On 31-Mar-06, at 11:17 AM, bayerj wrote: Mind, that XML documents are not more flexible than RDBMS.
You can represent any XML document in a RDBMS. You cannot represent any RDBMS in an XML document. RDBMS are (strictly spoken) relations and XML documents are trees. Relations are superior to trees, at least mathematically speaking.
Once you have set up your system in a practicable way (e.G. not needing to create a new table via SQL Queries for a new type of node, which would be a pain) SQL is far superior to XML.
Anyway, cElementTree seems to be the best way to go for you now. Its performance is untopped by any other python xml library, as far as I know.
-- http://mail.python.org/mailman/listinfo/python-list
If I may hijack this thread for a bit, I'd like to dig deeper into
this issue :)
Currently my simulation program produces an XML log file with events
represented as nodes.
Often those files grow to multiple GB size. I like this setup because
the format is open
and easily parse-able with a variety of tools. So I have a bunch I
scripts that can analyze
different aspects of the simulation.
I have not much clue about databases, except that they exist,
somewhat complex, and often
use proprietary formats for efficiency. So any points on whether RDBM-
based setup
would be better would be greatly appreciated.
Even trivial aspects, such as whether to produce RDBM during the
simulation, or convert the complete XML log file into one, are not
entirely clear to me. I gather that RDBM would be much better suited
for analysis, but what about portability ? Is database file a
separate entity that may be passed around?
Apologies if this seems like a selfish question, perhaps consider it
a full disclosure, different set-ups/examples would be appreciated as
well.
--
Cheers, Ivan
Perhaps what you have said is correct. But XML is more direct for
programmers and readers in my view point.
bayerj 写道: Mind, that XML documents are not more flexible than RDBMS.
You can represent any XML document in a RDBMS. You cannot represent any RDBMS in an XML document. RDBMS are (strictly spoken) relations and XML documents are trees. Relations are superior to trees, at least mathematically speaking.
Once you have set up your system in a practicable way (e.G. not needing to create a new table via SQL Queries for a new type of node, which would be a pain) SQL is far superior to XML.
Anyway, cElementTree seems to be the best way to go for you now. Its performance is untopped by any other python xml library, as far as I know.
Ivan Vinogradov wrote: I have not much clue about databases, except that they exist, somewhat complex, and often use proprietary formats for efficiency.
Prorietary storage format, but a standardized API...
So any points on whether RDBM-based setup would be better would be greatly appreciated.
The typical use case for RDBMS is that you have a number
of record types (classes/relations/tables) with a regular
structure, and all data fits into these structures. When
you want to search for something, you know exactly in what
field of what table to look (but not which item of course).
You also typically have multiple users who need to be able
to update the same database simultaneously without getting
in each others way.
Even trivial aspects, such as whether to produce RDBM during the simulation, or convert the complete XML log file into one, are not entirely clear to me.
Most databases as suited at writing data in fairly small chunks,
although it's typically much faster to write 100 items in a
transaction, than to write 100 transactions with one item each.
I gather that RDBM would be much better suited for analysis, but what about portability ? Is database file a separate entity that may be passed around?
Who says that a database needs to reside in a file? Most
databases reside on disk, but it might well be in raw
partitions.
In general, you should see the database as a persistent
representation of data in a system. It's not a transport
mechanism.
Sullivan WxPyQtKinter wrote: My search needs: 1. Search and return all the record (an element) with specific id. 2. Search and return all the record whose child nodes has a specific id or attribute.
Try lxml, which is based on the libxml2 library. The current SVN version has
support for xml:id through the XMLDTDID function. It simply returns an XML
tree and an ID dictionary. http://codespeak.net/lxml
Stefan This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: leegold2 |
last post by:
Commonly done, eg. you enter a word in a search engine and when a
hit-page comes up the search word(s) are highlighted.
I'm doing a fulltext search that works well but I've tried a few
"packaged...
|
by: j |
last post by:
Hi,
Anyone out there with binary search tree experience. Working on a
project due tomorrow and really stuck. We need a function that splits
a binary tree into a bigger one and smaller one(for a...
|
by: richard.pasco |
last post by:
Hey all
I am trying to write a script that allows users to search through a
database of names. But rather than give a search string and then return
all those that match, I would like it to...
|
by: Huihong |
last post by:
Please check out our source code search engine here, http://www.codase.com
e.g., search main method,...
|
by: jdworley |
last post by:
Howdy!
The search is a 1 GB+ file from a crash to be exact - had an
e-mail correspondence (window) hopefully in it so I saved
the file on reboot before windoze kicked in and want to open
and...
|
by: Huihong |
last post by:
Please check out our newly released source code search engine here,
http://www.codase.com
e.g., search socket method call,...
|
by: DC |
last post by:
Hi,
I am programming a search catalogue with 200000 items (and growing). I
am currently using the SQL Server 2000 fulltext engine for this task
but it does not fit the requirements anymore.
...
|
by: Huihong |
last post by:
Please check out our source code search engine here, http://www.codase.com
e.g., search the "main" method,...
|
by: Rick |
last post by:
I have a large list of objects where each object has a unique
(non-overlapping) date range. The list is sorted chronologically. What is
the most efficient way to search this list for a single...
|
by: DolphinDB |
last post by:
Tired of spending countless mintues downsampling your data? Look no further!
In this article, youll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
|
by: ryjfgjl |
last post by:
ExcelToDatabase: batch import excel into database automatically...
|
by: isladogs |
last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM).
In this month's session, we are pleased to welcome back...
|
by: Vimpel783 |
last post by:
Hello!
Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
|
by: jfyes |
last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
|
by: ArrayDB |
last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
|
by: PapaRatzi |
last post by:
Hello,
I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
|
by: Defcon1945 |
last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
|
by: Shllpp 09 |
last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
| |