472,353 Members | 2,019 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,353 software developers and data experts.

How to request data from a lazily-created tree structure ?

Problem:

- You have tree structure (XML-like) that you don't want to create
100% in memory, because it just takes too long (for instance, you need
a http request to request the information from a slow distant site).
- But you want to be able to request data from it, such has "give me
all nodes that are under a "//foo/bar" tree, and have a child with an
"baz" attribute of value "zzz".

Question :

Do you have any other idea to request data from a lazily-created tree
structure ?

And does it make sense to create a DOM-like structure and to use a
generic XPath engine to request the tree ? (and does this generic
XPath engine exist ?)

The idea is to have the tree structure created on the fly (we are in
python), only when the XPath engine requests the data. Hopefully the
XPath engine will not request all the data from the tree (if the
request is smart enough and does not contain **, for instance).

Thanks
Jun 27 '08 #1
8 1667
méchoui schrieb:
Problem:

- You have tree structure (XML-like) that you don't want to create
100% in memory, because it just takes too long (for instance, you need
a http request to request the information from a slow distant site).
- But you want to be able to request data from it, such has "give me
all nodes that are under a "//foo/bar" tree, and have a child with an
"baz" attribute of value "zzz".

Question :

Do you have any other idea to request data from a lazily-created tree
structure ?

And does it make sense to create a DOM-like structure and to use a
generic XPath engine to request the tree ? (and does this generic
XPath engine exist ?)

The idea is to have the tree structure created on the fly (we are in
python), only when the XPath engine requests the data. Hopefully the
XPath engine will not request all the data from the tree (if the
request is smart enough and does not contain **, for instance).
Generic XPath works only with a DOM(like) structure. How else would you
e.g. evaluate an expression like foo[last()]?
So if you really need lazy evaluation, you will need to specifically
analyze the query of interest and see if it can be coded in a way that
allows to forget as much of the tree as possible, or even better not
query it.

Diez
Jun 27 '08 #2
On Jun 16, 11:16 pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
méchoui schrieb:
Problem:
- You have tree structure (XML-like) that you don't want to create
100% in memory, because it just takes too long (for instance, you need
a http request to request the information from a slow distant site).
- But you want to be able to request data from it, such has "give me
all nodes that are under a "//foo/bar" tree, and have a child with an
"baz" attribute of value "zzz".
Question :
Do you have any other idea to request data from a lazily-created tree
structure ?
And does it make sense to create a DOM-like structure and to use a
generic XPath engine to request the tree ? (and does this generic
XPath engine exist ?)
The idea is to have the tree structure created on the fly (we are in
python), only when the XPath engine requests the data. Hopefully the
XPath engine will not request all the data from the tree (if the
request is smart enough and does not contain **, for instance).

Generic XPath works only with a DOM(like) structure. How else would you
e.g. evaluate an expression like foo[last()]?

So if you really need lazy evaluation, you will need to specifically
analyze the query of interest and see if it can be coded in a way that
allows to forget as much of the tree as possible, or even better not
query it.

Diez
Yes, I need to make sure my requests are properly written so that the
generic XPath engine does not need all the structure in memory.

There are quite a few cases where you really don't need to load
everything at all. /a/b/*/c/d is an example. But even with an example
like /x/z[last()]/t, you don't need to load everything under the
every /x/z nodes. You just need to check for the latest one, and make
sure there is a t node under it.

Anyway, if I need to make requests that need all the data... that
means that the need for lazy instantiation of nodes disappears,
right ?
Jun 27 '08 #3
>
Yes, I need to make sure my requests are properly written so that the
generic XPath engine does not need all the structure in memory.

There are quite a few cases where you really don't need to load
everything at all. /a/b/*/c/d is an example. But even with an example
like /x/z[last()]/t, you don't need to load everything under the
every /x/z nodes. You just need to check for the latest one, and make
sure there is a t node under it.

Anyway, if I need to make requests that need all the data... that
means that the need for lazy instantiation of nodes disappears,
right ?

Yes. And unless you have memory-constraints I have to admit that I
really doubt that the parsing overhead isn't by far exceeded by the
network latency.

Diez
Jun 27 '08 #4
On Jun 17, 9:08 am, "Diez B. Roggisch" <de...@nospam.web.dewrote:
Yes, I need to make sure my requests are properly written so that the
generic XPath engine does not need all the structure in memory.
There are quite a few cases where you really don't need to load
everything at all. /a/b/*/c/d is an example. But even with an example
like /x/z[last()]/t, you don't need to load everything under the
every /x/z nodes. You just need to check for the latest one, and make
sure there is a t node under it.
Anyway, if I need to make requests that need all the data... that
means that the need for lazy instantiation of nodes disappears,
right ?

Yes. And unless you have memory-constraints I have to admit that I
really doubt that the parsing overhead isn't by far exceeded by the
network latency.

Diez
Do you know if there is such XPath engine that can be applied to a DOM-
like structure ?

One way would be to take an XPath engine from an existing XML engine
(ElementTree, or any other), and see what APIs it calls... and see if
we cannot create a DOM-like structure that has the same API. Duck
typing, really...
Jun 27 '08 #5
Do you know if there is such XPath engine that can be applied to a DOM-
like structure ?
No. But I toyed with the idea to write one :)
One way would be to take an XPath engine from an existing XML engine
(ElementTree, or any other), and see what APIs it calls... and see if
we cannot create a DOM-like structure that has the same API. Duck
typing, really...

Why can't you create a *real* DOM?

Diez
Jun 27 '08 #6
On Jun 17, 10:54*pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
Do you know if there is such XPath engine that can be applied to a DOM-
like structure ?

No. But I toyed with the idea to write one :)
One way would be to take an XPath engine from an existing XML engine
(ElementTree, or any other), and see what APIs it calls... and see if
we cannot create a DOM-like structure that has the same API. Duck
typing, really...

Why can't you create a *real* DOM?

Diez
I don't know what "real" means, in fact. In python, being a "real" sg
is all about having the same interface, right? May be I did not
undertand what you meant.

I cannot load all the data in memory before I request it, because it
would take too long. If using XPath-like tools requires that I load
the data in memory, I'd better create my own algorithm instead. It
will be much faster.

What I mean it: if I have a XPath engine that works well on a specific
DOM-like structure... may be I can create my own DOM-lile structure to
fool the XPath engine; so that I can use it on my own structure.
Jun 27 '08 #7
On Jun 17, 11:54 pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
Do you know if there is suchXPathengine that can be applied to a DOM-
like structure ?

No. But I toyed with the idea to write one :)
One way would be to take anXPathengine from an existing XML engine
(ElementTree, or any other), and see what APIs it calls... and see if
we cannot create a DOM-like structure that has the same API. Duck
typing, really...

Why can't you create a *real* DOM?

Diez
I may have found sg: http://sourceforge.net/projects/pdis-xpath/

A XPath 1.0, in pure python, on top of ElementTree. I'll have a look.
Jun 27 '08 #8
On 17 juin, 13:53, méchoui <laurent.pl...@gmail.comwrote:
On Jun 17, 9:08 am, "Diez B. Roggisch" <de...@nospam.web.dewrote:
Yes, I need to make sure my requests are properly written so that the
generic XPath engine does not need all the structure in memory.
There are quite a few cases where you really don't need to load
everything at all. /a/b/*/c/d is an example. But even with an example
like /x/z[last()]/t, you don't need to load everything under the
every /x/z nodes. You just need to check for the latest one, and make
sure there is a t node under it.
Anyway, if I need to make requests that need all the data... that
means that the need for lazy instantiation of nodes disappears,
right ?
Yes. And unless you have memory-constraints I have to admit that I
really doubt that the parsing overhead isn't by far exceeded by the
network latency.
Diez

Do you know if there is such XPath engine that can be applied to a DOM-
like structure ?

One way would be to take an XPath engine from an existing XML engine
(ElementTree, or any other), and see what APIs it calls... and see if
we cannot create a DOM-like structure that has the same API. Duck
typing, really...
I have something that works. http://lauploix.blogspot.com/2008/07...-my-trees.html
It has the pro and cons of the ElementTree 1.3 XPath engine, but it
works quite nice.

Laurent Ploix
Aug 2 '08 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Pavils Jurjans | last post by:
Hallo, I am working on multilingual web-application, and I have to be very sure about how the international characters are encoded and decoded in...
8
by: turnit \(removethis\) | last post by:
I have a login form that uses the post method to carry the information to the next page. The form works just fine in ie6.0, but fails in mozilla...
5
by: Jack | last post by:
Hi, I am trying to get a thorough understanding of a code where a addition or deletion of records can be done from a list of records. For addition...
6
by: Drew | last post by:
I have been trying to figure out why I am getting this message... I have an application that is 20 different ASP pages that works in sequential...
3
by: ExclusiveResorts | last post by:
Can the CallContext be used reliably for storing request specific data? We are developing an application library that uses the CallContext to keep...
19
by: Larry Lard | last post by:
In the old days (VB3 era), there was a thing called the Data Control, and you could use it to databind controls on forms to datasources, and so (as...
15
by: CMOS | last post by:
one of the projects im working in currently requires use of ultra large sized maps, lists, vector, etc. (basically stl containers). Sizes might...
2
by: MDANH2002 | last post by:
Hi From VB.NET I want to simulate the POST request of the following HTML form <html> <title>HTTP Post Testing</title> <body> ...
5
by: james | last post by:
To give an example of what I mean I have already altered the code: def output_random_lesson_of_type(self, type=None): """Output a lesson of a...
1
by: Chris Ashley | last post by:
Hi, If I create a Bitmap object from a stream, is the data making up the image from that stream duplicated in the Bitmap object, or does the...
1
by: Kemmylinns12 | last post by:
Blockchain technology has emerged as a transformative force in the business world, offering unprecedented opportunities for innovation and...
0
by: Naresh1 | last post by:
What is WebLogic Admin Training? WebLogic Admin Training is a specialized program designed to equip individuals with the skills and knowledge...
0
by: antdb | last post by:
Ⅰ. Advantage of AntDB: hyper-convergence + streaming processing engine In the overall architecture, a new "hyper-convergence" concept was...
0
by: AndyPSV | last post by:
HOW CAN I CREATE AN AI with an .executable file that would suck all files in the folder and on my computerHOW CAN I CREATE AN AI with an .executable...
0
by: Arjunsri | last post by:
I have a Redshift database that I need to use as an import data source. I have configured the DSN connection using the server, port, database, and...
0
hi
by: WisdomUfot | last post by:
It's an interesting question you've got about how Gmail hides the HTTP referrer when a link in an email is clicked. While I don't have the specific...
0
Oralloy
by: Oralloy | last post by:
Hello Folks, I am trying to hook up a CPU which I designed using SystemC to I/O pins on an FPGA. My problem (spelled failure) is with the...
0
BLUEPANDA
by: BLUEPANDA | last post by:
At BluePanda Dev, we're passionate about building high-quality software and sharing our knowledge with the community. That's why we've created a SaaS...
0
by: Rahul1995seven | last post by:
Introduction: In the realm of programming languages, Python has emerged as a powerhouse. With its simplicity, versatility, and robustness, Python...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.