473,581 Members | 6,653 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

How to request data from a lazily-created tree structure ?

Problem:

- You have tree structure (XML-like) that you don't want to create
100% in memory, because it just takes too long (for instance, you need
a http request to request the information from a slow distant site).
- But you want to be able to request data from it, such has "give me
all nodes that are under a "//foo/bar" tree, and have a child with an
"baz" attribute of value "zzz".

Question :

Do you have any other idea to request data from a lazily-created tree
structure ?

And does it make sense to create a DOM-like structure and to use a
generic XPath engine to request the tree ? (and does this generic
XPath engine exist ?)

The idea is to have the tree structure created on the fly (we are in
python), only when the XPath engine requests the data. Hopefully the
XPath engine will not request all the data from the tree (if the
request is smart enough and does not contain **, for instance).

Thanks
Jun 27 '08 #1
8 1789
méchoui schrieb:
Problem:

- You have tree structure (XML-like) that you don't want to create
100% in memory, because it just takes too long (for instance, you need
a http request to request the information from a slow distant site).
- But you want to be able to request data from it, such has "give me
all nodes that are under a "//foo/bar" tree, and have a child with an
"baz" attribute of value "zzz".

Question :

Do you have any other idea to request data from a lazily-created tree
structure ?

And does it make sense to create a DOM-like structure and to use a
generic XPath engine to request the tree ? (and does this generic
XPath engine exist ?)

The idea is to have the tree structure created on the fly (we are in
python), only when the XPath engine requests the data. Hopefully the
XPath engine will not request all the data from the tree (if the
request is smart enough and does not contain **, for instance).
Generic XPath works only with a DOM(like) structure. How else would you
e.g. evaluate an expression like foo[last()]?
So if you really need lazy evaluation, you will need to specifically
analyze the query of interest and see if it can be coded in a way that
allows to forget as much of the tree as possible, or even better not
query it.

Diez
Jun 27 '08 #2
On Jun 16, 11:16 pm, "Diez B. Roggisch" <de...@nospam.w eb.dewrote:
méchoui schrieb:
Problem:
- You have tree structure (XML-like) that you don't want to create
100% in memory, because it just takes too long (for instance, you need
a http request to request the information from a slow distant site).
- But you want to be able to request data from it, such has "give me
all nodes that are under a "//foo/bar" tree, and have a child with an
"baz" attribute of value "zzz".
Question :
Do you have any other idea to request data from a lazily-created tree
structure ?
And does it make sense to create a DOM-like structure and to use a
generic XPath engine to request the tree ? (and does this generic
XPath engine exist ?)
The idea is to have the tree structure created on the fly (we are in
python), only when the XPath engine requests the data. Hopefully the
XPath engine will not request all the data from the tree (if the
request is smart enough and does not contain **, for instance).

Generic XPath works only with a DOM(like) structure. How else would you
e.g. evaluate an expression like foo[last()]?

So if you really need lazy evaluation, you will need to specifically
analyze the query of interest and see if it can be coded in a way that
allows to forget as much of the tree as possible, or even better not
query it.

Diez
Yes, I need to make sure my requests are properly written so that the
generic XPath engine does not need all the structure in memory.

There are quite a few cases where you really don't need to load
everything at all. /a/b/*/c/d is an example. But even with an example
like /x/z[last()]/t, you don't need to load everything under the
every /x/z nodes. You just need to check for the latest one, and make
sure there is a t node under it.

Anyway, if I need to make requests that need all the data... that
means that the need for lazy instantiation of nodes disappears,
right ?
Jun 27 '08 #3
>
Yes, I need to make sure my requests are properly written so that the
generic XPath engine does not need all the structure in memory.

There are quite a few cases where you really don't need to load
everything at all. /a/b/*/c/d is an example. But even with an example
like /x/z[last()]/t, you don't need to load everything under the
every /x/z nodes. You just need to check for the latest one, and make
sure there is a t node under it.

Anyway, if I need to make requests that need all the data... that
means that the need for lazy instantiation of nodes disappears,
right ?

Yes. And unless you have memory-constraints I have to admit that I
really doubt that the parsing overhead isn't by far exceeded by the
network latency.

Diez
Jun 27 '08 #4
On Jun 17, 9:08 am, "Diez B. Roggisch" <de...@nospam.w eb.dewrote:
Yes, I need to make sure my requests are properly written so that the
generic XPath engine does not need all the structure in memory.
There are quite a few cases where you really don't need to load
everything at all. /a/b/*/c/d is an example. But even with an example
like /x/z[last()]/t, you don't need to load everything under the
every /x/z nodes. You just need to check for the latest one, and make
sure there is a t node under it.
Anyway, if I need to make requests that need all the data... that
means that the need for lazy instantiation of nodes disappears,
right ?

Yes. And unless you have memory-constraints I have to admit that I
really doubt that the parsing overhead isn't by far exceeded by the
network latency.

Diez
Do you know if there is such XPath engine that can be applied to a DOM-
like structure ?

One way would be to take an XPath engine from an existing XML engine
(ElementTree, or any other), and see what APIs it calls... and see if
we cannot create a DOM-like structure that has the same API. Duck
typing, really...
Jun 27 '08 #5
Do you know if there is such XPath engine that can be applied to a DOM-
like structure ?
No. But I toyed with the idea to write one :)
One way would be to take an XPath engine from an existing XML engine
(ElementTree, or any other), and see what APIs it calls... and see if
we cannot create a DOM-like structure that has the same API. Duck
typing, really...

Why can't you create a *real* DOM?

Diez
Jun 27 '08 #6
On Jun 17, 10:54*pm, "Diez B. Roggisch" <de...@nospam.w eb.dewrote:
Do you know if there is such XPath engine that can be applied to a DOM-
like structure ?

No. But I toyed with the idea to write one :)
One way would be to take an XPath engine from an existing XML engine
(ElementTree, or any other), and see what APIs it calls... and see if
we cannot create a DOM-like structure that has the same API. Duck
typing, really...

Why can't you create a *real* DOM?

Diez
I don't know what "real" means, in fact. In python, being a "real" sg
is all about having the same interface, right? May be I did not
undertand what you meant.

I cannot load all the data in memory before I request it, because it
would take too long. If using XPath-like tools requires that I load
the data in memory, I'd better create my own algorithm instead. It
will be much faster.

What I mean it: if I have a XPath engine that works well on a specific
DOM-like structure... may be I can create my own DOM-lile structure to
fool the XPath engine; so that I can use it on my own structure.
Jun 27 '08 #7
On Jun 17, 11:54 pm, "Diez B. Roggisch" <de...@nospam.w eb.dewrote:
Do you know if there is suchXPathengine that can be applied to a DOM-
like structure ?

No. But I toyed with the idea to write one :)
One way would be to take anXPathengine from an existing XML engine
(ElementTree, or any other), and see what APIs it calls... and see if
we cannot create a DOM-like structure that has the same API. Duck
typing, really...

Why can't you create a *real* DOM?

Diez
I may have found sg: http://sourceforge.net/projects/pdis-xpath/

A XPath 1.0, in pure python, on top of ElementTree. I'll have a look.
Jun 27 '08 #8
On 17 juin, 13:53, méchoui <laurent.pl...@ gmail.comwrote:
On Jun 17, 9:08 am, "Diez B. Roggisch" <de...@nospam.w eb.dewrote:
Yes, I need to make sure my requests are properly written so that the
generic XPath engine does not need all the structure in memory.
There are quite a few cases where you really don't need to load
everything at all. /a/b/*/c/d is an example. But even with an example
like /x/z[last()]/t, you don't need to load everything under the
every /x/z nodes. You just need to check for the latest one, and make
sure there is a t node under it.
Anyway, if I need to make requests that need all the data... that
means that the need for lazy instantiation of nodes disappears,
right ?
Yes. And unless you have memory-constraints I have to admit that I
really doubt that the parsing overhead isn't by far exceeded by the
network latency.
Diez

Do you know if there is such XPath engine that can be applied to a DOM-
like structure ?

One way would be to take an XPath engine from an existing XML engine
(ElementTree, or any other), and see what APIs it calls... and see if
we cannot create a DOM-like structure that has the same API. Duck
typing, really...
I have something that works. http://lauploix.blogspot.com/2008/07...-my-trees.html
It has the pro and cons of the ElementTree 1.3 XPath engine, but it
works quite nice.

Laurent Ploix
Aug 2 '08 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
10547
by: Pavils Jurjans | last post by:
Hallo, I am working on multilingual web-application, and I have to be very sure about how the international characters are encoded and decoded in the client-server form requests. There's a great article about the issue: http://ppewww.ph.gla.ac.uk/~flavell/charset/form-i18n.html Generally, that states that this are is filled with...
8
10632
by: turnit \(removethis\) | last post by:
I have a login form that uses the post method to carry the information to the next page. The form works just fine in ie6.0, but fails in mozilla and fails in ie5.2 on a mac. "HTTP/1.1 400 Bad Request" was the original error msg now that I reinstalled iislockdown and urscan I get "The parameter is incorrect." If I use the get method with this...
5
2191
by: Jack | last post by:
Hi, I am trying to get a thorough understanding of a code where a addition or deletion of records can be done from a list of records. For addition part of the form, data is being obtained from set of input boxes. The following is the code being used: If Len(Request.Form("ID_0"))>0 Then AuthorID=Request.Form("ID_0")...
6
2416
by: Drew | last post by:
I have been trying to figure out why I am getting this message... I have an application that is 20 different ASP pages that works in sequential order (step1, step2, step3...). After each step the app writes the entered text to a cookie. This feature is so that if there is a problem, we can still retrieve the data. This Bad Request error...
3
2791
by: ExclusiveResorts | last post by:
Can the CallContext be used reliably for storing request specific data? We are developing an application library that uses the CallContext to keep an IdentityMap (hashtable of business objects that have been loaded from the DB) and a collection of business objects that have been modified during the current request. These object maps need to...
19
2217
by: Larry Lard | last post by:
In the old days (VB3 era), there was a thing called the Data Control, and you could use it to databind controls on forms to datasources, and so (as the marketing speak goes), 'create database applications without writing a single line of code!!!'. Personally, and I know I wasn't alone in this, I was always suspicious of this claim, because one...
15
2579
by: CMOS | last post by:
one of the projects im working in currently requires use of ultra large sized maps, lists, vector, etc. (basically stl containers). Sizes might grow up to 1000 Million entries. since it is impossible to have all this data in memory, im planning to implement these containers to hold data both in memory and disk at the same time. im not sure...
2
6292
by: MDANH2002 | last post by:
Hi From VB.NET I want to simulate the POST request of the following HTML form <html> <title>HTTP Post Testing</title> <body> <form action=http://www.example.com/postdata enctype="multipart/form-data" method="post" name="testform">
5
1623
by: james | last post by:
To give an example of what I mean I have already altered the code: def output_random_lesson_of_type(self, type=None): """Output a lesson of a specific type. If no type is passed in then output any type.""" output_lessons = self.lesson_data if type: filtered_lessons = filter(lambda x: x == type, self.lesson_data) if filtered_lessons:
1
1160
by: Chris Ashley | last post by:
Hi, If I create a Bitmap object from a stream, is the data making up the image from that stream duplicated in the Bitmap object, or does the Bitmap just point to the stream somehow without holding any image data? I did notice that if I populated a Bitmap object from a MemoryStream, if I closed the MemoryStream I got a GDI error when I...
0
7854
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
8134
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8296
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7880
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
8157
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
1
5665
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5350
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
1
2295
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1394
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.