Connecting Tech Pros Worldwide Forums | Help | Site Map

quickest way to read xml from the web

Emmanuel
Guest
 
Posts: n/a
#1: Dec 19 '05
Hi there,

My client would like to process an xml file. the structure of which is as
below.
<xml>
<stockitem>
<releaseddate>.....date value...</releaseddate>
<...aditional tags for additional info>
</stockitem>
<stockitem>
<releaseddate>.....date value...</releaseddate>
<...aditional tags for additional info>
</stockitem>
<stockitem>
<releaseddate>.....date value...</releaseddate>
<...aditional tags for additional info>
</stockitem>
<stockitem>
<releaseddate>.....date value...</releaseddate>
<...aditional tags for additional info>
</stockitem>
<stockitem>
<releaseddate>.....date value...</releaseddate>
<...aditional tags for additional info>
</stockitem>
<stockitem>
<releaseddate>.....date value...</releaseddate>
<...aditional tags for additional info>
</stockitem>
<xml>


Now, the latest stock item will always be on top, thus if today i've process
all of the above xml stockitems, then tomorrow, all i need
to do is to process the first couple of xml stockitem elements (tags) which
are greater than yesterdays <releaseddate>.
This xml file can and does grow pretty fast (couple of megs) and is cleaned
up by the vendor only every couple of months (if not weeks).
I need to process the xml file such that as soon as i find an xml stockitem
which has the releaseddate as something that i've processed, then to finish
processing the xml file.
Which means i do not want to download the entire xml file for processing,
rather bring down only chunks (bits) of the xml file and read it line by
line (or stockitem by stockitem).

Can this be done?

(Currently i load the entire xml into an xmldocument class and then process
it).



Karl Seguin
Guest
 
Posts: n/a
#2: Dec 19 '05

re: quickest way to read xml from the web


Well, the quickest way to process an XML file is with the XML Reader. The
simplest is with the XMLDocument. You could use the XMLDocument with little
effort by retrieving all new items based on an xPath. However, this would
load the entire document into memory. If it's just a couple megs, I think
I'd start off with the XMLDocument and only move to the XMLReader if
necessary.

You mention that you want to "download" only the necessary part of the file.
I'm not sure what your question is anymore. Is it "what's the quickest way
to process part of an XML file" or "How to download only part of the file"?
Is the file not local to your code and must be downloaded whenever you want
to process it? If it's the latter, then the solution is more complicated.
I believe that even if you use the System.Net classes to get a stream to
your resource, once you call GetResponse() the entire response is on your
local computer. You can give it a try, but you'll really be limited by
whatever protocol you use to access the file.

Karl

--
MY ASP.Net tutorials
http://www.openmymind.net/
http://openmymind.net/redirector.aspx?documentId=51 - Learn about AJAX!



"Emmanuel" <em@enforge.net.ij> wrote in message
news:eMCVHqHBGHA.2036@TK2MSFTNGP14.phx.gbl...[color=blue]
> Hi there,
>
> My client would like to process an xml file. the structure of which is as
> below.
> <xml>
> <stockitem>
> <releaseddate>.....date value...</releaseddate>
> <...aditional tags for additional info>
> </stockitem>
> <stockitem>
> <releaseddate>.....date value...</releaseddate>
> <...aditional tags for additional info>
> </stockitem>
> <stockitem>
> <releaseddate>.....date value...</releaseddate>
> <...aditional tags for additional info>
> </stockitem>
> <stockitem>
> <releaseddate>.....date value...</releaseddate>
> <...aditional tags for additional info>
> </stockitem>
> <stockitem>
> <releaseddate>.....date value...</releaseddate>
> <...aditional tags for additional info>
> </stockitem>
> <stockitem>
> <releaseddate>.....date value...</releaseddate>
> <...aditional tags for additional info>
> </stockitem>
> <xml>
>
>
> Now, the latest stock item will always be on top, thus if today i've
> process
> all of the above xml stockitems, then tomorrow, all i need
> to do is to process the first couple of xml stockitem elements (tags)
> which
> are greater than yesterdays <releaseddate>.
> This xml file can and does grow pretty fast (couple of megs) and is
> cleaned
> up by the vendor only every couple of months (if not weeks).
> I need to process the xml file such that as soon as i find an xml
> stockitem
> which has the releaseddate as something that i've processed, then to
> finish
> processing the xml file.
> Which means i do not want to download the entire xml file for processing,
> rather bring down only chunks (bits) of the xml file and read it line by
> line (or stockitem by stockitem).
>
> Can this be done?
>
> (Currently i load the entire xml into an xmldocument class and then
> process
> it).
>
>
>[/color]


Emmanuel
Guest
 
Posts: n/a
#3: Dec 19 '05

re: quickest way to read xml from the web


Thank you for your help on this matter.
I need this entire process to be quick,
- processing the file
- downloading the xml file.

As you've mentioned the XMLDocument will load the entire document into the
memory (which probably means that the XML file will first have to be
downloaded from the site), which could take some time.

But i'll give this a go,
Once again, thanks.

"Karl Seguin" <karl REMOVE @ REMOVE openmymind REMOVEMETOO . ANDME net>
wrote in message news:ee%23WTlJBGHA.1180@TK2MSFTNGP09.phx.gbl...[color=blue]
> Well, the quickest way to process an XML file is with the XML Reader. The
> simplest is with the XMLDocument. You could use the XMLDocument with
> little effort by retrieving all new items based on an xPath. However,
> this would load the entire document into memory. If it's just a couple
> megs, I think I'd start off with the XMLDocument and only move to the
> XMLReader if necessary.
>
> You mention that you want to "download" only the necessary part of the
> file. I'm not sure what your question is anymore. Is it "what's the
> quickest way to process part of an XML file" or "How to download only part
> of the file"? Is the file not local to your code and must be downloaded
> whenever you want to process it? If it's the latter, then the solution is
> more complicated. I believe that even if you use the System.Net classes to
> get a stream to your resource, once you call GetResponse() the entire
> response is on your local computer. You can give it a try, but you'll
> really be limited by whatever protocol you use to access the file.
>
> Karl
>
> --
> MY ASP.Net tutorials
> http://www.openmymind.net/
> http://openmymind.net/redirector.aspx?documentId=51 - Learn about AJAX!
>
>
>
> "Emmanuel" <em@enforge.net.ij> wrote in message
> news:eMCVHqHBGHA.2036@TK2MSFTNGP14.phx.gbl...[color=green]
>> Hi there,
>>
>> My client would like to process an xml file. the structure of which is as
>> below.
>> <xml>
>> <stockitem>
>> <releaseddate>.....date value...</releaseddate>
>> <...aditional tags for additional info>
>> </stockitem>
>> <stockitem>
>> <releaseddate>.....date value...</releaseddate>
>> <...aditional tags for additional info>
>> </stockitem>
>> <stockitem>
>> <releaseddate>.....date value...</releaseddate>
>> <...aditional tags for additional info>
>> </stockitem>
>> <stockitem>
>> <releaseddate>.....date value...</releaseddate>
>> <...aditional tags for additional info>
>> </stockitem>
>> <stockitem>
>> <releaseddate>.....date value...</releaseddate>
>> <...aditional tags for additional info>
>> </stockitem>
>> <stockitem>
>> <releaseddate>.....date value...</releaseddate>
>> <...aditional tags for additional info>
>> </stockitem>
>> <xml>
>>
>>
>> Now, the latest stock item will always be on top, thus if today i've
>> process
>> all of the above xml stockitems, then tomorrow, all i need
>> to do is to process the first couple of xml stockitem elements (tags)
>> which
>> are greater than yesterdays <releaseddate>.
>> This xml file can and does grow pretty fast (couple of megs) and is
>> cleaned
>> up by the vendor only every couple of months (if not weeks).
>> I need to process the xml file such that as soon as i find an xml
>> stockitem
>> which has the releaseddate as something that i've processed, then to
>> finish
>> processing the xml file.
>> Which means i do not want to download the entire xml file for processing,
>> rather bring down only chunks (bits) of the xml file and read it line by
>> line (or stockitem by stockitem).
>>
>> Can this be done?
>>
>> (Currently i load the entire xml into an xmldocument class and then
>> process
>> it).
>>
>>
>>[/color]
>
>[/color]


Closed Thread