473,324 Members | 2,531 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,324 software developers and data experts.

webservice based on large XML file as datasource

Hi,
I am trying to create a WebService which uses data stored in a large
xml file (200-400 MB). Obviously parsing the xml for every webservice
request is not feasible since that would make the response very slow.

What should I do to make sure that I can load the xml file once and
keep it in memory. Then I should be able to use the same in-memory
parsed xml document.

Thanks

Tarun
-=-
Nov 18 '05 #1
3 1844
SF
Tarun,

have you considered reading it in a DataSet and then use DataViews
to do the querying? Or you could read it in a XMLDocument and then
use XPath to access the specific elements you are looking for.

In any case I'd put the data structure you use for in-memory storage
in the application cache and use a CacheDependency to keep
the original XML file and the in-memory data in sync.

Hope that helps
Stefan

"Tarun Jain" <tj***@hotmail.com> wrote in message
news:a3*************************@posting.google.co m...
Hi,
I am trying to create a WebService which uses data stored in a large
xml file (200-400 MB). Obviously parsing the xml for every webservice
request is not feasible since that would make the response very slow.

What should I do to make sure that I can load the xml file once and
keep it in memory. Then I should be able to use the same in-memory
parsed xml document.

Thanks

Tarun
-=-

Nov 18 '05 #2
you just store in a static (shared in vb). while accessing a dom is
threadsafe, updating is not. you will need to supply locking for your
threads. if you need to a lot of updates. then you will not want to lock the
whole doms, but rather lock at the lowest node you need.

-- bruce (sqlwork.com)

"Tarun Jain" <tj***@hotmail.com> wrote in message
news:a3*************************@posting.google.co m...
Hi,
I am trying to create a WebService which uses data stored in a large
xml file (200-400 MB). Obviously parsing the xml for every webservice
request is not feasible since that would make the response very slow.

What should I do to make sure that I can load the xml file once and
keep it in memory. Then I should be able to use the same in-memory
parsed xml document.

Thanks

Tarun
-=-

Nov 18 '05 #3
Thanks for your input so far.

A more complete description of the problem is that

--> I have to only read the data. There are no updates to
the xml document that are required.

--> I am planning to use XPath to retrieve the data.

--> I am planning to use the cache. But the problem is
that this web service will not see a lot of traffic,
something like 10 queries a day. I know that ASP.NET shuts
down the application when there are no more sessions.
So what will happen is that if I cache the XmlDocument
object, once the application shuts down the cache will
disappear along with it.

What can I do to get around these problems ??

Tarun
-=-

-----Original Message-----
Tarun,

have you considered reading it in a DataSet and then use DataViewsto do the querying? Or you could read it in a XMLDocument and thenuse XPath to access the specific elements you are looking for.
In any case I'd put the data structure you use for in- memory storagein the application cache and use a CacheDependency to keep
the original XML file and the in-memory data in sync.

Hope that helps
Stefan

"Tarun Jain" <tj***@hotmail.com> wrote in message
news:a3*************************@posting.google.c om...
Hi,
I am trying to create a WebService which uses data stored in a large xml file (200-400 MB). Obviously parsing the xml for every webservice request is not feasible since that would make the response very slow.
What should I do to make sure that I can load the xml file once and keep it in memory. Then I should be able to use the same in-memory parsed xml document.

Thanks

Tarun
-=-

.

Nov 18 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: ohaya | last post by:
Hi, I'm a real newbie, but have been asked to try to fix a problem in one of our JSP pages that is suppose to read in a text file and display it. From my testing thus far, it appears this page...
6
by: guillaume | last post by:
I have to read and process a large ASCII file containing a mesh : a list of points and triangles. The file is 100 MBytes. I first tried to do it in memory but I think I am running out of memory...
1
by: DJTB | last post by:
zodb-dev@zope.org] Hi, I'm having problems storing large amounts of objects in a ZODB. After committing changes to the database, elements are not cleared from memory. Since the number of...
0
by: pruebauno | last post by:
Hello all, I am having issues compiling Python with large file support. I tried forcing the configure script to add it but then it bombs in the make process. Any help will be appreciated. ...
7
by: Joseph | last post by:
Hi, I'm having bit of questions on recursive pointer. I have following code that supports upto 8K files but when i do a file like 12K i get a segment fault. I Know it is in this line of code. ...
3
by: Charlie | last post by:
Dear all, I am currently writting a trace analyzer in C++. It always fails to open a very large input file (3.7Gb). I tried on a simple program, same thing happens:...
6
by: Thomas Due | last post by:
Hi, I am writing an ASP.NET project where I allow users to upload files to the server. I have changed to web.config to allow a total file size of 100MB. My problem is that if the total file size...
1
by: David | last post by:
Hello. I can't upload large file with HtmlInputFile control:( Is there any file size limitation in HtmlInputFile control? If yes how can I upload to server large size file? Than you.
4
by: coldy | last post by:
Hi, I have a large txt file (1GB) which I need to break into smaller files based on the contents of a column in the file. The values in the column of intrest starts low, then increases, then...
2
by: robert | last post by:
Somebody who uses my app gets a error : os.stat('/path/filename') OSError: Value too large for defined data type: '/path/filename' on a big file >4GB ( Python 2.4.4 / Linux )
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.