473,320 Members | 1,940 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Loading large file in memory or not?

I'm writing a program that will query an XML file with the XmlTextReader
class (the question is not specifically about XML, however). This file is
very large (maybe 20 MB at most), and during extreme conditions, it'll have
to be queried thousands of times (maybe 20,000 times at most). Small
fragments of this file are used to construct another file.

What is most economical, in terms of system resources? Opening and closing
the file + initializing and destroying the XmlTextReader thousands of times,
or loading all the contents in the file in memory, and pick data from a tree
in memory? Also, would it be more efficient to query a table in SQL Server
thousands of times?

Thanks,

Gustaf

Nov 15 '05 #1
2 3432
"Gustaf Liljegren" <gu**************@bredband.net> wrote in message
news:uj**************@TK2MSFTNGP09.phx.gbl...
I'm writing a program that will query an XML file with the XmlTextReader
class (the question is not specifically about XML, however). This file is
very large (maybe 20 MB at most), and during extreme conditions, it'll have to be queried thousands of times (maybe 20,000 times at most). Small
fragments of this file are used to construct another file.

What is most economical, in terms of system resources? Opening and closing
the file + initializing and destroying the XmlTextReader thousands of times, or loading all the contents in the file in memory, and pick data from a tree in memory? Also, would it be more efficient to query a table in SQL Server
thousands of times?

Thanks,

Gustaf


Do it in memory unless you're really strapped for memory - continually
opening, parsing and closing a file will be really slow and unless each time
you're only using extremely small fragments to construct your other file,
you'll probably end up reading a lot of the original in anyway. 20MB really
isn't a lot of memory nowadays, and a tree representation of it in memory
won't be a lot larger. In terms of SQL, again, you'll trade off speed for
memory usage, but unless your machine really has no spare memory (or you
don't care about speed) just read it in once to a tree you can search
efficiently.

Steve
Nov 15 '05 #2
"Steve McLellan" <sj*@fixerlabs.com.NOSPAM> wrote:
Do it in memory unless you're really strapped for memory - continually
opening, parsing and closing a file will be really slow and unless each time you're only using extremely small fragments to construct your other file,
you'll probably end up reading a lot of the original in anyway. 20MB really isn't a lot of memory nowadays, and a tree representation of it in memory
won't be a lot larger.


Thank you very much. That's a convincing answer.

Gustaf

Nov 15 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Philipp K. Janert, Ph.D. | last post by:
Dear All! I am trying to load a relatively large table (about 1 Million rows) into an sqlite table, which is kept in memory. The load process is very slow - on the order of 15 minutes or so. ...
2
by: Wizfrog | last post by:
Hello, I'm working with a pretty large XML file, but I really only need to display a few things that requires quite a few transforms. I already limited to the transforms to the data i need to...
4
by: =?Utf-8?B?UGhpbCBKb2huc29u?= | last post by:
Hello, I am working on a peice of code that takes a file and loads it into a SQL Server DB. When I load large files (over 70Mb) I get a system out of memory exception. My machine has 2G RAM ...
5
by: s0suk3 | last post by:
Hi, I wanted to know how cautious it is to do something like: f = file("filename", "rb") f.read() for a possibly huge file. When calling f.read(), and not doing anything with the return...
17
by: byte8bits | last post by:
How does C++ safely open and read very large files? For example, say I have 1GB of physical memory and I open a 4GB file and attempt to read it like so: #include <iostream> #include <fstream>...
0
by: =?Utf-8?B?UGhpbCBKb2huc29u?= | last post by:
Hi, I am working on an ASP.NET 1.1 application that creates thumbnails using the code below. Problem is that the code loads an entire file into memory to instantiate the image (file.Data is a...
6
by: Terry Carroll | last post by:
I am trying to do something with a very large tarfile from within Python, and am running into memory constraints. The tarfile in question is a 4-gigabyte datafile from freedb.org,...
2
by: Kris Kennaway | last post by:
I would like to MIME encode a message from a large file without first loading the file into memory. Assume the file has been pre-encoded on disk (actually I am using encode_7or8bit, so the...
5
by: DR | last post by:
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped file) then loading 50GB unzipped data? im using System.IO.Compression.GZipStream and its not maxing out the cpu while...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.