472,145 Members | 1,568 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,145 software developers and data experts.

Dealing with large files (random access)


This post is the 'sequel' ;) of the "Data Oriented vs Object Oriented
Design" post, but it can be read and treated apart from that one. I
will just quote the beginning of my previous message to expose the

New to .NET, I'm working on an Winforms client application using VS
2005 beta2. My needs considering data storage are the followings:

(1) Small files (0 < length < 10 mb), containing lots of small
'objects' that need to be loaded in memory at runtime, in order to
garantee small access time to each 'object'. Each of these 'objects'
collection should be able to easily bind to a DataGridView, AND to
provide *filtering* and *sorting* capability.

(2) 'Mid-large' files (0 < length < 100 mb), containing lots of
'mid-large objects'. Such files shouldn't be fully loaded in memory,
but the class that deals with those files must expose random
accessing/adding/removing/modifying 'object' functionalities.

In facts, a file of type (1) contains an index of the large 'objects'
contained in a corresponding file of type (2). Each index will be used
to fill a DataGridView, and each entry (row) will provide a
'reference' to the 'big object' stored in the corresponding type (2)
file (in addition to some other object-specific details that will be
displayed on the DGView of course).

Since this is a Windows client application, using a database with a
data provider is out of the question.
This post deals with type (2) files.

My question will go straight to the point: what is the most efficient
approach to deal with such files?

I haven't much thought about it yet. Again, using a serialized List<T>
(or any object Collection) would be fine, but randomly accessing an
object without loading the whole list into memory is easier said than

I'm sure you experienced people will have some great starting ideas ;)

Nov 17 '05 #1
1 2292
interesting assumption:
Since this is a Windows client application, using a database with a
data provider is out of the question.

Why in the world would you say that? Access databases and MSDE databases
are both completely free to distribute with your windows client application.
Download WebMatrix from the Microsoft site and you get MSDE.

For a good writeup of MSDE and comparisons with other tools, See

To see how this plays with the next version of MS software, See
Honestly, the idea of writing something new to do all this is difficult for
me to rationalize.

Your notion of packing all your small files into a big one is not better.
Your small files cannot grow without rewriting the large one. Why not use
the file system... that's what it's for!

--- Nick Malik [Microsoft]
MCSD, CFPS, Certified Scrummaster

Disclaimer: Opinions expressed in this forum are my own, and not
representative of my employer.
I do not answer questions on behalf of my employer. I'm just a
programmer helping programmers.
Nov 17 '05 #2

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

19 posts views Thread by Brad Tilley | last post: by
24 posts views Thread by Joerg Schuster | last post: by
2 posts views Thread by jdev8080 | last post: by
57 posts views Thread by Chris Foote | last post: by
3 posts views Thread by sebastian.harko | last post: by
25 posts views Thread by tekctrl | last post: by
reply views Thread by leo001 | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.