By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,814 Members | 1,050 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,814 IT Pros & Developers. It's quick & easy.

too much memory use

P: n/a
hi ,
i am making a program for desktop search.
in order to make it more effective im implementing the records in the
form of a tree(files starting with 'a','b','c'....have different
trees ..so 26 trees in all) in memory and writing it down in file.
the max size file has 16000 records...now to implement the tree in
list i'm using line no as index ..and empty child nodes are
represented as "\n"
all this work is going on in the memory..
problem is the system eats up my 512 mb RAM +1gb virtual store n hangs
cant think of an effective way to implement tree in memory(i can
compact it on disk by writing just the index no..along with the record
from which tree in memory can be reconstructed, but i have to
implement tree as previous to implement random access)
please help..
rohit sethi
delhi college of engineering

May 22 '07 #1
Share this Question
Share on Google+
2 Replies


P: n/a
In <11**********************@a26g2000pre.googlegroups .com>, rohit wrote:
i am making a program for desktop search.
in order to make it more effective im implementing the records in the
form of a tree(files starting with 'a','b','c'....have different
trees ..so 26 trees in all) in memory and writing it down in file.
the max size file has 16000 records...now to implement the tree in
list i'm using line no as index ..and empty child nodes are
represented as "\n"
all this work is going on in the memory..
problem is the system eats up my 512 mb RAM +1gb virtual store n hangs
cant think of an effective way to implement tree in memory(i can
compact it on disk by writing just the index no..along with the record
from which tree in memory can be reconstructed, but i have to
implement tree as previous to implement random access)
I'm not quite sure what exactly you have as in-memory data structures and
how many "records" -- are you sure you don't keep references to objects
you don't really need anymore? Or maybe you have object cycles and
implemented the `__del__()` method on those objects?

Anyway… If the data doesn't fit into memory anymore it's time to put
them into a database. Either a `shelve`, an SQL database like SQLite or
maybe an object database like zodb or Durus.

Ciao,
Marc 'BlackJack' Rintsch
May 22 '07 #2

P: n/a
rohit <ro***********@gmail.comwrote:
>
i am making a program for desktop search.
in order to make it more effective im implementing the records in the
form of a tree(files starting with 'a','b','c'....have different
trees ..so 26 trees in all) in memory and writing it down in file.
the max size file has 16000 records...now to implement the tree in
list i'm using line no as index ..and empty child nodes are
represented as "\n"
all this work is going on in the memory..
problem is the system eats up my 512 mb RAM +1gb virtual store n hangs
cant think of an effective way to implement tree in memory(i can
compact it on disk by writing just the index no..along with the record
from which tree in memory can be reconstructed, but i have to
implement tree as previous to implement random access)
Why don't you just use a database? That's what they're designed for.
--
Tim Roberts, ti**@probo.com
Providenza & Boekelheide, Inc.
May 24 '07 #3

This discussion thread is closed

Replies have been disabled for this discussion.