too much memory use

[rohit]

hi ,
i am making a program for desktop search.
in order to make it more effective im implementing the records in the
form of a tree(files starting with 'a','b','c'....have different
trees ..so 26 trees in all) in memory and writing it down in file.
the max size file has 16000 records...now to implement the tree in
list i'm using line no as index ..and empty child nodes are
represented as "\n"
all this work is going on in the memory..
problem is the system eats up my 512 mb RAM +1gb virtual store n hangs
cant think of an effective way to implement tree in memory(i can
compact it on disk by writing just the index no..along with the record
from which tree in memory can be reconstructed, but i have to
implement tree as previous to implement random access)
please help..
rohit sethi
delhi college of engineering

[Marc 'BlackJack' Rintsch]

In <11**********************@a26g2000pre.googlegroups .com>, rohit wrote:

i am making a program for desktop search.
in order to make it more effective im implementing the records in the
form of a tree(files starting with 'a','b','c'....have different
trees ..so 26 trees in all) in memory and writing it down in file.
the max size file has 16000 records...now to implement the tree in
list i'm using line no as index ..and empty child nodes are
represented as "\n"
all this work is going on in the memory..
problem is the system eats up my 512 mb RAM +1gb virtual store n hangs
cant think of an effective way to implement tree in memory(i can
compact it on disk by writing just the index no..along with the record
from which tree in memory can be reconstructed, but i have to
implement tree as previous to implement random access)

I'm not quite sure what exactly you have as in-memory data structures and
how many "records" -- are you sure you don't keep references to objects
you don't really need anymore? Or maybe you have object cycles and
implemented the `__del__()` method on those objects?

Anyway… If the data doesn't fit into memory anymore it's time to put
them into a database. Either a `shelve`, an SQL database like SQLite or
maybe an object database like zodb or Durus.

Ciao,
Marc 'BlackJack' Rintsch

[Tim Roberts]

rohit <ro***********@gmail.comwrote:

>
i am making a program for desktop search.
in order to make it more effective im implementing the records in the
form of a tree(files starting with 'a','b','c'....have different
trees ..so 26 trees in all) in memory and writing it down in file.
the max size file has 16000 records...now to implement the tree in
list i'm using line no as index ..and empty child nodes are
represented as "\n"
all this work is going on in the memory..
problem is the system eats up my 512 mb RAM +1gb virtual store n hangs
cant think of an effective way to implement tree in memory(i can
compact it on disk by writing just the index no..along with the record
from which tree in memory can be reconstructed, but i have to
implement tree as previous to implement random access)

Why don't you just use a database? That's what they're designed for.
--
Tim Roberts, ti**@probo.com
Providenza & Boekelheide, Inc.