470,820 Members | 1,901 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 470,820 developers. It's quick & easy.

i have a big dictionary...:)

hi,

i have a big dictionary...

well...
i have a 6mb textfile,
from which i build a dictionary the following way:
the first word of a line is the key, the rest is the value.
and i fill all the lines into the dictionary (it contains around
100*1000 lines)

now when it's in memory, it uses around 60megabytes of memory (measured
from "ps aux | grep python" :)

how could i lower this memory requirement?

is there a simple database-format, something small.... so i don't have
to keep the whole dictionary in memory?

thanks,
gabor
Jul 18 '05 #1
2 1460
gabor farkas wrote:
is there a simple database-format, something small.... so i don't have
to keep the whole dictionary in memory?


take a look at sqlite.

--
Maciej "Fiedzia" Dziardziel (fiedzia (at) fiedzia (dot) prv (dot) pl)
www.fiedzia.prv.pl

A corpse is a corpse, of course, of course, and no-one can talk to a
corpse, of course. That is, of course, unless the corpse is the famous
Mr. Dead!
Jul 18 '05 #2
gabor farkas <ga***@z10n.net> wrote in message news:<ma**************************************@pyt hon.org>...

Have you considered keeping only the key in memory, plus an offset of
its location in the text file?

So it would look like:
key offset
key1 14
key2 275
key3 64004

Then as you need to access individual lines, hop to the right spot in
the file based on the offset, and read the line.
Jul 18 '05 #3

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

1 post views Thread by none | last post: by
4 posts views Thread by brianobush | last post: by
1 post views Thread by john wright | last post: by
8 posts views Thread by Brian L. Troutwine | last post: by
3 posts views Thread by JamesB | last post: by
3 posts views Thread by wildThought | last post: by
8 posts views Thread by Bob Altman | last post: by
reply views Thread by mihailmihai484 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.