very large dictionary

[Simon Strobl]

Hello,

I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?

SImon

[Marc 'BlackJack' Rintsch]

On Fri, 01 Aug 2008 00:46:09 -0700, Simon Strobl wrote:

I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?

What does "load a dictionary" mean? Was it saved with the `pickle`
module?

How about using a database instead of a dictionary?

Ciao,
Marc 'BlackJack' Rintsch

[Simon Strobl]

What does "load a dictionary" mean?

I had a file bigrams.py with a content like below:

bigrams = {
", djy" : 75 ,
", djz" : 57 ,
", djzoom" : 165 ,
", dk" : 28893 ,
", dk.au" : 854 ,
", dk.b." : 3668 ,
....

}

In another file I said:

from bigrams import bigrams

How about using a database instead of a dictionary?

If there is no other way to do it, I will have to learn how to use
databases in Python. I would prefer to be able to use the same type of
scripts with data of all sizes, though.

[bearophileHUGS]

Simon Strobl:

I had a file bigrams.py with a content like below:
bigrams = {
", djy" : 75 ,
", djz" : 57 ,
", djzoom" : 165 ,
", dk" : 28893 ,
", dk.au" : 854 ,
", dk.b." : 3668 ,
...
}
In another file I said:
from bigrams import bigrams

Probably there's a limit in the module size here. You can try to
change your data format on disk, creating a text file like this:
", djy" 75
", djz" 57
", djzoom" 165
....
Then in a module you can create an empty dict, read the lines of the
data with:
for line in somefile:
part, n = .rsplit(" ", 1)
somedict[part.strip('"')] = int(n)

Otherwise you may have to use a BigTable, a DB, etc.

If there is no other way to do it, I will have to learn how to use
databases in Python. I would prefer to be able to use the same type of
scripts with data of all sizes, though.

I understand, I don't know if there are documented limits for the
dicts of the 64-bit Python.

Bye,
bearophile

[Sion Arrowsmith]

Simon Strobl <Si**********@g mail.comwrote:

>I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?

Let's just eliminate one thing here: this server is running a
64-bit OS, isn't it? Because if it's a 32-bit OS, the blunt
answer is "You can't, no matter how much physical memory you
have" and you're going to have to go down the database route
(or some approach which stores the mapping on disk and only
loads items into memory on demand).

--
\S -- si***@chiark.gr eenend.org.uk -- http://www.chaos.org.uk/~sion/
"Frankly I have no feelings towards penguins one way or the other"
-- Arthur C. Clarke
her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump

[Raja Baz]

On Fri, 01 Aug 2008 14:47:17 +0100, Sion Arrowsmith wrote:

Simon Strobl <Si**********@g mail.comwrote:

>>I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?

Let's just eliminate one thing here: this server is running a 64-bit OS,
isn't it? Because if it's a 32-bit OS, the blunt answer is "You can't,
no matter how much physical memory you have" and you're going to have to
go down the database route (or some approach which stores the mapping on
disk and only loads items into memory on demand).

I very highly doubt he has 128GB of main memory and is running a 32bit OS.

[Raja Baz]

On Fri, 01 Aug 2008 14:47:17 +0100, Sion Arrowsmith wrote:

Simon Strobl <Si**********@g mail.comwrote:

>>I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?

Let's just eliminate one thing here: this server is running a 64-bit OS,
isn't it? Because if it's a 32-bit OS, [etc...]

I very highly doubt he has 128GB of main memory and is running a 32bit OS.

[Sean]

Simon Strobl wrote:

Hello,

I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?

SImon

Take a look at the python bsddb module. Uing btree tables is fast, and
it has the benefit that once the table is open, the programing interface
is identical to a normal dictionary.

http://docs.python.org/lib/bsddb-objects.html

Sean

[Steven D'Aprano]

On Fri, 01 Aug 2008 00:46:09 -0700, Simon Strobl wrote:

Hello,

I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?

How do you know the dictionary takes 6.8G?

I'm going to guess an answer to my own question. In a later post, Simon
wrote:

[quote]
I had a file bigrams.py with a content like below:

bigrams = {
", djy" : 75 ,
", djz" : 57 ,
", djzoom" : 165 ,
", dk" : 28893 ,
", dk.au" : 854 ,
", dk.b." : 3668 ,
....

}
[end quote]
I'm guessing that the file is 6.8G of *text*. How much memory will it
take to import that? I don't know, but probably a lot more than 6.8G. The
compiler has to read the whole file in one giant piece, analyze it,
create all the string and int objects, and only then can it create the
dict. By my back-of-the-envelope calculations, the pointers alone will
require about 5GB, nevermind the objects they point to.

I suggest trying to store your data as data, not as Python code. Create a
text file "bigrams.tx t" with one key/value per line, like this:

djy : 75
djz : 57
djzoom : 165
dk : 28893
....

Then import it like such:

bigrams = {}
for line in open('bigrams.t xt', 'r'):
key, value = line.split(':')
bigrams[key.strip()] = int(value.strip ())
This will be slower, but because it only needs to read the data one line
at a time, it might succeed where trying to slurp all 6.8G in one piece
will fail.

--
Steven

[Jorgen Grahn]

On Fri, 1 Aug 2008 01:05:07 -0700 (PDT), Simon Strobl <Si**********@g mail.comwrote:

>What does "load a dictionary" mean?

I had a file bigrams.py with a content like below:

bigrams = {
", djy" : 75 ,
", djz" : 57 ,
", djzoom" : 165 ,
", dk" : 28893 ,
", dk.au" : 854 ,
", dk.b." : 3668 ,
...

}

In another file I said:

from bigrams import bigrams

>How about using a database instead of a dictionary?

If there is no other way to do it, I will have to learn how to use
databases in Python.

If you use Berkeley DB ("import bsddb"), you don't have to learn much.
These databases look very much like dictionaries string:string, only
they are disk-backed.

(I assume here that Berkeley DB supports 7GB data sets.)

/Jorgen

--
// Jorgen Grahn <grahn@ Ph'nglui mglw'nafh Cthulhu
\X/ snipabacken.se R'lyeh wgah'nagl fhtagn!