how to build a dict including a large number of data

[wanzathe]

hi everyone
i'm a newbie to python :)
i have a binary file named test.dat including 9600000 records.
the record format is int a + int b + int c + int d
i want to build a dict like this: key=int a,int b values=int c,int d
i choose using bsddb and it takes about 140 seconds to build the dict.
what can i do if i want to make my program run faster?
or is there another way i can choose?
Thanks in advance.

My Code:
-----------------------------------------------------------------------------------
my_file = file('test.dat' ,'rb')
content = my_file.read()
record_number = len(content) / 16

db = bsddb.btopen('t est.dat.db','n' ,cachesize=5000 00000)
for i in range(0,record_ number):
a = struct.unpack(" IIII",content[i*16:i*16+16])
db['%d_%d' % (a[0],a[1])] = '%d_%d' % (a[2],a[3])

db.close()
my_file.close()

[Chris]

On Jan 4, 3:57 pm, wanzathe <wanza...@gmail .comwrote:

hi everyone
i'm a newbie to python :)
i have a binary file named test.dat including 9600000 records.
the record format is int a + int b + int c + int d
i want to build a dict like this: key=int a,int b values=int c,int d
i choose using bsddb and it takes about 140 seconds to build the dict.
what can i do if i want to make my program run faster?
or is there another way i can choose?
Thanks in advance.

My Code:
-----------------------------------------------------------------------------------
my_file = file('test.dat' ,'rb')
content = my_file.read()
record_number = len(content) / 16

db = bsddb.btopen('t est.dat.db','n' ,cachesize=5000 00000)
for i in range(0,record_ number):
a = struct.unpack(" IIII",content[i*16:i*16+16])
db['%d_%d' % (a[0],a[1])] = '%d_%d' % (a[2],a[3])

db.close()
my_file.close()

my_file = file('test.dat' ,'rb')
db = bsddb.btopen('t est.dat.db','n' ,cachesize=5000 00000)
content = myfile.read(16)
while content:
a = struct.unpack(' IIII',content)
db['%d_%d' % (a[0],a[1])] = '%d_%d' % (a[2],a[3])
content = myfile.read(16)

db.close()
my_file.close()

That would be more memory efficient, as for speed you would need to
time it on your side.

[Fredrik Lundh]

wanzathe wrote:

i have a binary file named test.dat including 9600000 records.
the record format is int a + int b + int c + int d
i want to build a dict like this: key=int a,int b values=int c,int d
i choose using bsddb and it takes about 140 seconds to build the dict.

you're not building a dict, you're populating a persistent database.
storing ~70000 records per second isn't that bad, really...

what can i do if i want to make my program run faster?
or is there another way i can choose?

why not just use a real Python dictionary, and the marshal module for
serialization?

</F>

[wanzathe]

On 1ÔÂ4ÈÕ, ÏÂÎç10ʱ17·Ö, Fredrik Lundh <fred...@python ware.comwrote:

wanzathe wrote:

i have a binary file named test.dat including 9600000 records.
the record format is int a + int b + int c + int d
i want to build a dict like this: key=int a,int b values=int c,int d
i choose using bsddb and it takes about 140 seconds to build the dict.

you're not building a dict, you're populating a persistent database.
storing ~70000 records per second isn't that bad, really...

what can i do if i want to make my program run faster?
or is there another way i can choose?

why not just use a real Python dictionary, and the marshal module for
serialization?

</F>

hi,Fredrik Lundn
you are right, i'm populating a persistent database.
i plan to use a real Python dictionary and use cPickle for
serialization at first, but it did not work because the number of
records is too large.
Thanks