469,610 Members | 1,649 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,610 developers. It's quick & easy.

need optimizing help

I have a dictionary with a very very large(possibly millions) of
key/value pairs.
The key is a tuple that looks like (id,date)
What is the fastest way to get out all of the values that match any
key given that they individual key elements are coming from two
seperate lists?
The approach of
for id in IDS:
for date in dates:
data=myDict[(id,date)]

seems to just take too long. Is there a speedier, more pythonic, way
of doing this? Any help speeding this up would be much appreciated!!
Jul 18 '05 #1
3 989
On Sat, 13 Mar 2004 10:07:54 -0800, rabbits77 wrote:
I have a dictionary with a very very large(possibly millions) of
key/value pairs.
The key is a tuple that looks like (id,date)
What is the fastest way to get out all of the values that match any
key given that they individual key elements are coming from two
seperate lists?
The approach of
for id in IDS:
for date in dates:
data=myDict[(id,date)]

seems to just take too long. Is there a speedier, more pythonic, way
of doing this? Any help speeding this up would be much appreciated!!


if IDS and dates lenght is the same, you could write
for i in enumerate(IDS):
data=myDict[(IDS[i],dates[i])]

which is more good looking, but I think performances are similar.

my 2 cents.

Riccardo
Jul 18 '05 #2
In article <68*************************@posting.google.com> ,
rabbits77 <ra*******@bigmailbox.net> wrote:

I have a dictionary with a very very large(possibly millions) of
key/value pairs.


If it's literally millions, you probably want to use a database. Just
the dictionary itself (not even talking about the objects used for keys
and values) will take at least 12MB per million entries on a machine with
32-bit addresses.
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

"usenet imitates usenet" --Darkhawk
Jul 18 '05 #3
rabbits77 <ra*******@bigmailbox.net> wrote:
I have a dictionary with a very very large(possibly millions) of
key/value pairs.
The key is a tuple that looks like (id,date)
What is the fastest way to get out all of the values that match any
key given that they individual key elements are coming from two
seperate lists?
The approach of
for id in IDS:
for date in dates:
data=myDict[(id,date)]

seems to just take too long. Is there a speedier, more pythonic, way
of doing this? Any help speeding this up would be much appreciated!!


Hmm... since you're looking up every possible combination of 'id' and
'date', there is no short cut of algorithm. As for speed, it may be
faster to use external database.

--
William Park, Open Geometry Consulting, <op**********@yahoo.ca>
Linux solution for data processing and document management.
Jul 18 '05 #4

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

6 posts views Thread by A Future Computer Scientist | last post: by
16 posts views Thread by JustSomeGuy | last post: by
5 posts views Thread by ArShAm | last post: by
35 posts views Thread by Thomas Matthews | last post: by
4 posts views Thread by Flashman | last post: by
12 posts views Thread by Mark E. Fenner | last post: by
reply views Thread by gheharukoh7 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.