469,623 Members | 1,422 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,623 developers. It's quick & easy.

Speed Freak

This is a conceptual question rather than a specific coding one, but
hopefully someone might have played around with something similar. In
a nutshell, I have around 10 million information entries with lots of
data points. My current script has two stages - the first organises
certain data points of the data into large (huge) hashes, and the
second stage forks off lots of children and does the subsequent
processing to produce ldifs, using the information in the hashes
(thanks to copy-on-write, and the fact that the children don't need to
update the hashes this doesn't use a great deal of memory).

My current problem is with stage one - it is, by current necessity, a
single process, since it needs to refer to information within the
hashes as it builds them, and the processing required by the single
processor is the choke point here. I would like to cut down the
current time it takes to do the first stage processing (~50 minutes)
and I am at liberty to use any interesting techniques in order to do
so - my hardware is somewhat above spec (24 CPU 6800, 48G RAM etc),
and can be dedicated 100% to the script when it runs, so unusual and
incredibly memory / CPU wasteful techniques are more than welcome.

I've thought of threading (no real experience, but I could probably
figure something out), parent hash-controller with multiple forked
children etc, I'm just curious if anyone has done something similar
and already knows the most efficient way of doing this.

Jul 19 '05 #1
0 1268

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

13 posts views Thread by Yang Li Ke | last post: by
8 posts views Thread by Rob Ristroph | last post: by
34 posts views Thread by Jacek Generowicz | last post: by
28 posts views Thread by Maboroshi | last post: by
52 posts views Thread by Neuruss | last post: by
7 posts views Thread by YAZ | last post: by
6 posts views Thread by Ham | last post: by
6 posts views Thread by Jassim Rahma | last post: by
11 posts views Thread by kyosohma | last post: by
reply views Thread by gheharukoh7 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.