440,235 Members | 1,008 Online Need help? Post your question and get tips & solutions from a community of 440,235 IT Pros & Developers. It's quick & easy.

 P: n/a I want to do something like this: for i = 1 in range(0,N): for j = 1 in range(0,N): D[i][j] = calculate(i,j) I would like to now do this using a fixed number of threads, say 10 threads. What is the easiest way to do the "parfor" in python? Thanks in advance for your help, --j Jan 13 '07 #1
12 Replies

 P: n/a JohnI want to do something like this: Johnfor i = 1 in range(0,N): John for j = 1 in range(0,N): John D[i][j] = calculate(i,j) JohnI would like to now do this using a fixed number of threads, say John10 threads. What is the easiest way to do the "parfor" in python? I'd create a queue containing 10 tokens. Pull a token off the queue, invoke the thread with the parameters for its chunk, have it compute its bit, lock D, update it, unlock it, then return the token to the token queue. Sketching (and completely untested): # Calculate one row of D def calcrow(i, N, token, Tqueue, Dqueue): d = [0.0] * N for j in range(N): d[j] = calculate(i, j) D = Dqueue.get() D[i][:] = d Dqueue.put(D) Tqueue.put(token) # This queue limits the number of simultaneous threads Tqueue = Queue.Queue() for i in range(10): Tqueue.put(i) # This queue guards the shared matrix, D Dqueue = Queue.Queue() D = [] for i in range(N): D.append([0.0] * N) Dqueue.put(D) for i in range(N): token = Tqueue.get() t = threading.Thread(target=calcrow, args=(i, N, token, Tqueue, Dqueue)) t.start() Skip Jan 13 '07 #2

 P: n/a "John"

 P: n/a Dennis Lee Bieber wrote: On 13 Jan 2007 12:15:44 -0800, "John"

 P: n/a Damn! That is bad news. So even if caclulate is independent for (i,j) and is computable on separate CPUs (parts of it are CPU bound, parts are IO bound) python cant take advantage of this? Surprised, --Tom Paul Rubin wrote: "John"

 P: n/a Damn! That is bad news. So even if caclulate is independent for (i,j) and is computable on separate CPUs (parts of it are CPU bound, parts are IO bound) python cant take advantage of this? Surprised, --j Paul Rubin wrote: "John"

 P: n/a "John"

 P: n/a John wrote: Thanks. Does it matter if I call shell commands os.system...etc in calculate? Thanks, --j The os.system command neglects important changes in the environment (redirected streams) and would not work with current version of ppsmp. Although there is a very simple workaround: print os.popen("yourcommand").read() instead of os.system("yourcommand") Here is a complete working example of that code: http://www.parallelpython.com/compon...,29/topic,13.0 Jan 14 '07 #10

 P: n/a JohnDamn! That is bad news. So even if caclulate is independent for John(i,j) and is computable on separate CPUs (parts of it are CPU Johnbound, parts are IO bound) python cant take advantage of this? It will help if parts are I/O bound, presuming the threads which block release the global interpreter lock (GIL). There is a module in development (processing.py) that provides an API like the threading module but that uses processes under the covers: http://mail.python.org/pipermail/pyt...er/069297.html You might find that an interesting alternative. Skip Jan 14 '07 #11 