Threaded for loop

[John]

I want to do something like this:

for i = 1 in range(0,N):
for j = 1 in range(0,N):
D[i][j] = calculate(i,j)

I would like to now do this using a fixed number of threads, say 10
threads.
What is the easiest way to do the "parfor" in python?

Thanks in advance for your help,
--j

[skip]

JohnI want to do something like this:

Johnfor i = 1 in range(0,N):
John for j = 1 in range(0,N):
John D[i][j] = calculate(i,j)

JohnI would like to now do this using a fixed number of threads, say
John10 threads. What is the easiest way to do the "parfor" in python?

I'd create a queue containing 10 tokens. Pull a token off the queue, invoke
the thread with the parameters for its chunk, have it compute its bit, lock
D, update it, unlock it, then return the token to the token queue. Sketching
(and completely untested):

# Calculate one row of D
def calcrow(i, N, token, Tqueue, Dqueue):
d = [0.0] * N
for j in range(N):
d[j] = calculate(i, j)
D = Dqueue.get()
D[i][:] = d
Dqueue.put(D)
Tqueue.put(token)

# This queue limits the number of simultaneous threads
Tqueue = Queue.Queue()
for i in range(10):
Tqueue.put(i)

# This queue guards the shared matrix, D
Dqueue = Queue.Queue()
D = []
for i in range(N):
D.append([0.0] * N)
Dqueue.put(D)

for i in range(N):
token = Tqueue.get()
t = threading.Thread(target=calcrow, args=(i, N, token, Tqueue,
Dqueue))
t.start()

Skip

[Paul Rubin]

"John" <we**********@yahoo.comwrites:

I want to do something like this:

for i = 1 in range(0,N):
for j = 1 in range(0,N):
D[i][j] = calculate(i,j)

I would like to now do this using a fixed number of threads, say 10
threads. What is the easiest way to do the "parfor" in python?

It won't help in terms of actual parallelism. Python only lets one
thread run at a time, even on a multi-cpu computer.

[Carl Banks]

Dennis Lee Bieber wrote:

On 13 Jan 2007 12:15:44 -0800, "John" <we**********@yahoo.comdeclaimed
the following in comp.lang.python:

I want to do something like this:

for i = 1 in range(0,N):
for j = 1 in range(0,N):
D[i][j] = calculate(i,j)

I would like to now do this using a fixed number of threads, say 10
threads.
What is the easiest way to do the "parfor" in python?

Thanks in advance for your help,
--j

Don't know if it's the easiest -- and if "calculate" is a CPU-bound
number cruncher with no I/O or other OS-blocking calls, it won't be
faster either as the GIL will only let one run at a time, even on
multi-core processors.

It could still be helpful if you'd like to get as much done as possible
in as short a time as possible, and you suspect that one or two cases
are likely to hold everything up.
Carl Banks

[parallelpython]

John wrote:

I want to do something like this:

for i = 1 in range(0,N):
for j = 1 in range(0,N):
D[i][j] = calculate(i,j)

I would like to now do this using a fixed number of threads, say 10
threads.
What is the easiest way to do the "parfor" in python?

Thanks in advance for your help,

As it was already mentioned before threads will not help in terms of
parallelism (only one thread will be actually working). If you want to
calculate this in parallel here is an easy solution:

import ppsmp

#start with 10 processes
srv = ppsmp.Server(10)

f = []

for i = 1 in range(0,N):
for j = 1 in range(0,N):
#it might be a little bit more complex if 'calculate' depends on
other modules or calls functions
f.append(srv.submit(calculate, (i,j)))

for i = 1 in range(0,N):
for j = 1 in range(0,N):
D[i][j] = f.pop(0)

You can get the latest version of ppsmp module here:
http://www.parallelpython.com/

[John]

Damn! That is bad news. So even if caclulate is independent for (i,j)
and
is computable on separate CPUs (parts of it are CPU bound, parts are IO
bound)
python cant take advantage of this?

Surprised,
--Tom

Paul Rubin wrote:

"John" <we**********@yahoo.comwrites:

I want to do something like this:

for i = 1 in range(0,N):
for j = 1 in range(0,N):
D[i][j] = calculate(i,j)

I would like to now do this using a fixed number of threads, say 10
threads. What is the easiest way to do the "parfor" in python?

It won't help in terms of actual parallelism. Python only lets one
thread run at a time, even on a multi-cpu computer.

[John]

Damn! That is bad news. So even if caclulate is independent for (i,j)
and
is computable on separate CPUs (parts of it are CPU bound, parts are IO
bound)
python cant take advantage of this?

Surprised,
--j

Paul Rubin wrote:

"John" <we**********@yahoo.comwrites:

I want to do something like this:

for i = 1 in range(0,N):
for j = 1 in range(0,N):
D[i][j] = calculate(i,j)

I would like to now do this using a fixed number of threads, say 10
threads. What is the easiest way to do the "parfor" in python?

It won't help in terms of actual parallelism. Python only lets one
thread run at a time, even on a multi-cpu computer.

[John]

Thanks. Does it matter if I call shell commands os.system...etc in
calculate?

Thanks,
--j

pa************@gmail.com wrote:

John wrote:

I want to do something like this:

for i = 1 in range(0,N):
for j = 1 in range(0,N):
D[i][j] = calculate(i,j)

I would like to now do this using a fixed number of threads, say 10
threads.
What is the easiest way to do the "parfor" in python?

Thanks in advance for your help,

As it was already mentioned before threads will not help in terms of
parallelism (only one thread will be actually working). If you want to
calculate this in parallel here is an easy solution:

import ppsmp

#start with 10 processes
srv = ppsmp.Server(10)

f = []

for i = 1 in range(0,N):
for j = 1 in range(0,N):
#it might be a little bit more complex if 'calculate' depends on
other modules or calls functions
f.append(srv.submit(calculate, (i,j)))

for i = 1 in range(0,N):
for j = 1 in range(0,N):
D[i][j] = f.pop(0)

You can get the latest version of ppsmp module here:
http://www.parallelpython.com/

[Paul Rubin]

"John" <we**********@yahoo.comwrites:

Damn! That is bad news. So even if caclulate is independent for
(i,j) and is computable on separate CPUs (parts of it are CPU bound,
parts are IO bound) python cant take advantage of this?

Not at the moment, unless you write C extensions that release the
global interpreter lock (GIL). One of these days. Meanwhile there
are various extension modules that let you use multiple processes,
look up POSH and Pyro.

[parallelpython]

John wrote:

Thanks. Does it matter if I call shell commands os.system...etc in
calculate?

Thanks,
--j

The os.system command neglects important changes in the environment
(redirected streams) and would not work with current version of ppsmp.
Although there is a very simple workaround:
print os.popen("yourcommand").read()
instead of os.system("yourcommand")
Here is a complete working example of that code:
http://www.parallelpython.com/compon...,29/topic,13.0

[skip]

JohnDamn! That is bad news. So even if caclulate is independent for
John(i,j) and is computable on separate CPUs (parts of it are CPU
Johnbound, parts are IO bound) python cant take advantage of this?

It will help if parts are I/O bound, presuming the threads which block
release the global interpreter lock (GIL).

There is a module in development (processing.py) that provides an API like
the threading module but that uses processes under the covers:

http://mail.python.org/pipermail/pyt...er/069297.html

You might find that an interesting alternative.

Skip

[sturlamolden]

John wrote:

I want to do something like this:

for i = 1 in range(0,N):
for j = 1 in range(0,N):
D[i][j] = calculate(i,j)

I would like to now do this using a fixed number of threads, say 10
threads.

Why do you want to run this in 10 threads? Do you have 10 CPUs?

If you are concerned about CPU time, you should not be using threads
(regardless of language) as they are often implemented with the
assumption that they stay idle most of the time (e.g. win32 threads and
pthreads). In addition, CPython has a global interpreter lock (GIL)
that prevents the interpreter from running on several processors in
parallel. It means that python threads are a tool for things like
writing non-blocking i/o and maintaining responsiveness in a GUI'. But
that is what threads are implemented to do anyway, so it doesn't
matter. IronPython and Jython do not have a GIL.

In order to speed up computation you should run multiple processes and
do some sort of IPC. Take a look at MPI (e.g. mpi4py.scipy.org) or
'parallel python'. MPI is the de facto industry standard for dealing
with CPU bound problems on systems with multiple processors, whether
the memory is shared or distributed does not matter. Contrary to common
belief, this approach is more efficient than running multiple threads,
sharing memory and synchronizong with mutexes and event objects - even
if you are using a system unimpeded by a GIL.

The number of parallel tasks should be equal to the number of available
CPU units, not more, as you will get excessive context shifts if the
number of busy threads or processes exceed the number of computational
units. If you only have two logical CPUs (e.g. one dual-core processor)
you should only run two parallel tasks - not ten. If you try to
parallelize using additional tasks (e.g. 8 more), you will just waste
time doing more context shifts, more cache misses, etc. But if you are
a lucky bastard with access to a 10-way server, sure run 10 tasks in
parallel.

[Paul Boddie]

sk**@pobox.com wrote:

>
There is a module in development (processing.py) that provides an API like
the threading module but that uses processes under the covers:

http://mail.python.org/pipermail/pyt...er/069297.html

You might find that an interesting alternative.

See the promised parallel processing overview on the python.org Wiki
for a selection of different solutions:

http://wiki.python.org/moin/ParallelProcessing

Paul