469,645 Members | 1,174 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,645 developers. It's quick & easy.

numarray and SMP

In a quest to speed up numarray computations, I tried writing a 'threaded
array' class for use on SMP systems that would distribute its workload
across the processors. I hit a snag when I found out that since the Python
interpreter is not reentrant, this effectively disables parallel
processing in Python. I've come up with two solutions to this problem,
both involving numarray's C functions that perform the actual vector
operations:

1) Surround the C vector operations with Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS, thus allowing the vector operations (which don't
access Python structures) to run in parallel with the interpreter.
Python glue code would take care of threading and locking.

2) Move the parallelization into the C vector functions themselves. This
would likely get poorer performance (a chain of vector operations
couldn't be combined into one threaded operation).

I'd much rather do #1, but will playing around with the interpreter state
like that cause any problems?

Jul 18 '05 #1
4 1374
Christopher T King wrote:
In a quest to speed up numarray computations, I tried writing a 'threaded
array' class for use on SMP systems that would distribute its workload
across the processors. I hit a snag when I found out that since the Python
interpreter is not reentrant, this effectively disables parallel
processing in Python. I've come up with two solutions to this problem,
both involving numarray's C functions that perform the actual vector
operations:


[...]

I suggest you repost this to the numpy list as well. Not only are the
developers there, but this issue interests many of us, so you'd get an eager
audience and more discussion. Not that I don't think c.l.py is a good forum,
quite the contrary: many threading experts live here and not in numpy. I
meant posting to both places, since the combined expertise of 'generic
threading' experts from c.l.py and numeric/numarray users/developers will
likely be a good thing.

Best,

f
Jul 18 '05 #2
On Thu, 1 Jul 2004, Fernando Perez wrote:
I suggest you repost this to the numpy list as well. Not only are the
developers there, but this issue interests many of us, so you'd get an eager
audience and more discussion. Not that I don't think c.l.py is a good forum,
quite the contrary: many threading experts live here and not in numpy. I
meant posting to both places, since the combined expertise of 'generic
threading' experts from c.l.py and numeric/numarray users/developers will
likely be a good thing.


Thanks, and done.

Jul 18 '05 #3
Christopher T King wrote:
On Thu, 1 Jul 2004, Fernando Perez wrote:
I suggest you repost this to the numpy list as well. Not only are the
[...]
Thanks, and done.


Just saw it. I don't really have an answer for you, but I'm curious about what
others on that list may say. We'll see.

Cheers,

f
Jul 18 '05 #4
In article <Pi**************************************@ccc1.wpi .edu>,
Christopher T King <sq******@WPI.EDU> wrote:

1) Surround the C vector operations with Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS, thus allowing the vector operations (which don't
access Python structures) to run in parallel with the interpreter.
Python glue code would take care of threading and locking.

2) Move the parallelization into the C vector functions themselves. This
would likely get poorer performance (a chain of vector operations
couldn't be combined into one threaded operation).

I'd much rather do #1, but will playing around with the interpreter state
like that cause any problems?


Not at all -- that's precisely what they're there for. All you have to
do is make sure you're not calling back into Python before doing
Py_END_ALLOW_THREADS. Traditionally, Python has only done this for I/O
operations; I'm very happy to see someone trying to take a whack at the
computational side. (Although I ended up mostly dropping the ball, that
was the original impetus for the Decimal project, to encourage more
computational threading.)
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

"Typing is cheap. Thinking is expensive." --Roy Smith, c.l.py
Jul 18 '05 #5

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

reply views Thread by RJS | last post: by
3 posts views Thread by Alexander Schwaigkofler | last post: by
4 posts views Thread by Marco Bubke | last post: by
4 posts views Thread by Marco Bubke | last post: by
2 posts views Thread by Marc Schellens | last post: by
3 posts views Thread by SunX | last post: by
3 posts views Thread by Mizrandir | last post: by
reply views Thread by andrewfelch | last post: by
10 posts views Thread by Bryan | last post: by
reply views Thread by robert | last post: by
reply views Thread by gheharukoh7 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.