473,320 Members | 2,124 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

No speedup on multi-processor machine?

Hi,
I am using Python Thread library for my parallel processing course
project. I am doing matrix convolution on a multi-processor machine
running Solaris. I just found out that no speed-up is obtained with
threading. It is probably because of something called GIL in Python.
How can I get around
that GIL and get speed-up?
Thanks in advance.
Daniel

Apr 21 '07 #1
14 1722
On Apr 21, 10:53 pm, Dennis Lee Bieber <wlfr...@ix.netcom.comwrote:
On 21 Apr 2007 14:02:12 -0700, danfan1...@yahoo.com declaimed the
following in comp.lang.python:
Hi,
I am using Python Thread library for my parallel processing course
project. I am doing matrix convolution on a multi-processor machine
running Solaris. I just found out that no speed-up is obtained with
threading. It is probably because of something called GIL in Python.
How can I get around
that GIL and get speed-up?

Threading in Python is optimized for I/O bound processing, wherein
the threads spend most of their lives sleeping (blocked waiting for some
I/O to complete, or some lock/event/condition to change state). It is
not optimized for parallel number crunching.

Options:

Don't use the common "CPython" (eg, the Python built from C-language
source using the C-runtime library). Jython (a version that runs on the
JVM, using Java libraries) may not be afflicted with the GIL.
IronPython is *definitely* not restricted by the GIL.

Fuzzyman
http://www.voidspace.org.uk/ironpython/index.shtml

Apr 21 '07 #2
On Apr 21, 11:02 pm, danfan1...@yahoo.com wrote:
Hi,
I am using Python Thread library for my parallel processing course
project. I am doing matrix convolution on a multi-processor machine
running Solaris. I just found out that no speed-up is obtained with
threading. It is probably because of something called GIL in Python.
How can I get around
that GIL and get speed-up?
Thanks in advance.
Daniel
Perhaps try

http://www.parallelpython.com/

or

http://www.its.caltech.edu/~astraw/seppo.html

Caleb

Apr 21 '07 #3
Caleb Hattingh wrote:
On Apr 21, 11:02 pm, danfan1...@yahoo.com wrote:
>>Hi,
I am using Python Thread library for my parallel processing course
project. I am doing matrix convolution on a multi-processor machine
running Solaris. I just found out that no speed-up is obtained with
threading. It is probably because of something called GIL in Python.
How can I get around
that GIL and get speed-up?
Thanks in advance.
Daniel
If you're actually doing the convolution in Python, you need
optimization before you need more CPUs. There's a numerics library
for Python called NumPy, but it doesn't have a convolution function,
although it has an FFT, which may be useful.

But this is just homework. Do something reasonable and turn it
in. A high performance solution to this problem is probably more
work than it's worth.

John Nagle
Apr 21 '07 #4
Fuzzyman:
IronPython is *definitely* not restricted by the GIL.
IronPython is currently mostly slower than CPython although the
particular problem should be tested to see if IronPython helps.

Some recent benchmarks between IronPython and CPython:
http://sparcs.kaist.ac.kr/~tinuviel/pybench/

Neil
Apr 22 '07 #5
On Apr 22, 1:03 am, Neil Hodgson <nyamatongwe+thun...@gmail.com>
wrote:
Fuzzyman:
IronPython is *definitely* not restricted by the GIL.

IronPython is currently mostly slower than CPython although the
particular problem should be tested to see if IronPython helps.

Some recent benchmarks between IronPython and CPython:http://sparcs.kaist.ac.kr/~tinuviel/pybench/
Yep, I've seen that. :-)

http://www.voidspace.org.uk/python/w..._21.shtml#e687

It's not entirely slower, but mainly slower.

OTOH, if you split a job into two threads on a twin-core machine,
IronPython performance will increase dramatically whilst CPython will
go down...

Additionally, extending IronPython from C# is orders of magnitude
easier than extending CPython from C.

Fuzzyman
http://www.voidspace.org.uk/ironpython/index.shtml
Neil

Apr 22 '07 #6
On 2007-04-21, da********@yahoo.com <da********@yahoo.comwrote:
I am using Python Thread library for my parallel processing
course project. I am doing matrix convolution on a
multi-processor machine running Solaris. I just found out that
no speed-up is obtained with threading. It is probably because
of something called GIL in Python. How can I get around that
GIL and get speed-up?
Not much of a parallel processing course. It appears they
haven't taught you anything about parallel processing.

http://wiki.python.org/moin/ParallelProcessing
--
Grant Edwards grante Yow! There's a SALE on
at STRETCH SOCKS down at the
visi.com "7-11"!!
Apr 22 '07 #7
John Nagle wrote:
There's a numerics library
for Python called NumPy, but it doesn't have a convolution function,
although it has an FFT, which may be useful.
In [1]: from numpy import *

In [2]: convolve?
Type: function
Base Class: <type 'function'>
Namespace: Interactive
File:
/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy-1.0.3.dev3714-py2.5-macosx-10.3-fat.egg/numpy/core/numeric.py
Definition: convolve(a, v, mode='full')
Docstring:
Returns the discrete, linear convolution of 1-D sequences a and v; mode
can be 'valid', 'same', or 'full' to specify size of the resulting sequence.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

Apr 22 '07 #8
Thanks guys. But I think IronPython only works on Windows machine, but
I am using a Sun machine. I was suggested to use Jython, which can run
on Sun. But I need to use Numpy for matrix operations, which is only
available to CPython.

Apr 22 '07 #9
On Apr 22, 1:03 am, Neil Hodgson <nyamatongwe+thun...@gmail.com>
wrote:
Fuzzyman:
IronPython is *definitely* not restricted by the GIL.

IronPython is currently mostly slower than CPython although the
particular problem should be tested to see if IronPython helps.

Some recent benchmarks between IronPython and CPython:http://sparcs.kaist.ac.kr/~tinuviel/pybench/
Not when running on the .NET platform, there is not a lot of
difference overall:

http://www.voidspace.org.uk/python/w..._21.shtml#e688

Fuzzyman
http://www.voidspace.org.uk/ironpython/index.shtml
>
Neil

Apr 22 '07 #10
da********@yahoo.com schrieb:
Thanks guys. But I think IronPython only works on Windows machine, but
I am using a Sun machine.
Isn't there a mono port for Sun?

Stefan
Apr 22 '07 #11
In article <Ed*****************@newssvr14.news.prodigy.net> ,
John Nagle <na***@animats.comwrote:
>Caleb Hattingh wrote:
>On Apr 21, 11:02 pm, danfan1...@yahoo.com wrote:
>>>Hi,
I am using Python Thread library for my parallel processing course
project. I am doing matrix convolution on a multi-processor machine
running Solaris. I just found out that no speed-up is obtained with
threading. It is probably because of something called GIL in Python.
How can I get around
that GIL and get speed-up?
Thanks in advance.
Daniel

If you're actually doing the convolution in Python, you need
optimization before you need more CPUs. There's a numerics library
for Python called NumPy, but it doesn't have a convolution function,
although it has an FFT, which may be useful.

But this is just homework. Do something reasonable and turn it
in. A high performance solution to this problem is probably more
work than it's worth.

John Nagle
Along with the excellent advice given by Dennis, John, and the
rest, please be aware that *process*-level parallelization of
a problem sometimes is a benefit. As already recommended, <URL:
http://wiki.python.org/moin/ParallelProcessing touches on most
of the pertinent concepts.
Apr 22 '07 #12
On Apr 21, 5:14 pm, Fuzzyman <fuzzy...@gmail.comwrote:
Additionally, extending IronPython from C# is orders of magnitude
easier than extending CPython from C.
Given the existence of Pyrex, that statement is pretty difficult to
substantiate.

-Mike

Apr 23 '07 #13
On Apr 23, 9:52 pm, Klaas <mike.kl...@gmail.comwrote:
On Apr 21, 5:14 pm, Fuzzyman <fuzzy...@gmail.comwrote:
Additionally, extending IronPython from C# is orders of magnitude
easier than extending CPython from C.

Given the existence of Pyrex, that statement is pretty difficult to
substantiate.
With Pyrex you still need to do memory management for non-Python
types. Additionally compiling C# from within IronPython is *very*easy.

C# is a much easier language to use than C - even with the help of
Pyrex. You use types defined in C# *natively* within IronPython.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/articles.shtml
-Mike

Apr 24 '07 #14
On 23 Apr 2007 13:52:52 -0700, Klaas <mi********@gmail.comwrote:
On Apr 21, 5:14 pm, Fuzzyman <fuzzy...@gmail.comwrote:
>Additionally, extending IronPython from C# is orders of magnitude
easier than extending CPython from C.

Given the existence of Pyrex, that statement is pretty difficult to
substantiate.
IMHO, you don't even need Pyrex. When you've done it manually once,
you have the blueprint for your other extension modules. Most of the
work should be normal "fake OOP in C" programming.

Doing it for the first time is another matter, however.

/Jorgen

--
// Jorgen Grahn <grahn@ Ph'nglui mglw'nafh Cthulhu
\X/ snipabacken.dyndns.org R'lyeh wgah'nagl fhtagn!
Apr 24 '07 #15

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

12
by: * ProteanThread * | last post by:
but depends upon the clique: ...
0
by: frankenberry | last post by:
I have multi-page tiff files. I need to extract individual frames from the multi-page tiffs and save them as single-page tiffs. 95% of the time I receive multi-page tiffs containing 1 or more black...
6
by: cody | last post by:
What are multi file assemblies good for? What are the advantages of using multiple assemblies (A.DLL+B.DLL) vs. a single multi file assembly (A.DLL+A.NETMODULE)?
4
by: mimmo | last post by:
Hi! I should convert the accented letters of a string in the correspondent letters not accented. But when I compile with -Wall it give me: warning: multi-character character constant Do the...
8
by: TM | last post by:
I have a small application that displays records from an access mdb into two datagrids and am looking to see if it is possible to speedup the loadtime somehow. In my formload I am filling my...
5
by: Shane Story | last post by:
I can seem to get the dimensions of a frame in a multiframe tiff. After selecting activeframe, the Width/Height is still really much larger than the page's actual dimensions. When I split a...
5
by: bobwansink | last post by:
Hi, I'm relatively new to programming and I would like to create a C++ multi user program. It's for a project for school. This means I will have to write a paper about the theory too. Does anyone...
12
by: Lars Schouw | last post by:
All, Does anyone know how much performance speedup I can expect by using 64 bit C++ / Windows XP 64 bit over the 32 bit versions? Did anyone test this under Visual Studio 2005 or Intel C++...
4
by: Galen Somerville | last post by:
My VB2005 app gets real time Heart sounds and an ECG from a USB device. I'm looking for a way to speed up the drawing of the traces on the screen. In the following code the routine GetSounds...
5
by: Johnny Blonde | last post by:
Hello Group! I really tried hard for two hours to rewrite the following expression (python 2.4): -------------------------- teilnehmer = for r in Reisen.select(AND(Reisen.q.RESVON <= datum,...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.