473,320 Members | 1,945 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Python with MPI enable C-module

Hello All,

A software project that I am working on currently has a C++ library
that runs on a Beowulf cluster for parallel computation. The library
code uses MPI for its implementation. We are currently developing
python wrappers for the library. Our current design uses one Python
process as a front end with processes created from the library code on
the back end (it is processes of this type that run on all the nodes of
the cluster).

What I'm considering is delving into the possibility of having Python
processes on all of the nodes of the cluster. These processes would
wrap our C++ library. What I'm wondering is:

1) Would setting up an environment like this require modifying the
Python interpreter or the C++ module that is being wrapped? What I'm
hoping is that the C++ module can go on happily doing MPI operations
despite the fact that it is actually being called by the Python
interpreter.

2) Would it be possible to spawn these Python processes using mpiexec
(or something similar), or would I need to use some of the MPI-2
features to dynamically set up the MPI environment?

3) Has anyone accomplished something like this already? I know there
are extensions and modules that add MPI functionality to Python, but
I'm hoping they could be avoided, since the Python code itself should
never really have to be aware of MPI, only the C++ module that has
already been written.

Let me apologize up front if this has already been discussed before.
I've done a few searches in this group and it seems that most of the
discussion on this front took place many years ago, so the relevance is
quite low.

Thank you in advance for your attention.

~doug

Oct 19 '06 #1
2 1794
On 19 Oct 2006 06:51:00 -0700
"df****@gmail.com" <df****@gmail.comwrote:
1) Would setting up an environment like this require modifying the
Python interpreter or the C++ module that is being wrapped? What I'm
hoping is that the C++ module can go on happily doing MPI operations
despite the fact that it is actually being called by the Python
interpreter.
If I am understanding you correctly, you have a Python module which
calls into the C++ library to do the actual MPI operations. If that is
the case, then there should be any problems just calling the Python
wrappers. The MPI library should be unaffected. It would be the same as
having your node programs call into the C++ library, now you just have
one more layer of abstraction.
2) Would it be possible to spawn these Python processes using mpiexec
(or something similar), or would I need to use some of the MPI-2
features to dynamically set up the MPI environment?
If you are writing a Python wrapper to the MPI library, then the
wrapper will be just like a module and not a self sustained program. If
that is the case, mpiexec/mpirun/mpd would not be able to do anything
since they expect to start a program that has main()/__main__.
3) Has anyone accomplished something like this already? I know there
are extensions and modules that add MPI functionality to Python, but
I'm hoping they could be avoided, since the Python code itself should
never really have to be aware of MPI, only the C++ module that has
already been written.
Haven't done it but from what I know about MPI and Python, it is
possible.

--
Mitko Haralanov mi***@qlogic.com
Senior Software Engineer 650.934.8064
System Interconnect Group http://www.qlogic.com

Oct 20 '06 #2
Mitko,

Thank you for your response.

You understand the problem correctly. The Python module uses MPI calls
in its implementation. The idea of the project is to create an
abstraction for a distributed memory computer. In doing so, the user is
mostly isolated from a need to understand and use MPI calls.

After my post, I was able to find some more information on the subject.
I think you are right in that mpiexec (or equivalents) would not work
without some extra work. However it seems that using an additional
module (such as <a
href="http://datamining.anu.edu.au/~ole/pypar/">Pypar</a>) would allow
me to use Python with mpiexec.

Thank you,
~doug

On Oct 20, 7:13 pm, Mitko Haralanov <m...@qlogic.comwrote:
On 19 Oct 2006 06:51:00 -0700

"dfj...@gmail.com" <dfj...@gmail.comwrote:
1) Would setting up an environment like this require modifying the
Python interpreter or the C++ module that is being wrapped? What I'm
hoping is that the C++ module can go on happily doing MPI operations
despite the fact that it is actually being called by the Python
interpreter.If I am understanding you correctly, you have a Python module which
calls into the C++ library to do the actual MPI operations. If that is
the case, then there should be any problems just calling the Python
wrappers. The MPI library should be unaffected. It would be the same as
having your node programs call into the C++ library, now you just have
one more layer of abstraction.
2) Would it be possible to spawn these Python processes using mpiexec
(or something similar), or would I need to use some of the MPI-2
features to dynamically set up the MPI environment?If you are writing a Python wrapper to the MPI library, then the
wrapper will be just like a module and not a self sustained program. If
that is the case, mpiexec/mpirun/mpd would not be able to do anything
since they expect to start a program that has main()/__main__.
3) Has anyone accomplished something like this already? I know there
are extensions and modules that add MPI functionality to Python, but
I'm hoping they could be avoided, since the Python code itself should
never really have to be aware of MPI, only the C++ module that has
already been written.Haven't done it but from what I know about MPI and Python, it is
possible.

--
Mitko Haralanov m...@qlogic.com
Senior Software Engineer 650.934.8064
System Interconnect Group http://www.qlogic.com
Oct 21 '06 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Logan | last post by:
Several people asked me for the following HOWTO, so I decided to post it here (though it is still very 'alpha' and might contain many (?) mistakes; didn't test what I wrote, but wrote it - more or...
3
by: Equis Uno | last post by:
Hi, I'm trying to run some Python software I downloaded off of sf.net. It's called Boa. It uses wxPython. It appears my install of Python cannot see my install of wxPython.
2
by: Russell E. Owen | last post by:
I'm trying to build Python 2.3.4 from source on a RedHat Enterprise machine for installation in a net-wide accessible directory /net/python. I tried all of the following variants of ./configure...
0
by: Adam McCarthy | last post by:
I'm trying to get a cross compiler working for arm-wince-pe. This is the output for the primes Pyrex example. If I compile simple Hello, World's etc, it works fine, but for some reason Python...
14
by: Mark Dufour | last post by:
After nine months of hard work, I am proud to introduce my baby to the world: an experimental Python-to-C++ compiler. It can convert many Python programs into optimized C++ code, without any user...
2
by: Dean | last post by:
I've been trying to make python a dynamic library. I downloaded Python 2.4.3 Final from the Python web site and I cannot get it to create the library. I've tried using the directive: ...
1
by: J. Jeffrey Close | last post by:
Hi all, I have been trying for some time to build Python 2.4.x from source on OS X 10.4.6. I've found *numerous* postings on various mailing lists and web pages documenting the apparently...
83
by: Licheng Fang | last post by:
Hi, I'm learning STL and I wrote some simple code to compare the efficiency of python and STL. //C++ #include <iostream> #include <string> #include <vector> #include <set> #include...
4
by: Phoe6 | last post by:
Hi all, I am trying to disable the NIC card (and other cards) enabled in my machine to test diagnostics on that card. I am trying to disable it programmatic using python. I checked python wmi and...
3
by: Mathias Waack | last post by:
After switching my development environment to 64 bit I've got a problem with a python extension for a 32 bit application. Compiling the app under Linux results in the following error: g++ -m32...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.