473,700 Members | 2,648 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Python with MPI enable C-module

Hello All,

A software project that I am working on currently has a C++ library
that runs on a Beowulf cluster for parallel computation. The library
code uses MPI for its implementation. We are currently developing
python wrappers for the library. Our current design uses one Python
process as a front end with processes created from the library code on
the back end (it is processes of this type that run on all the nodes of
the cluster).

What I'm considering is delving into the possibility of having Python
processes on all of the nodes of the cluster. These processes would
wrap our C++ library. What I'm wondering is:

1) Would setting up an environment like this require modifying the
Python interpreter or the C++ module that is being wrapped? What I'm
hoping is that the C++ module can go on happily doing MPI operations
despite the fact that it is actually being called by the Python
interpreter.

2) Would it be possible to spawn these Python processes using mpiexec
(or something similar), or would I need to use some of the MPI-2
features to dynamically set up the MPI environment?

3) Has anyone accomplished something like this already? I know there
are extensions and modules that add MPI functionality to Python, but
I'm hoping they could be avoided, since the Python code itself should
never really have to be aware of MPI, only the C++ module that has
already been written.

Let me apologize up front if this has already been discussed before.
I've done a few searches in this group and it seems that most of the
discussion on this front took place many years ago, so the relevance is
quite low.

Thank you in advance for your attention.

~doug

Oct 19 '06 #1
2 1808
On 19 Oct 2006 06:51:00 -0700
"df****@gmail.c om" <df****@gmail.c omwrote:
1) Would setting up an environment like this require modifying the
Python interpreter or the C++ module that is being wrapped? What I'm
hoping is that the C++ module can go on happily doing MPI operations
despite the fact that it is actually being called by the Python
interpreter.
If I am understanding you correctly, you have a Python module which
calls into the C++ library to do the actual MPI operations. If that is
the case, then there should be any problems just calling the Python
wrappers. The MPI library should be unaffected. It would be the same as
having your node programs call into the C++ library, now you just have
one more layer of abstraction.
2) Would it be possible to spawn these Python processes using mpiexec
(or something similar), or would I need to use some of the MPI-2
features to dynamically set up the MPI environment?
If you are writing a Python wrapper to the MPI library, then the
wrapper will be just like a module and not a self sustained program. If
that is the case, mpiexec/mpirun/mpd would not be able to do anything
since they expect to start a program that has main()/__main__.
3) Has anyone accomplished something like this already? I know there
are extensions and modules that add MPI functionality to Python, but
I'm hoping they could be avoided, since the Python code itself should
never really have to be aware of MPI, only the C++ module that has
already been written.
Haven't done it but from what I know about MPI and Python, it is
possible.

--
Mitko Haralanov mi***@qlogic.co m
Senior Software Engineer 650.934.8064
System Interconnect Group http://www.qlogic.com

Oct 20 '06 #2
Mitko,

Thank you for your response.

You understand the problem correctly. The Python module uses MPI calls
in its implementation. The idea of the project is to create an
abstraction for a distributed memory computer. In doing so, the user is
mostly isolated from a need to understand and use MPI calls.

After my post, I was able to find some more information on the subject.
I think you are right in that mpiexec (or equivalents) would not work
without some extra work. However it seems that using an additional
module (such as <a
href="http://datamining.anu. edu.au/~ole/pypar/">Pypar</a>) would allow
me to use Python with mpiexec.

Thank you,
~doug

On Oct 20, 7:13 pm, Mitko Haralanov <m...@qlogic.co mwrote:
On 19 Oct 2006 06:51:00 -0700

"dfj...@gmail.c om" <dfj...@gmail.c omwrote:
1) Would setting up an environment like this require modifying the
Python interpreter or the C++ module that is being wrapped? What I'm
hoping is that the C++ module can go on happily doing MPI operations
despite the fact that it is actually being called by the Python
interpreter.If I am understanding you correctly, you have a Python module which
calls into the C++ library to do the actual MPI operations. If that is
the case, then there should be any problems just calling the Python
wrappers. The MPI library should be unaffected. It would be the same as
having your node programs call into the C++ library, now you just have
one more layer of abstraction.
2) Would it be possible to spawn these Python processes using mpiexec
(or something similar), or would I need to use some of the MPI-2
features to dynamically set up the MPI environment?If you are writing a Python wrapper to the MPI library, then the
wrapper will be just like a module and not a self sustained program. If
that is the case, mpiexec/mpirun/mpd would not be able to do anything
since they expect to start a program that has main()/__main__.
3) Has anyone accomplished something like this already? I know there
are extensions and modules that add MPI functionality to Python, but
I'm hoping they could be avoided, since the Python code itself should
never really have to be aware of MPI, only the C++ module that has
already been written.Haven't done it but from what I know about MPI and Python, it is
possible.

--
Mitko Haralanov m...@qlogic.com
Senior Software Engineer 650.934.8064
System Interconnect Group http://www.qlogic.com
Oct 21 '06 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
3846
by: Logan | last post by:
Several people asked me for the following HOWTO, so I decided to post it here (though it is still very 'alpha' and might contain many (?) mistakes; didn't test what I wrote, but wrote it - more or less - during my own installation of Python 2.3 on Fedora Core 1 Linux for a friend of mine). Anyway, HTH, L.
3
2815
by: Equis Uno | last post by:
Hi, I'm trying to run some Python software I downloaded off of sf.net. It's called Boa. It uses wxPython. It appears my install of Python cannot see my install of wxPython.
2
3748
by: Russell E. Owen | last post by:
I'm trying to build Python 2.3.4 from source on a RedHat Enterprise machine for installation in a net-wide accessible directory /net/python. I tried all of the following variants of ./configure (the first was required for Python 2.3.3 on RedHat 9): ../configure --prefix=/net/python --enable-unicode=ucs4 ../configure --prefix=/net/python ../configure --prefix=/net/python --enable-unicode=ucs2 All of these result in the ominous message...
0
2024
by: Adam McCarthy | last post by:
I'm trying to get a cross compiler working for arm-wince-pe. This is the output for the primes Pyrex example. If I compile simple Hello, World's etc, it works fine, but for some reason Python libraries/headers seem to produce this. I have libpython2.3.a in /usr/local/lib/gcc-lib/arm-wince-pe/3.3.3/../../../../arm-wince-pe/lib/ and /usr/lib/python2.3/config. I've checked it using arm-wince-pe-objdump. I also extracted it with...
14
4331
by: Mark Dufour | last post by:
After nine months of hard work, I am proud to introduce my baby to the world: an experimental Python-to-C++ compiler. It can convert many Python programs into optimized C++ code, without any user intervention such as adding type declarations. It uses rather advanced static type inference techniques to deduce type information by itself. In addition, it determines whether deduced types may be parameterized, and if so, it generates...
2
3578
by: Dean | last post by:
I've been trying to make python a dynamic library. I downloaded Python 2.4.3 Final from the Python web site and I cannot get it to create the library. I've tried using the directive: --enable-shared and
1
1700
by: J. Jeffrey Close | last post by:
Hi all, I have been trying for some time to build Python 2.4.x from source on OS X 10.4.6. I've found *numerous* postings on various mailing lists and web pages documenting the apparently well-known problems of doing so. Various problems arise either in the ../configure step, with configure arguments that don't work, or in the compile, or in my case in the link step with libtool.
83
3646
by: Licheng Fang | last post by:
Hi, I'm learning STL and I wrote some simple code to compare the efficiency of python and STL. //C++ #include <iostream> #include <string> #include <vector> #include <set> #include <algorithm> using namespace std;
4
14227
by: Phoe6 | last post by:
Hi all, I am trying to disable the NIC card (and other cards) enabled in my machine to test diagnostics on that card. I am trying to disable it programmatic using python. I checked python wmi and i could not find ways to disable/enable, (listing is however, possible). Where should I look for to enable/disable devices in python. Thanks,
3
3021
by: Mathias Waack | last post by:
After switching my development environment to 64 bit I've got a problem with a python extension for a 32 bit application. Compiling the app under Linux results in the following error: g++ -m32 -Wall -g -O2 -I. -Idb -DPYTHON=25 -o mappy.o -c mappy.cpp In file included from /usr/include/python2.5/Python.h:57, from mappy.cpp:29: /usr/include/python2.5/pyport.h:732:2: error: #error "LONG_BIT definition appears wrong for platform (bad...
0
8725
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, weíll explore What is ONU, What Is Router, ONU & Routerís main usage, and What is the difference between ONU and Router. Letís take a closer look ! Part I. Meaning of...
0
8644
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9214
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
5902
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4403
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4656
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3088
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2392
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2027
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.