473,587 Members | 2,483 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Extending Python with C: Cannot find MPI library

I am writing some Python code using the Message Passing Interface
(MPI), an API used in parallel computing. There exist a number of
Python implementations of MPI, but apparently they all rely on the
Numeric Python (numpy) package. I need to run my code on a particular
machine made available by my university, which does not have numpy and
will not be getting it.

So I am trying to write my own mini-wrapper for MPI in C to extend
my Python program.

There exists a special C compiler to use when compiling C programs
that use MPI, called mpicc.

Here is what I have tried:

I have written a minimal C program that uses MPI. It works correctly
when compiled with mpicc.

Then I've written Python boilerplate code and a Python script to
compile my C program into a Python-includable module. This does not
work a priori because the Python compilation script uses gcc rather
than mpicc. So I have taken the exact gcc command that the script uses
and replaced "gcc" with "mpicc".

This produces an *.so file that compiles without errors.
Unfortunately, it fails when I try to import it in Python. It can't
find the MPI library.

My MPI code looks like this (plus some boilerplate code):

/* Return the MPI rank of the current process. */
int rank(){
int argc;
char **argv;
int rank, size;
MPI_Init( &argc, &argv );
MPI_Comm_rank( MPI_COMM_WORLD, &rank );
MPI_Finalize();
return rank;
}

/* Main. A 'Hello World' function. */
int hello() {
int rankNumber = rank();
printf ("Hello, World. I am process %d.\n", rankNumber);
return rankNumber;
}

My Python program that includes it looks like this:

import ctest

ctest.hello()


My error message is this:

[ore@localhost Opgave03]$ mpiexec -n 1 python ctest.py
Traceback (most recent call last):
File "ctest.py", line 1, in <module>
import ctest
ImportError: /big/School/Cluster/Opgave03/ctest.so: undefined
symbol: ompi_mpi_comm_w orld
[ore@localhost Opgave03]$

Can anyone suggest anything? Can I get MPI to work in Python?

Last time I asked a similar question, someone recommended that I
check out Cython instead of C. Do MPI bindings for Cython exist?

Thanks in advance.

- Claus Appel
Jun 27 '08 #1
7 4077
-On [20080619 13:53], Spectrum (sp********@gma il.com) wrote:
ImportError: /big/School/Cluster/Opgave03/ctest.so: undefined
symbol: ompi_mpi_comm_w orld
[ore@localhost Opgave03]$

Can anyone suggest anything? Can I get MPI to work in Python?
Sounds like a typical case of not specifying any -L and/or -l options. In
this case the ctest.so module has an undefined reference to the symbol
ompi_mpi_comm_w orld and cannot resolve it. So either it was not linked
properly or the loader cannot find the required library.

Apply nm -D and ldd to the .so and see what ails.

--
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org/ asmodai
イェルーン ラウフ*ッ ク ヴァン デル ウェルヴェ ン
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
Under this standard shalt thou conquer...
Jun 27 '08 #2
On Jun 19, 2:10*pm, Jeroen Ruigrok van der Werven <asmo...@in-
nomine.orgwrote :
-On [20080619 13:53], Spectrum (spectru...@gma il.com) wrote:
*ImportError: /big/School/Cluster/Opgave03/ctest.so: undefined
symbol: ompi_mpi_comm_w orld
*[ore@localhost Opgave03]$
*Can anyone suggest anything? Can I get MPI to work in Python?

Sounds like a typical case of not specifying any -L and/or -l options. In
this case the ctest.so module has an undefined reference to the symbol
ompi_mpi_comm_w orld and cannot resolve it. So either it was not linked
properly or the loader cannot find the required library.

Apply nm -D and ldd to the .so and see what ails.
Thanks for the reply.

It is very likely that I am not compiling properly. I am a sucky C
programmer.

The commands you recommended give the following output:

[ore@localhost Opgave03]$ nm -D ctest.so
U MPI_Comm_rank
U MPI_Finalize
U MPI_Init
w _Jv_RegisterCla sses
000018cc A __bss_start
w __cxa_finalize
w __gmon_start__
U __printf_chk
000018cc A _edata
000018d4 A _end
000006c4 T _fini
00000414 T _init
00000600 T hello
00000650 T main
U ompi_mpi_comm_w orld
000005b0 T rank
[ore@localhost Opgave03]$
[ore@localhost Opgave03]$
[ore@localhost Opgave03]$ ldd ctest.so
linux-gate.so.1 = (0x00110000)
libpython2.5.so .1.0 =/usr/lib/libpython2.5.so .1.0
(0x00113000)
libpthread.so.0 =/lib/libpthread.so.0 (0x0025a000)
libc.so.6 =/lib/libc.so.6 (0x00342000)
libdl.so.2 =/lib/libdl.so.2 (0x00273000)
libutil.so.1 =/lib/libutil.so.1 (0x00278000)
libm.so.6 =/lib/libm.so.6 (0x0027c000)
/lib/ld-linux.so.2 (0x00325000)
[ore@localhost Opgave03]$

The same commands, when applied to the mpicc-compiled, correctly-
functioning C program (in this case just named 'a.out') give this:

[ore@localhost Opgave03]$ ./a.out
Hello, World. I am process 0.
[ore@localhost Opgave03]$ nm -D a.out
U MPI_Comm_rank
U MPI_Finalize
U MPI_Init
08048818 R _IO_stdin_used
w _Jv_RegisterCla sses
080499fc A __bss_start
080499f8 D __data_start
w __gmon_start__
08048750 T __libc_csu_fini
08048760 T __libc_csu_init
U __libc_start_ma in
080499fc A _edata
08049b10 A _end
080487f8 T _fini
08048814 R _fp_hw
08048560 T _init
080485f0 T _start
080499f8 W data_start
080486f9 T hello
0804871f T main
08049a00 B ompi_mpi_comm_w orld
U printf
080486c4 T rank
[ore@localhost Opgave03]$ ldd a.out
linux-gate.so.1 = (0x00110000)
libmpi.so.0 =/usr/lib/openmpi/1.2.4-gcc/libmpi.so.0
(0x0042f000)
libopen-rte.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
rte.so.0 (0x003d4000)
libopen-pal.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
pal.so.0 (0x00344000)
libdl.so.2 =/lib/libdl.so.2 (0x00df0000)
libnsl.so.1 =/lib/libnsl.so.1 (0x003b9000)
libutil.so.1 =/lib/libutil.so.1 (0x004d8000)
libm.so.6 =/lib/libm.so.6 (0x0025c000)
libpthread.so.0 =/lib/libpthread.so.0 (0x00287000)
libc.so.6 =/lib/libc.so.6 (0x004dc000)
/lib/ld-linux.so.2 (0x00325000)
[ore@localhost Opgave03]$

I don't know how to interpret these, so can I get some more advice?

Thanks in advance.

- Claus Appel

Jun 27 '08 #3
-On [20080619 16:21], Spectrum (sp********@gma il.com) wrote:
libmpi.so.0 =/usr/lib/openmpi/1.2.4-gcc/libmpi.so.0
(0x0042f000)
libopen-rte.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
rte.so.0 (0x003d4000)
libopen-pal.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
pal.so.0 (0x00344000)
These libraries are what your binaries (.so and such) needs to link against
as well. I am not sure if you have to add a flag (-mpi) to your compiler or
if you need to add a -L/usr/lib/openmpi/1.2.4-gcc -lmpi -lopen-rte
-lopen-pal to the incantations.

--
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org/ asmodai
イェルーン ラウフ*ッ ク ヴァン デル ウェルヴェ ン
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
The distance to here is infinite...
Jun 27 '08 #4
On Jun 19, 4:39*pm, Jeroen Ruigrok van der Werven <asmo...@in-
nomine.orgwrote :
-On [20080619 16:21], Spectrum (spectru...@gma il.com) wrote:
* * * * *libmpi.so.0 =/usr/lib/openmpi/1.2.4-gcc/libmpi.so.0
(0x0042f000)
* * * * *libopen-rte.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
rte.so.0 (0x003d4000)
* * * * *libopen-pal.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
pal.so.0 (0x00344000)

These libraries are what your binaries (.so and such) needs to link against
as well. I am not sure if you have to add a flag (-mpi) to your compiler or
if you need to add a -L/usr/lib/openmpi/1.2.4-gcc -lmpi -lopen-rte
-lopen-pal to the incantations.
Thanks for the reply.

I did some more experimentation . The compiling commands I use are:

mpicc -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-
D_FORTIFY_SOURC E=2 -fexceptions -fstack-protector --param=ssp-buffer-
size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -
D_GNU_SOURCE -fPIC -fPIC -I/usr/include/python2.5 -c ctest.c -o
ctest.o
gcc -pthread -shared build/temp.linux-i686-2.5/ctest.o -L/usr/lib -
lpython2.5 -o ctest.so

It turns out that changing the second "gcc" to "mpicc" causes the
module to no longer be includable. Python will give this error
message:

ore@localhost Opgave03]$ python ctest.py
Traceback (most recent call last):
File "ctest.py", line 1, in <module>
import ctest
ImportError: dynamic module does not define init function
(initctest)
[ore@localhost Opgave03]$

The same happens if I try to add "-L/usr/lib/openmpi/1.2.4-gcc -lmpi
-lopen-rte -lopen-pal" as options to the second compilation command,
as you recommended.

Looks like MPI doesn't like Python...
Jun 27 '08 #5
-On [20080619 17:11], Spectrum (sp********@gma il.com) wrote:
ImportError: dynamic module does not define init function
Might be it's looking, but not finding, something like crti.S or the likes,
the C runtime files that specify stuff like _init and _fini.

--
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org/ asmodai
イェルーン ラウフ*ッ ク ヴァン デル ウェルヴェ ン
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
If you understand, things are as they are; if not, things are as they are...
Jun 27 '08 #6
* Jeroen Ruigrok van der Werven wrote:
-On [20080619 17:11], Spectrum (sp********@gma il.com) wrote:
> ImportError: dynamic module does not define init function
Might be it's looking, but not finding, something like crti.S or the
likes, the C runtime files that specify stuff like _init and _fini.
Nah. It's just looking for the init function required by the python module
loader. See: http://docs.python.org/ext/methodTable.html

nd
--
die (eval q-qq:Just Another Perl Hacker
:-)

# André Malo, <http://www.perlig.de/#
Jun 27 '08 #7
On Jun 20, 5:56 am, Jeroen Ruigrok van der Werven <asmo...@in-
nomine.orgwrote :
-On [20080619 17:11], Spectrum (spectru...@gma il.com) wrote:
ImportError: dynamic module does not define init function

Might be it's looking, but not finding, something like crti.S or the likes,
the C runtime files that specify stuff like _init and _fini.
No, it means what it says. The OP seems to have stumbled past all the
hurdles imposed by using the mpicc compiler and by his C code calling
other code. Now he needs to put in the glue code so that his
blahblah.so acts like a Python module that defines a python function.
He (inter alia) needs to read http://docs.python.org/ext/ext.html --
starting at the beginning, not at the section that addresses the init
function.

HTH,
John
Jun 27 '08 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
2809
by: Equis Uno | last post by:
Hi, I'm trying to run some Python software I downloaded off of sf.net. It's called Boa. It uses wxPython. It appears my install of Python cannot see my install of wxPython.
0
1133
by: fu | last post by:
hi there, i got the following problem: I have a set of compiled C files, providing a API for a special encryption / decrytion of messages to be exchanged over the internet, well documented and with a lot of programming examples of how to work with it and its functions (in this case the examples are all in C). Now i want to use it with twisted/python together, to have all the advantages of twisted (especially the reactor instead of...
3
1993
by: stefan | last post by:
Hi Folks, I currenty extended some of my C++ functionality to python and also embedded python to use python functionality in my C++ system (and use as well these extended functions). While this works fine with the core python functionality, as soon as I run a script (on the embedded system) which tries to import modules which are not in the core system, like "xml" or "re", it fails and says it cannot find the related dll (for example...
24
3126
by: Jean-Baptiste PERIN | last post by:
Hi, I'm trying to make a windows dll reachable from a python script .. and I'm encountering troubles during link step .. I use "lcc" to compile and link I use python 2.3 under win XP Here's the compile command which seems to work well : -----------------------------------------------------
3
13490
by: ch424 | last post by:
Hi there, I'm using Python 2.4.1 on Ubuntu Linux, and I'm having problems extending python in C: The C code is below: #include <Python.h> #include "ni488.h"
6
2993
by: Qun Cao | last post by:
Hi Everyone, I am a beginner on cross language development. My problem at hand is to build a python interface for a C++ application built on top of a 3D game engine. The purpose of this python interface is providing a convenient scripting toolkit for the application. One example is that a user can write a python script like: player = Player() game.loadPlayer(player) player.moveTo(location)
0
1131
by: Carl Douglas | last post by:
Hi Python fans, I am developing a DLL that is loaded by a host application on windows. I'm using python 2.5. My DLL uses an embedded python interpreter which can access the host application through an API which I have exposed using SWIG 1.3.31. Therefore I have both extended and embedded Python at once: the SWIG generated code is statically linked into my DLL, and one of the
7
3382
by: Maximus Decimus | last post by:
HI all, I am using python v2.5 and I am an amateur working on python. I am extending python for my research work and would like some help and guidance w.r.t this matter from you experienced python developers. II want to add some more KEYWORDS and DATATYPES into the python script apart from the existing ones. It would be really great if anybody could guide me as which files and
0
2100
by: Tim Spens | last post by:
--- On Fri, 6/27/08, Tim Spens <t_spens@yahoo.comwrote: I think I know where the problem is but I'm unsure how to fix it. When I call Register_Handler(...) from python via callback.setHandler1(callback1) this only seems to affect pythons ability to trigger an "event" in c. PyObject *Handler is always NULL even after I call Register_Handler(...). I thought there was some magic here that was assigning the pointer *Handler to my python...
0
7924
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, well explore What is ONU, What Is Router, ONU & Routers main usage, and What is the difference between ONU and Router. Lets take a closer look ! Part I. Meaning of...
0
7854
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
1
7978
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8221
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
5722
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupr who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5395
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
3845
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
2364
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1455
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.