By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
464,822 Members | 1,008 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 464,822 IT Pros & Developers. It's quick & easy.

Extending Python with C: Cannot find MPI library

P: n/a
I am writing some Python code using the Message Passing Interface
(MPI), an API used in parallel computing. There exist a number of
Python implementations of MPI, but apparently they all rely on the
Numeric Python (numpy) package. I need to run my code on a particular
machine made available by my university, which does not have numpy and
will not be getting it.

So I am trying to write my own mini-wrapper for MPI in C to extend
my Python program.

There exists a special C compiler to use when compiling C programs
that use MPI, called mpicc.

Here is what I have tried:

I have written a minimal C program that uses MPI. It works correctly
when compiled with mpicc.

Then I've written Python boilerplate code and a Python script to
compile my C program into a Python-includable module. This does not
work a priori because the Python compilation script uses gcc rather
than mpicc. So I have taken the exact gcc command that the script uses
and replaced "gcc" with "mpicc".

This produces an *.so file that compiles without errors.
Unfortunately, it fails when I try to import it in Python. It can't
find the MPI library.

My MPI code looks like this (plus some boilerplate code):

/* Return the MPI rank of the current process. */
int rank(){
int argc;
char **argv;
int rank, size;
MPI_Init( &argc, &argv );
MPI_Comm_rank( MPI_COMM_WORLD, &rank );
MPI_Finalize();
return rank;
}

/* Main. A 'Hello World' function. */
int hello() {
int rankNumber = rank();
printf ("Hello, World. I am process %d.\n", rankNumber);
return rankNumber;
}

My Python program that includes it looks like this:

import ctest

ctest.hello()


My error message is this:

[ore@localhost Opgave03]$ mpiexec -n 1 python ctest.py
Traceback (most recent call last):
File "ctest.py", line 1, in <module>
import ctest
ImportError: /big/School/Cluster/Opgave03/ctest.so: undefined
symbol: ompi_mpi_comm_world
[ore@localhost Opgave03]$

Can anyone suggest anything? Can I get MPI to work in Python?

Last time I asked a similar question, someone recommended that I
check out Cython instead of C. Do MPI bindings for Cython exist?

Thanks in advance.

- Claus Appel
Jun 27 '08 #1
Share this Question
Share on Google+
7 Replies

P: n/a
-On [20080619 13:53], Spectrum (sp********@gmail.com) wrote:
ImportError: /big/School/Cluster/Opgave03/ctest.so: undefined
symbol: ompi_mpi_comm_world
[ore@localhost Opgave03]$

Can anyone suggest anything? Can I get MPI to work in Python?
Sounds like a typical case of not specifying any -L and/or -l options. In
this case the ctest.so module has an undefined reference to the symbol
ompi_mpi_comm_world and cannot resolve it. So either it was not linked
properly or the loader cannot find the required library.

Apply nm -D and ldd to the .so and see what ails.

--
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org/ asmodai
イェルーン ラウフ*ック ヴァン デル ウェルヴェン
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
Under this standard shalt thou conquer...
Jun 27 '08 #2

P: n/a
On Jun 19, 2:10*pm, Jeroen Ruigrok van der Werven <asmo...@in-
nomine.orgwrote:
-On [20080619 13:53], Spectrum (spectru...@gmail.com) wrote:
*ImportError: /big/School/Cluster/Opgave03/ctest.so: undefined
symbol: ompi_mpi_comm_world
*[ore@localhost Opgave03]$
*Can anyone suggest anything? Can I get MPI to work in Python?

Sounds like a typical case of not specifying any -L and/or -l options. In
this case the ctest.so module has an undefined reference to the symbol
ompi_mpi_comm_world and cannot resolve it. So either it was not linked
properly or the loader cannot find the required library.

Apply nm -D and ldd to the .so and see what ails.
Thanks for the reply.

It is very likely that I am not compiling properly. I am a sucky C
programmer.

The commands you recommended give the following output:

[ore@localhost Opgave03]$ nm -D ctest.so
U MPI_Comm_rank
U MPI_Finalize
U MPI_Init
w _Jv_RegisterClasses
000018cc A __bss_start
w __cxa_finalize
w __gmon_start__
U __printf_chk
000018cc A _edata
000018d4 A _end
000006c4 T _fini
00000414 T _init
00000600 T hello
00000650 T main
U ompi_mpi_comm_world
000005b0 T rank
[ore@localhost Opgave03]$
[ore@localhost Opgave03]$
[ore@localhost Opgave03]$ ldd ctest.so
linux-gate.so.1 = (0x00110000)
libpython2.5.so.1.0 =/usr/lib/libpython2.5.so.1.0
(0x00113000)
libpthread.so.0 =/lib/libpthread.so.0 (0x0025a000)
libc.so.6 =/lib/libc.so.6 (0x00342000)
libdl.so.2 =/lib/libdl.so.2 (0x00273000)
libutil.so.1 =/lib/libutil.so.1 (0x00278000)
libm.so.6 =/lib/libm.so.6 (0x0027c000)
/lib/ld-linux.so.2 (0x00325000)
[ore@localhost Opgave03]$

The same commands, when applied to the mpicc-compiled, correctly-
functioning C program (in this case just named 'a.out') give this:

[ore@localhost Opgave03]$ ./a.out
Hello, World. I am process 0.
[ore@localhost Opgave03]$ nm -D a.out
U MPI_Comm_rank
U MPI_Finalize
U MPI_Init
08048818 R _IO_stdin_used
w _Jv_RegisterClasses
080499fc A __bss_start
080499f8 D __data_start
w __gmon_start__
08048750 T __libc_csu_fini
08048760 T __libc_csu_init
U __libc_start_main
080499fc A _edata
08049b10 A _end
080487f8 T _fini
08048814 R _fp_hw
08048560 T _init
080485f0 T _start
080499f8 W data_start
080486f9 T hello
0804871f T main
08049a00 B ompi_mpi_comm_world
U printf
080486c4 T rank
[ore@localhost Opgave03]$ ldd a.out
linux-gate.so.1 = (0x00110000)
libmpi.so.0 =/usr/lib/openmpi/1.2.4-gcc/libmpi.so.0
(0x0042f000)
libopen-rte.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
rte.so.0 (0x003d4000)
libopen-pal.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
pal.so.0 (0x00344000)
libdl.so.2 =/lib/libdl.so.2 (0x00df0000)
libnsl.so.1 =/lib/libnsl.so.1 (0x003b9000)
libutil.so.1 =/lib/libutil.so.1 (0x004d8000)
libm.so.6 =/lib/libm.so.6 (0x0025c000)
libpthread.so.0 =/lib/libpthread.so.0 (0x00287000)
libc.so.6 =/lib/libc.so.6 (0x004dc000)
/lib/ld-linux.so.2 (0x00325000)
[ore@localhost Opgave03]$

I don't know how to interpret these, so can I get some more advice?

Thanks in advance.

- Claus Appel

Jun 27 '08 #3

P: n/a
-On [20080619 16:21], Spectrum (sp********@gmail.com) wrote:
libmpi.so.0 =/usr/lib/openmpi/1.2.4-gcc/libmpi.so.0
(0x0042f000)
libopen-rte.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
rte.so.0 (0x003d4000)
libopen-pal.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
pal.so.0 (0x00344000)
These libraries are what your binaries (.so and such) needs to link against
as well. I am not sure if you have to add a flag (-mpi) to your compiler or
if you need to add a -L/usr/lib/openmpi/1.2.4-gcc -lmpi -lopen-rte
-lopen-pal to the incantations.

--
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org/ asmodai
イェルーン ラウフ*ック ヴァン デル ウェルヴェン
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
The distance to here is infinite...
Jun 27 '08 #4

P: n/a
On Jun 19, 4:39*pm, Jeroen Ruigrok van der Werven <asmo...@in-
nomine.orgwrote:
-On [20080619 16:21], Spectrum (spectru...@gmail.com) wrote:
* * * * *libmpi.so.0 =/usr/lib/openmpi/1.2.4-gcc/libmpi.so.0
(0x0042f000)
* * * * *libopen-rte.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
rte.so.0 (0x003d4000)
* * * * *libopen-pal.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
pal.so.0 (0x00344000)

These libraries are what your binaries (.so and such) needs to link against
as well. I am not sure if you have to add a flag (-mpi) to your compiler or
if you need to add a -L/usr/lib/openmpi/1.2.4-gcc -lmpi -lopen-rte
-lopen-pal to the incantations.
Thanks for the reply.

I did some more experimentation. The compiling commands I use are:

mpicc -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-
D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-
size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -
D_GNU_SOURCE -fPIC -fPIC -I/usr/include/python2.5 -c ctest.c -o
ctest.o
gcc -pthread -shared build/temp.linux-i686-2.5/ctest.o -L/usr/lib -
lpython2.5 -o ctest.so

It turns out that changing the second "gcc" to "mpicc" causes the
module to no longer be includable. Python will give this error
message:

ore@localhost Opgave03]$ python ctest.py
Traceback (most recent call last):
File "ctest.py", line 1, in <module>
import ctest
ImportError: dynamic module does not define init function
(initctest)
[ore@localhost Opgave03]$

The same happens if I try to add "-L/usr/lib/openmpi/1.2.4-gcc -lmpi
-lopen-rte -lopen-pal" as options to the second compilation command,
as you recommended.

Looks like MPI doesn't like Python...
Jun 27 '08 #5

P: n/a
-On [20080619 17:11], Spectrum (sp********@gmail.com) wrote:
ImportError: dynamic module does not define init function
Might be it's looking, but not finding, something like crti.S or the likes,
the C runtime files that specify stuff like _init and _fini.

--
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org/ asmodai
イェルーン ラウフ*ック ヴァン デル ウェルヴェン
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
If you understand, things are as they are; if not, things are as they are...
Jun 27 '08 #6

P: n/a
* Jeroen Ruigrok van der Werven wrote:
-On [20080619 17:11], Spectrum (sp********@gmail.com) wrote:
> ImportError: dynamic module does not define init function
Might be it's looking, but not finding, something like crti.S or the
likes, the C runtime files that specify stuff like _init and _fini.
Nah. It's just looking for the init function required by the python module
loader. See: http://docs.python.org/ext/methodTable.html

nd
--
die (eval q-qq:Just Another Perl Hacker
:-)

# André Malo, <http://www.perlig.de/#
Jun 27 '08 #7

P: n/a
On Jun 20, 5:56 am, Jeroen Ruigrok van der Werven <asmo...@in-
nomine.orgwrote:
-On [20080619 17:11], Spectrum (spectru...@gmail.com) wrote:
ImportError: dynamic module does not define init function

Might be it's looking, but not finding, something like crti.S or the likes,
the C runtime files that specify stuff like _init and _fini.
No, it means what it says. The OP seems to have stumbled past all the
hurdles imposed by using the mpicc compiler and by his C code calling
other code. Now he needs to put in the glue code so that his
blahblah.so acts like a Python module that defines a python function.
He (inter alia) needs to read http://docs.python.org/ext/ext.html --
starting at the beginning, not at the section that addresses the init
function.

HTH,
John
Jun 27 '08 #8

This discussion thread is closed

Replies have been disabled for this discussion.