473,320 Members | 2,012 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Extending Python with C: Cannot find MPI library

I am writing some Python code using the Message Passing Interface
(MPI), an API used in parallel computing. There exist a number of
Python implementations of MPI, but apparently they all rely on the
Numeric Python (numpy) package. I need to run my code on a particular
machine made available by my university, which does not have numpy and
will not be getting it.

So I am trying to write my own mini-wrapper for MPI in C to extend
my Python program.

There exists a special C compiler to use when compiling C programs
that use MPI, called mpicc.

Here is what I have tried:

I have written a minimal C program that uses MPI. It works correctly
when compiled with mpicc.

Then I've written Python boilerplate code and a Python script to
compile my C program into a Python-includable module. This does not
work a priori because the Python compilation script uses gcc rather
than mpicc. So I have taken the exact gcc command that the script uses
and replaced "gcc" with "mpicc".

This produces an *.so file that compiles without errors.
Unfortunately, it fails when I try to import it in Python. It can't
find the MPI library.

My MPI code looks like this (plus some boilerplate code):

/* Return the MPI rank of the current process. */
int rank(){
int argc;
char **argv;
int rank, size;
MPI_Init( &argc, &argv );
MPI_Comm_rank( MPI_COMM_WORLD, &rank );
MPI_Finalize();
return rank;
}

/* Main. A 'Hello World' function. */
int hello() {
int rankNumber = rank();
printf ("Hello, World. I am process %d.\n", rankNumber);
return rankNumber;
}

My Python program that includes it looks like this:

import ctest

ctest.hello()


My error message is this:

[ore@localhost Opgave03]$ mpiexec -n 1 python ctest.py
Traceback (most recent call last):
File "ctest.py", line 1, in <module>
import ctest
ImportError: /big/School/Cluster/Opgave03/ctest.so: undefined
symbol: ompi_mpi_comm_world
[ore@localhost Opgave03]$

Can anyone suggest anything? Can I get MPI to work in Python?

Last time I asked a similar question, someone recommended that I
check out Cython instead of C. Do MPI bindings for Cython exist?

Thanks in advance.

- Claus Appel
Jun 27 '08 #1
7 4067
-On [20080619 13:53], Spectrum (sp********@gmail.com) wrote:
ImportError: /big/School/Cluster/Opgave03/ctest.so: undefined
symbol: ompi_mpi_comm_world
[ore@localhost Opgave03]$

Can anyone suggest anything? Can I get MPI to work in Python?
Sounds like a typical case of not specifying any -L and/or -l options. In
this case the ctest.so module has an undefined reference to the symbol
ompi_mpi_comm_world and cannot resolve it. So either it was not linked
properly or the loader cannot find the required library.

Apply nm -D and ldd to the .so and see what ails.

--
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org/ asmodai
イェルーン ラウフãƒ*ック ヴァン デル ウェルヴェン
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
Under this standard shalt thou conquer...
Jun 27 '08 #2
On Jun 19, 2:10*pm, Jeroen Ruigrok van der Werven <asmo...@in-
nomine.orgwrote:
-On [20080619 13:53], Spectrum (spectru...@gmail.com) wrote:
*ImportError: /big/School/Cluster/Opgave03/ctest.so: undefined
symbol: ompi_mpi_comm_world
*[ore@localhost Opgave03]$
*Can anyone suggest anything? Can I get MPI to work in Python?

Sounds like a typical case of not specifying any -L and/or -l options. In
this case the ctest.so module has an undefined reference to the symbol
ompi_mpi_comm_world and cannot resolve it. So either it was not linked
properly or the loader cannot find the required library.

Apply nm -D and ldd to the .so and see what ails.
Thanks for the reply.

It is very likely that I am not compiling properly. I am a sucky C
programmer.

The commands you recommended give the following output:

[ore@localhost Opgave03]$ nm -D ctest.so
U MPI_Comm_rank
U MPI_Finalize
U MPI_Init
w _Jv_RegisterClasses
000018cc A __bss_start
w __cxa_finalize
w __gmon_start__
U __printf_chk
000018cc A _edata
000018d4 A _end
000006c4 T _fini
00000414 T _init
00000600 T hello
00000650 T main
U ompi_mpi_comm_world
000005b0 T rank
[ore@localhost Opgave03]$
[ore@localhost Opgave03]$
[ore@localhost Opgave03]$ ldd ctest.so
linux-gate.so.1 = (0x00110000)
libpython2.5.so.1.0 =/usr/lib/libpython2.5.so.1.0
(0x00113000)
libpthread.so.0 =/lib/libpthread.so.0 (0x0025a000)
libc.so.6 =/lib/libc.so.6 (0x00342000)
libdl.so.2 =/lib/libdl.so.2 (0x00273000)
libutil.so.1 =/lib/libutil.so.1 (0x00278000)
libm.so.6 =/lib/libm.so.6 (0x0027c000)
/lib/ld-linux.so.2 (0x00325000)
[ore@localhost Opgave03]$

The same commands, when applied to the mpicc-compiled, correctly-
functioning C program (in this case just named 'a.out') give this:

[ore@localhost Opgave03]$ ./a.out
Hello, World. I am process 0.
[ore@localhost Opgave03]$ nm -D a.out
U MPI_Comm_rank
U MPI_Finalize
U MPI_Init
08048818 R _IO_stdin_used
w _Jv_RegisterClasses
080499fc A __bss_start
080499f8 D __data_start
w __gmon_start__
08048750 T __libc_csu_fini
08048760 T __libc_csu_init
U __libc_start_main
080499fc A _edata
08049b10 A _end
080487f8 T _fini
08048814 R _fp_hw
08048560 T _init
080485f0 T _start
080499f8 W data_start
080486f9 T hello
0804871f T main
08049a00 B ompi_mpi_comm_world
U printf
080486c4 T rank
[ore@localhost Opgave03]$ ldd a.out
linux-gate.so.1 = (0x00110000)
libmpi.so.0 =/usr/lib/openmpi/1.2.4-gcc/libmpi.so.0
(0x0042f000)
libopen-rte.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
rte.so.0 (0x003d4000)
libopen-pal.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
pal.so.0 (0x00344000)
libdl.so.2 =/lib/libdl.so.2 (0x00df0000)
libnsl.so.1 =/lib/libnsl.so.1 (0x003b9000)
libutil.so.1 =/lib/libutil.so.1 (0x004d8000)
libm.so.6 =/lib/libm.so.6 (0x0025c000)
libpthread.so.0 =/lib/libpthread.so.0 (0x00287000)
libc.so.6 =/lib/libc.so.6 (0x004dc000)
/lib/ld-linux.so.2 (0x00325000)
[ore@localhost Opgave03]$

I don't know how to interpret these, so can I get some more advice?

Thanks in advance.

- Claus Appel

Jun 27 '08 #3
-On [20080619 16:21], Spectrum (sp********@gmail.com) wrote:
libmpi.so.0 =/usr/lib/openmpi/1.2.4-gcc/libmpi.so.0
(0x0042f000)
libopen-rte.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
rte.so.0 (0x003d4000)
libopen-pal.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
pal.so.0 (0x00344000)
These libraries are what your binaries (.so and such) needs to link against
as well. I am not sure if you have to add a flag (-mpi) to your compiler or
if you need to add a -L/usr/lib/openmpi/1.2.4-gcc -lmpi -lopen-rte
-lopen-pal to the incantations.

--
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org/ asmodai
イェルーン ラウフãƒ*ック ヴァン デル ウェルヴェン
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
The distance to here is infinite...
Jun 27 '08 #4
On Jun 19, 4:39*pm, Jeroen Ruigrok van der Werven <asmo...@in-
nomine.orgwrote:
-On [20080619 16:21], Spectrum (spectru...@gmail.com) wrote:
* * * * *libmpi.so.0 =/usr/lib/openmpi/1.2.4-gcc/libmpi.so.0
(0x0042f000)
* * * * *libopen-rte.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
rte.so.0 (0x003d4000)
* * * * *libopen-pal.so.0 =/usr/lib/openmpi/1.2.4-gcc/libopen-
pal.so.0 (0x00344000)

These libraries are what your binaries (.so and such) needs to link against
as well. I am not sure if you have to add a flag (-mpi) to your compiler or
if you need to add a -L/usr/lib/openmpi/1.2.4-gcc -lmpi -lopen-rte
-lopen-pal to the incantations.
Thanks for the reply.

I did some more experimentation. The compiling commands I use are:

mpicc -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-
D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-
size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -
D_GNU_SOURCE -fPIC -fPIC -I/usr/include/python2.5 -c ctest.c -o
ctest.o
gcc -pthread -shared build/temp.linux-i686-2.5/ctest.o -L/usr/lib -
lpython2.5 -o ctest.so

It turns out that changing the second "gcc" to "mpicc" causes the
module to no longer be includable. Python will give this error
message:

ore@localhost Opgave03]$ python ctest.py
Traceback (most recent call last):
File "ctest.py", line 1, in <module>
import ctest
ImportError: dynamic module does not define init function
(initctest)
[ore@localhost Opgave03]$

The same happens if I try to add "-L/usr/lib/openmpi/1.2.4-gcc -lmpi
-lopen-rte -lopen-pal" as options to the second compilation command,
as you recommended.

Looks like MPI doesn't like Python...
Jun 27 '08 #5
-On [20080619 17:11], Spectrum (sp********@gmail.com) wrote:
ImportError: dynamic module does not define init function
Might be it's looking, but not finding, something like crti.S or the likes,
the C runtime files that specify stuff like _init and _fini.

--
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org/ asmodai
イェルーン ラウフãƒ*ック ヴァン デル ウェルヴェン
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
If you understand, things are as they are; if not, things are as they are...
Jun 27 '08 #6
* Jeroen Ruigrok van der Werven wrote:
-On [20080619 17:11], Spectrum (sp********@gmail.com) wrote:
> ImportError: dynamic module does not define init function
Might be it's looking, but not finding, something like crti.S or the
likes, the C runtime files that specify stuff like _init and _fini.
Nah. It's just looking for the init function required by the python module
loader. See: http://docs.python.org/ext/methodTable.html

nd
--
die (eval q-qq:Just Another Perl Hacker
:-)

# André Malo, <http://www.perlig.de/#
Jun 27 '08 #7
On Jun 20, 5:56 am, Jeroen Ruigrok van der Werven <asmo...@in-
nomine.orgwrote:
-On [20080619 17:11], Spectrum (spectru...@gmail.com) wrote:
ImportError: dynamic module does not define init function

Might be it's looking, but not finding, something like crti.S or the likes,
the C runtime files that specify stuff like _init and _fini.
No, it means what it says. The OP seems to have stumbled past all the
hurdles imposed by using the mpicc compiler and by his C code calling
other code. Now he needs to put in the glue code so that his
blahblah.so acts like a Python module that defines a python function.
He (inter alia) needs to read http://docs.python.org/ext/ext.html --
starting at the beginning, not at the section that addresses the init
function.

HTH,
John
Jun 27 '08 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Equis Uno | last post by:
Hi, I'm trying to run some Python software I downloaded off of sf.net. It's called Boa. It uses wxPython. It appears my install of Python cannot see my install of wxPython.
0
by: fu | last post by:
hi there, i got the following problem: I have a set of compiled C files, providing a API for a special encryption / decrytion of messages to be exchanged over the internet, well documented and...
3
by: stefan | last post by:
Hi Folks, I currenty extended some of my C++ functionality to python and also embedded python to use python functionality in my C++ system (and use as well these extended functions). While...
24
by: Jean-Baptiste PERIN | last post by:
Hi, I'm trying to make a windows dll reachable from a python script .. and I'm encountering troubles during link step .. I use "lcc" to compile and link I use python 2.3 under win XP ...
3
by: ch424 | last post by:
Hi there, I'm using Python 2.4.1 on Ubuntu Linux, and I'm having problems extending python in C: The C code is below: #include <Python.h> #include "ni488.h"
6
by: Qun Cao | last post by:
Hi Everyone, I am a beginner on cross language development. My problem at hand is to build a python interface for a C++ application built on top of a 3D game engine. The purpose of this python...
0
by: Carl Douglas | last post by:
Hi Python fans, I am developing a DLL that is loaded by a host application on windows. I'm using python 2.5. My DLL uses an embedded python interpreter which can access the host application...
7
by: Maximus Decimus | last post by:
HI all, I am using python v2.5 and I am an amateur working on python. I am extending python for my research work and would like some help and guidance w.r.t this matter from you experienced...
0
by: Tim Spens | last post by:
--- On Fri, 6/27/08, Tim Spens <t_spens@yahoo.comwrote: I think I know where the problem is but I'm unsure how to fix it. When I call Register_Handler(...) from python via...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.