469,336 Members | 5,570 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,336 developers. It's quick & easy.

segfault in extension module


I've written a function in C to perform protein sequence alignment. This
works fine in a standalone C program. I've added the necessary packaging
to use it in Python; it returns three strings and an integer. However, as
soon as the function is complete, I get a segfault and the interpreter
dies.

If I run Python interactively, just calling the function causes a
segfault. If I'm running a script, I can actually print out the return
values (which are what I'd expect - so something's working) but as soon as
the script is done I get the segfault again. I can even call the function
twice, with different arguments - and it works both times. So it appears
that the problem is with tying up loose ends.

How do I determine what is going wrong? I do not get any problem like
this in the C version. I am not using free() anywhere - I will eventually
need to fix this, but I cannot find any place where I might be accessing
unavailable memory. (Adding in free() does not make any difference in the
module, for what it's worth, but I've had some issues with the C program
so I've left it out.)

[I've also tried using PyMem_Malloc instead, throughout the C code.
Doesn't help.]

thanks,
Nat
(please reply directly!)
Jul 18 '05 #1
12 2834
Nathaniel Echols <ec****@OCF.Berkeley.EDU> writes:
How do I determine what is going wrong?


I recommend to run your code (or the interactive python) in a debugger.
For example, with gdb, you'd get

gdb /usr/bin/python
(gdb) run
import foo

[program crashes]
(gdb) bt

The latter command will give a backtrace, which should tell you where
it crashes. If you don't get enough detail, make sure you compile your
module with debugging information. If you are on Windows, make then
sure that Python is compiled for debugging as well.

Regards,
Martin
Jul 18 '05 #2
Hello Nat,
I've written a function in C to perform protein sequence alignment. This
works fine in a standalone C program. I've added the necessary packaging
to use it in Python; Which one? Are you using the C API?
it returns three strings and an integer. However, as
soon as the function is complete, I get a segfault and the interpreter
dies.

Looks like refcount problems, check out
http://www.python.org/doc/current/ext/refcounts.html .

IMO just avoid all this stuff and use SWIG/Boost.Python/Pyrex.

HTH.
Miki
Jul 18 '05 #3
> > I've written a function in C to perform protein sequence alignment. This
works fine in a standalone C program. I've added the necessary packaging
to use it in Python; Which one? Are you using the C API?


Yup - I'm reading right out of the manual.
Looks like refcount problems, check out
http://www.python.org/doc/current/ext/refcounts.html .
I read this before and couldn't figure out what it meant. This does seem
like it would relate, but I can't figure out what I'm doing incorrectly.
I just have one function which calls a pure C function and returns a
tuple of strings from it. I'm guessing I need to add a Py_INCREF()
somewhere but so far this just makes it segfault sooner. (I'm
not sure what argument to use for Py_INCREF(), either.)

I've looked at several other pages, and they all seem to involve setups
more complicated than what I'm doing. I'm already using Py_BuildValue()
to generate the returned tuple, and my understanding is that this should
avoid major problems. . .
IMO just avoid all this stuff and use SWIG/Boost.Python/Pyrex.


I'll look at these, but I only have a tiny little bit of code I need to do
this with - I coded it from scratch with the intention of using it this
way, and could have written it in Python if I didn't care about speed.
Would I really benefit from using one of the other methods? The goal here
is explicitly to put only the very time-dependent code in C; everything
else stays in Python.

thanks,
Nat
Jul 18 '05 #4
> The latter command will give a backtrace, which should tell you where it
crashes. If you don't get enough detail, make sure you compile your
module with debugging information. If you are on Windows, make then sure
that Python is compiled for debugging as well. Regards, Martin


Okay:

#0 0x420744fe in _int_free () from /lib/tls/libc.so.6
#1 0x420734d6 in free () from /lib/tls/libc.so.6
#2 0x0809dc0d in _PyObject_GC_Del ()
#3 0x080ce86f in PyDict_Next ()
#4 0x080d191d in _PyModule_Clear ()

I guess this makes sense, but I'm still not sure how to fix it. . .
Jul 18 '05 #5
Nathaniel Echols <ec****@OCF.Berkeley.EDU> writes:
[...]
IMO just avoid all this stuff and use SWIG/Boost.Python/Pyrex.
I'll look at these, but I only have a tiny little bit of code I need to do

[...] Would I really benefit from using one of the other methods?
Why not take advantage of it? You've already discovered how
hand-writing extensions can be painful. SWIG is probably best for
you. Ignore all the fancy SWIG features, just ask it to wrap the
function, it's very easy.

The goal here
is explicitly to put only the very time-dependent code in C; everything
else stays in Python.


SWIG should be able to do that. There's probably some overhead above
a hand-written wrapper, but it's so trivial to use SWIG in this simple
way that you probably shouldn't begin to worry about that -- it's
unlikely to be a problem.
John
Jul 18 '05 #6
Nathaniel Echols <ec****@OCF.Berkeley.EDU> writes:
Looks like refcount problems, check out
http://www.python.org/doc/current/ext/refcounts.html .
I read this before and couldn't figure out what it meant.


Then I am pretty sure this is your problem :-)
This does seem like it would relate, but I can't figure out what I'm
doing incorrectly. I just have one function which calls a pure C
function and returns a tuple of strings from it. I'm guessing I
need to add a Py_INCREF() somewhere but so far this just makes it
segfault sooner. (I'm not sure what argument to use for
Py_INCREF(), either.)

I've looked at several other pages, and they all seem to involve setups
more complicated than what I'm doing. I'm already using Py_BuildValue()
to generate the returned tuple, and my understanding is that this should
avoid major problems. . .


Post some code.

Cheers,
mwh

--
If Unicode is a horde of zombies with flaming dung sticks,
the hideous intricacies of JIS, Chinese Big-5, Chinese
Traditional, KOI-8, et cetera are at least an army of ogres
with salt and flensing knives. -- Eric S. Raymond, python-dev
Jul 18 '05 #7
Nathaniel Echols <ec****@OCF.Berkeley.EDU> writes:
Okay:

#0 0x420744fe in _int_free () from /lib/tls/libc.so.6
#1 0x420734d6 in free () from /lib/tls/libc.so.6
#2 0x0809dc0d in _PyObject_GC_Del ()
#3 0x080ce86f in PyDict_Next ()
#4 0x080d191d in _PyModule_Clear ()

I guess this makes sense, but I'm still not sure how to fix it. . .


Ah, tls/libc.so.6. I think you lose, being confronted with a buggy C
library. Try making it not use /lib/tls.

If that does not change the behaviour, you probably have a
ref-counting bug somewhere. Try building a debugging version of
Python.

Regards,
Martin
Jul 18 '05 #8
> Post some code.

may god have mercy on my soul:

#include <Python.h>
#include "nw.h"
static PyObject *nw_align (PyObject *self, PyObject *args) {
char *seq1, *seq2, *mfile;
char *out1, *out2, *match;
int penalty, status, score;
PyObject *results;

if (! PyArg_ParseTuple(args, "sssi", &seq1, &seq2, &mfile, &penalty)) {
return NULL;
}

status = nw(seq1, seq2, mfile, penalty, &out1, &out2, &match, &score);

if (status == -1) {
PyErr_NoMemory();
return NULL;
}

results = Py_BuildValue("(sssi)", out1, out2, match, score);
return results;
}

static PyMethodDef nwMethods[] = {
{"align", nw_align, METH_VARARGS,
"Perform Needleman-Wunsch alignment of two protein sequences."},
{NULL, NULL, 0, NULL} /* This is required! No idea why. */
};

void initnw (void) {
(void) Py_InitModule("nw", nwMethods);
}

This is just the wrapper; the function that it calls is defined such:
int nw (char *seq1, char *seq2, char *matrixfile, int penalty,
char **_out1, char **_out2, char **_match, int *_score);
(I can supply this too, but I already know this works fine in a standalone
C program.)

In Python, I simply do this:
(out1, out2, match, score) = nw.align(seq1, seq2, "BLOSUM62", 0)

This is it; should be simple to fix, no?
Jul 18 '05 #9
Nat Echols <ec****@uclink.berkeley.edu> writes:
Post some code.


may god have mercy on my soul:


Well, I can't see anything flagrantly wrong with that.

Does a debugger provide any hints?

Cheers,
mwh

--
[3] Modem speeds being what they are, large .avi files were
generally downloaded to the shell server instead[4].
[4] Where they were usually found by the technical staff, and
burned to CD. -- Carlfish, asr
Jul 18 '05 #10
> Well, I can't see anything flagrantly wrong with that.

Does a debugger provide any hints?


Someone else suggested this, and this is what the backtrace from gdb
indicated:

#0 0x420744fe in _int_free () from /lib/tls/libc.so.6
#1 0x420734d6 in free () from /lib/tls/libc.so.6
#2 0x0809dc0d in _PyObject_GC_Del ()
#3 0x080ce86f in PyDict_Next ()
#4 0x080d191d in _PyModule_Clear ()
(It goes on like this for a while. . .)

As far as the tls C library, which someone told me was buggy - I don't
know how to force use of a different library; Python appears to be linked
to the same one. At any rate, I seem to have the same problem on two
different machines, one running SuSE 8.1, one running RedHat 9.0.

thanks,
Nat
Jul 18 '05 #11
Nat Echols <ec****@uclink.berkeley.edu> writes:
Well, I can't see anything flagrantly wrong with that.

Does a debugger provide any hints?
Someone else suggested this, and this is what the backtrace from gdb
indicated:

#0 0x420744fe in _int_free () from /lib/tls/libc.so.6
#1 0x420734d6 in free () from /lib/tls/libc.so.6
#2 0x0809dc0d in _PyObject_GC_Del ()
#3 0x080ce86f in PyDict_Next ()
#4 0x080d191d in _PyModule_Clear ()

^^^^^^^^^^^^^^^
*That's* very odd.
(It goes on like this for a while. . .)


Can we see a little more?

Cheers,
mwh

--
ZAPHOD: Listen three eyes, don't try to outwierd me, I get stranger
things than you free with my breakfast cereal.
-- The Hitch-Hikers Guide to the Galaxy, Episode 7
Jul 18 '05 #12
> *That's* very odd.

Can we see a little more?


Yup:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1074141248 (LWP 5500)]
0x420744b0 in _int_free () from /lib/tls/libc.so.6
(gdb) bt
#0 0x420744b0 in _int_free () from /lib/tls/libc.so.6
#1 0x420734d6 in free () from /lib/tls/libc.so.6
#2 0x08055a1f in _PyObject_Del ()
#3 0x080593c5 in PyString_AsEncodedString ()
#4 0x0807db19 in _PyEval_SliceIndex ()
#5 0x080c41c9 in PyFunction_SetClosure ()
#6 0x080cdf49 in PyDict_New ()
#7 0x080ce35e in PyDict_SetItem ()
#8 0x080d1799 in _PyModule_Clear ()
#9 0x0808e81a in PyImport_Cleanup ()
#10 0x08096114 in Py_Finalize ()
#11 0x080539bf in Py_Main ()
#12 0x08053469 in main ()
#13 0x42015574 in __libc_start_main () from /lib/tls/libc.so.6

Now, if I run it interactively instead (still within gdb), I get this
backtrace when it segfaults after I try to exit Python:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1074141248 (LWP 5502)]
0x420744b0 in _int_free () from /lib/tls/libc.so.6
(gdb) bt
#0 0x420744b0 in _int_free () from /lib/tls/libc.so.6
#1 0x420734d6 in free () from /lib/tls/libc.so.6
#2 0x08055a1f in _PyObject_Del ()
#3 0x080593c5 in PyString_AsEncodedString ()
#4 0x080ce4d9 in PyDict_DelItem ()
#5 0x0805f1f2 in PyString_Fini ()
#6 0x08096153 in Py_Finalize ()
#7 0x080539bf in Py_Main ()
#8 0x08053469 in main ()
#9 0x42015574 in __libc_start_main () from /lib/tls/libc.so.6

Jul 18 '05 #13

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

reply views Thread by Robert Frunzke | last post: by
reply views Thread by David Eger | last post: by
6 posts views Thread by Juho Saarikko | last post: by
8 posts views Thread by Torsten Mohr | last post: by
3 posts views Thread by Travis Berg | last post: by
4 posts views Thread by klappnase | last post: by
14 posts views Thread by Donn Ingle | last post: by
3 posts views Thread by Mitko Haralanov | last post: by
18 posts views Thread by Prasad | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by zhoujie | last post: by
reply views Thread by Marylou17 | last post: by
1 post views Thread by Marylou17 | last post: by
1 post views Thread by Marylou17 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.