Unexpected extension module behaviour

[rimmer]

I'm writing an extension module in C in which I'm passing an array of
floats from C to python. The code below illustrates a simple C
function designed to output an array of floats.

---------
extTest.c
---------
#include <stdio.h>

double *testArray(int nsamp) {

double nums[10000];
int i;
double cumdata = 0.0;

printf("%d\n", nsamp);
for (i=0; i<=nsamp; i++) {
printf("%d\n", i);
nums[i] = cumdata;
cumdata += 0.5;
printf("%f\n", nums[i]);
}
return nums;
}

Then I write a wrapper function to pass the data back and forth between
C and Python.

----------------
extTestWrapper.c
----------------

#include "/usr/include/python2.4/Python.h"
#include <stdio.h>

// external declarations
extern float *testArray(int);

// Python wrapper for the testArray function
PyObject *extTest_testArray(PyObject *self, PyObject *args) {

double *nums;
int nsamp;
int i;
PyObject *pynums;

if (!PyArg_ParseTuple(args, "i", &nsamp)) {
return NULL;
}

// call the C function
nums = testArray(nsamp);

// build a Python list object containing the array values
pynums = PyList_New(nsamp);
for (i=0; i<=nsamp; i++){
PyList_SetItem(pynums, i, PyFloat_FromDouble(nums[i]));
}
return Py_BuildValue("O", pynums);
}

// method table mapping names to wrappers
static PyMethodDef extTestMethods [] = {
{"testArray", extTest_testArray, METH_VARARGS},
{NULL, NULL}
};

//module init function
void initextTest() {
Py_InitModule("extTest", extTestMethods);
}

I then run the following setup.py script using python setup.py install
--install-lib=.

--------------------------------------------------------------------------------------------
# setup.py for extTest

from distutils.core import setup, Extension

setup(name="extTest", version="0.0.1",
ext_modules=[Extension("extTest", ["extTest.c", "extTestWrapper.c"])])
--------------------------------------------------------------------------------------------

The library builds and installs ok. When I invoke the testArray
function, it appears to work correctly (the output is as expected).

For example,

import extTest
a = extTest.testArray(5)

yields the following output:

5
0
0.000000
1
0.500000
2
1.000000
3
1.500000
4
2.000000
5
2.500000
Exception exceptions.IndexError: 'list assignment index out of range'
in 'garbage collection' ignored
Fatal Python error: unexpected exception during garbage collection
Aborted

Here is where I'm stumped. I must be doing something wrong during the
PyList_SetItem or the Py_BuildValue.

Any ideas on fixing this problem ?

Regards,

Rimmer

[John Machin]

On 25/05/2006 12:09 PM, rimmer wrote:

I'm writing an extension module in C in which I'm passing an array of
floats from C to python. The code below illustrates a simple C
function designed to output an array of floats.

---------
extTest.c
---------
#include <stdio.h>

double *testArray(int nsamp) {

double nums[10000];
int i;
double cumdata = 0.0;

printf("%d\n", nsamp);
for (i=0; i<=nsamp; i++) {
printf("%d\n", i);
nums[i] = cumdata;
cumdata += 0.5;
printf("%f\n", nums[i]);
}
return nums;
Your problem is right here. The array nums is local to the function.
You are returning a pointer to memory whose contents are utterly useless
once you return from the function. Depending on the architecture and the
compiler, the pointer may point outside the stack, maybe causing the
hardware to take exception when the pointer is dereferenced, or it may
be inside the stack, in which case the next few function calls are
liable to trash the contents.
}

Then I write a wrapper function to pass the data back and forth between
C and Python.
Before you do that, test it with a simple C main()!!!

[snip]
Here is where I'm stumped. I must be doing something wrong during the
PyList_SetItem or the Py_BuildValue.
Yes, you may have problems there too, but I didn't bother reading that
far :-)

Any ideas on fixing this problem ?

Regards,

Rimmer

[John Machin]

On 25/05/2006 12:09 PM, rimmer wrote:

I'm writing an extension module in C in which I'm passing an array of
floats from C to python. The code below illustrates a simple C
function designed to output an array of floats.
[snip]
Couldn't restrain myself from reading further :-)

Then I write a wrapper function to pass the data back and forth between
C and Python.

----------------
extTestWrapper.c
----------------

#include "/usr/include/python2.4/Python.h"
#include <stdio.h>

// external declarations
extern float *testArray(int);
Um, shouldn't that be "double", not "float"?

// Python wrapper for the testArray function
PyObject *extTest_testArray(PyObject *self, PyObject *args) {

double *nums;
int nsamp;
int i;
PyObject *pynums;

if (!PyArg_ParseTuple(args, "i", &nsamp)) {
return NULL;
}

// call the C function
nums = testArray(nsamp);

// build a Python list object containing the array values
pynums = PyList_New(nsamp);
Test for errors!
for (i=0; i<=nsamp; i++){
Um, shouldn't that be "<", not "<="???
Note, you have the same problem in the C function.
"nsamp" is presumed in the absence of any docs to mean "number of
samples". A caller passing in 5 expects to get 5 values, NOT 6.
But you are calling PyList_New with 5.
PyList_SetItem(pynums, i, PyFloat_FromDouble(nums[i]));
Given you are trying to stuff one extra item into the list, you should
definitely test for errors here!!!

I suggest that you change this incrementally. First, just change the
above line to test for errors. Then run it again so you can see what
happens. Second, fix the other problems.
}
return Py_BuildValue("O", pynums);

Rather unnecessary; you can just return pynums.

[read before snipping :-)]
HTH,
John

[rimmer]

Thanks for the help John.

Indeed, changing <= to < has it licked.