473,545 Members | 2,639 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

exposing C array to python namespace: NumPy and array module.

Dear list,

I am writing a Python extension module that needs a way to expose pieces
of a big C array to python. Currently, I am using NumPy like the following:

PyObject* res = PyArray_FromDim sAndData(1, int*dim, PyArray_DOUBLE,
char*buf);

Users will get a Numeric Array object and can change its values (and
actually change the underlying C array).

This works fine. However, when I deliver my module, I find NumPy is
unnecessarily large for this simple task. As a matter of fact, I had to
build from source NumPy, ATLAS etc on Solaris, Linux, Mac.... and if a
user would like to use my module, he has to do the same thing!

Python's array module is built-in, easy to use, but *without* a
FromLenAndData function! Even the buffer interface provides only 'get
buffer' but no 'set buffer' functions. Could anyone tell me how I can
create an array object from existing data? Some vague ideas might be
used: 1. PyCObject (I do not really understand the manual), 2. copy and
modify arraymodule.c to my project (doable at all? License issue?) 3.
Create an array object and hack it. (no api to do this.)

I would strongly suggest an arraymodule.h with Array_FromLenAn dData.

Many thanks in advance.
Bo
Jul 18 '05 #1
8 3844
On Sat, 2005-01-01 at 08:18, Bo Peng wrote:
Python's array module is built-in, easy to use, but *without* a
FromLenAndData function! Even the buffer interface provides only 'get
buffer' but no 'set buffer' functions. Could anyone tell me how I can
create an array object from existing data?


Python has no array objects in the core language, only lists. The
distinction is important when discussing numarray etc, because Python
lists and NumPy etc arrays are very different.

While you can build a Python list from a subsection of your C array,
changes made in Python won't be pushed back to the C array it was
created from. If this is OK, you can probably build the list using just
a for loop - I'm not sure if there are any more efficient methods for
variable length lists.

If the Python user needs to be able to change the underlying array, I'd
probably drop the use of the built-in list class entirely and write my
own class that looks like a list (and smells like a list, and tastes
like a list - lucky we didn't step in it!). It can be pretty simple,
providing as few of the list protocol methods as:

__getitem__ (a PyList_GetItem equivalent)
__setitem__ (a PyList_SetItem equivalent)

and preferably:

__len__
__iter__

or as much of the list protocol as documented on the Python/C API page
as you need.

I'd probably implement the class in Python, and have my extension module
provide a couple of simple functions to the underlying C array. These
could be considered private to your list class. That'd make writing
things like the __iter__ method much nicer, while still letting you
implement __len__, __getitem__, __setitem__, etc in C. For example, I
might write:

class CArray(object):
def __init__(self, ...):
...

def __getitem__(sel f, index):
_carray_getitem (self, index)

def __len__(self):
_carray_len(sel f, index)

def __iter__(self):
# build and return an interator using Python
...
If you want to write part of your extension module in Python and part in
C, there are two main ways to do it. The usual way is to write a
'wrapper' in Python that imports the C parts, wraps them where necessary
or just pushes them into its own namespace, etc.

The less common way is to import __builtins__ and __main__ into your C
extension module's namespace then PyRun_String() python code in it to
set things up. I find this approach MUCH more useful when embedding
Python in an app and I only want to write small bits of my module in
Python.
The other alternative is to code your class entirely in C, implementing
all the __methods__ as C functions. Unattractive as far as I'm
concerned, but then I find constructing classes using Python's C API
irritating and less clear than it could be.

Here's the code -- hideously reformatted to avoid wrapping in the mail -
in my initmodule() function that I use to set up the module so that
Python code can execute in its namespace. You can ignore the
const_cast<> stuff, chances are your compiler will ignore the const
problems.

----
// 'd' is the dictionary of the extension module, as obtained
// with PyModule_GetDic t(module)

PyObject* builtinModule = PyImport_Import ModuleEx(
const_cast<char *>("__builtin__ "),
d, d, Py_BuildValue(c onst_cast<char* >("[]"))
);
if (builtinModule == NULL)
{
// Error handling will not be shown; it'll depend on your module anyway.
}
PyDict_SetItemS tring(d, const_cast<char *>("__builtin__ "),
builtinModule);

PyObject* exceptionsModul e = PyImport_Import ModuleEx(
const_cast<char *>("exceptions" ), d, d,
Py_BuildValue(c onst_cast<char* >("[]"))
);
if (exceptionsModu le == NULL) {}
PyDict_SetItemS tring(d, const_cast<char *>("exceptions" ),
exceptionsModul e);

// We can now run Python code in the module's namespace. For
// example (untested), as my real examples wouldn't do you any
// good, they're too bound to the internal API of my module:

QString python_code = "";
python_code += "def sample_function ():\n";
python_code += " print \"See, it worked\"\n";
// My app sets sysdefaultencod ing to utf-8, hence:
char* python_code_cst ring = python_code.utf 8();

// Note that we pass our module dictionary as both
// locals and globals. This makes the code effectively
// run "in" the extension module, as if it was being
// run during loading of a Python module after an
// 'import' statement.
PyObject* result = PyRun_String(py thon_code_cstri ng,
Py_file_input,
d,d);
if (result == NULL)
{
qDebug("Python code to declare sample_function failed!");
PyErr_Print(); // also clears the exception
}
// Because 'result' may be NULL, not a PyObject*, we must call PyXDECREF
not Py_DECREF
Py_XDECREF(resu lt);
--

Ugh - I'd forgotten how ugly C code limited to 80 cols and without
syntax highlighting really was. Especially when the reformatting is done
as badly as I've done it. I hope you can make some sense out of that,
anyway. Note that once the setup is done you can run as many python code
snippets as you want, for declaring variables, functions, classes, etc.

In my case, its easier to execute snippets as shown above than it is to
worry about the module search path and wrapping things using a Python
module. If you're doing substantial amounts of Python coding for your
module, you'll almost certainly be better off writing a Python module
that uses your C module internally (see PIL for a good example of this).

--
Craig Ringer

Jul 18 '05 #2
Craig Ringer wrote:
On Sat, 2005-01-01 at 08:18, Bo Peng wrote:

Python's array module is built-in, easy to use, but *without* a
FromLenAndDat a function! Even the buffer interface provides only 'get
buffer' but no 'set buffer' functions. Could anyone tell me how I can
create an array object from existing data?

Python has no array objects in the core language, only lists. The
distinction is important when discussing numarray etc, because Python
lists and NumPy etc arrays are very different.


Thank you very much for the detailed reply!

Sorry if I was not clear enough. I was talking about the differece
between python array module
(http://docs.python.org/lib/module-array.html, Modules/arraymodule.c in
the source tree) and NumPy array. They both use C-style memory block
arrangement for efficient memory access. While NumPy has both, the array
module is designed to be used purely in Python so there is no header
file and no function to build an array from a pointer.

One of the methods you suggested (creating a new type) already
implemented in arraymodule.c. I am not sure if it is appropriate to add
the file into my project and add a 'CreateFromLenA ndBuf' function.

Bo
Jul 18 '05 #3
Craig Ringer wrote:
On Sat, 2005-01-01 at 08:18, Bo Peng wrote:

Python's array module is built-in, easy to use, but *without* a
FromLenAndDat a function! Even the buffer interface provides only 'get
buffer' but no 'set buffer' functions. Could anyone tell me how I can
create an array object from existing data?

Python has no array objects in the core language, only lists. The
distinction is important when discussing numarray etc, because Python
lists and NumPy etc arrays are very different.


Thank you very much for the detailed reply!

Sorry if I was not clear enough. I was talking about the differece
between python array module
(http://docs.python.org/lib/module-array.html, Modules/arraymodule.c in
the source tree) and NumPy array. They both use C-style memory block
arrangement for efficient memory access. While NumPy has both, the array
module is designed to be used purely in Python so there is no header
file and no function to build an array from a pointer.

One of the methods you suggested (creating a new type) already
implemented in arraymodule.c. I am not sure if it is appropriate to add
the file into my project and add a 'CreateFromLenA ndBuf' function.

Bo
Jul 18 '05 #4
On Sat, 2005-01-01 at 10:27 -0600, Bo Peng wrote:
Sorry if I was not clear enough. I was talking about the differece
between python array module
(http://docs.python.org/lib/module-array.html, Modules/arraymodule.c in
the source tree) and NumPy array. They both use C-style memory block
arrangement for efficient memory access. While NumPy has both, the array
module is designed to be used purely in Python so there is no header
file and no function to build an array from a pointer.
Thanks for clarifying that - I had misunderstood your reference to
arraymodule.c .

I guess the core language doesn't have an array type, but as there's a
standard lib module that does (I'd forgotten it was there), it hardly
matters.

It would seem sensible to extend that module with a C API for mapping an
existing array. That would be a rather handy thing to have in the
standard library.
One of the methods you suggested (creating a new type) already
implemented in arraymodule.c. I am not sure if it is appropriate to add
the file into my project and add a 'CreateFromLenA ndBuf' function.


That sounds like a reasonable approach to me, but I'm hardly an expert.
The code's license permits you to do so, and it's hardly worth repeating
the work if you don't have to.

--
Craig Ringer

Jul 18 '05 #5
Bo Peng wrote:
Dear list,

I am writing a Python extension module that needs a way to expose pieces
of a big C array to python. Currently, I am using NumPy like the following:

PyObject* res = PyArray_FromDim sAndData(1, int*dim, PyArray_DOUBLE,
char*buf);

Users will get a Numeric Array object and can change its values (and
actually change the underlying C array).

This works fine. However, when I deliver my module, I find NumPy is
unnecessarily large for this simple task. As a matter of fact, I had to
build from source NumPy, ATLAS etc on Solaris, Linux, Mac.... and if a
user would like to use my module, he has to do the same thing!

Python's array module is built-in, easy to use, but *without* a
FromLenAndData function! Even the buffer interface provides only 'get
buffer' but no 'set buffer' functions. Could anyone tell me how I can
create an array object from existing data? Some vague ideas might be
used: 1. PyCObject (I do not really understand the manual), 2. copy and
modify arraymodule.c to my project (doable at all? License issue?) 3.
Create an array object and hack it. (no api to do this.)

I would strongly suggest an arraymodule.h with Array_FromLenAn dData.

Many thanks in advance.
Bo

I don't know how much this will help but when I am faced with a problem
like this, I use Pyrex and look at the generated code. All you need to
do in Pyrex is import the array module and create your array like you
would in Python. To get the data into the array you will need to use
the buffer interface and fill it in from your C code.
Jul 18 '05 #6
Bo Peng wrote:
Dear list,
I am writing a Python extension module that needs a way to expose pieces
of a big C array to python. Currently, I [use] NumPy.... Users ... actually
change the underlying C array.

Python's array module is built-in, easy to use, but *without* a
FromLenAndData function!

Python's array module is not built to do this well. It can re-size the
array, delete elements inside the array, and other things that don't
work very well with C-managed data. I wrote "blocks and views" to
overcome this problem. A "View" of data can be pointed at data, and
the "view" behaves much like a Python array (except that you cannot
affect the array's size). You can even take "slices" of the view,
which will produce a new view referring to the same base memory. There
are two kinds of views available, read-only views and writable views.

Have a look at:
http://members.dsl-only.net/~daniels/Block.html

to see if it addresses your problem. It is MIT-licensed (give credit,
but feel free to use). Let me know if it works OK, could use a tweak,
or is completely useless. I'll be more than happy to respond to
questions.

--Scott David Daniels
Sc***********@A cm.Org
Jul 18 '05 #7
Scott David Daniels wrote:
Python's array module is not built to do this well. It can re-size the
array, delete elements inside the array, and other things that don't
work very well with C-managed data. I wrote "blocks and views" to
overcome this problem.


As always the case, this problem have been encountered and solved! Your
code is exactly what I needed. However, I was too impatient to wait for
your reply. :-) I have realized the problems of arraymodule.c and added
a ob_ownmem member in the object structure. Existing operations have
been modified (mostly unchanged) according to this flag and the new
module works well.

Thank everyone for the help. especially in thie new year's day.

Bo
Jul 18 '05 #8
Bo Peng wrote:
Scott David Daniels wrote:
I wrote "blocks and views" to overcome this problem.

I was too impatient to wait for your reply. :-)

I call 21-hour turnaround over New Year's Eve pretty good. Clearly I
will never be quick enough for you ;-). Since I presented this at
the Vancouver Python Workshop last August, I'll claim a negative
five months response time (possibly a personal best).

--Scott David Daniels
Sc***********@A cm.Org
Jul 18 '05 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
1972
by: youngdubliner | last post by:
I'm having a problem ........ I've stripped all my code to help isolate the problem. Its seems to be with importing numarray when python is embedded in C. I have a simple C program it Opens Python imports a script and then Closes Python like so .......
2
3264
by: Pepijn Kenter | last post by:
Hi all. I'm new to python and want to use it for an assignment. I have succesfully implemented a galois field class including the _repr__, __mul__, __div__, __add__ and __sub__ methods. Basically a galois field is an integer modulo a prime number. Now I want to make a matrix of these galois field objects using the numpy library....
5
7310
by: Krish | last post by:
Hello People I hope I am On Topic. Anyways, here is my problem. Any insights would be really appreciated. I have wrapped a C IO module using SWIG -> Python Module. Suppose the name of the module is "imageio" and the reader function from the file is image_read() which returns an object ( "filled" C structure) with a lot of members.
6
3810
by: Summercoolness | last post by:
i used C too much and haven't used Python for a while... like in C, if we want an array of array of float, we use float a; now in Python, seems like we have to do something like a = ] * 200
2
2265
by: goetzie | last post by:
I am using Python 2.4.1 and Numeric 23.8 and running on Windows XP. I am passing a Numeric array of strings (objects) to a C Extension module using the following python code: import Numeric import TestDLL # my C Extension Module a = Numeric.array(, 'O' ) print 'a =', a print 'type a =', type(a)
3
3268
by: jefishman | last post by:
I have a Python (2.3.x) interpreter running embedded in a C++ application on a host machine. I would like to run a specific package on that host machine (numpy). I have managed to compile (cross-compile) the library, so that I have the python modules and the compiled .so files. I am relatively sure that these would import normally in a...
2
5157
by: Holger | last post by:
Dear all, I need to do a FFT on an array of 20k real values. Origin of the sampled data is a sinus wave with light harmonics. The goal is an frequency spectrum with the magnitudes of the first 50. harmonics. I addressed python like: test_arr = src_data_dict
2
1681
by: Davy | last post by:
Hi all, I have Python 2.4 and 2.5 in my PC. And PythonWin is installed as IDE. When I tried to use site-packages "Numpy", I installed the both version (i.e. for 2.4 and 2.5). Python 2.4 and Numpy for it work together well. But when I type "from numpy import *" in Python 2.5. It throw out an
3
5943
by: Sean Davis | last post by:
I have a set of numpy arrays which I would like to save to a gzip file. Here is an example without gzip: b=numpy.ones(1000000,dtype=numpy.uint8) a=numpy.zeros(1000000,dtype=numpy.uint8) fd = file('test.dat','wb') a.tofile(fd) b.tofile(fd) fd.close()
0
7425
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7682
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
7935
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7449
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
7780
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
1
5351
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
3479
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
1
1037
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
734
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.