473,718 Members | 1,947 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Wrapping a C library in Python

I've got a C library with about 50 calls in it that I want to wrap in
Python. I know I could use some tool like SWIG, but that will give me a
too-literal translation; I want to make some modifications along the way
to make the interface more Pythonic.

For example, all of these functions return an error code (typically just
errno passed along, but not always). They all accept as one of their
arguments a pointer to someplace to store their result. I want to
change all that to returning the result directly and throwing exceptions.

I also want to mutate some of the return types. A common way these
functions return a set of values is a pair of arrays of strings, forming
key-value pairs. In Python, it would make sense to return this as a
dictionary.

I know what I'm describing is kind of vague, but are there tools around
which might help automate (at least in part) this translation process?
Jul 18 '05 #1
13 3856

Take a close look at Pyrex at

<http://nz.cosc.canterb ury.ac.nz/~greg/python/Pyrex/>
/Jean Brouwers
ProphICy Semiconductor, Inc.
In article <ro************ ***********@rea der1.panix.com> , Roy Smith
<ro*@panix.co m> wrote:
I've got a C library with about 50 calls in it that I want to wrap in
Python. I know I could use some tool like SWIG, but that will give me a
too-literal translation; I want to make some modifications along the way
to make the interface more Pythonic.

For example, all of these functions return an error code (typically just
errno passed along, but not always). They all accept as one of their
arguments a pointer to someplace to store their result. I want to
change all that to returning the result directly and throwing exceptions.

I also want to mutate some of the return types. A common way these
functions return a set of values is a pair of arrays of strings, forming
key-value pairs. In Python, it would make sense to return this as a
dictionary.

I know what I'm describing is kind of vague, but are there tools around
which might help automate (at least in part) this translation process?

Jul 18 '05 #2
[Roy Smith]
I've got a C library with about 50 calls in it that I want to wrap in
Python. I know I could use some tool like SWIG, but that will give me a
too-literal translation; I want to make some modifications along the way
to make the interface more Pythonic.


I used Pyrex with both pleasure and success for wrapping C libraries
while giving the API a more Pythonic flavour. I found Pyrex to be a
wonderful tool for easily doing such things.

--
François Pinard http://pinard.progiciels-bpi.ca
Jul 18 '05 #3
Roy Smith wrote:
I've got a C library with about 50 calls in it that I want to wrap in
Python.


I'd recommend ctypes (http://starship.python.net/crew/theller/ctypes/).
It is very easy to use and multi-platform.

I have a small project that provides a DB API 2.0 interface to the ODBTP
(http://odbtp.sf.net) library. See http://benjiyork.com/odbtp.html
(beware programmer-web-design ahead) for the code (LGPL). It might
serve as an example to get you on your way.

BTW, if anyone is interested, I hope to have a much improved version of
the wrapper ready in a few weeks to be included in the official ODBTP
distribution.
--
Benji York
be***@benjiyork .com

Jul 18 '05 #4
Roy Smith <ro*@panix.co m> wrote in message news:<ro******* *************** *@reader1.panix .com>...
I've got a C library with about 50 calls in it that I want to wrap in
Python. I know I could use some tool like SWIG, but that will give me a
too-literal translation; I want to make some modifications along the way
to make the interface more Pythonic.

For example, all of these functions return an error code (typically just
errno passed along, but not always). They all accept as one of their
arguments a pointer to someplace to store their result. I want to
change all that to returning the result directly and throwing exceptions.

I also want to mutate some of the return types. A common way these
functions return a set of values is a pair of arrays of strings, forming
key-value pairs. In Python, it would make sense to return this as a
dictionary.

I know what I'm describing is kind of vague, but are there tools around
which might help automate (at least in part) this translation process?


While SWIG is a strong tool for wrapping C code into many different
langauges, there are definitely better tools out there for producing
python modules from C code.

I recommend both boost::python
(http://www.boost.org/libs/python/doc/index.html) and ctypes
(http://starship.python.net/crew/theller/ctypes/), but for very
different reasons.

If wrapped code speed and production of a binary is a goal in creating
your python extensions, use boost::python. The price you pay for
creating an fast binary python extension is coding your translation
from python in C/C++ in C/C++ (I suppose you could get around this by
creating a boost::python module, and then wrapping it with a python
module to do the type translation). And in boost::python, C++ to
python exception translation is supported.

If speed of development is your main goal, then I'd use ctypes.
ctypes will dynamically load the library in python code. At the
python code level, you should then do the translation to python types.
No C/C++ coding required.

Hope this helps!

Michael Loritsch
Jul 18 '05 #5
>>>>> "Michael" == Michael Loritsch <lo******@gmail .com> writes:

Roy> For example, all of these functions return an error code
Roy> (typically just errno passed along, but not always). They
Roy> all accept as one of their arguments a pointer to someplace
Roy> to store their result. I want to change all that to
Roy> returning the result directly and throwing exceptions.

Roy> I also want to mutate some of the return types. A common way
Roy> these functions return a set of values is a pair of arrays of
Roy> strings, forming key-value pairs. In Python, it would make
Roy> sense to return this as a dictionary.

SWIG can do all this - see the section on typemaps and call policies
in the SWIG manual.

For example, it is easy to tell SWIG that pointers are used for output

void somefunc(double *OUTPUT, double *OUTPUT)

will be called from python like

x, y = o.somefunc()

And you need to do no more than add the one declaration line. INPUT,
OUTPUT and INOUT are special tokens that SWIG recognizes and applies
translation rules too. You can define your own such tokens to do more
complicated things (like turning a double *array into a python list,
etc)

Michael> I recommend both boost::python
Michael> (http://www.boost.org/libs/python/doc/index.html) and
Michael> ctypes (http://starship.python.net/crew/theller/ctypes/),
Michael> but for very different reasons.

Michael> If wrapped code speed and production of a binary is a
Michael> goal in creating your python extensions, use
Michael> boost::python. The price you pay for creating an fast
Michael> binary python extension is coding your translation from
Michael> python in C/C++ in C/C++ (I suppose you could get around
Michael> this by creating a boost::python module, and then
Michael> wrapping it with a python module to do the type
Michael> translation). And in boost::python, C++ to python
Michael> exception translation is supported.

I don't fully agree with this. I've been working on a wrapper for
antigrain, a C++ library that makes heavy use of templates. I started
off using pyste (a boost::python generator) and boost. It worked
reasonably well - pyste is not being actively maintained right but you
can usually work around the limitations by writing boost code where
you need to. I was reasonably happy, until I saw my *.so files
ballooning. After wrapping a small fraction of the library, and
having instantiated only a few of the many templates I ultimately
wanted, my extension files were at 20MB, which is *much larger* than
the agg library or moderately sophisticated applications built around
it. And I still had *a lot* left to expose!

I started over in SWIG. First, SWIG has excellent support for C++ and
templates, and in many cases could auto-wrap and entire header with

%include "someheader .h"

Second, by the time I had the SWIG wrapping to a comparable point that
the boost wrapping was at when I started over, I had only a 500K
extension module. Same functionality, 40x smaller. Since ultimately
I may want to distribute my software in compiled form (eg a windows
installer) this is an important difference. Third, the compile times
were much shorter in SWIG. Fourth, SWIG produces c and cxx files as
its output, which you can distribute with your app (user doesn't need
to have SWIG to compile). This is not true for boost.

In a nutshell, for wrapping a large C++ library (not the original
poster's question, I know), I found SWIG more suitable for the reasons
above. I don't want to slam boost - I think it is an awesome package
-- but you should be aware of these potential problems. The ease of
wrapping agg was comparable in boost and SWIG.

Where boost (and pycxx) really shines above SWIG is when you want to
write functions and methods yourself that interact with python objects
-- for example if you were writing a python extension largely from
scratch rather than wrapping an existing library. The ability to use
friendly boost C++ classes like dict and list that manage memory for
you and have a pythonic feel is great.

JDH
Jul 18 '05 #6
John Hunter <jd******@ace.b sd.uchicago.edu > writes:
>> "Michael" == Michael Loritsch <lo******@gmail .com> writes:

Michael> I recommend both boost::python
Michael> (http://www.boost.org/libs/python/doc/index.html) and
Michael> ctypes (http://starship.python.net/crew/theller/ctypes/),
Michael> but for very different reasons.

Michael> If wrapped code speed and production of a binary is a
Michael> goal in creating your python extensions, use
Michael> boost::python. The price you pay for creating an fast
Michael> binary python extension is coding your translation from
Michael> python in C/C++ in C/C++ (I suppose you could get around
Michael> this by creating a boost::python module, and then
Michael> wrapping it with a python module to do the type
Michael> translation). And in boost::python, C++ to python
Michael> exception translation is supported.

I don't fully agree with this. I've been working on a wrapper for
antigrain, a C++ library that makes heavy use of templates. I started
off using pyste (a boost::python generator) and boost. It worked
reasonably well - pyste is not being actively maintained right but you
can usually work around the limitations by writing boost code where
you need to. I was reasonably happy, until I saw my *.so files
ballooning. After wrapping a small fraction of the library, and
having instantiated only a few of the many templates I ultimately
wanted, my extension files were at 20MB, which is *much larger* than
the agg library or moderately sophisticated applications built around
it. And I still had *a lot* left to expose!


Did you strip the extension modules (run 'strip' on the .so file)? I
know, there's nothing in distutils that will do that automatically. I
just did this on an extension module of mine using boost::python.
Before stripping, it was about 1.3 MB, afterwards, 50 kB.

The problem is the symbol table is kept by default, and it seems with
GNU C++ (at least) that can get *huge* when templates are involved.

Compile times are pain still.

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)phy sics(dot)mcmast er(dot)ca
Jul 18 '05 #7
lo******@gmail. com (Michael Loritsch) wrote:
If speed of development is your main goal, then I'd use ctypes.
ctypes will dynamically load the library in python code. At the
python code level, you should then do the translation to python types.
No C/C++ coding required.


OK, I decided to give ctypes a try. After a few false steps (mostly
related to sorting out LD_LIBRARY_PATH issues), I got it to work for
simple functions. Once you get your head around how it works, it's
pretty neat.

The problem is, I'm at a loss what to do for a slightly more complex
case. The API I'm working with requires that you create a dm_handle
which you then pass into all the other calls to establish a context.
You start with (approximately) :

/*
* Create a handle. The caller must have allocated memory for
* the object pointed to by dmh.
*/
create_dm_handl e (struct dm_handle *dmh);

I don't see how I can do the required memory allocation. Calling
malloc() directly seems kind of scary. Even if I could do that, doing
sizeof (struct dm_handle) won't work in the Python environment.

Am I missing something obvious here?
Jul 18 '05 #8
Roy Smith wrote:
The problem is, I'm at a loss what to do for a slightly more complex
case.


The ctypes tutorial talks about this issue:
http://starship.python.net/crew/thel.../tutorial.html.
Especially the section s "Passing pointers" and "Structures and Unions".

I've also used the built-in struct module for simple cases.
--
Benji York
be***@benjiyork .com

Jul 18 '05 #9
Roy Smith <ro*@panix.co m> writes:
lo******@gmail. com (Michael Loritsch) wrote:
If speed of development is your main goal, then I'd use ctypes.
ctypes will dynamically load the library in python code. At the
python code level, you should then do the translation to python types.
No C/C++ coding required.


OK, I decided to give ctypes a try. After a few false steps (mostly
related to sorting out LD_LIBRARY_PATH issues), I got it to work for
simple functions. Once you get your head around how it works, it's
pretty neat.

The problem is, I'm at a loss what to do for a slightly more complex
case. The API I'm working with requires that you create a dm_handle
which you then pass into all the other calls to establish a context.
You start with (approximately) :

/*
* Create a handle. The caller must have allocated memory for
* the object pointed to by dmh.
*/
create_dm_handl e (struct dm_handle *dmh);

I don't see how I can do the required memory allocation. Calling
malloc() directly seems kind of scary. Even if I could do that, doing
sizeof (struct dm_handle) won't work in the Python environment.

Am I missing something obvious here?


It's simple.
First, you define the structure:

import ctypes
class dm_handle(ctype s.Structure):
_fields_ = [....] # whatever is is

Then, create an instance (this will allocate memory internally)

my_handle = cm_handle()

and finally call the function, pssing it a pointer to the structure:

mydll.create_dm _handle(ctypes. byref(my_handle ))

Thomas
Jul 18 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
2423
by: USCode | last post by:
Does Python have facilities for wrapping up the interpreter, any necessary modules, GUI library, python scripts, image files, etc. up into a single executable like exists for Tcl/Tk? e.g. http://freewrap.sourceforge.net/ http://www.equi4.com/starkit.html I've seen py2exe but it's not quite the same thing and doesn't appear to be as comprehensive as the above 2 are for Tcl/Tk. Thanks!
0
1337
by: Terry Hancock | last post by:
Hi all, Suppose you have found a really nice module that does what you want (or will do, it still being in development), and you don't want to waste a lot of duplicated effort, but this module is in Perl rather than Python (and yes, the author is not open to conversion -- already tried that ;-) ). If it were written in C, I'd have a couple of answers, such
1
1482
by: TPJ | last post by:
I'm trying to get a wrapper for my C code in order to be able to use it as a module in Python. I'm doing it as follows: C code (file_c.c): ------------------------------- #include <stdio.h> void hello( int size ) { printf("Hello! %d\n", size);
2
7574
by: Jacob Cohen | last post by:
Under VC7.1, I am trying to wrap a native-C++ DLL that contains C++ objects in a Managed-C++ class library for use in a C# project. I created and compiled the native DLL under VC7.1 as a Win32 DLL C++ (unmanaged) project. I found the decorated names of the symbols I wanted to export and created a .def file, so that it generated an import library. I took this import library and had my Managed-C++ class library link against it. The...
10
2376
by: Tom the Canuck | last post by:
What would be the best way to proceed? Should I make a pure virtual class and then derive from that? I want the base class to have functions defined so that I don't have to do the work all over again in the derived class. Is this not what objects are for? Please illuminate on how to do this properly. I don't want code, just a pointer on how to do the job right. Any comments or links will help.
3
2882
by: gabriel.becedillas | last post by:
Hi, I'm having problems wrapping a hierarchy of classes, actually having problems wrapping the base class. I don't need to use the WrapClass mechanism since I don't want to override classes in Python. My code boils down to: class Base { public: virtual ~Base()
0
214
by: Paul Anton Letnes | last post by:
>
8
2279
by: Anish Chapagain | last post by:
Hi!! I tried wrapping a simple C code suing SWIG to Python, but am having problem, my .c file is, Step 1: example.c -------------- double val=3.0; int fact(int n)
5
4305
by: Charlie | last post by:
Hi All, I am new to using swig/C++/python. I got some problem with function pointers. I posted in swig-user, but got no response. So I forwarded it here. You help is greatly appreciated. Thanks! ---------- Forwarded message ----------
0
8827
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9352
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
9118
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
7985
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
5971
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4481
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4740
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3180
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
2122
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.