473,739 Members | 6,651 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Pyrex list/array

I'm trying to move a function into pyrex for speed. The python side
needs to pass a list to the pyrex function. Do I need to convert to
array or something so pyrex can generate tight code? I'm not clear how
to do this.

Jun 3 '06 #1
7 2047
On 4/06/2006 4:56 AM, Jim Lewis wrote:
I'm trying to move a function into pyrex for speed.
You probably didn't expect the Inquisition; nobody does. But here it is,
nice red uniforms and all:

1. What is your speed requirement and how far short of that are you at
the moment?
2. Are you sure there is no Python or third-party module that does what
you want?
3. Is your algorithm the best possible?
4. Is your Python implementation of that algorithm the best possible?
Have you exposed it to the critical gaze of the speed-freaks in this
newsgroup?
5. Does your architecture support psyco? If so, have you tried that and
what were the results?
The python side
needs to pass a list to the pyrex function. Do I need to convert to
array or something so pyrex can generate tight code? I'm not clear how
to do this.


The question might be better asked on the Pyrex mailing list.

You don't need to convert a list to a C array, and it may not even be
possible, depending on what type(s) of data you have in the list.

Almost any Python code is also valid Pyrex code. For a start, just
compile your function with Pyrex and compare the speed. What you do next
is going to depend very much on what operations you are performing on
the list and the objects it contains. Watch out for Python built-ins
like range, xrange, ord, chr, abs, bool, int(a_number), float(a_number) ,
divmod, max/min (two_numeric_ar gs). In almost all cases you get cheap
wins by replacing use of these by simple C-like code -- provided of
course you are absolutely sure you know what types you are dealing with.

HTH,
John
Jun 4 '06 #2
Thanks for your comments.
You probably didn't expect the Inquisition...
Correct ;-)
1. What is your speed requirement and how far short of that are you at the moment?
~10 times faster.
2. Are you sure there is no Python or third-party module that does what you want?
Yes.
3. Is your algorithm the best possible?
I think so although of course one can never be certain.
4. Is your Python implementation of that algorithm the best possible? Have you exposed it to the critical gaze of the speed-freaks in this newsgroup?
Thanks for the good suggestion but I want to try pyrex first.
5. Does your architecture support psyco? If so, have you tried that and what were the results?
Already using psyco.
The question might be better asked on the Pyrex mailing list.
I did not find it - where is it?
Almost any Python code is also valid Pyrex code. For a start, just compile your function with Pyrex and compare the speed.
It's slower.
What you do next is going to depend very much on what operations you are performing on the list and the objects it contains.


Simple list of ints. Comparing sections of lists between each other.

Jun 4 '06 #3
5. Does your architecture support psyco? If so, have you tried that
and what were the results?


Jim> Already using psyco.

Is it substantially faster with psyco than without? If psyco is performing
its magic on the critical section of code already, you are going to lose
that when switching to Pyrex.

Skip
Jun 4 '06 #4
> Is it substantially faster with psyco than without? If psyco is performing
its magic on the critical section of code already, you are going to lose
that when switching to Pyrex.


Yes but from what I read Pyrex can be a lot faster than psyco under the
right circumstances.

Jun 4 '06 #5
Is it substantially faster with psyco than without? If psyco is
performing its magic on the critical section of code already, you are
going to lose that when switching to Pyrex.


Jim> Yes but from what I read Pyrex can be a lot faster than psyco under
Jim> the right circumstances.

I'm sure that's true. That also means under the wrong circumstances it
might not. ;-) Can you post the code that's running too slowly?

Also, note that psyco learned some new tricks at the recent NeedForSpeed
sprint. You might want to check out the latest version from Subversion and
give it a whirl.

Skip
Jun 4 '06 #6
On 4/06/2006 7:59 PM, Jim Lewis wrote:
Thanks for your comments.
You probably didn't expect the Inquisition...
Correct ;-)


Nobody does :-)
The question might be better asked on the Pyrex mailing list.
I did not find it - where is it?


Evidently somewhere near the Hall of the Mountain King. A reference to
it is cunningly concealed in the last place one would think of finding
it: under the heading "Mailing List" on the Pyrex home page :-) Here:
http://lists.copyleft.no/mailman/listinfo/pyrex ... get in quick before
the pirate moves it again.
Almost any Python code is also valid Pyrex code. For a start, just compile your function with Pyrex and compare the speed.


It's slower.
What you do next is going to depend very much on what operations you are performing on the list and the objects it contains.


Simple list of ints. Comparing sections of lists between each other.


Do you mean alist[x:x+n] == alist[y:y+n] ?
If so, that's creating two new lists each of size n, and then comparing
those two lists. I doubt that psyco would recognize that it didn't need
to copy the two slices. The first step might be to write functions to
compare without copying, e.g.:

def py_slices_cmp_e q(py_list, start1, start2, size):
"""Return 1 if py_list[start1+size] == py_list[start2+size]
else 0"""
offset = start2 - start1
for i in xrange(start1, start1+size):
if py_list[i] != py_list[i+offset]:
return 0
return 1

See what psyco makes of that.
Then turn that into a cdef function for Pyrex.

If that's still not fast enough, then you might be in for some harder work:

Allocate memory for a C array, unpack your list into it, write
comparison functions c_slices_cmp_* that operate on your array of ints.
There should be no Python stuff in there, only C constructs. You can
even use memcmp() for the cmp_eq function.

Which brings us back to your original question "Do I need to convert to
array or something so pyrex can generate tight code?" ...
1. Some of the above may help you to determine whether you need to.
2. Without any knowledge of the size of your list or what you are doing,
we can't help you much more on whether you need to.
3. AFAICT, Pyrex doesn't do much in the way of optimisation, leaving
that up to the C compiler. Generating tight code would depend more on
you replacing appropriately-chosen Pythonisms with C-isms.

As for *how* to make your C array, something like this:

cdef extern from "Python.h":
void PyMem_Free(void *p)
void* PyMem_Malloc(in t n) except NULL
# untested -- compiles OK :-)
cdef int * int_array_from_ list(object ilist):
cdef int n, i
cdef int * arrayp
n = len(ilist)
arrayp = <int *>PyMem_Malloc( n * sizeof(int))
for i from 0 <= i < n:
arrayp[i] = ilist[i]
return &arrayp[0]

Hoping some of this helps,
John
Jun 4 '06 #7
> cunningly concealed in the last place one would think of finding it: under the heading "Mailing List" on the Pyrex home page :-)

Hmmm - maybe I should try the scroll bar occassionally ;-)
Do you mean alist[x:x+n] == alist[y:y+n] ?
OK, probably you an Skip are right - let's see if I missed something at
the Python level.

There are essentially two differences from your snip above. I am trying
to compute n and there are multiple (under 10) lists. Size of lists are
typically under 100 ints.
...See what psyco makes of that.


I'm doing a similar straightforward loop approach but it's too slow.

Jim

Jun 4 '06 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
2309
by: Gary Stephenson | last post by:
I'm getting a clean generate, compile and link from my .pyx script, but when I attempt to run the resultant .exe, I get: "The procedure entry point Py_NoneStruct could not be located in the dynamic link library python23.dll" Can anybody provide me with a clue here? I've already cut down my script to remove all references to None, Py_None, NULLs, etc, but to no avail. fwiw I'm using latest Pyrex and ActiveState build 232.
10
2038
by: Kyler Laird | last post by:
I need to submit C/C++ code for a class. (It's not a programming class. The choice of language is inertial. I think that it mostly serves to distract students from the course subject.) I'm fairly fluent with C but it hurts to think about writing in C when Python is *so* much more appropriate for these operations. I'd like to keep my sanity and satisfy the course requirements by programming in Python and converting the code to C. It...
1
1679
by: Paul Prescod | last post by:
PyCon 2004 Slides on "Extending Python with Pyrex" PDF: http://www.prescod.net/pyrex/ExtendingPythonWithPyrex.pdf PPT: http://www.prescod.net/pyrex/ExtendingPythonWithPyrex.ppt Pycon 2004 Slides on "Optimizing Python code with Pyrex" PDF: http://www.prescod.net/pyrex/OptimizingPythonWithPyrex.pdf PPT: http://www.prescod.net/pyrex/OptimizingPythonWithPyrex.ppt I hope this helps me. Please report any mistakes.
0
1202
by: Pawel Oleksik | last post by:
Hi, I have no idea, how to properly declare/use Numeric functions in pyrex. For example the code below causes segmentation fault: --in pyrex-------------- cdef extern from "Numeric/arrayobject.h":
0
2836
by: Chris Lambacher | last post by:
Hi, I have to do some data manipulation that needs to be fast. I had a generator approach (that was faster than a list approch) which was taking approximately 5 seconds to run both encode and decode. I have done a conversion to pyrex and have gotten it down to about 2 seconds to run both encode and decode. An an excerpt from the pyrex code is included below. My question is, is there a better way to allocate the memory for the...
11
1997
by: Jim Lewis | last post by:
Has anyone found a good link on exactly how to speed up code using pyrex? I found various info but the focus is usually not on code speedup.
0
992
by: Georg Grabler | last post by:
Hello everybody. I finally decided to use pyrex for my tasks wrapping and creating new python objects. Anyway, i ran into struggles. I want an array to be passed to a function, so basically i started the function as follows: def addToList (self, char *array):
1
1340
by: Hans Terlouw | last post by:
Hi, When trying to wrap C code using Pyrex, I encountered a strange problem with a piece of pure Python code. I tried to isolate the problem. The following code causes Pyrex to generate C code which gcc cannot compile: class Axis: axtype = "unknown type of axis" def __init__(self):
3
1495
by: skip | last post by:
I'm using Pyrex 0.9.5.1a. I have this simple Pyrex module: cdef class Foo: cdef public char attr def __init__(self): self.attr = 0 class Bar(Foo): def __init__(self):
0
8969
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8794
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9484
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9342
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9211
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
8218
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
4826
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3286
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2748
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.