473,549 Members | 2,627 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Pyrex list/array

I'm trying to move a function into pyrex for speed. The python side
needs to pass a list to the pyrex function. Do I need to convert to
array or something so pyrex can generate tight code? I'm not clear how
to do this.

Jun 3 '06 #1
7 2035
On 4/06/2006 4:56 AM, Jim Lewis wrote:
I'm trying to move a function into pyrex for speed.
You probably didn't expect the Inquisition; nobody does. But here it is,
nice red uniforms and all:

1. What is your speed requirement and how far short of that are you at
the moment?
2. Are you sure there is no Python or third-party module that does what
you want?
3. Is your algorithm the best possible?
4. Is your Python implementation of that algorithm the best possible?
Have you exposed it to the critical gaze of the speed-freaks in this
newsgroup?
5. Does your architecture support psyco? If so, have you tried that and
what were the results?
The python side
needs to pass a list to the pyrex function. Do I need to convert to
array or something so pyrex can generate tight code? I'm not clear how
to do this.


The question might be better asked on the Pyrex mailing list.

You don't need to convert a list to a C array, and it may not even be
possible, depending on what type(s) of data you have in the list.

Almost any Python code is also valid Pyrex code. For a start, just
compile your function with Pyrex and compare the speed. What you do next
is going to depend very much on what operations you are performing on
the list and the objects it contains. Watch out for Python built-ins
like range, xrange, ord, chr, abs, bool, int(a_number), float(a_number) ,
divmod, max/min (two_numeric_ar gs). In almost all cases you get cheap
wins by replacing use of these by simple C-like code -- provided of
course you are absolutely sure you know what types you are dealing with.

HTH,
John
Jun 4 '06 #2
Thanks for your comments.
You probably didn't expect the Inquisition...
Correct ;-)
1. What is your speed requirement and how far short of that are you at the moment?
~10 times faster.
2. Are you sure there is no Python or third-party module that does what you want?
Yes.
3. Is your algorithm the best possible?
I think so although of course one can never be certain.
4. Is your Python implementation of that algorithm the best possible? Have you exposed it to the critical gaze of the speed-freaks in this newsgroup?
Thanks for the good suggestion but I want to try pyrex first.
5. Does your architecture support psyco? If so, have you tried that and what were the results?
Already using psyco.
The question might be better asked on the Pyrex mailing list.
I did not find it - where is it?
Almost any Python code is also valid Pyrex code. For a start, just compile your function with Pyrex and compare the speed.
It's slower.
What you do next is going to depend very much on what operations you are performing on the list and the objects it contains.


Simple list of ints. Comparing sections of lists between each other.

Jun 4 '06 #3
5. Does your architecture support psyco? If so, have you tried that
and what were the results?


Jim> Already using psyco.

Is it substantially faster with psyco than without? If psyco is performing
its magic on the critical section of code already, you are going to lose
that when switching to Pyrex.

Skip
Jun 4 '06 #4
> Is it substantially faster with psyco than without? If psyco is performing
its magic on the critical section of code already, you are going to lose
that when switching to Pyrex.


Yes but from what I read Pyrex can be a lot faster than psyco under the
right circumstances.

Jun 4 '06 #5
Is it substantially faster with psyco than without? If psyco is
performing its magic on the critical section of code already, you are
going to lose that when switching to Pyrex.


Jim> Yes but from what I read Pyrex can be a lot faster than psyco under
Jim> the right circumstances.

I'm sure that's true. That also means under the wrong circumstances it
might not. ;-) Can you post the code that's running too slowly?

Also, note that psyco learned some new tricks at the recent NeedForSpeed
sprint. You might want to check out the latest version from Subversion and
give it a whirl.

Skip
Jun 4 '06 #6
On 4/06/2006 7:59 PM, Jim Lewis wrote:
Thanks for your comments.
You probably didn't expect the Inquisition...
Correct ;-)


Nobody does :-)
The question might be better asked on the Pyrex mailing list.
I did not find it - where is it?


Evidently somewhere near the Hall of the Mountain King. A reference to
it is cunningly concealed in the last place one would think of finding
it: under the heading "Mailing List" on the Pyrex home page :-) Here:
http://lists.copyleft.no/mailman/listinfo/pyrex ... get in quick before
the pirate moves it again.
Almost any Python code is also valid Pyrex code. For a start, just compile your function with Pyrex and compare the speed.


It's slower.
What you do next is going to depend very much on what operations you are performing on the list and the objects it contains.


Simple list of ints. Comparing sections of lists between each other.


Do you mean alist[x:x+n] == alist[y:y+n] ?
If so, that's creating two new lists each of size n, and then comparing
those two lists. I doubt that psyco would recognize that it didn't need
to copy the two slices. The first step might be to write functions to
compare without copying, e.g.:

def py_slices_cmp_e q(py_list, start1, start2, size):
"""Return 1 if py_list[start1+size] == py_list[start2+size]
else 0"""
offset = start2 - start1
for i in xrange(start1, start1+size):
if py_list[i] != py_list[i+offset]:
return 0
return 1

See what psyco makes of that.
Then turn that into a cdef function for Pyrex.

If that's still not fast enough, then you might be in for some harder work:

Allocate memory for a C array, unpack your list into it, write
comparison functions c_slices_cmp_* that operate on your array of ints.
There should be no Python stuff in there, only C constructs. You can
even use memcmp() for the cmp_eq function.

Which brings us back to your original question "Do I need to convert to
array or something so pyrex can generate tight code?" ...
1. Some of the above may help you to determine whether you need to.
2. Without any knowledge of the size of your list or what you are doing,
we can't help you much more on whether you need to.
3. AFAICT, Pyrex doesn't do much in the way of optimisation, leaving
that up to the C compiler. Generating tight code would depend more on
you replacing appropriately-chosen Pythonisms with C-isms.

As for *how* to make your C array, something like this:

cdef extern from "Python.h":
void PyMem_Free(void *p)
void* PyMem_Malloc(in t n) except NULL
# untested -- compiles OK :-)
cdef int * int_array_from_ list(object ilist):
cdef int n, i
cdef int * arrayp
n = len(ilist)
arrayp = <int *>PyMem_Malloc( n * sizeof(int))
for i from 0 <= i < n:
arrayp[i] = ilist[i]
return &arrayp[0]

Hoping some of this helps,
John
Jun 4 '06 #7
> cunningly concealed in the last place one would think of finding it: under the heading "Mailing List" on the Pyrex home page :-)

Hmmm - maybe I should try the scroll bar occassionally ;-)
Do you mean alist[x:x+n] == alist[y:y+n] ?
OK, probably you an Skip are right - let's see if I missed something at
the Python level.

There are essentially two differences from your snip above. I am trying
to compute n and there are multiple (under 10) lists. Size of lists are
typically under 100 ints.
...See what psyco makes of that.


I'm doing a similar straightforward loop approach but it's too slow.

Jim

Jun 4 '06 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
2285
by: Gary Stephenson | last post by:
I'm getting a clean generate, compile and link from my .pyx script, but when I attempt to run the resultant .exe, I get: "The procedure entry point Py_NoneStruct could not be located in the dynamic link library python23.dll" Can anybody provide me with a clue here? I've already cut down my script to remove all references to None,...
10
2021
by: Kyler Laird | last post by:
I need to submit C/C++ code for a class. (It's not a programming class. The choice of language is inertial. I think that it mostly serves to distract students from the course subject.) I'm fairly fluent with C but it hurts to think about writing in C when Python is *so* much more appropriate for these operations. I'd like to keep my...
1
1664
by: Paul Prescod | last post by:
PyCon 2004 Slides on "Extending Python with Pyrex" PDF: http://www.prescod.net/pyrex/ExtendingPythonWithPyrex.pdf PPT: http://www.prescod.net/pyrex/ExtendingPythonWithPyrex.ppt Pycon 2004 Slides on "Optimizing Python code with Pyrex" PDF: http://www.prescod.net/pyrex/OptimizingPythonWithPyrex.pdf PPT:...
0
1194
by: Pawel Oleksik | last post by:
Hi, I have no idea, how to properly declare/use Numeric functions in pyrex. For example the code below causes segmentation fault: --in pyrex-------------- cdef extern from "Numeric/arrayobject.h":
0
2822
by: Chris Lambacher | last post by:
Hi, I have to do some data manipulation that needs to be fast. I had a generator approach (that was faster than a list approch) which was taking approximately 5 seconds to run both encode and decode. I have done a conversion to pyrex and have gotten it down to about 2 seconds to run both encode and decode. An an excerpt from the pyrex...
11
1974
by: Jim Lewis | last post by:
Has anyone found a good link on exactly how to speed up code using pyrex? I found various info but the focus is usually not on code speedup.
0
987
by: Georg Grabler | last post by:
Hello everybody. I finally decided to use pyrex for my tasks wrapping and creating new python objects. Anyway, i ran into struggles. I want an array to be passed to a function, so basically i started the function as follows: def addToList (self, char *array):
1
1329
by: Hans Terlouw | last post by:
Hi, When trying to wrap C code using Pyrex, I encountered a strange problem with a piece of pure Python code. I tried to isolate the problem. The following code causes Pyrex to generate C code which gcc cannot compile: class Axis: axtype = "unknown type of axis" def __init__(self):
3
1486
by: skip | last post by:
I'm using Pyrex 0.9.5.1a. I have this simple Pyrex module: cdef class Foo: cdef public char attr def __init__(self): self.attr = 0 class Bar(Foo): def __init__(self):
0
7521
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7720
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
7959
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7473
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
6044
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5369
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
3483
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
1944
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
0
764
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.