473,325 Members | 2,870 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,325 software developers and data experts.

Pyrex list/array

I'm trying to move a function into pyrex for speed. The python side
needs to pass a list to the pyrex function. Do I need to convert to
array or something so pyrex can generate tight code? I'm not clear how
to do this.

Jun 3 '06 #1
7 2019
On 4/06/2006 4:56 AM, Jim Lewis wrote:
I'm trying to move a function into pyrex for speed.
You probably didn't expect the Inquisition; nobody does. But here it is,
nice red uniforms and all:

1. What is your speed requirement and how far short of that are you at
the moment?
2. Are you sure there is no Python or third-party module that does what
you want?
3. Is your algorithm the best possible?
4. Is your Python implementation of that algorithm the best possible?
Have you exposed it to the critical gaze of the speed-freaks in this
newsgroup?
5. Does your architecture support psyco? If so, have you tried that and
what were the results?
The python side
needs to pass a list to the pyrex function. Do I need to convert to
array or something so pyrex can generate tight code? I'm not clear how
to do this.


The question might be better asked on the Pyrex mailing list.

You don't need to convert a list to a C array, and it may not even be
possible, depending on what type(s) of data you have in the list.

Almost any Python code is also valid Pyrex code. For a start, just
compile your function with Pyrex and compare the speed. What you do next
is going to depend very much on what operations you are performing on
the list and the objects it contains. Watch out for Python built-ins
like range, xrange, ord, chr, abs, bool, int(a_number), float(a_number),
divmod, max/min (two_numeric_args). In almost all cases you get cheap
wins by replacing use of these by simple C-like code -- provided of
course you are absolutely sure you know what types you are dealing with.

HTH,
John
Jun 4 '06 #2
Thanks for your comments.
You probably didn't expect the Inquisition...
Correct ;-)
1. What is your speed requirement and how far short of that are you at the moment?
~10 times faster.
2. Are you sure there is no Python or third-party module that does what you want?
Yes.
3. Is your algorithm the best possible?
I think so although of course one can never be certain.
4. Is your Python implementation of that algorithm the best possible? Have you exposed it to the critical gaze of the speed-freaks in this newsgroup?
Thanks for the good suggestion but I want to try pyrex first.
5. Does your architecture support psyco? If so, have you tried that and what were the results?
Already using psyco.
The question might be better asked on the Pyrex mailing list.
I did not find it - where is it?
Almost any Python code is also valid Pyrex code. For a start, just compile your function with Pyrex and compare the speed.
It's slower.
What you do next is going to depend very much on what operations you are performing on the list and the objects it contains.


Simple list of ints. Comparing sections of lists between each other.

Jun 4 '06 #3
5. Does your architecture support psyco? If so, have you tried that
and what were the results?


Jim> Already using psyco.

Is it substantially faster with psyco than without? If psyco is performing
its magic on the critical section of code already, you are going to lose
that when switching to Pyrex.

Skip
Jun 4 '06 #4
> Is it substantially faster with psyco than without? If psyco is performing
its magic on the critical section of code already, you are going to lose
that when switching to Pyrex.


Yes but from what I read Pyrex can be a lot faster than psyco under the
right circumstances.

Jun 4 '06 #5
Is it substantially faster with psyco than without? If psyco is
performing its magic on the critical section of code already, you are
going to lose that when switching to Pyrex.


Jim> Yes but from what I read Pyrex can be a lot faster than psyco under
Jim> the right circumstances.

I'm sure that's true. That also means under the wrong circumstances it
might not. ;-) Can you post the code that's running too slowly?

Also, note that psyco learned some new tricks at the recent NeedForSpeed
sprint. You might want to check out the latest version from Subversion and
give it a whirl.

Skip
Jun 4 '06 #6
On 4/06/2006 7:59 PM, Jim Lewis wrote:
Thanks for your comments.
You probably didn't expect the Inquisition...
Correct ;-)


Nobody does :-)
The question might be better asked on the Pyrex mailing list.
I did not find it - where is it?


Evidently somewhere near the Hall of the Mountain King. A reference to
it is cunningly concealed in the last place one would think of finding
it: under the heading "Mailing List" on the Pyrex home page :-) Here:
http://lists.copyleft.no/mailman/listinfo/pyrex ... get in quick before
the pirate moves it again.
Almost any Python code is also valid Pyrex code. For a start, just compile your function with Pyrex and compare the speed.


It's slower.
What you do next is going to depend very much on what operations you are performing on the list and the objects it contains.


Simple list of ints. Comparing sections of lists between each other.


Do you mean alist[x:x+n] == alist[y:y+n] ?
If so, that's creating two new lists each of size n, and then comparing
those two lists. I doubt that psyco would recognize that it didn't need
to copy the two slices. The first step might be to write functions to
compare without copying, e.g.:

def py_slices_cmp_eq(py_list, start1, start2, size):
"""Return 1 if py_list[start1+size] == py_list[start2+size]
else 0"""
offset = start2 - start1
for i in xrange(start1, start1+size):
if py_list[i] != py_list[i+offset]:
return 0
return 1

See what psyco makes of that.
Then turn that into a cdef function for Pyrex.

If that's still not fast enough, then you might be in for some harder work:

Allocate memory for a C array, unpack your list into it, write
comparison functions c_slices_cmp_* that operate on your array of ints.
There should be no Python stuff in there, only C constructs. You can
even use memcmp() for the cmp_eq function.

Which brings us back to your original question "Do I need to convert to
array or something so pyrex can generate tight code?" ...
1. Some of the above may help you to determine whether you need to.
2. Without any knowledge of the size of your list or what you are doing,
we can't help you much more on whether you need to.
3. AFAICT, Pyrex doesn't do much in the way of optimisation, leaving
that up to the C compiler. Generating tight code would depend more on
you replacing appropriately-chosen Pythonisms with C-isms.

As for *how* to make your C array, something like this:

cdef extern from "Python.h":
void PyMem_Free(void *p)
void* PyMem_Malloc(int n) except NULL
# untested -- compiles OK :-)
cdef int * int_array_from_list(object ilist):
cdef int n, i
cdef int * arrayp
n = len(ilist)
arrayp = <int *>PyMem_Malloc(n * sizeof(int))
for i from 0 <= i < n:
arrayp[i] = ilist[i]
return &arrayp[0]

Hoping some of this helps,
John
Jun 4 '06 #7
> cunningly concealed in the last place one would think of finding it: under the heading "Mailing List" on the Pyrex home page :-)

Hmmm - maybe I should try the scroll bar occassionally ;-)
Do you mean alist[x:x+n] == alist[y:y+n] ?
OK, probably you an Skip are right - let's see if I missed something at
the Python level.

There are essentially two differences from your snip above. I am trying
to compute n and there are multiple (under 10) lists. Size of lists are
typically under 100 ints.
...See what psyco makes of that.


I'm doing a similar straightforward loop approach but it's too slow.

Jim

Jun 4 '06 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Gary Stephenson | last post by:
I'm getting a clean generate, compile and link from my .pyx script, but when I attempt to run the resultant .exe, I get: "The procedure entry point Py_NoneStruct could not be located in the...
10
by: Kyler Laird | last post by:
I need to submit C/C++ code for a class. (It's not a programming class. The choice of language is inertial. I think that it mostly serves to distract students from the course subject.) I'm...
1
by: Paul Prescod | last post by:
PyCon 2004 Slides on "Extending Python with Pyrex" PDF: http://www.prescod.net/pyrex/ExtendingPythonWithPyrex.pdf PPT: http://www.prescod.net/pyrex/ExtendingPythonWithPyrex.ppt Pycon 2004 Slides...
0
by: Pawel Oleksik | last post by:
Hi, I have no idea, how to properly declare/use Numeric functions in pyrex. For example the code below causes segmentation fault: --in pyrex-------------- cdef extern from...
0
by: Chris Lambacher | last post by:
Hi, I have to do some data manipulation that needs to be fast. I had a generator approach (that was faster than a list approch) which was taking approximately 5 seconds to run both encode and...
11
by: Jim Lewis | last post by:
Has anyone found a good link on exactly how to speed up code using pyrex? I found various info but the focus is usually not on code speedup.
0
by: Georg Grabler | last post by:
Hello everybody. I finally decided to use pyrex for my tasks wrapping and creating new python objects. Anyway, i ran into struggles. I want an array to be passed to a function, so basically i...
1
by: Hans Terlouw | last post by:
Hi, When trying to wrap C code using Pyrex, I encountered a strange problem with a piece of pure Python code. I tried to isolate the problem. The following code causes Pyrex to generate C code...
3
by: skip | last post by:
I'm using Pyrex 0.9.5.1a. I have this simple Pyrex module: cdef class Foo: cdef public char attr def __init__(self): self.attr = 0 class Bar(Foo): def __init__(self):
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.