472,354 Members | 2,059 Online

# Efficient custom checksum of one-dim numpy arrays of uint16

Hi,

This is my first post here, I am looking forward to being here.

I have actually posted almost the same question on comp.lang.python:

and got some hints at what i could do, but it is a little bit too complicated for me (yet), and i was wondering if there is a simpler way to do this.

I have implemented two functions for making a complementary ones addition of uint16 one-dimensional arrays, because I need this for computing a checksum on very many arrays with 50-10000 elements (difrerent size for each array)

Here it goes

from numpy import *

c = a + b
c += c >> 16
return c & 0xFFFF

def compl_one_checksum(uint16s):

This works correct, but it is also *the* bottleneck in the several ksloc apllication as 88% of the time is spend doing the reduce, and 95% of that time is spend in the compl_add_uint16 function.

I does not surprise me that it is slow as the interpreter has to walk through the three lines in compl_add_uint on each addition.

However, i cannot figure out how to make it the right pythonic and efficient way?

I prefer a pure numpy implementation, but I am also willing to go into some weave inlining and blitz, although I have never tried that before and I do not have a blitz compatible C++ compiler installed on my MS Server 2003 OS.

How can I make that faster?

From looking around it seems like i need to go in some ufunc reduce direction to do this the numpy way, but how?

Thank you for you time,

-- Slaunger
Nov 15 '08 #1
1 3570
I got the answer on comp.python.numeric.general from Charles R. Harris.

This does the trick:
Expand|Select|Wrap|Line Numbers
2.     y = sum(x, dtype=uint64)
3.     while y >= 2**16 :
4.         y = (y & uint64(0xffff)) + (y >> uint64(16))
5.     return y
6.
That just boosted my checksum processing speed from 413 kB/s to, hold on, 47400 kB/s, which is more than a 100-fold increase. numpy rules!

-- Slaunger
Nov 15 '08 #2