By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,551 Members | 1,159 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,551 IT Pros & Developers. It's quick & easy.

Efficient custom checksum of one-dim numpy arrays of uint16

P: 3

This is my first post here, I am looking forward to being here.

I have actually posted almost the same question on comp.lang.python:

and got some hints at what i could do, but it is a little bit too complicated for me (yet), and i was wondering if there is a simpler way to do this.

I have implemented two functions for making a complementary ones addition of uint16 one-dimensional arrays, because I need this for computing a checksum on very many arrays with 50-10000 elements (difrerent size for each array)

Here it goes

from numpy import *

def compl_add_uint16(a, b):
c = a + b
c += c >> 16
return c & 0xFFFF

def compl_one_checksum(uint16s):
return reduce(compl_add_uint16, uint16s, 0x0000)

This works correct, but it is also *the* bottleneck in the several ksloc apllication as 88% of the time is spend doing the reduce, and 95% of that time is spend in the compl_add_uint16 function.

I does not surprise me that it is slow as the interpreter has to walk through the three lines in compl_add_uint on each addition.

However, i cannot figure out how to make it the right pythonic and efficient way?

I prefer a pure numpy implementation, but I am also willing to go into some weave inlining and blitz, although I have never tried that before and I do not have a blitz compatible C++ compiler installed on my MS Server 2003 OS.

How can I make that faster?

From looking around it seems like i need to go in some ufunc reduce direction to do this the numpy way, but how?

Thank you for you time,

-- Slaunger
Nov 15 '08 #1
Share this Question
Share on Google+
1 Reply

P: 3
I got the answer on comp.python.numeric.general from Charles R. Harris.

This does the trick:
Expand|Select|Wrap|Line Numbers
  1. def complement_add_uint16(x) :
  2.     y = sum(x, dtype=uint64)    
  3.     while y >= 2**16 :
  4.         y = (y & uint64(0xffff)) + (y >> uint64(16))
  5.     return y
That just boosted my checksum processing speed from 413 kB/s to, hold on, 47400 kB/s, which is more than a 100-fold increase. numpy rules!

-- Slaunger
Nov 15 '08 #2

Post your reply

Sign in to post your reply or Sign up for a free account.