472,354 Members | 2,059 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,354 software developers and data experts.

Efficient custom checksum of one-dim numpy arrays of uint16


This is my first post here, I am looking forward to being here.

I have actually posted almost the same question on comp.lang.python:


and got some hints at what i could do, but it is a little bit too complicated for me (yet), and i was wondering if there is a simpler way to do this.

I have implemented two functions for making a complementary ones addition of uint16 one-dimensional arrays, because I need this for computing a checksum on very many arrays with 50-10000 elements (difrerent size for each array)

Here it goes

from numpy import *

def compl_add_uint16(a, b):
c = a + b
c += c >> 16
return c & 0xFFFF

def compl_one_checksum(uint16s):
return reduce(compl_add_uint16, uint16s, 0x0000)

This works correct, but it is also *the* bottleneck in the several ksloc apllication as 88% of the time is spend doing the reduce, and 95% of that time is spend in the compl_add_uint16 function.

I does not surprise me that it is slow as the interpreter has to walk through the three lines in compl_add_uint on each addition.

However, i cannot figure out how to make it the right pythonic and efficient way?

I prefer a pure numpy implementation, but I am also willing to go into some weave inlining and blitz, although I have never tried that before and I do not have a blitz compatible C++ compiler installed on my MS Server 2003 OS.

How can I make that faster?

From looking around it seems like i need to go in some ufunc reduce direction to do this the numpy way, but how?

Thank you for you time,

-- Slaunger
Nov 15 '08 #1
1 3570
I got the answer on comp.python.numeric.general from Charles R. Harris.

This does the trick:
Expand|Select|Wrap|Line Numbers
  1. def complement_add_uint16(x) :
  2.     y = sum(x, dtype=uint64)    
  3.     while y >= 2**16 :
  4.         y = (y & uint64(0xffff)) + (y >> uint64(16))
  5.     return y
That just boosted my checksum processing speed from 413 kB/s to, hold on, 47400 kB/s, which is more than a 100-fold increase. numpy rules!

-- Slaunger
Nov 15 '08 #2

Sign in to post your reply or Sign up for a free account.

Similar topics

by: Mercuro | last post by:
Hello i'm looking for a simple way to checksum my data. The data is 70 bytes long per record, so a 32 byte hex md5sum would increase the size of my mysql db a lot. I'm looking for something...
by: Chris Tanger | last post by:
I am creating a class that has a method "Write" that I wish to make threadsafe. The method must block calling threads until the task performed in write is complete. Only 1 thread at a time can...
by: pmatos | last post by:
Hi all, I am trying to create a simple but efficient C++ logging class. I know there are lots of them out there but I want something simple and efficient. The number one requirement is the...
by: s.subbarayan | last post by:
Dear all, I have one peculiar problem with me for which I need ur inputs how to implement it: I have 2 registers each 8 bit wide. The first register stores: Register Map:...
by: Chua Wen Ching | last post by:
Hi there, I had been wondering for a while. I am building a very efficient Custom Exception Library. But i wasn't that sure whether i should be using XML or not. There are 3 ways which i...
by: Christopher Kimbell | last post by:
I have created my own WebPart using code, it is not derived from a UserControl. Inside this WebPart I create a Calendar Control and add it to the WebParts Control collection. Now I want to apply...
by: Ed Sonneveld | last post by:
Hi, I have hosted my webservice at a hosting company and it has been working fine for 2 years now. The webservice is called by winforms clients over the internet, using the proxy class generated...
by: rizwanahmed24 | last post by:
Hello i have made a custom control. i have placed a panel on it. I want this panel to behave just like the normal panel. The problem i was having is that the panel on my custom control doesnt...
by: =?Utf-8?B?YmlsbCB0aWU=?= | last post by:
Activator.CreateInstance vs. CunstructorInfo.Invoke - If I may use either of them, which one is more efficient? - When is one preferred to the other? Thank you.
by: Kemmylinns12 | last post by:
Blockchain technology has emerged as a transformative force in the business world, offering unprecedented opportunities for innovation and efficiency. While initially associated with cryptocurrencies...
by: Matthew3360 | last post by:
Hi there. I have been struggling to find out how to use a variable as my location in my header redirect function. Here is my code. header("Location:".$urlback); Is this the right layout the...
by: Matthew3360 | last post by:
Hi, I have a python app that i want to be able to get variables from a php page on my webserver. My python app is on my computer. How would I make it so the python app could use a http request to get...
by: AndyPSV | last post by:
HOW CAN I CREATE AN AI with an .executable file that would suck all files in the folder and on my computerHOW CAN I CREATE AN AI with an .executable file that would suck all files in the folder and...
by: WisdomUfot | last post by:
It's an interesting question you've got about how Gmail hides the HTTP referrer when a link in an email is clicked. While I don't have the specific technical details, Gmail likely implements measures...
by: Matthew3360 | last post by:
Hi, I have been trying to connect to a local host using php curl. But I am finding it hard to do this. I am doing the curl get request from my web server and have made sure to enable curl. I get a...
by: Oralloy | last post by:
Hello Folks, I am trying to hook up a CPU which I designed using SystemC to I/O pins on an FPGA. My problem (spelled failure) is with the synthesis of my design into a bitstream, not the C++...
by: Carina712 | last post by:
Setting background colors for Excel documents can help to improve the visual appeal of the document and make it easier to read and understand. Background colors can be used to highlight important...
by: BLUEPANDA | last post by:
At BluePanda Dev, we're passionate about building high-quality software and sharing our knowledge with the community. That's why we've created a SaaS starter kit that's not only easy to use but also...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.