473,320 Members | 1,965 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Easy way to make Numarray Bool use 1 bit?

I need to be able to store thousands of one-million-length bit arrays.
The first operations use the & operator. But eventually I must perform
an inner product with a one-million-length integer array (summing at
all the indices of the integer operand where the boolean operand has a
"1").

If I use the Boolean type, I end up using 1-byte, per needed bit, which
is not efficient enough (only 1,000 arrays per gigabyte of memory).
I'm limited by 32-bit windows to use only 2 GB total.

Anybody know of a fast way to use only 125 kilobytes of memory per
array and yet be able to inner product with an integer array quickly?
Don't tell me I have to write C/C++ ! (ugh!)

TIA,
AndrewF

Feb 7 '06 #1
1 1044
Ahh, I figured it out. The key is to set up the shapes so they
broadcast like an outerproduct, but using bitwise_and as the operation
and the second operand as the bit filter. So we temporarily expand
each bit to a byte for numarray to be able to use it for dot products.
If you use a temporary Boolean variable to store the bitwise_and ufunc
output, the answer is automatically converted to boolean.

Set up the shapes for an innerproduct-like operation to go from byte
format back to bit compact Uint8 format. Go numarray!

Feb 7 '06 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: RJS | last post by:
Hi all, I can't get a py2exe compiled app to run with numarray (numarray-0.5.win32- py2.2). Also wxPythonWIN32-2.3.3.1-Py22 and ActivePython-2.2.1-222. In the sample below, commenting out...
3
by: Alexander Schwaigkofler | last post by:
Hi! I have the following problem with numarray. I read the install.txt manual, but it doesn't already work. OS: Microsoft Windows 2000 python: Python 2.2.3 (#42, May 30 2003, 18:12:08) on...
4
by: Marco Bubke | last post by:
Hi I have tried to include numarray in Pyrex but I get allways this error: Traceback (most recent call last): File "gears.py", line 9, in ? import gl File...
4
by: Marco Bubke | last post by:
Hi Ok, I get a reproduceable seqmentation fault. Ok, fist the code: cdef NumArray array_to_double(NumArray array): # maybe here is memoty leak! cdef NumArray flat_array_obj flat_array_obj...
2
by: Marc Schellens | last post by:
Following the NumPy documentation, I took over some C code, but run into an error. Does anybody have a suggestion? Thanks, marc gdlpython.cpp:225: `PyArray_Type' undeclared (first use this...
3
by: SunX | last post by:
I tried to initialize a float point array by: import numarray xur = numarray.fromfunction(lambda x,y,z:x*y*z, (2, 2, 2)) but I ended up with an integer array even though x, y, and z are all...
0
by: andrewfelch | last post by:
Below is the code to/from Boolean arrays and Unsigned integers. On my Pentium 4, functions such as "bitwise_and" are 32 times faster when run on 32-bit integers instead of the...
10
by: Bryan | last post by:
hi, what is the difference among numeric, numpy and numarray? i'm going to start using matplotlib soon and i'm not sure which one i should use. this page says, "Numarray is a...
0
by: robert | last post by:
just a note - some speed comparisons : 0.60627370238398726 0.42836673376223189 0.36965815487747022 0.016557970357098384 0.15692469294117473 0.01951756438393204
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.