473,698 Members | 2,305 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

bitshift a large amount of data

I am reading in 1bit data to a buffer that contains over 30000 bits and
I would like to beable to bitshift the entire buffer and AND and OR it
with other buffers of the same size. I though I could use the stl
bitset class, but I cannot find a way to load the buffer into the
bitset. I have been trying with this test program (with the size
scaled down to 1024 bits) to load a buffer into the bitset, but prints
out all zeros.

#include <iostream>
#include <bitset>

int main(){
int const size = 1024;
unsigned char* data = new unsigned char[size];
*(data+1000) = 0xEA;
std::bitset<siz e> bits(*data);
std::cout<<bits .count()<<std:: endl;
std::cout<<bits <<std::endl;
delete[] data;
return 0;
}

Feb 9 '06 #1
8 4898

Er*********@gma il.com wrote:
I am reading in 1bit data to a buffer that contains over 30000 bits and
I would like to beable to bitshift the entire buffer and AND and OR it
with other buffers of the same size. I though I could use the stl
bitset class, but I cannot find a way to load the buffer into the
bitset. I have been trying with this test program (with the size
scaled down to 1024 bits) to load a buffer into the bitset, but prints
out all zeros.

#include <iostream>
#include <bitset>

int main(){
int const size = 1024;
unsigned char* data = new unsigned char[size];
*(data+1000) = 0xEA;
std::bitset<siz e> bits(*data);


Who knows what will be in this data. All 0 is just one possibility.
You never assign that area of memory any value. The compiler is being
nice to you.

Try bits(0xEA);

You're using the unsigned long constructor. I'm not positive how the
conversion is done but my bet is you are actually only passing the
first byte of "data" into the constructor. The data you set is of
course 1000 bytes away.

Feb 9 '06 #2
I don't care what is in the data right now because this is just a test.
My compiler is giving me zeros and I assigne data+1000 0xEA just to
puts some ones in the buffer. I believe you are right about the
unsigned long constructor being used. Is there away to make bitset use
a buffer I give it?

Feb 9 '06 #3

Er*********@gma il.com wrote:
I don't care what is in the data right now because this is just a test.
My compiler is giving me zeros and I assigne data+1000 0xEA just to
puts some ones in the buffer. I believe you are right about the
unsigned long constructor being used. Is there away to make bitset use
a buffer I give it?


No, there are a few shortcommings of bitset and the inability to define
one in terms of actual bits is one of those things. You can only do so
up to the size of an unsigned long.

Your other option is to define a string of 1's and 0's and pass it in.
You could also do something like:

unsigned long array[size/sizeof(unsigned long)] = {####L, etc};
bitset<size> bits;

for (int i = 0; i < size/sizeof(unsigned long); ++i)
bits |= (bitset<size>(a rray[i]) << i * sizeof(unsigned long));

That is untested btw...

Feb 9 '06 #4
That is what I was afraid of. No way to define a bitset based on a
random memory buffer. I saw how you could define it as string, but
that will not help me. Strangly, I just wrote a test program similar
to yours before I read your response. I used unsigned char instead of
unsigned longs though because my data is byte aligned not 4 byte
aligned. I may change that though so I can load 4 bytes at a time
instead of 1 byte at a time.

1 #include <iostream>
2 #include <bitset>
3
4 int main(){
5 int const size = 256;
6 unsigned char* data = new unsigned char[size/8];
7 unsigned char* dataBegin = data;
8 *(data) = 0xEA;
9 *(data+1) = 0x1;
10 *(data+8) = 0xFF;
11 std::bitset<siz e> bits;
12 for(int i=0; i<size/8; ++i){
13 bits <<= 8;
14 bits |= std::bitset<siz e>(*data);
15 ++data;
16 }
17 std::cout<<bits .count()<<std:: endl;
18 std::cout<<bits <<std::endl;
19 delete[] dataBegin;
20 return 0;
21 }

Feb 9 '06 #5
* Eric Medlin:
I am reading in 1bit data to a buffer that contains over 30000 bits and
I would like to beable to bitshift the entire buffer and AND and OR it
with other buffers of the same size. I though I could use the stl
bitset class, but I cannot find a way to load the buffer into the
bitset. I have been trying with this test program (with the size
scaled down to 1024 bits) to load a buffer into the bitset, but prints
out all zeros.

#include <iostream>
#include <bitset>

int main(){
int const size = 1024;
unsigned char* data = new unsigned char[size];
*(data+1000) = 0xEA;
std::bitset<siz e> bits(*data);
std::cout<<bits .count()<<std:: endl;
std::cout<<bits <<std::endl;
delete[] data;
return 0;
}


No standard library container can use external storage for its elements;
it makes a copy of the data (that's why container elements must be
copyable).

You can use a std::bitset directly, or you can copy the data to and from
a std::bitset, or you can roll your own.

For the latter, if measurements tell you that you have to optimize, it
might be a good idea to look up your system's available bitblt
functions; typically they employ graphics hardware to do the operations.
--
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
Feb 9 '06 #6
Well even loading the data 4 bytes at a time it is just to slow for me
to load the data that way. Using a size 30000 it takes 0.015 secs when
a memcpy only takes 0.00005 secs. I have to load 30000 rows of data at
a size of 30000. That makes this way of loading bitset to slow for me.
So unless its bitshift and AND and OR operators are way faster than
what I can write for a big chunk of memory then I won't be able to use
it. And it looks like such an easy solution to my problem to if you
could only loaded it up faster.

Feb 9 '06 #7
<Er*********@gm ail.com> wrote in message
news:11******** *************@g 43g2000cwa.goog legroups.com...
I am reading in 1bit data to a buffer that contains over 30000 bits and
I would like to beable to bitshift the entire buffer and AND and OR it
with other buffers of the same size. I though I could use the stl
bitset class, but I cannot find a way to load the buffer into the
bitset. I have been trying with this test program (with the size
scaled down to 1024 bits) to load a buffer into the bitset, but prints
out all zeros.

#include <iostream>
#include <bitset>

int main(){
int const size = 1024;
unsigned char* data = new unsigned char[size];
*(data+1000) = 0xEA;
std::bitset<siz e> bits(*data);
std::cout<<bits .count()<<std:: endl;
std::cout<<bits <<std::endl;
delete[] data;
return 0;
}


If I had to do this, I don't think I'd actually shift the bits until a
bit/byte value was requested.

I would create a vector of unsigned char and allocate enought memory for
3000 bytes and load the data into the vector. Shifting right or left by any
multiple of 8 is trivial. Shift right by 8, delete the last char, push 0
onto the front. Shift left by 8, delete the first element, push 0 onto the
back.

Your only concern comes in then when you actually want to read the bits
(perhaps to AND or OR them). Encapsulate the vector with a class, over ride
the >> and << to do your custom shifting (only shift by mulitples of 8, any
remainder keep in a variable). Override []. When [] is used build the byte
to return. The override of [] would be a little more difficult, but not
much.

Say, for instance, they had shifted right 12. You would delete the last
element, push a 0 onto the front. That leaves a remainder of 4 to shift
right. So if the [] method requests byte 100, you would know to build the
byte by shifting the 100 element right 4, and making the high 4 bits the low
bits of element 99.

return (( BitVector[100] >> 4 ) & 0x0F ) | (( BitVector[99] << 4 ) & 0xF0)

That is untested, but I'm fairly sure it's right. Of course the constant 4
would be in a variable, and you would do something similar if it was shifted
left, and look out for your first and last elements.

The ANDing and ORing is trivial, just do it a byte at a time (with the
returned shifted value of course) and build a new vector to return.
Feb 9 '06 #8
Er*********@gma il.com wrote:
I am reading in 1bit data to a buffer that contains over 30000 bits and
I would like to beable to bitshift the entire buffer and AND and OR it
with other buffers of the same size. I though I could use the stl
bitset class, but I cannot find a way to load the buffer into the
bitset. I have been trying with this test program (with the size
scaled down to 1024 bits) to load a buffer into the bitset, but prints
out all zeros.

#include <iostream>
#include <bitset>

int main(){
int const size = 1024;
unsigned char* data = new unsigned char[size];
*(data+1000) = 0xEA;
std::bitset<siz e> bits(*data);
std::cout<<bits .count()<<std:: endl;
std::cout<<bits <<std::endl;
delete[] data;
return 0;
}


See page 122 ('Sets' chapter) in "Applying C++" by
Scott Robert Ladd.

That chapter contains a complete implemenation of
a ShortBitSet class that handles up to 65k bits and
a LongBitSet class that handles up to 4 billion bits.
'or', 'and', etc operators are included, but shift
operators are not - you'll have to add your own.

Regards,
Larry
Feb 10 '06 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
2733
by: steve | last post by:
Hi, I have researched but have not found a good solution to this problem. I am importing large amounts of data (over 50 Meg) into a new mysql db that I set up. I use >mysql dbname < importfile.txt But I keep getting timeouts and errors due to the data being too large. I know that since if I break the imported data into multiple chuncks (by importing a few tables at a time) then everything works.
1
1625
by: Robert May | last post by:
Hi, I am trying to execute some code compiled by g++ on Linux and have found that after some time, the program allocates a huge amount of swap space (250MB on my machine which has 512MB physical) and (700MB on another server with 1GB physical RAM). I have used vmstat to trend the amount of swap and observed that the memory is not being "thrashed" and there is simply a large amount of data that has been swapped out. This still slows...
5
2182
by: Mike | last post by:
This is a general question on the best way to import a large amount of data to a MS-SQL DB. I can have the data in just about any format I need to, I just don't know how to import the data. I some experience with SQL but not much. There is about 1500 to 2000 lines of data. I am looking for the best way to get this amount of data in on a monthly basis. Any help is greatly thanked!!
11
4127
by: Macca | last post by:
Hi, I'm writing an application that will pass a large amount of data between classes/functions. In C++ it was more efficient to send a pointer to the object, e.g structure rather than passing the actual structure itself. Is this true of C# also?
3
1241
by: kamran | last post by:
Hi, I have a web service that may return a very large amount of data. I want that data to return in chunks, like first return 10% of data than return the next 10% and so on, until all is finished. How are other people doing this? I have come up with the following way: While inside the webmethod, fetch the data from the db and store it in a file called "someguid.data" and return the guid. The client will receive a guid and request again,...
7
10820
by: =?Utf-8?B?TW9iaWxlTWFu?= | last post by:
Hello everyone: I am looking for everyone's thoughts on moving large amounts (actually, not very large, but large enough that I'm throwing exceptions using the default configurations). We're doing a proof-of-concept on WCF whereby we have a Windows form client and a Server. Our server is a middle-tier that interfaces with our SQL 05 database server.
4
1926
by: bcomeara | last post by:
I am writing a program which needs to include a large amount of data. Basically, the data are p values for different possible outcomes from trials with different number of observations (the p values are necessarily based on slow simulations rather than on a standard function, so I estimated them once and want the program to include this information). Currently, I have this stored as a vector of vectors of varying sizes (first vector is...
16
2575
by: Jack | last post by:
I need to process large amount of data. The data structure fits well in a dictionary but the amount is large - close to or more than the size of physical memory. I wonder what will happen if I try to load the data into a dictionary. Will Python use swap memory or will it fail? Thanks.
0
2357
by: jcatubay | last post by:
I have a function that returns a list more than 200000 objects and the object has 37 fields. I added the wcf as a web reference so i dont have to add any configuration item in my web apps config file. WCF works when I only have small amount of data. SInce I did'nt add the config file, how can I allow the wcf to send large amount of data. It sems to be timing out. When I debugged my wcf I'm getting this error "An error occurred while receiving...
0
8674
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8604
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
1
8895
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8861
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7728
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6518
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
4619
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3046
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
2001
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.