473,320 Members | 1,810 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

recording massive data stream

i have a very large data stream, on the order of 1.5 million records an
hour (32 bytes per/record), which i would like to try and record to
disk. it doesn't need to be iserted into a db, so flat files or
binaries will probably be the end result. what would be the most
efficient way of acheiving this with c++? the data is arriving via the
internet on a broadband connection 3mbps and the hardware is an
unraided uata-133 with a 1.4ghz athlon and 512mb of ram. if any more
information is needed to base your input on, please let me know. any
help would be greatly appreciated.

many thanks,
lorn

Jul 23 '05 #1
4 1715
Lorn wrote:
i have a very large data stream, on the order of 1.5 million records an
hour (32 bytes per/record), which i would like to try and record to
disk. it doesn't need to be iserted into a db, so flat files or
binaries will probably be the end result. what would be the most
efficient way of acheiving this with c++? the data is arriving via the
internet on a broadband connection 3mbps and the hardware is an
unraided uata-133 with a 1.4ghz athlon and 512mb of ram. if any more
information is needed to base your input on, please let me know. any
help would be greatly appreciated.

many thanks,
lorn


1500000*32 bytes ~= 45.7 MB/hour. This is a slow stream even for
slowest PC that you can find :). I think you can use any method to
store this to disk. I guess it would be efficient enough to buffer the
data and append it to the disk file only once in a while.

Jul 23 '05 #2
ok, thanks for your reply.

assuming a much more aggresive stream. 500-1000MB/hr., is a write
buffer still the most efficient method?

thanks

Jul 23 '05 #3
Lorn wrote:
ok, thanks for your reply.

assuming a much more aggresive stream. 500-1000MB/hr., is a write
buffer still the most efficient method?

thanks


Bigger the buffer, the less you would write to disk. But, on power
failure you could loose that buffer. So it's up to you to come with a
number that seems feet to your needs.

Also, you could chose to use a big buffer and flush it to disk ever X
seconds / minutes or when it's full. Maybe this would be efficient for
all the possibilities.

Jul 23 '05 #4


Lorn schreef:
ok, thanks for your reply.

assuming a much more aggresive stream. 500-1000MB/hr., is a write
buffer still the most efficient method?


Still no technical challenge, still no C++ subject
(what efficiency? power?battery? CPU? IO bytes/sec?)

Jul 23 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: Sugapablo | last post by:
I had an idea for something that I can't find any evidence if it exists, or if it can be done. I assume it can be done. What I'd like to be able to do, is to allow people who come to my website,...
26
by: Chris Lasher | last post by:
Hello, I have a rather large (100+ MB) FASTA file from which I need to access records in a random order. The FASTA format is a standard format for storing molecular biological sequences. Each...
3
by: Jussi Rasku | last post by:
Hello, I'm working on a small web-script that would allow me to remotely setup recordings to my tv-tuner (on my linux box). The idea is simple, but I'm unsure how to accomplish it. The idea:...
6
by: Dovelet | last post by:
Hi all, I would like to write DOS program to change the MS Windows Sound Recording source. When I run it with the parameter, it will change the recording source as follow: C:\> abc.exe...
1
by: ACaunter | last post by:
Hi all, Have been getting no luck lately, but there must be a way i can recording audio from the website and save it.. It's no problem on applications (VB.Net), and I know other have done it...
1
by: Sakharam Phapale | last post by:
Hi All, I am developing an application like sound recorder. While recording if there is a silence for more than given time (say 5 seconds), Recording should be paused.
0
by: azdavid | last post by:
hi how i am cause the client(client-size) to recording himself and afterward sending the file(stream) to the server. how i am doning that in ASP NET?? Thank you
8
by: jer | last post by:
I've noticed you've helped a lot of people that were trying to do sound recording through VB.net. I'm searched all over the internet, and I'm still having problems write sound to a wav file. My...
9
by: fevos | last post by:
Hi i want to record voice and send it throw Bluetooth stream , the problem is the recorder accepts only I.O Stream And write at the 'stream' without using (stream.write), ''using openNETCF...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.