473,323 Members | 1,537 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,323 software developers and data experts.

Binary files and 100% CPU usage haunts me

AaronL
99
Hello,

I've recently started to program my own file structure in visual basic 6 and I found out the way to do this of course is binary files. However, my problem is, I don't know anything lol! I know the basics of reading and writing binary files. I came up with a method of pulling records with a comma seperation but it isn't near efficiant because everytime I go read a record, I get 100% CPU usage and that isn't exactly what I want my software to do. Especially when reading multiple records in. Long story short, why does this cause 100% cpu usage and what is a better way to manipulate binary files? The bigger the file, the worse it is...

Expand|Select|Wrap|Line Numbers
  1. Dim getloop As Long
  2. Dim binaryin As Byte
  3. getloop = 1
  4. Open "c:\test\test.alr" For Binary As 1
  5.     While Not EOF(1)
  6.         Get #1, getloop, binaryin
  7.         getloop = getloop + 1
  8.     Wend
  9. Close 1
I tried this too

Expand|Select|Wrap|Line Numbers
  1. For getloop = 1 to FileLen("c:\test\test.alr") 
  2.      Get #1, getloop, binaryin
  3. Next getloop
I know these processes are going through the entire file, however my question is, how do you know where a record starts and ends without something like

If binaryin = <ascii code for comma or something> Then
Record = Record + 1
End If

I know I'm probably going about this whole thing wrong, I suck. I hope someone can shed some light on this in layman's terms.
Jan 29 '07 #1
6 2075
willakawill
1,646 1GB
Hi, a couple of things you need when dealing with binary file input and output.
Firstly you should avoid variable length records where possible.
e.g. if you are storing a string it could be
Expand|Select|Wrap|Line Numbers
  1. Dim mystr1 As String
which is a variable length or
Expand|Select|Wrap|Line Numbers
  1. Dim mystr2 As String * 50
which has a fixed length of 50

You also need to know if the number of records will change or not in which case you may need to store the number of records at the beginning of the binary file
Jan 30 '07 #2
Killer42
8,435 Expert 8TB
There may be more to come, as I'm very busy right now. But I think your basic problem is that you're reading one byte at a time.

You can read in just about any size chunk you want. Have a play with this variation...
Expand|Select|Wrap|Line Numbers
  1. Dim DataChunk As String
  2. Open "c:\test\test.alr" For Binary Access Read Shared As #1
  3. DataChunk = Space$(LOF(1)) ' Set size of buffer to read into.
  4. Get #1, , DataChunk ' Read entire buffer in one go.
  5. Close 1
Note, I believe you can also use a binary array, and there may be many reasons why it's better than a string. I just don't remember how, as I started using strings way back before the binary data type existed.

As you can see for yourself, since the size of the buffer you are reading into determines how much you actually read, you can adjust your buffer size to tweak performance against memory usage. Generally speaking, the bigger the chunks you deal with in each operation, the quicker the read/write will be.
Jan 30 '07 #3
Killer42
8,435 Expert 8TB
...You also need to know if the number of records will change or not in which case you may need to store the number of records at the beginning of the binary file
Unless they're fixed length, of course. In which case you can easily calculate the number of records from the file size.

Also, RANDOM may be useful in such a case rather than BINARY.
Jan 30 '07 #4
AaronL
99
Cool, I'm storing the data as a byte though for my file encryption method, how do I allocate memory to a byte variable type

There may be more to come, as I'm very busy right now. But I think your basic problem is that you're reading one byte at a time.

You can read in just about any size chunk you want. Have a play with this variation...
Expand|Select|Wrap|Line Numbers
  1. Dim DataChunk As String
  2. Open "c:\test\test.alr" For Binary Access Read Shared As #1
  3. DataChunk = Space$(LOF(1)) ' Set size of buffer to read into.
  4. Get #1, , DataChunk ' Read entire buffer in one go.
  5. Close 1
Note, I believe you can also use a binary array, and there may be many reasons why it's better than a string. I just don't remember how, as I started using strings way back before the binary data type existed.

As you can see for yourself, since the size of the buffer you are reading into determines how much you actually read, you can adjust your buffer size to tweak performance against memory usage. Generally speaking, the bigger the chunks you deal with in each operation, the quicker the read/write will be.
Jan 31 '07 #5
Killer42
8,435 Expert 8TB
Cool, I'm storing the data as a byte though for my file encryption method, how do I allocate memory to a byte variable type
A Byte variable can only hold a single byte (hence the name), but you can make an array of Byte type, just like any other data type.

Defining an array is as simple as adding the number of occurrences when you define the variable (Eg. Dim DataByte(1 To 1024) As Byte) but I'm afraid I haven't used a byte array in this way for years, and have forgotten how.

I expect willakawill can probably help with this.

One thing to note - the calls to do the IO are the really slow part, so even taking the time to stuff the bytes into a string and then writing the string in one chunk would make things a bit faster. Likewise, to read it back, you could read it into a string and then pick that apart into a Byte array. However, if you're already dealing with Byte variables, you shouldn't have to jump through hoops like that, as you can just work directly with the array.

In fact I've just tried it, and you can read an entire Byte array from a binary file like so...
Expand|Select|Wrap|Line Numbers
  1. Dim Data(1 To 50) As Byte
  2. Open "c:\test\test.alr" For Binary Access Read Shared As #1
  3. ' I don't recall how to control the number of bytes read/written.
  4. Get #1, , Data ' Read entire array in one go.
  5. Close 1
Jan 31 '07 #6
AaronL
99
Thank you so much, problem solved :)
Jan 31 '07 #7

Sign in to post your reply or Sign up for a free account.

Similar topics

14
by: Kevin Knorpp | last post by:
Hello. I need to be able to extract the data from the attached file (or any file in the same format) so that I can work with the data in PHP. I'm fairly comfortable with using PHP with...
21
by: Sami Viitanen | last post by:
Hello, How can I check if a file is binary or text? There was some easy way but I forgot it.. Thanks in adv.
13
by: yaipa | last post by:
What would be the common sense way of finding a binary pattern in a ..bin file, say some 200 bytes, and replacing it with an updated pattern of the same length at the same offset? Also, the...
3
by: GMane Python | last post by:
Hello All. I have a program that downloads 'gigabytes' of Axis NetCam photos per day. Right now, I set up the process to put the images into a queue, and every 30 or so seconds, 'pop' them from...
11
by: ajou_king | last post by:
I was running some tests on my Win32 1GHZ processor to see how long it would take to transmit objects numerous times via TCP/IP using C# ..NET Remoting vs the C++ trustworthy method of binary...
10
by: Fabuloussites | last post by:
I'm considering deploying an application that will us an IP address locaiton database provided by Ip2location.com... http://www.ip2location.net/ip2location-dotnet-component.aspx their .net...
4
by: FreewareGuy | last post by:
I want to add binary files to my VB 2005 executable to make a single exe with all the required files which I can extract at will through my code, how do I do this pls. guide o how to add binary...
6
by: =?Utf-8?B?VGhvbWFzWg==?= | last post by:
Hi, Is it possible to read a file in reverse and only get the last 100 bytes in the file without reading the whole file from the begining? I have to get info from files that are in the last 100...
3
by: masood.iqbal | last post by:
Hi, Kindly excuse my novice question. In all the literature on ifstream that I have seen, nowhere have I read what happens if you try to read a binary file using the ">>" operator. I ran into...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.