473,320 Members | 2,109 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Binary file writing problem

Hi,
I've got a program (see source below) that makes a file and fills it with
random binary values (from 0 to 255). The source below works, however the
program creates files at a rate of about 0.5MB per second. There is a
serious performance issue with this program.
There is a loop within the main loop that generates a buffer of 500 Bytes
that are then all written at once.
Can anybody tell me how I can improve the performance of the program? Idealy
I could with a rate of about 25Mb/s.

Thanx Josh

('reps' is the number of repetitions. e.g. Number of bytes)

FileStream fs = new FileStream(fileLocation, FileMode.CreateNew);

// Create the writer for data.

BinaryWriter w = new BinaryWriter(fs);

Random r =new Random();

for (int i = 0; i < Int32.Parse(reps.ToString()); )

{

byte[] bits = new byte[500];

for (int j = 0; j <= 499; j++)

{

bits[j] = byte.Parse(r.Next(0, 255).ToString());

i++;

if(i >= Int32.Parse(reps.ToString()))

{

break;

}

}

w.Write(bits);

}
Nov 16 '05 #1
3 2699
Not sure what disk IO subsystem you are running this on, but don't ever
expect to achieve 25MB write throughput on anything less than a RAID0 (HW)
with a minimum of 7-8 spindles.
As a simple rule take a max. sustained IO rate of ~80-100 IO writes per
second per (7200 RPM) spindle.
So 500Kb per sec. is a fair number on a single drive with a block size of
500 bytes, what you should try first is to increase the buffer size to
something like 4-8 Kb, but don't expect higher throughputs than 2 - 3 Mb per
sec.

Willy.

source below) that makes a file and fills it with
random binary values (from 0 to 255). The source below works, however the
program creates files at a rate of about 0.5MB per second. There is a
serious performance issue with this program.
There is a loop within the main loop that generates a buffer of 500 Bytes
that are then all written at once.
Can anybody tell me how I can improve the performance of the program?
Idealy
I could with a rate of about 25Mb/s.

Thanx Josh

('reps' is the number of repetitions. e.g. Number of bytes)

FileStream fs = new FileStream(fileLocation, FileMode.CreateNew);

// Create the writer for data.

BinaryWriter w = new BinaryWriter(fs);

Random r =new Random();

for (int i = 0; i < Int32.Parse(reps.ToString()); )

{

byte[] bits = new byte[500];

for (int j = 0; j <= 499; j++)

{

bits[j] = byte.Parse(r.Next(0, 255).ToString());

i++;

if(i >= Int32.Parse(reps.ToString()))

{

break;

}

}

w.Write(bits);

}

Nov 16 '05 #2
Hello
for (int i = 0; i < Int32.Parse(reps.ToString()); ) I understand that reps is a variable of type int. why convert it to a string
and back to int again? and this takes place at every iteration.
for(int i = 0; i < reps; i++) will do
bits[j] = byte.Parse(r.Next(0, 255).ToString()); Again converting to string and back to byte the following is much better.
bits[j] = (byte)r.Next(0, 255);

i++;
if(i >= Int32.Parse(reps.ToString()))
{
break;
}

The above is not needed, the check is redundant since you do the check in
the for statement, and again you convert to string and parse the string.
Also although i++ would work here, but its most appropriate place would be
in the for loop itself.

Converting an integer to a string includes allocation of a new string, and
the process writing the string representation of int in the allocated
string. The allocation will make garbage collections occur more frequently.
Converting from string to int is parsing each character in the string, and
checking if it is a valid number and calculating the resulting string. These
2 steps harm the performance badly in your case, and

The bits array is allocated reps times, I would allocate it once outside the
outer loop and reuse it to avoid allocations and garbage collections. Also
the BinaryReader itself is not usefull in your case, because you are writing
a byte array which can be done with FileStream. And since BinaryReader and
FileStream classes provides buffers, so you don't need your own buffer. You
can use WriteByte and FileStream will take care of buffering.

The program would be

FileStream fs = new FileStream(fileLocation, FileMode.CreateNew);
Random r =new Random();
int iterations = reps * 500;
for (int i = 0; i < iterations;i++ )
{
fs.WriteByte((byte)r.Next(0, 255));
}
Nov 16 '05 #3
Forget this, as Sherif correctly states the Filestream is buffered, so you
shouldn't buffer yourself.
Applying the changes as pointed out by Sherif, a sustained transfer rate of
10MB/sec should be possible on a single drive.
25 MB will however require a RAID0 stripe set of 2-3 drives.

Willy.

"Willy Denoyette [MVP]" <wi*************@pandora.be> wrote in message
news:%2***************@TK2MSFTNGP11.phx.gbl...
Not sure what disk IO subsystem you are running this on, but don't ever
expect to achieve 25MB write throughput on anything less than a RAID0 (HW)
with a minimum of 7-8 spindles.
As a simple rule take a max. sustained IO rate of ~80-100 IO writes per
second per (7200 RPM) spindle.
So 500Kb per sec. is a fair number on a single drive with a block size of
500 bytes, what you should try first is to increase the buffer size to
something like 4-8 Kb, but don't expect higher throughputs than 2 - 3 Mb
per sec.

Willy.

Nov 16 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

103
by: Steven T. Hatton | last post by:
§27.4.2.1.4 Type ios_base::openmode Says this about the std::ios::binary openmode flag: *binary*: perform input and output in binary mode (as opposed to text mode) And that is basically _all_ it...
3
by: John R. Delaney | last post by:
I am running in debugging mode after a clean C++ compilation under .NET 2003. In a BIG loop (controlled many levels up in the call stack), I open a file with fopen using the "a" option. Then I write...
5
by: Neo | last post by:
Hello: I am receiving a Binary File in a Request from a application. The stream which comes to me has the boundary (Something like "---------------------------39<WBR>­0C0F3E0099" without the...
4
by: knapak | last post by:
Hello I'm a self instructed amateur attempting to read a huge file from disk... so bear with me please... I just learned that reading a file in binary is faster than text. So I wrote the...
3
by: nicolasg | last post by:
Hi, I'm trying to open a file (any file) in binary mode and save it inside a new text file. After that I want to read the source from the text file and save it back to the disk with its...
15
by: Jacques | last post by:
Hi I am an dotNet newby, so pardon my ignorance. I am looking for a method of saving/copying a managed class to a stream/file WITHOUT saving the object's state, eg. if I have a ref class with...
15
by: JoeC | last post by:
I am writing a program that I am trying to learn and save binary files. This is the page I found as a source: http://www.angelfire.com/country/aldev0/cpphowto/cpp_BinaryFileIO.html I have...
2
by: Thomi Aurel RUAG A | last post by:
Hy I'm using Python 2.4.2 on an ARM (PXA-270) platform (linux-2.6.17). My Goal is to write a list of bytes down to a file (opened in binary mode) in one cycle. The crux is that a '0x0a' (line...
13
by: zach | last post by:
Can someone help me out, I can't figure out what I'm doing wrong to write to a file in binary mode. What's wrong with my code? <?php $fileName = "something.dat"; $string = "This is a...
5
by: zehra.mb | last post by:
Hi, I had written application for storing employee data in binary file and reading those data from binary file and display it in C language. But I face some issue with writing data to binary file....
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.