473,230 Members | 1,930 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,230 software developers and data experts.

Binary file writing problem

Hi,
I've got a program (see source below) that makes a file and fills it with
random binary values (from 0 to 255). The source below works, however the
program creates files at a rate of about 0.5MB per second. There is a
serious performance issue with this program.
There is a loop within the main loop that generates a buffer of 500 Bytes
that are then all written at once.
Can anybody tell me how I can improve the performance of the program? Idealy
I could with a rate of about 25Mb/s.

Thanx Josh

('reps' is the number of repetitions. e.g. Number of bytes)

FileStream fs = new FileStream(fileLocation, FileMode.CreateNew);

// Create the writer for data.

BinaryWriter w = new BinaryWriter(fs);

Random r =new Random();

for (int i = 0; i < Int32.Parse(reps.ToString()); )

{

byte[] bits = new byte[500];

for (int j = 0; j <= 499; j++)

{

bits[j] = byte.Parse(r.Next(0, 255).ToString());

i++;

if(i >= Int32.Parse(reps.ToString()))

{

break;

}

}

w.Write(bits);

}
Nov 16 '05 #1
3 2691
Not sure what disk IO subsystem you are running this on, but don't ever
expect to achieve 25MB write throughput on anything less than a RAID0 (HW)
with a minimum of 7-8 spindles.
As a simple rule take a max. sustained IO rate of ~80-100 IO writes per
second per (7200 RPM) spindle.
So 500Kb per sec. is a fair number on a single drive with a block size of
500 bytes, what you should try first is to increase the buffer size to
something like 4-8 Kb, but don't expect higher throughputs than 2 - 3 Mb per
sec.

Willy.

source below) that makes a file and fills it with
random binary values (from 0 to 255). The source below works, however the
program creates files at a rate of about 0.5MB per second. There is a
serious performance issue with this program.
There is a loop within the main loop that generates a buffer of 500 Bytes
that are then all written at once.
Can anybody tell me how I can improve the performance of the program?
Idealy
I could with a rate of about 25Mb/s.

Thanx Josh

('reps' is the number of repetitions. e.g. Number of bytes)

FileStream fs = new FileStream(fileLocation, FileMode.CreateNew);

// Create the writer for data.

BinaryWriter w = new BinaryWriter(fs);

Random r =new Random();

for (int i = 0; i < Int32.Parse(reps.ToString()); )

{

byte[] bits = new byte[500];

for (int j = 0; j <= 499; j++)

{

bits[j] = byte.Parse(r.Next(0, 255).ToString());

i++;

if(i >= Int32.Parse(reps.ToString()))

{

break;

}

}

w.Write(bits);

}

Nov 16 '05 #2
Hello
for (int i = 0; i < Int32.Parse(reps.ToString()); ) I understand that reps is a variable of type int. why convert it to a string
and back to int again? and this takes place at every iteration.
for(int i = 0; i < reps; i++) will do
bits[j] = byte.Parse(r.Next(0, 255).ToString()); Again converting to string and back to byte the following is much better.
bits[j] = (byte)r.Next(0, 255);

i++;
if(i >= Int32.Parse(reps.ToString()))
{
break;
}

The above is not needed, the check is redundant since you do the check in
the for statement, and again you convert to string and parse the string.
Also although i++ would work here, but its most appropriate place would be
in the for loop itself.

Converting an integer to a string includes allocation of a new string, and
the process writing the string representation of int in the allocated
string. The allocation will make garbage collections occur more frequently.
Converting from string to int is parsing each character in the string, and
checking if it is a valid number and calculating the resulting string. These
2 steps harm the performance badly in your case, and

The bits array is allocated reps times, I would allocate it once outside the
outer loop and reuse it to avoid allocations and garbage collections. Also
the BinaryReader itself is not usefull in your case, because you are writing
a byte array which can be done with FileStream. And since BinaryReader and
FileStream classes provides buffers, so you don't need your own buffer. You
can use WriteByte and FileStream will take care of buffering.

The program would be

FileStream fs = new FileStream(fileLocation, FileMode.CreateNew);
Random r =new Random();
int iterations = reps * 500;
for (int i = 0; i < iterations;i++ )
{
fs.WriteByte((byte)r.Next(0, 255));
}
Nov 16 '05 #3
Forget this, as Sherif correctly states the Filestream is buffered, so you
shouldn't buffer yourself.
Applying the changes as pointed out by Sherif, a sustained transfer rate of
10MB/sec should be possible on a single drive.
25 MB will however require a RAID0 stripe set of 2-3 drives.

Willy.

"Willy Denoyette [MVP]" <wi*************@pandora.be> wrote in message
news:%2***************@TK2MSFTNGP11.phx.gbl...
Not sure what disk IO subsystem you are running this on, but don't ever
expect to achieve 25MB write throughput on anything less than a RAID0 (HW)
with a minimum of 7-8 spindles.
As a simple rule take a max. sustained IO rate of ~80-100 IO writes per
second per (7200 RPM) spindle.
So 500Kb per sec. is a fair number on a single drive with a block size of
500 bytes, what you should try first is to increase the buffer size to
something like 4-8 Kb, but don't expect higher throughputs than 2 - 3 Mb
per sec.

Willy.

Nov 16 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

103
by: Steven T. Hatton | last post by:
§27.4.2.1.4 Type ios_base::openmode Says this about the std::ios::binary openmode flag: *binary*: perform input and output in binary mode (as opposed to text mode) And that is basically _all_ it...
3
by: John R. Delaney | last post by:
I am running in debugging mode after a clean C++ compilation under .NET 2003. In a BIG loop (controlled many levels up in the call stack), I open a file with fopen using the "a" option. Then I write...
5
by: Neo | last post by:
Hello: I am receiving a Binary File in a Request from a application. The stream which comes to me has the boundary (Something like "---------------------------39<WBR>­0C0F3E0099" without the...
4
by: knapak | last post by:
Hello I'm a self instructed amateur attempting to read a huge file from disk... so bear with me please... I just learned that reading a file in binary is faster than text. So I wrote the...
3
by: nicolasg | last post by:
Hi, I'm trying to open a file (any file) in binary mode and save it inside a new text file. After that I want to read the source from the text file and save it back to the disk with its...
15
by: Jacques | last post by:
Hi I am an dotNet newby, so pardon my ignorance. I am looking for a method of saving/copying a managed class to a stream/file WITHOUT saving the object's state, eg. if I have a ref class with...
15
by: JoeC | last post by:
I am writing a program that I am trying to learn and save binary files. This is the page I found as a source: http://www.angelfire.com/country/aldev0/cpphowto/cpp_BinaryFileIO.html I have...
2
by: Thomi Aurel RUAG A | last post by:
Hy I'm using Python 2.4.2 on an ARM (PXA-270) platform (linux-2.6.17). My Goal is to write a list of bytes down to a file (opened in binary mode) in one cycle. The crux is that a '0x0a' (line...
13
by: zach | last post by:
Can someone help me out, I can't figure out what I'm doing wrong to write to a file in binary mode. What's wrong with my code? <?php $fileName = "something.dat"; $string = "This is a...
5
by: zehra.mb | last post by:
Hi, I had written application for storing employee data in binary file and reading those data from binary file and display it in C language. But I face some issue with writing data to binary file....
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.