IO optimization when copying bytes from one file to another

[Skwerl]

Hi guys. I've written code to embed an ICC profile in a TIFF image, and I
think my IO operations are slowing things down. It is taking about a second
to embed each tag in 7-meg TIFF files. Doesn't sound too bad until you try
doing it to 500 files. Basically what I am doing is this:

1. Read header from image, add 12-byte tag to header, and write to a new
TIFF file
2. Update all the offset pointers in other tags in TIFF header to reflect
the change that adding the 12-bytes made and write them to the new TIFF file
3. Copy the rest of the original TIFF to the new one.
4. Append ICC profile (around 100K) to new TIFF file.
5. Delete original TIFF and rename new one to the name of the original

I believe the roblem to be in the method I am using the copy the data from
the original file to the new one. I have pasted it below. Any suggestions
on how to squeeze some more speed out, either in my main algorithm or the
following function? Thanks a bunch!

Josh

private void copyBytes(FileS tream source, FileStream destination, long
fromIndex, long length)
{
const int chunkSize = 1024;
long currentIndex = fromIndex;
long endIndex = fromIndex + length;
long bytesToCopy = 0;
byte[] bytes = new byte[chunkSize];
byte[] endLump;
byte[] twoBytes = new byte[2];

source.Seek(fro mIndex, SeekOrigin.Begi n);
//Copy a chunk at a time
for(bytesToCopy = length; bytesToCopy >= chunkSize; bytesToCopy -=
chunkSize)
{
source.Read(byt es, 0, chunkSize);
destination.Wri te(bytes, 0, chunkSize);
currentIndex += bytes.Length;
}

//Copy the rest now
endLump = new byte[bytesToCopy];
source.Read(end Lump, 0, endLump.Length) ;
destination.Wri te(endLump, 0, endLump.Length) ;
destination.Flu sh();
}

[Jon Skeet [C# MVP]]

Skwerl <ju******@anoth erretarded.com> wrote:

Hi guys. I've written code to embed an ICC profile in a TIFF image, and I
think my IO operations are slowing things down. It is taking about a second
to embed each tag in 7-meg TIFF files. Doesn't sound too bad until you try
doing it to 500 files. Basically what I am doing is this:

1. Read header from image, add 12-byte tag to header, and write to a new
TIFF file
2. Update all the offset pointers in other tags in TIFF header to reflect
the change that adding the 12-bytes made and write them to the new TIFF file
3. Copy the rest of the original TIFF to the new one.
4. Append ICC profile (around 100K) to new TIFF file.
5. Delete original TIFF and rename new one to the name of the original

I believe the roblem to be in the method I am using the copy the data from
the original file to the new one. I have pasted it below. Any suggestions
on how to squeeze some more speed out, either in my main algorithm or the
following function? Thanks a bunch!

You can make your method simpler and more reliable (by using the return
value of Read) quite easily. I've also increased the chunk size to
possibly speed things up a bit.

const int BufferSize = 32768;

void CopyBytes (Stream source, Stream dest, long fromIndex,
long length)
{
source.Seek(fro mIndex, SeekOrigin.Begi n);
byte[] buffer = new byte[BufferSize];

while (length > 0)
{
int read = source.Read(buf fer, 0,
Math.Min (length, BufferSize));

if (read <= 0)
{
throw new IOException ("Insufficie nt data remaining");
}

dest.Write (buffer, 0, read);
length -= read;
}
}

Assuming you're going to call Close or Dispose on the destination
stream, chances are you don't need to call Flush by the way.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Skwerl]

Thanks, Jon. I appreciate the response. I tried your code, and found it was
about 15% slower than what I tried doing last night, which was to just read
it all in and then write all of it out. I'm guessing it was just because of
the small (7-meg) file size. here's my code:
private void copyBytes(FileS tream source, FileStream destination, long
fromIndex, long length)
{
byte[] buffer = new byte[length];
source.Seek(fro mIndex, SeekOrigin.Begi n);
source.Read(buf fer, 0, buffer.Length);
destination.Wri te(buffer, 0, buffer.Length);
}
"Jon Skeet [C# MVP]" wrote:

Skwerl <ju******@anoth erretarded.com> wrote:

Hi guys. I've written code to embed an ICC profile in a TIFF image, and I
think my IO operations are slowing things down. It is taking about a second
to embed each tag in 7-meg TIFF files. Doesn't sound too bad until you try
doing it to 500 files. Basically what I am doing is this:

1. Read header from image, add 12-byte tag to header, and write to a new
TIFF file
2. Update all the offset pointers in other tags in TIFF header to reflect
the change that adding the 12-bytes made and write them to the new TIFF file
3. Copy the rest of the original TIFF to the new one.
4. Append ICC profile (around 100K) to new TIFF file.
5. Delete original TIFF and rename new one to the name of the original

I believe the roblem to be in the method I am using the copy the data from
the original file to the new one. I have pasted it below. Any suggestions
on how to squeeze some more speed out, either in my main algorithm or the
following function? Thanks a bunch!

You can make your method simpler and more reliable (by using the return
value of Read) quite easily. I've also increased the chunk size to
possibly speed things up a bit.

const int BufferSize = 32768;

void CopyBytes (Stream source, Stream dest, long fromIndex,
long length)
{
source.Seek(fro mIndex, SeekOrigin.Begi n);
byte[] buffer = new byte[BufferSize];

while (length > 0)
{
int read = source.Read(buf fer, 0,
Math.Min (length, BufferSize));

if (read <= 0)
{
throw new IOException ("Insufficie nt data remaining");
}

dest.Write (buffer, 0, read);
length -= read;
}
}

Assuming you're going to call Close or Dispose on the destination
stream, chances are you don't need to call Flush by the way.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Jon Skeet [C# MVP]]

Skwerl <ju******@anoth erretarded.com> wrote:

Thanks, Jon. I appreciate the response. I tried your code, and found it was
about 15% slower than what I tried doing last night, which was to just read
it all in and then write all of it out. I'm guessing it was just because of
the small (7-meg) file size. here's my code:
private void copyBytes(FileS tream source, FileStream destination, long
fromIndex, long length)
{
byte[] buffer = new byte[length];
source.Seek(fro mIndex, SeekOrigin.Begi n);
source.Read(buf fer, 0, buffer.Length);
destination.Wri te(buffer, 0, buffer.Length);
}

Did you run mine and then run yours? If so, things would be buffered.
You should either flush all buffers before running either of them, or
run both several times.

I'd be surprised if my method was really 15% slower (a little bit, but
not 15%). Of course, I've been known to be surprised before :)

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Skwerl]

Actually, I ran mine a few times last night, and it consistently ran
around 730 ms per image in trials of 30 images each. I changed the code
altogether to yours this morning, ran it and got about 860 ms per image on
several trials of the same 30 images. I changed it back to mine as it was
previously, and again ended up with about 730 ms per image. I'm a real
novice when it comes to this sort of operation, and I've always seen these
things done in pieces like with your method, so I'm surprised the simplistic
way I did it is actually a little faster. Just FYI, the system I am running
this on is an XP SP2 with NTFS partitions. Performance aside, is your method
a better way to do this?
Thanks once again,
Josh

"Jon Skeet [C# MVP]" wrote:

Skwerl <ju******@anoth erretarded.com> wrote:

Thanks, Jon. I appreciate the response. I tried your code, and found it was
about 15% slower than what I tried doing last night, which was to just read
it all in and then write all of it out. I'm guessing it was just because of
the small (7-meg) file size. here's my code:
private void copyBytes(FileS tream source, FileStream destination, long
fromIndex, long length)
{
byte[] buffer = new byte[length];
source.Seek(fro mIndex, SeekOrigin.Begi n);
source.Read(buf fer, 0, buffer.Length);
destination.Wri te(buffer, 0, buffer.Length);
}

Did you run mine and then run yours? If so, things would be buffered.
You should either flush all buffers before running either of them, or
run both several times.

I'd be surprised if my method was really 15% slower (a little bit, but
not 15%). Of course, I've been known to be surprised before :)

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Jon Skeet [C# MVP]]

Skwerl <ju******@anoth erretarded.com> wrote:

Actually, I ran mine a few times last night, and it consistently ran
around 730 ms per image in trials of 30 images each. I changed the code
altogether to yours this morning, ran it and got about 860 ms per image on
several trials of the same 30 images. I changed it back to mine as it was
previously, and again ended up with about 730 ms per image. I'm a real
novice when it comes to this sort of operation, and I've always seen these
things done in pieces like with your method, so I'm surprised the simplistic
way I did it is actually a little faster. Just FYI, the system I am running
this on is an XP SP2 with NTFS partitions. Performance aside, is your method
a better way to do this?

Yes, in terms of memory consumption. Consider an image which is several
hundred megs in size - with my code, you never need to have more than
32K in memory at a time. With yours, you read the whole thing into
memory, and then write the whole thing out. You're also assuming that
one call to Read will read the whole file, ignoring the return value,
which is never a good idea.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Skwerl]

Yes, I am assuming that it will read it all. Yikes, I didn't realize that it
wouldn't always read it all, unless an excepteion were thrown. Under what
conditions would it not read all of the file? Performance is a big issue
here, so I want to try to guage whether or not I need to worry about this.
The code will never need to handle anything but 3-7 meg TIFF files. Thanks
once again, Jon.

Josh
"Jon Skeet [C# MVP]" wrote:

Skwerl <ju******@anoth erretarded.com> wrote:

Actually, I ran mine a few times last night, and it consistently ran
around 730 ms per image in trials of 30 images each. I changed the code
altogether to yours this morning, ran it and got about 860 ms per image on
several trials of the same 30 images. I changed it back to mine as it was
previously, and again ended up with about 730 ms per image. I'm a real
novice when it comes to this sort of operation, and I've always seen these
things done in pieces like with your method, so I'm surprised the simplistic
way I did it is actually a little faster. Just FYI, the system I am running
this on is an XP SP2 with NTFS partitions. Performance aside, is your method
a better way to do this?

Yes, in terms of memory consumption. Consider an image which is several
hundred megs in size - with my code, you never need to have more than
32K in memory at a time. With yours, you read the whole thing into
memory, and then write the whole thing out. You're also assuming that
one call to Read will read the whole file, ignoring the return value,
which is never a good idea.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Jon Skeet [C# MVP]]

Skwerl <ju******@anoth erretarded.com> wrote:

Yes, I am assuming that it will read it all. Yikes, I didn't realize that it
wouldn't always read it all, unless an excepteion were thrown. Under what
conditions would it not read all of the file?
I don't know, for sure. I would imagine that some network file systems
might give data in chunks, like NetworkStreams do. It could be that
FileStreams will always read however much you ask for - but that's not
true for streams in general. Basically it's good practice not to ignore
the return value of Read :)
Performance is a big issue
here, so I want to try to guage whether or not I need to worry about this.
The code will never need to handle anything but 3-7 meg TIFF files.

Why not try increasing the buffer size of the code I gave you to, say,
1MB. That way you won't need to change the code if you ever get a huge
file, and you don't need to worry about whether or not FileStream will
always return the whole of the data. The performance difference should
be trivial at that stage.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Skwerl]

That sounds perfectly reasonable. Thanks a bunch!

Josh

"Jon Skeet [C# MVP]" wrote:

Skwerl <ju******@anoth erretarded.com> wrote:

Yes, I am assuming that it will read it all. Yikes, I didn't realize that it
wouldn't always read it all, unless an excepteion were thrown. Under what
conditions would it not read all of the file?

I don't know, for sure. I would imagine that some network file systems
might give data in chunks, like NetworkStreams do. It could be that
FileStreams will always read however much you ask for - but that's not
true for streams in general. Basically it's good practice not to ignore
the return value of Read :)

Performance is a big issue
here, so I want to try to guage whether or not I need to worry about this.
The code will never need to handle anything but 3-7 meg TIFF files.

Why not try increasing the buffer size of the code I gave you to, say,
1MB. That way you won't need to change the code if you ever get a huge
file, and you don't need to worry about whether or not FileStream will
always return the whole of the data. The performance difference should
be trivial at that stage.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too