"Stephan Steiner" <st*****@isuisse.com> wrote in message
news:%2****************@TK2MSFTNGP12.phx.gbl...
So what would be a suitable buffer size? I need to make a copy of one file
and append another file to it. The first file can be anywhere from a 100
MB to 2 GB, whereas the file to be appended will more likely be in the
10 - 100 MB area.
All you can do is measure, however, you should keep in mind that ALL file IO
using the Framework IO classes are buffered IO's, that means that
irrespective the buffer size you specify at the API level, the File System
will buffer reads and writes from/to disk in the FS cache. The amount of
bytes buffered depends on the used FS type (NTFS, FAT32, ....) and the usage
pattern (sequential, random, mixed).
So whether you read a byte or 256 KB at a time, the FS will always transfer
a block of data from the disk device to the FS cache. That means that the
transfer rate is theoretically determined by the speed of the physical IO
path, however transferring the data blocks to the FS cache and from the FS
cache further to the IO buffer in your application, means CPU overhead. It's
obvious that the smaller the buffers at the API level the larger the
overhead, buffers below a certain size will saturate the CPU, at which point
the IO rate becomes CPU bound.
So basically we have four determining factors for IO transfer rate:
1. Physical IO system (FS type, disk rotational speed, disk cache size, RAID
level ...)
2. CPU speed and number of...
3. Sequential or Random IO.
4. Buffer size.
I wrote a program to measure the impact of buffer size on the sequential IO
rate and measured the CPU consumption and IO count (logical) and transfer
speed.
Following are the results obtained reading a single large file (10GB) from a
single 10.000RPM, SATA drive (of course your mileage may vary).
blocksize = 16 bytes, cpu = 99,99% speed = 10,44 MB/s, IO = 684444/s
blocksize = 128 bytes, cpu = 78,31% speed = 57,89 MB/s, IO = 474199/s
blocksize = 256 bytes, cpu = 43,42% speed = 56,37 MB/s, IO = 230906/s
blocksize = 2 KB, cpu = 16,98% speed = 57,84 MB/s, IO = 29613/s
blocksize = 4 KB, cpu = 14,63% speed = 56,37 MB/s, IO = 14432/s
blocksize = 8 KB, cpu = 12,76% speed = 56,4 MB/s, IO = 7219/s
blocksize = 16 KB, cpu = 13,23% speed = 57,84 MB/s, IO = 3702/s
What does this tell us:
- The transfer speed is optimal at buffer sizes > 128 bytes, anything
smaller reduces the transfer speed to ~10MB/sec due to CPU saturation.
- Anything larger than 128 bytes doesn't increase the IO rate but reduces
the CPU consumption.
- CPU consumption stabilizes with >4KB buffers (Cache managers overhead). If
you want to further reduce CPU consumption, you will have to perform
unbuffered IO using PInvoke.
Conclusion, anything between 2KB and 8 KB gives you optimal results for both
IO transfer and CPU consumption. Bigger buffers are only a waste of memory,
too small buffers are a waste of CPU resources.
Willy.
