Improving perf of .NET 1.1 app that opens hundreds of files...

[David]

I have a VB.NET 1.1 app that must open/reopen and read hundreds (or sometimes
thousands) of large, multi-GB files and create one extremely large output
file (this is seismic data). I'm using the FileStream class for reading and
writing.

I need to improve its performance, so I first thought I would add my own
app-level buffering on the read-side. But that made it slower (and then I
realized that FileStream already did buffering...duh!)

Next I tried implementing a cache of open FileStream objects. In creating
its output, the app must close and re-open the same input files repeatedly
(extracting different data each time), so I was thinking that I could keep
some number of previously-opened FileStream objects (or references thereto)
in a list, in order to reduce the amount of closing/re-opening. But
surprisingly, that change also made the app slower. It's almost as if the CLR
or NTFS or Win (XP) is keeping recently-opened files - or at least
information about the files - around for a while, so when you re-reference
them, the re-open occurs very quickly (any opinions on whether this
conclusion is true???)

After a couple of weeks of experimentation and performance tests (and posts
to this and other ng's), I'm running out of ideas on how to make this app run
faster.

ANY advice would be greatly appreciated!

DT

[Kim Greenlee]

Hi David,

I’m guessing that you are running into a page fault problem. Run your
application and then open Task Manager.

1. On the Processes page select your application.
2. If you can’t see the Page Faults counter then...
3. From the File Menu select View -Select Columns and make sure the Page
Faults check box is checked.
4. Return to the Processes page and watch what is happening.

As your application is running if the Page Faults are going up steady and
quickly that is likely where your problem is.

Read this article for an overview of how files are read into memory:
http://www.microsoft.com/technet/pro.../wperfch7.mspx

If you have a page fault problem you have a couple choices to reduce the
number of page faults: change the software, upgrade the hardware, or both.
The choices available to you to programmatically reduce page faults are
application dependent. But basically you want to do as little file opening
and closing as possible. Especially with these large files. As for
hardware, you can load up on RAM, get a really fast hard drive, and make sure
that your buses are fast. The memory is pretty inexpensive these days, but
the bus is on the motherboard, so you maybe looking at a new machine.

Hope that helps you,

Kim Greenlee

--
digipede - Many legs make light work.
Grid computing for the real world.
http://www.digipede.net
http://krgreenlee.blogspot.net