What's the best way to stream data to disk in enterprise application?

[Mike]

Hi All,

I'm writing an application that is accepting data from possibly 100's
of clients. The amount of data I get is around 3 or 4 KB per second
per client. The data that is streamed from the client can last
anywhere from a few seconds to maybe an hour.

My question is, with all this data coming in what is the best way to
write this information to disk. At the moment I am simply storing up
the data in a memory buffer and when the client disconnects I write
all the data to disk in one go. I am worried that writing it all in
one go might prove to be quite cpu intensive and also I am worried
that this won't scale because I am storing everything in memory until
the end of the session which means I could have alot of data in
memory.

I have thought about having smaller buffers which write the data out
when they get full, IE after every 1MB of data for example. Another
idea was to just stream the data out to disk as soon as it arrives.
Another idea I had was to add the session to a queue when it has
finished and have a low priority thread service this queue to write
the data to disk.

Any ideas would be appreciated and maybe some idea on the nest way to
code the idea would be good also.

Many thanks