Multithreaded file access

[Dan Kelley]

We have a multithreaded app that responds to events, and writes these
events to a text file. This text file is used by an external system
for further processing.

We want to be able to write multiple entries to the file, and then
rename it and copy it to a new location when the file reaches a
certian size, or a certain time span elapses (which ever comes first).

I have created a static/shared timer member variable in class that
writes to the file. When the file is written to, I check the size of
the file. If it is greater than the limit a funtion is called to
rename the file, and the timer is "reset".

My question is, in a multithreaded environment, how do I detect if the
file is being written to in the event the timer fires at the same time
and tries to rename the file. I could swallow the IO exception, sleep
for a while, and try again, but this cannot be the best solution.

Anyone have any ideas?

Thanks
Dan

[Miha Markic]

Hi Dan,

You should use some sort synchronization mechanism, such as using Monitor or
ReaderWriterLock or some other class.

--
Miha Markic - RightHand .NET consulting & software development
miha at rthand com

"Dan Kelley" <da***********@pershing.co.uk> wrote in message
news:7a**************************@posting.google.c om...

We have a multithreaded app that responds to events, and writes these
events to a text file. This text file is used by an external system
for further processing.

We want to be able to write multiple entries to the file, and then
rename it and copy it to a new location when the file reaches a
certian size, or a certain time span elapses (which ever comes first).

I have created a static/shared timer member variable in class that
writes to the file. When the file is written to, I check the size of
the file. If it is greater than the limit a funtion is called to
rename the file, and the timer is "reset".

My question is, in a multithreaded environment, how do I detect if the
file is being written to in the event the timer fires at the same time
and tries to rename the file. I could swallow the IO exception, sleep
for a while, and try again, but this cannot be the best solution.

Anyone have any ideas?

Thanks
Dan

[Dmitriy Lapshin [C# / .NET MVP]]

Dan,

I am afraid your architecture is wrong in the first place. Timers are
definitely not the best solution in a multi-threaded environment. I'd
suggest creating a separate thread instead. This thread would be responsible
*only* for managing the file. It would sleep most of the time, until a
dedicated AutoResetEvent is set indicating that an entry has arrived. The
thread will then wake up and check the file size first. If the size limit
has been reached, the thread will back up the old file and start a new one.
Then, the entry will be written to the appropriate file.

To enable time-controlled log file backup, the thread will be waiting for
the wake-up event with a timeout. If the timeout has occured, the thread
will check whether it's about time the file should have been backed up, and
if yes, backs up the file and starts a new one. Either way, it goes to the
sleep mode again until an entry arrives or the timeout occurs.

--
Dmitriy Lapshin [C# / .NET MVP]
X-Unity Test Studio
http://x-unity.miik.com.ua/teststudio.aspx
Bring the power of unit testing to VS .NET IDE

"Dan Kelley" <da***********@pershing.co.uk> wrote in message
news:7a**************************@posting.google.c om...

We have a multithreaded app that responds to events, and writes these
events to a text file. This text file is used by an external system
for further processing.

We want to be able to write multiple entries to the file, and then
rename it and copy it to a new location when the file reaches a
certian size, or a certain time span elapses (which ever comes first).

I have created a static/shared timer member variable in class that
writes to the file. When the file is written to, I check the size of
the file. If it is greater than the limit a funtion is called to
rename the file, and the timer is "reset".

My question is, in a multithreaded environment, how do I detect if the
file is being written to in the event the timer fires at the same time
and tries to rename the file. I could swallow the IO exception, sleep
for a while, and try again, but this cannot be the best solution.

Anyone have any ideas?

Thanks
Dan

[Leon Lambert]

I have responded to a question similar to this in the past. Do google
search for "lambertl File locking question"

Hope this helps
Leon Lambert

Dan Kelley wrote:

We have a multithreaded app that responds to events, and writes these
events to a text file. This text file is used by an external system
for further processing.

We want to be able to write multiple entries to the file, and then
rename it and copy it to a new location when the file reaches a
certian size, or a certain time span elapses (which ever comes first).

I have created a static/shared timer member variable in class that
writes to the file. When the file is written to, I check the size of
the file. If it is greater than the limit a funtion is called to
rename the file, and the timer is "reset".

My question is, in a multithreaded environment, how do I detect if the
file is being written to in the event the timer fires at the same time
and tries to rename the file. I could swallow the IO exception, sleep
for a while, and try again, but this cannot be the best solution.

Anyone have any ideas?

Thanks
Dan

[A Hirsi]

da***********@pershing.co.uk (Dan Kelley) wrote in message news:<7a**************************@posting.google. com>...

We have a multithreaded app that responds to events, and writes these
events to a text file. This text file is used by an external system
for further processing.

We want to be able to write multiple entries to the file, and then
rename it and copy it to a new location when the file reaches a
certian size, or a certain time span elapses (which ever comes first).

I have created a static/shared timer member variable in class that
writes to the file. When the file is written to, I check the size of
the file. If it is greater than the limit a funtion is called to
rename the file, and the timer is "reset".

My question is, in a multithreaded environment, how do I detect if the
file is being written to in the event the timer fires at the same time
and tries to rename the file. I could swallow the IO exception, sleep
for a while, and try again, but this cannot be the best solution.

Anyone have any ideas?

Thanks
Dan

Dan:
If you are not using the FileSystemWatcher class then I recommend it
for
checking the status of the file instead of writing your own routines.

You may need to synchronize your threads so that one fires off only
after
the other has completed its job. Look in MSDN for synchronizing
threads.
Another alternative may be writing to a file or table to keep track of
when
the writing operation is complete. The timer will only be fired if the
status of the file is complete.

Here is a pseudo code

--
Write to the text file the first time before the timer fires off
set timer.enable = true
timer.interval = 5000 ' 5 seconds
iStatus = GetStatus() ' read from db or some ini file
if iStatus = 1 then ' File has been written to
do something
remember to reset the value of iStatus
else
'do nothing until timer fires off again
end if
'

Hirsi

[Alvin Bruney]

This is still not an effective solution as the file system class uses
windows messaging to notify the monitor about file changing events. Needless
to say, with a queue system in a multithreaded environment, you are in for a
nightmare which grows as the queue extends - basically your file change
notification may be late, but this doesn't stop your threads from writing.
The architecture is faulty. Follow Dmitriy's guide. Putting bandaids on the
problem doesn't make it go away.

--
Regards,
Alvin Bruney
Got DotNet? Get it here...
http://www.networkip.net/dotnet/tidbits/default.htm
"A Hirsi" <mo********@yahoo.com> wrote in message
news:aa**************************@posting.google.c om...

da***********@pershing.co.uk (Dan Kelley) wrote in message

news:<7a**************************@posting.google. com>...

We have a multithreaded app that responds to events, and writes these
events to a text file. This text file is used by an external system
for further processing.

We want to be able to write multiple entries to the file, and then
rename it and copy it to a new location when the file reaches a
certian size, or a certain time span elapses (which ever comes first).

I have created a static/shared timer member variable in class that
writes to the file. When the file is written to, I check the size of
the file. If it is greater than the limit a funtion is called to
rename the file, and the timer is "reset".

My question is, in a multithreaded environment, how do I detect if the
file is being written to in the event the timer fires at the same time
and tries to rename the file. I could swallow the IO exception, sleep
for a while, and try again, but this cannot be the best solution.

Anyone have any ideas?

Thanks
Dan

Dan:
If you are not using the FileSystemWatcher class then I recommend it
for
checking the status of the file instead of writing your own routines.

You may need to synchronize your threads so that one fires off only
after
the other has completed its job. Look in MSDN for synchronizing
threads.
Another alternative may be writing to a file or table to keep track of
when
the writing operation is complete. The timer will only be fired if the
status of the file is complete.

Here is a pseudo code

--
Write to the text file the first time before the timer fires off
set timer.enable = true
timer.interval = 5000 ' 5 seconds
iStatus = GetStatus() ' read from db or some ini file
if iStatus = 1 then ' File has been written to
do something
remember to reset the value of iStatus
else
'do nothing until timer fires off again
end if
'

Hirsi

[Dan Kelley]

Thanks for all the suggestions. I have now figured this out. I cannot
see what the problem is with using System.Threading.Timers - we
currently make use of them in other multithreaded applications with no
problems.

I understood about the need to synchronise, but my problem was I was
originally creating an instance of a StreamWriter, passing in an
existing file name, and getting confused how to lock this method call
between the different threads. Instead what I have opted for is a
member level streamwriter, and when I need to write to it I call
Mutex.WaitOne (with an instance of a member level mutex) and perform
my writes, and then release.

The timer also uses the mutex to close the file, set the member
streamwriter to a new instance of a streamwriter and move the old
file.

I have tested this on a dual hyperthreaded processor and get
consistent results, so seems to work.

Thanks for all your help.
Dan

mo********@yahoo.com (A Hirsi) wrote in message news:<aa**************************@posting.google. com>...

da***********@pershing.co.uk (Dan Kelley) wrote in message news:<7a**************************@posting.google. com>...

We have a multithreaded app that responds to events, and writes these
events to a text file. This text file is used by an external system
for further processing.

We want to be able to write multiple entries to the file, and then
rename it and copy it to a new location when the file reaches a
certian size, or a certain time span elapses (which ever comes first).

I have created a static/shared timer member variable in class that
writes to the file. When the file is written to, I check the size of
the file. If it is greater than the limit a funtion is called to
rename the file, and the timer is "reset".

My question is, in a multithreaded environment, how do I detect if the
file is being written to in the event the timer fires at the same time
and tries to rename the file. I could swallow the IO exception, sleep
for a while, and try again, but this cannot be the best solution.

Anyone have any ideas?

Thanks
Dan

Dan:
If you are not using the FileSystemWatcher class then I recommend it
for
checking the status of the file instead of writing your own routines.

You may need to synchronize your threads so that one fires off only
after
the other has completed its job. Look in MSDN for synchronizing
threads.
Another alternative may be writing to a file or table to keep track of
when
the writing operation is complete. The timer will only be fired if the
status of the file is complete.

Here is a pseudo code

--
Write to the text file the first time before the timer fires off
set timer.enable = true
timer.interval = 5000 ' 5 seconds
iStatus = GetStatus() ' read from db or some ini file
if iStatus = 1 then ' File has been written to
do something
remember to reset the value of iStatus
else
'do nothing until timer fires off again
end if
'

Hirsi