473,326 Members | 2,182 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

Multithreaded file access

We have a multithreaded app that responds to events, and writes these
events to a text file. This text file is used by an external system
for further processing.

We want to be able to write multiple entries to the file, and then
rename it and copy it to a new location when the file reaches a
certian size, or a certain time span elapses (which ever comes first).

I have created a static/shared timer member variable in class that
writes to the file. When the file is written to, I check the size of
the file. If it is greater than the limit a funtion is called to
rename the file, and the timer is "reset".

My question is, in a multithreaded environment, how do I detect if the
file is being written to in the event the timer fires at the same time
and tries to rename the file. I could swallow the IO exception, sleep
for a while, and try again, but this cannot be the best solution.

Anyone have any ideas?

Thanks
Dan
Jul 21 '05 #1
6 2757
Hi Dan,

You should use some sort synchronization mechanism, such as using Monitor or
ReaderWriterLock or some other class.

--
Miha Markic - RightHand .NET consulting & software development
miha at rthand com

"Dan Kelley" <da***********@pershing.co.uk> wrote in message
news:7a**************************@posting.google.c om...
We have a multithreaded app that responds to events, and writes these
events to a text file. This text file is used by an external system
for further processing.

We want to be able to write multiple entries to the file, and then
rename it and copy it to a new location when the file reaches a
certian size, or a certain time span elapses (which ever comes first).

I have created a static/shared timer member variable in class that
writes to the file. When the file is written to, I check the size of
the file. If it is greater than the limit a funtion is called to
rename the file, and the timer is "reset".

My question is, in a multithreaded environment, how do I detect if the
file is being written to in the event the timer fires at the same time
and tries to rename the file. I could swallow the IO exception, sleep
for a while, and try again, but this cannot be the best solution.

Anyone have any ideas?

Thanks
Dan

Jul 21 '05 #2
Dan,

I am afraid your architecture is wrong in the first place. Timers are
definitely not the best solution in a multi-threaded environment. I'd
suggest creating a separate thread instead. This thread would be responsible
*only* for managing the file. It would sleep most of the time, until a
dedicated AutoResetEvent is set indicating that an entry has arrived. The
thread will then wake up and check the file size first. If the size limit
has been reached, the thread will back up the old file and start a new one.
Then, the entry will be written to the appropriate file.

To enable time-controlled log file backup, the thread will be waiting for
the wake-up event with a timeout. If the timeout has occured, the thread
will check whether it's about time the file should have been backed up, and
if yes, backs up the file and starts a new one. Either way, it goes to the
sleep mode again until an entry arrives or the timeout occurs.

--
Dmitriy Lapshin [C# / .NET MVP]
X-Unity Test Studio
http://x-unity.miik.com.ua/teststudio.aspx
Bring the power of unit testing to VS .NET IDE

"Dan Kelley" <da***********@pershing.co.uk> wrote in message
news:7a**************************@posting.google.c om...
We have a multithreaded app that responds to events, and writes these
events to a text file. This text file is used by an external system
for further processing.

We want to be able to write multiple entries to the file, and then
rename it and copy it to a new location when the file reaches a
certian size, or a certain time span elapses (which ever comes first).

I have created a static/shared timer member variable in class that
writes to the file. When the file is written to, I check the size of
the file. If it is greater than the limit a funtion is called to
rename the file, and the timer is "reset".

My question is, in a multithreaded environment, how do I detect if the
file is being written to in the event the timer fires at the same time
and tries to rename the file. I could swallow the IO exception, sleep
for a while, and try again, but this cannot be the best solution.

Anyone have any ideas?

Thanks
Dan


Jul 21 '05 #3
I have responded to a question similar to this in the past. Do google
search for "lambertl File locking question"

Hope this helps
Leon Lambert

Dan Kelley wrote:
We have a multithreaded app that responds to events, and writes these
events to a text file. This text file is used by an external system
for further processing.

We want to be able to write multiple entries to the file, and then
rename it and copy it to a new location when the file reaches a
certian size, or a certain time span elapses (which ever comes first).

I have created a static/shared timer member variable in class that
writes to the file. When the file is written to, I check the size of
the file. If it is greater than the limit a funtion is called to
rename the file, and the timer is "reset".

My question is, in a multithreaded environment, how do I detect if the
file is being written to in the event the timer fires at the same time
and tries to rename the file. I could swallow the IO exception, sleep
for a while, and try again, but this cannot be the best solution.

Anyone have any ideas?

Thanks
Dan


Jul 21 '05 #4
da***********@pershing.co.uk (Dan Kelley) wrote in message news:<7a**************************@posting.google. com>...
We have a multithreaded app that responds to events, and writes these
events to a text file. This text file is used by an external system
for further processing.

We want to be able to write multiple entries to the file, and then
rename it and copy it to a new location when the file reaches a
certian size, or a certain time span elapses (which ever comes first).

I have created a static/shared timer member variable in class that
writes to the file. When the file is written to, I check the size of
the file. If it is greater than the limit a funtion is called to
rename the file, and the timer is "reset".

My question is, in a multithreaded environment, how do I detect if the
file is being written to in the event the timer fires at the same time
and tries to rename the file. I could swallow the IO exception, sleep
for a while, and try again, but this cannot be the best solution.

Anyone have any ideas?

Thanks
Dan

Dan:
If you are not using the FileSystemWatcher class then I recommend it
for
checking the status of the file instead of writing your own routines.

You may need to synchronize your threads so that one fires off only
after
the other has completed its job. Look in MSDN for synchronizing
threads.
Another alternative may be writing to a file or table to keep track of
when
the writing operation is complete. The timer will only be fired if the
status of the file is complete.

Here is a pseudo code

--
Write to the text file the first time before the timer fires off
set timer.enable = true
timer.interval = 5000 ' 5 seconds
iStatus = GetStatus() ' read from db or some ini file
if iStatus = 1 then ' File has been written to
do something
remember to reset the value of iStatus
else
'do nothing until timer fires off again
end if
'

Hirsi
Jul 21 '05 #5
This is still not an effective solution as the file system class uses
windows messaging to notify the monitor about file changing events. Needless
to say, with a queue system in a multithreaded environment, you are in for a
nightmare which grows as the queue extends - basically your file change
notification may be late, but this doesn't stop your threads from writing.
The architecture is faulty. Follow Dmitriy's guide. Putting bandaids on the
problem doesn't make it go away.

--
Regards,
Alvin Bruney
Got DotNet? Get it here...
http://www.networkip.net/dotnet/tidbits/default.htm
"A Hirsi" <mo********@yahoo.com> wrote in message
news:aa**************************@posting.google.c om...
da***********@pershing.co.uk (Dan Kelley) wrote in message

news:<7a**************************@posting.google. com>...
We have a multithreaded app that responds to events, and writes these
events to a text file. This text file is used by an external system
for further processing.

We want to be able to write multiple entries to the file, and then
rename it and copy it to a new location when the file reaches a
certian size, or a certain time span elapses (which ever comes first).

I have created a static/shared timer member variable in class that
writes to the file. When the file is written to, I check the size of
the file. If it is greater than the limit a funtion is called to
rename the file, and the timer is "reset".

My question is, in a multithreaded environment, how do I detect if the
file is being written to in the event the timer fires at the same time
and tries to rename the file. I could swallow the IO exception, sleep
for a while, and try again, but this cannot be the best solution.

Anyone have any ideas?

Thanks
Dan

Dan:
If you are not using the FileSystemWatcher class then I recommend it
for
checking the status of the file instead of writing your own routines.

You may need to synchronize your threads so that one fires off only
after
the other has completed its job. Look in MSDN for synchronizing
threads.
Another alternative may be writing to a file or table to keep track of
when
the writing operation is complete. The timer will only be fired if the
status of the file is complete.

Here is a pseudo code

--
Write to the text file the first time before the timer fires off
set timer.enable = true
timer.interval = 5000 ' 5 seconds
iStatus = GetStatus() ' read from db or some ini file
if iStatus = 1 then ' File has been written to
do something
remember to reset the value of iStatus
else
'do nothing until timer fires off again
end if
'

Hirsi

Jul 21 '05 #6
Thanks for all the suggestions. I have now figured this out. I cannot
see what the problem is with using System.Threading.Timers - we
currently make use of them in other multithreaded applications with no
problems.

I understood about the need to synchronise, but my problem was I was
originally creating an instance of a StreamWriter, passing in an
existing file name, and getting confused how to lock this method call
between the different threads. Instead what I have opted for is a
member level streamwriter, and when I need to write to it I call
Mutex.WaitOne (with an instance of a member level mutex) and perform
my writes, and then release.

The timer also uses the mutex to close the file, set the member
streamwriter to a new instance of a streamwriter and move the old
file.

I have tested this on a dual hyperthreaded processor and get
consistent results, so seems to work.

Thanks for all your help.
Dan

mo********@yahoo.com (A Hirsi) wrote in message news:<aa**************************@posting.google. com>...
da***********@pershing.co.uk (Dan Kelley) wrote in message news:<7a**************************@posting.google. com>...
We have a multithreaded app that responds to events, and writes these
events to a text file. This text file is used by an external system
for further processing.

We want to be able to write multiple entries to the file, and then
rename it and copy it to a new location when the file reaches a
certian size, or a certain time span elapses (which ever comes first).

I have created a static/shared timer member variable in class that
writes to the file. When the file is written to, I check the size of
the file. If it is greater than the limit a funtion is called to
rename the file, and the timer is "reset".

My question is, in a multithreaded environment, how do I detect if the
file is being written to in the event the timer fires at the same time
and tries to rename the file. I could swallow the IO exception, sleep
for a while, and try again, but this cannot be the best solution.

Anyone have any ideas?

Thanks
Dan

Dan:
If you are not using the FileSystemWatcher class then I recommend it
for
checking the status of the file instead of writing your own routines.

You may need to synchronize your threads so that one fires off only
after
the other has completed its job. Look in MSDN for synchronizing
threads.
Another alternative may be writing to a file or table to keep track of
when
the writing operation is complete. The timer will only be fired if the
status of the file is complete.

Here is a pseudo code

--
Write to the text file the first time before the timer fires off
set timer.enable = true
timer.interval = 5000 ' 5 seconds
iStatus = GetStatus() ' read from db or some ini file
if iStatus = 1 then ' File has been written to
do something
remember to reset the value of iStatus
else
'do nothing until timer fires off again
end if
'

Hirsi

Jul 21 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: Fernando Rodríguez | last post by:
Hi, I was trying to step through a first attemp of multithreaded script (see below), with PythonWin, but I'm gettign a very strange error: ...
3
by: Anthony | last post by:
Hi, How does (Can?) the stream mechanism work in multithreaded programs? I want to implement cout in my multithreaded program.
2
by: pradyumna | last post by:
In Project settins - C/C++ - Code Generation, what is the difference between the option "Multithreaded" and "Multithreaded DLL". I understand that on selecting multithreaded option, single and...
1
by: daniel.bron | last post by:
I'm maintaining a C++ application written by a developer who has now left. The app is a multithreaded client/server app for financial data. My compiler is MS visual C++ 6.0. I'm a C++ neophyte....
7
by: Pavils Jurjans | last post by:
Hello, I wanted to get some light in the subject. As I develop ASP.NET applications, it's necessary to understand how exactly the server- communication happens. It seems like initially...
6
by: Dan Kelley | last post by:
We have a multithreaded app that responds to events, and writes these events to a text file. This text file is used by an external system for further processing. We want to be able to write...
2
by: Alan Kemp | last post by:
Hi, I have a problem that is half python, half design. I have a multithreaded network server working, each client request spawns a new thread which deals with that client for as long as it is...
9
by: nicolas.michel.lava | last post by:
Hi there, I have some trouble using STL containers. I'm working on a multithreaded (pthread) application making heavy usage of STL containers. All accesses to containers are made in locked...
6
by: Stephen Carson | last post by:
I'm trying to build a Web Service that will kick off threads as logging requests come in. These threads will then log to the database. I have been able to make a simple Web Service. I have been...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.