Timer + High CPU Usage

[LBT]

I have a window service written using VB.NET. This window service will scan
folders for file and grab the file content to be inserted to SQL Server on
file detection. There are altogether 18 folders to be watched. Each folder is
assigned with a timer for watching purpose. Hence there will be 18 timers and
each timer is set to elapse on every second.

Problem here, once the window service is installed and started, the CPU
usage is very high (take about 10% of overall CPU usage at a four 2GHz
processors server even when there is no file being sent in for processing).
This highly comsumption of CPU power is because of the timers elapsed event?
Any idea to improve it? Any advice and comment would be much appreciated.

[Tom Shelton]

On 2004-12-01, LBT <LB*@discussions.microsoft.com> wrote:

I have a window service written using VB.NET. This window service will scan
folders for file and grab the file content to be inserted to SQL Server on
file detection. There are altogether 18 folders to be watched. Each folder is
assigned with a timer for watching purpose. Hence there will be 18 timers and
each timer is set to elapse on every second.

Problem here, once the window service is installed and started, the CPU
usage is very high (take about 10% of overall CPU usage at a four 2GHz
processors server even when there is no file being sent in for processing).
This highly comsumption of CPU power is because of the timers elapsed event?
More then likely what your doing in the event... How often are the
timers firing? Which timer class are you using?
Any idea to improve it? Any advice and comment would be much appreciated.

Have you looked at the FileSystemWatcher class in System.IO. You
shouldn't need timers and polling this way...

--
Tom Shelton [MVP]

[LBT]

Thanks for the reply.

I have used the Timer under "Components" tab located in the .NET IDE Toolbox.

I did use the FileSystemWatcher class before but the class somehow have
limitation. Whenever SQL Server is down or offline, the file that being
halfway processed or subsequent file sent in to the folder will not be
processed as FileSystemWatcher only provides method such as OnCreated,
OnModified, OnDeleted, etc. The file will be stuck inside the folder without
processed after all. Hence I have changed the mechanism by always scanning
folder for file using timer.

"Tom Shelton" wrote:

On 2004-12-01, LBT <LB*@discussions.microsoft.com> wrote:

I have a window service written using VB.NET. This window service will scan
folders for file and grab the file content to be inserted to SQL Server on
file detection. There are altogether 18 folders to be watched. Each folder is
assigned with a timer for watching purpose. Hence there will be 18 timers and
each timer is set to elapse on every second.

Problem here, once the window service is installed and started, the CPU
usage is very high (take about 10% of overall CPU usage at a four 2GHz
processors server even when there is no file being sent in for processing).
This highly comsumption of CPU power is because of the timers elapsed event?

More then likely what your doing in the event... How often are the
timers firing? Which timer class are you using?

Any idea to improve it? Any advice and comment would be much appreciated.

Have you looked at the FileSystemWatcher class in System.IO. You
shouldn't need timers and polling this way...

--
Tom Shelton [MVP]

[Tom Shelton]

In article <6F**********************************@microsoft.co m>, LBT wrote:

Thanks for the reply.

I have used the Timer under "Components" tab located in the .NET IDE Toolbox.

I did use the FileSystemWatcher class before but the class somehow have
limitation. Whenever SQL Server is down or offline, the file that being
halfway processed or subsequent file sent in to the folder will not be
processed as FileSystemWatcher only provides method such as OnCreated,
OnModified, OnDeleted, etc. The file will be stuck inside the folder without
processed after all. Hence I have changed the mechanism by always scanning
folder for file using timer.

You should still be able to use filesystemwatcher... You simply need to
queue failed requests - then you would only need one timer. That timer
would check periodically to see if there were files that needed to be
processed in the failed queue.

private sub watcher_Created (byval sender as object, byval e as
filesystemeventargs e)

try
attempt to process file

catch ex as exception
we couldn't process - so
failedqueue.add (file)
end try
end sub

private sub timer_elapsed (....)
try
attempt to process files in queue
catch ex as excpetion
add current file back to queue
failedqueue.add (file)
end try
end sub

make sense?

--
Tom Shelton [MVP]
OS Name: Microsoft Windows XP Professional
OS Version: 5.1.2600 Service Pack 2 Build 2600
System Up Time: 39 Days, 19 Hours, 23 Minutes, 47 Seconds

[LBT]

Appreciated for the suggestion :) But I still have doubt and wish that Tom
could provide me some valuable idea.

I think I will explain some logic flow of my window service in order to have
a better understanding to cope my hassle.

The window service needs to process files from 16 directories. Each folder
has its own file format. There will be no limitation on the number of files
that can be sent in to a folder at one time. But only 1 file is allowed to
process at one time (sequence of file based on creation datetime is very
important). I might need a queue table to stack the records (file info)
before each file get its turn to be processed. The role of the window service
is to obtain the file content and insert to a SQL table as a record. In the
SQL table, there is a INSERT trigger which will be invoked and execute a
stored procedure to carry out validation towards the record. At the last
portion of the stored procedure, it will update the queued record to
"FINISH". When WS detect for this "FINISH" status, it will move the file to
archive folder and processing for 1 file is considered finish.

Question here, if I use the FileSystemWatcher class, it will help me to put
the file info to the queue table with the help of OnCreated event. But I
might still need a timer to process the queue table to ensure that file is
processed in sequence and only one file is processed at one time.

Currently, I use a timer to scan directory. If no file is detected, the
process will stop and wait for next elapsed event. If file(s) detected, all
file info as grabbed at that time will be put to a queue table. All record(s)
inside the queue table should be finish processed before subsequent file
exists in a folder would be catered again. In order words, the timer elapsed
event will not scan directory for file until queue table is finish processed
(controlled using a boolean flag).

As what can be seen, timer should be there for first method
(FileSystemWatcher + Timer) and second method which I used for this moment.
Timer seem can't be escaped from using for both methods and the use of timer
would again increase the CPU usage when all timers elapsed at the same time.
For yuor information, timer interval is set to 1 second.

Any way I can improve so that the CPU usage will not reach the critical
level? What I've tested shows that the CPU usage is considered high when
there is no file being sent in to any of the folders. The CPU usage will
become higher when there is file sent in for processing. Any weakness in my
existing file processing mechanism? Any better workaround?

Thanks a lot

[Michael D. Ober]

Uset the FileSystemWatcher to tell you when a new file has been created.
Then use the same scanning algorithm you currently use. By using the
FileSystemWatcher, you eliminate both excessive scans and delayed scans.
Once you have completed scanning, then restart start the FileSystemWatcher.

Mike Ober.

"LBT" <LB*@discussions.microsoft.com> wrote in message
news:78**********************************@microsof t.com...

Appreciated for the suggestion :) But I still have doubt and wish that Tom
could provide me some valuable idea.

I think I will explain some logic flow of my window service in order to have a better understanding to cope my hassle.

The window service needs to process files from 16 directories. Each folder
has its own file format. There will be no limitation on the number of files that can be sent in to a folder at one time. But only 1 file is allowed to
process at one time (sequence of file based on creation datetime is very
important). I might need a queue table to stack the records (file info)
before each file get its turn to be processed. The role of the window service is to obtain the file content and insert to a SQL table as a record. In the SQL table, there is a INSERT trigger which will be invoked and execute a
stored procedure to carry out validation towards the record. At the last
portion of the stored procedure, it will update the queued record to
"FINISH". When WS detect for this "FINISH" status, it will move the file to archive folder and processing for 1 file is considered finish.

Question here, if I use the FileSystemWatcher class, it will help me to put the file info to the queue table with the help of OnCreated event. But I
might still need a timer to process the queue table to ensure that file is
processed in sequence and only one file is processed at one time.

Currently, I use a timer to scan directory. If no file is detected, the
process will stop and wait for next elapsed event. If file(s) detected, all file info as grabbed at that time will be put to a queue table. All record(s) inside the queue table should be finish processed before subsequent file
exists in a folder would be catered again. In order words, the timer elapsed event will not scan directory for file until queue table is finish processed (controlled using a boolean flag).

As what can be seen, timer should be there for first method
(FileSystemWatcher + Timer) and second method which I used for this moment. Timer seem can't be escaped from using for both methods and the use of timer would again increase the CPU usage when all timers elapsed at the same time. For yuor information, timer interval is set to 1 second.

Any way I can improve so that the CPU usage will not reach the critical
level? What I've tested shows that the CPU usage is considered high when
there is no file being sent in to any of the folders. The CPU usage will
become higher when there is file sent in for processing. Any weakness in my existing file processing mechanism? Any better workaround?

Thanks a lot

[LBT]

Currently I used the System.IO.Directory.GetFiles method to scan directory.
Is it the method will cause lot of undesired overhead? The use of
FileSystemWatcher will be able to eliminate this?

Thanks

"Michael D. Ober" wrote:

Uset the FileSystemWatcher to tell you when a new file has been created.
Then use the same scanning algorithm you currently use. By using the
FileSystemWatcher, you eliminate both excessive scans and delayed scans.
Once you have completed scanning, then restart start the FileSystemWatcher.

Mike Ober.

"LBT" <LB*@discussions.microsoft.com> wrote in message
news:78**********************************@microsof t.com...

Appreciated for the suggestion :) But I still have doubt and wish that Tom
could provide me some valuable idea.

I think I will explain some logic flow of my window service in order to

have

a better understanding to cope my hassle.

The window service needs to process files from 16 directories. Each folder
has its own file format. There will be no limitation on the number of

files

that can be sent in to a folder at one time. But only 1 file is allowed to
process at one time (sequence of file based on creation datetime is very
important). I might need a queue table to stack the records (file info)
before each file get its turn to be processed. The role of the window

service

is to obtain the file content and insert to a SQL table as a record. In

the

SQL table, there is a INSERT trigger which will be invoked and execute a
stored procedure to carry out validation towards the record. At the last
portion of the stored procedure, it will update the queued record to
"FINISH". When WS detect for this "FINISH" status, it will move the file

to

archive folder and processing for 1 file is considered finish.

Question here, if I use the FileSystemWatcher class, it will help me to

put

the file info to the queue table with the help of OnCreated event. But I
might still need a timer to process the queue table to ensure that file is
processed in sequence and only one file is processed at one time.

Currently, I use a timer to scan directory. If no file is detected, the
process will stop and wait for next elapsed event. If file(s) detected,

all

file info as grabbed at that time will be put to a queue table. All

record(s)

inside the queue table should be finish processed before subsequent file
exists in a folder would be catered again. In order words, the timer

elapsed

event will not scan directory for file until queue table is finish

processed

(controlled using a boolean flag).

As what can be seen, timer should be there for first method
(FileSystemWatcher + Timer) and second method which I used for this

moment.

Timer seem can't be escaped from using for both methods and the use of

timer

would again increase the CPU usage when all timers elapsed at the same

time.

For yuor information, timer interval is set to 1 second.

Any way I can improve so that the CPU usage will not reach the critical
level? What I've tested shows that the CPU usage is considered high when
there is no file being sent in to any of the folders. The CPU usage will
become higher when there is file sent in for processing. Any weakness in

my

existing file processing mechanism? Any better workaround?

Thanks a lot

[Fredrik Melin]

Hi,

I made a form, adding this code

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As
System.EventArgs) Handles Button1.Click
Dim oTimer(19) As MyTimer
For I As Integer = 0 To 19
oTimer(I) = New MyTimer
oTimer(I).Interval = 1000
oTimer(I).nID = I

AddHandler oTimer(I).Elapsed, AddressOf oTimer_Elapsed
oTimer(I).Start()
Next
End Sub

Public Sub oTimer_Elapsed(ByVal sender As Object, ByVal e As
System.Timers.ElapsedEventArgs)
Dim sArray() As String = System.IO.Directory.GetFiles("C:\T\" &
CType(sender, MyTimer).nID & "\", "*.*")
If Not sArray Is Nothing Then
If sArray.Length > 0 Then
Debug.WriteLine("Found file...")
End If
End If

End Sub

Public Class MyTimer
Inherits System.Timers.Timer

Private m_nID As Integer
Public Property nID() As Integer
Get
Return m_nID
End Get
Set(ByVal Value As Integer)
m_nID = Value
End Set
End Property
End Class

When running it (made 20 directories) it takes zero % processor power.

Also, a word of caution, a mistake that I made a while ago, if you create a
new timer and dont stop the old one (or run out of scope) it still elapes,
and when you have several thousand, the processor will start feeling it..
- Fredrik Melin
"LBT" <LB*@discussions.microsoft.com> wrote in message
news:E4**********************************@microsof t.com...

I have a window service written using VB.NET. This window service will scan
folders for file and grab the file content to be inserted to SQL Server on
file detection. There are altogether 18 folders to be watched. Each folder
is
assigned with a timer for watching purpose. Hence there will be 18 timers
and
each timer is set to elapse on every second.

Problem here, once the window service is installed and started, the CPU
usage is very high (take about 10% of overall CPU usage at a four 2GHz
processors server even when there is no file being sent in for
processing).
This highly comsumption of CPU power is because of the timers elapsed
event?
Any idea to improve it? Any advice and comment would be much appreciated.