473,545 Members | 2,688 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

System.IO.FileS ystemWatcher is missing alot of files.

I have a Windows Form application that monitors a directory using the
System.IO.FileS ystemWatcher.

On the event created I run a sub that sends an xml file over http to a
service.

When I copy 7000 files in the directory to be monitored only 123 are sent to
the service.

What might be going wrong?

More info:

The sending of the XML is more then just the http post. I also have to do
some stuff with xml first before sending it.
Feb 7 '06 #1
5 5126
"Philip Wagenaar" <ph************ *@online.nospam > wrote in message news:1A******** *************** ***********@mic rosoft.com...
I have a Windows Form application that monitors a directory using the
System.IO.FileS ystemWatcher.

On the event created I run a sub that sends an xml file over http to a
service.

When I copy 7000 files in the directory to be monitored only 123 are sent to
the service.

What might be going wrong?

More info:

The sending of the XML is more then just the http post. I also have to do
some stuff with xml first before sending it.


It's possible that you are missing events because it takes longer to process the files than it takes to generate the events and thus
some get lost. I had a similar situation using the FileSystemWatch er and I solved it by placing the event info into a FIFO and had
a separate worker thread unloading the FIFO and processing the events. This completely eliminated the problem. Perhaps this
approach would be appropriate here as well.
--
Al Reid
Feb 7 '06 #2
You mean working with threads?

Is it possible to post a sample?

"Al Reid" wrote:
"Philip Wagenaar" <ph************ *@online.nospam > wrote in message news:1A******** *************** ***********@mic rosoft.com...
I have a Windows Form application that monitors a directory using the
System.IO.FileS ystemWatcher.

On the event created I run a sub that sends an xml file over http to a
service.

When I copy 7000 files in the directory to be monitored only 123 are sent to
the service.

What might be going wrong?

More info:

The sending of the XML is more then just the http post. I also have to do
some stuff with xml first before sending it.


It's possible that you are missing events because it takes longer to process the files than it takes to generate the events and thus
some get lost. I had a similar situation using the FileSystemWatch er and I solved it by placing the event info into a FIFO and had
a separate worker thread unloading the FIFO and processing the events. This completely eliminated the problem. Perhaps this
approach would be appropriate here as well.
--
Al Reid

Feb 7 '06 #3
"Philip Wagenaar" <ph************ *@online.nospam > wrote in message news:5F******** *************** ***********@mic rosoft.com...
You mean working with threads?

Is it possible to post a sample?

"Al Reid" wrote:


Yes, I mean working with threads. If I get a chance, I'll see if I can't skinny down my working code and post the basics.

--
Al Reid

It's possible that you are missing events because it takes longer to process the files than it takes to generate the events and thus some get lost. I had a similar situation using the FileSystemWatch er and I solved it by placing the event info into a FIFO and had a separate worker thread unloading the FIFO and processing the events. This completely eliminated the problem. Perhaps this
approach would be appropriate here as well.
--
Al Reid

Feb 7 '06 #4
"Philip Wagenaar" <ph************ *@online.nospam > wrote in message news:5F******** *************** ***********@mic rosoft.com...
You mean working with threads?

Is it possible to post a sample?


I found my original proof of concept code. It was implemented as a Windows Service. Most of the concepts were obtained from Google
searches, etc.

=============== =============== =============== ====

Imports System

Imports System.Threadin g

Module General

Public goFIFO As New Collections.Que ue

Public gblnStopThreads As Boolean

Public gstrSourceDir As String

Public gstrDestination Dir As String

Public goLogger As New CLog

Public goEventlog As EventLog

Public gintIdle As Integer

Public gintInterval As Integer

End Module

-----------------------------

Imports System.Threadin g

Imports System.io

Public Class WorkerThread

Public Shared Sub ProcessFiles()

Dim e As Object

Dim strFilePath As String

Dim strDestDir As String

Do Until gblnStopThreads And (goFIFO.Count = 0)

Try

If goFIFO.Count > 0 Then

e = goFIFO.Dequeue

If TypeOf e Is RenamedEventArg s Then

Dim strOldName As String = CType(e, RenamedEventArg s).OldFullPath. Replace(gstrSou rceDir, gstrDestination Dir)

strDestDir = CType(e, RenamedEventArg s).FullPath.Rep lace(gstrSource Dir, gstrDestination Dir)

Rename(strOldNa me, strDestDir)

ElseIf TypeOf e Is FileSystemEvent Args Then

strFilePath = CType(e, FileSystemEvent Args).FullPath

strDestDir = strFilePath.Rep lace(gstrSource Dir, gstrDestination Dir)

If (GetAttr(strFil ePath) And FileAttribute.D irectory) = FileAttribute.D irectory Then

If Not Directory.Exist s(strDestDir) Then

strFilePath = strFilePath

strDestDir = strFilePath.Rep lace(gstrSource Dir, gstrDestination Dir)

Directory.Creat eDirectory(strD estDir)

End If

ElseIf New FileInfo(strDes tDir).Directory .Exists = False Then

strFilePath = strFilePath

strDestDir = strFilePath.Rep lace(gstrSource Dir, gstrDestination Dir)

Directory.Creat eDirectory(Path .GetDirectoryNa me(strDestDir))

End If

If Not (GetAttr(strFil ePath) And FileAttribute.D irectory) = FileAttribute.D irectory Then

FileCopy(strFil ePath, strDestDir)

End If

End If

Thread.Sleep(gi ntInterval)

Else

Thread.Sleep(gi ntIdle)

End If

Catch ex As Exception

goEventlog.Writ eEntry(ex.Messa ge, EventLogEntryTy pe.Error)

End Try

Loop

End Sub

End Class

-----------------------------

Imports System.ServiceP rocess

Imports System

Imports System.Threadin g

Imports System.io

Imports Microsoft.Win32

Public Class MirrorService

Inherits System.ServiceP rocess.ServiceB ase

Private WithEvents moWatcher As FileSystemWatch er

Protected Overrides Sub OnStart(ByVal args() As String)

Me.AutoLog = True

goEventlog = Me.EventLog()

Process.GetCurr entProcess().Pr iorityClass = ProcessPriority Class.High

gstrSourceDir = "C:\WatcherTest \Source"

gstrDestination Dir = "F:\WatcherTest \Destination"

gintIdle = 5000

gintInterval = 100

Dim oWorker As New WorkerThread

Dim oWorkerThread As New Thread(New ThreadStart(Add ressOf oWorker.Process Files))

' Start the thread.

oWorkerThread.S tart()

moWatcher = New FileSystemWatch er

moWatcher.Inter nalBufferSize = 16384

moWatcher.Path = gstrSourceDir

moWatcher.Inclu deSubdirectorie s = True

moWatcher.Enabl eRaisingEvents = True

End Sub

Protected Overrides Sub OnStop()

gblnStopThreads = True

End Sub

Private Sub moWatcher_Creat edOrChanged(ByV al sender As Object, ByVal e As System.IO.FileS ystemEventArgs) Handles moWatcher.Chang ed,
moWatcher.Creat ed

Try

If e.ChangeType <> WatcherChangeTy pes.Deleted Then

goFIFO.Enqueue( e)

End If

Catch ex As Exception

goEventlog.Writ eEntry(ex.Messa ge, EventLogEntryTy pe.Error)

End Try

End Sub

Private Sub moWatcher_Renam ed(ByVal sender As Object, ByVal e As System.IO.Renam edEventArgs) Handles moWatcher.Renam ed

goFIFO.Enqueue( CType(e, RenamedEventArg s))

End Sub

End Class

=============== =============== =============== ======

Sorry that the cut and paste lost the formatting. I hope this helps.

==

Al Reid
Feb 7 '06 #5
Thanks! This really helped me alot!

"Al Reid" wrote:
"Philip Wagenaar" <ph************ *@online.nospam > wrote in message news:5F******** *************** ***********@mic rosoft.com...
You mean working with threads?

Is it possible to post a sample?


I found my original proof of concept code. It was implemented as a Windows Service. Most of the concepts were obtained from Google
searches, etc.

=============== =============== =============== ====

Imports System

Imports System.Threadin g

Module General

Public goFIFO As New Collections.Que ue

Public gblnStopThreads As Boolean

Public gstrSourceDir As String

Public gstrDestination Dir As String

Public goLogger As New CLog

Public goEventlog As EventLog

Public gintIdle As Integer

Public gintInterval As Integer

End Module

-----------------------------

Imports System.Threadin g

Imports System.io

Public Class WorkerThread

Public Shared Sub ProcessFiles()

Dim e As Object

Dim strFilePath As String

Dim strDestDir As String

Do Until gblnStopThreads And (goFIFO.Count = 0)

Try

If goFIFO.Count > 0 Then

e = goFIFO.Dequeue

If TypeOf e Is RenamedEventArg s Then

Dim strOldName As String = CType(e, RenamedEventArg s).OldFullPath. Replace(gstrSou rceDir, gstrDestination Dir)

strDestDir = CType(e, RenamedEventArg s).FullPath.Rep lace(gstrSource Dir, gstrDestination Dir)

Rename(strOldNa me, strDestDir)

ElseIf TypeOf e Is FileSystemEvent Args Then

strFilePath = CType(e, FileSystemEvent Args).FullPath

strDestDir = strFilePath.Rep lace(gstrSource Dir, gstrDestination Dir)

If (GetAttr(strFil ePath) And FileAttribute.D irectory) = FileAttribute.D irectory Then

If Not Directory.Exist s(strDestDir) Then

strFilePath = strFilePath

strDestDir = strFilePath.Rep lace(gstrSource Dir, gstrDestination Dir)

Directory.Creat eDirectory(strD estDir)

End If

ElseIf New FileInfo(strDes tDir).Directory .Exists = False Then

strFilePath = strFilePath

strDestDir = strFilePath.Rep lace(gstrSource Dir, gstrDestination Dir)

Directory.Creat eDirectory(Path .GetDirectoryNa me(strDestDir))

End If

If Not (GetAttr(strFil ePath) And FileAttribute.D irectory) = FileAttribute.D irectory Then

FileCopy(strFil ePath, strDestDir)

End If

End If

Thread.Sleep(gi ntInterval)

Else

Thread.Sleep(gi ntIdle)

End If

Catch ex As Exception

goEventlog.Writ eEntry(ex.Messa ge, EventLogEntryTy pe.Error)

End Try

Loop

End Sub

End Class

-----------------------------

Imports System.ServiceP rocess

Imports System

Imports System.Threadin g

Imports System.io

Imports Microsoft.Win32

Public Class MirrorService

Inherits System.ServiceP rocess.ServiceB ase

Private WithEvents moWatcher As FileSystemWatch er

Protected Overrides Sub OnStart(ByVal args() As String)

Me.AutoLog = True

goEventlog = Me.EventLog()

Process.GetCurr entProcess().Pr iorityClass = ProcessPriority Class.High

gstrSourceDir = "C:\WatcherTest \Source"

gstrDestination Dir = "F:\WatcherTest \Destination"

gintIdle = 5000

gintInterval = 100

Dim oWorker As New WorkerThread

Dim oWorkerThread As New Thread(New ThreadStart(Add ressOf oWorker.Process Files))

' Start the thread.

oWorkerThread.S tart()

moWatcher = New FileSystemWatch er

moWatcher.Inter nalBufferSize = 16384

moWatcher.Path = gstrSourceDir

moWatcher.Inclu deSubdirectorie s = True

moWatcher.Enabl eRaisingEvents = True

End Sub

Protected Overrides Sub OnStop()

gblnStopThreads = True

End Sub

Private Sub moWatcher_Creat edOrChanged(ByV al sender As Object, ByVal e As System.IO.FileS ystemEventArgs) Handles moWatcher.Chang ed,
moWatcher.Creat ed

Try

If e.ChangeType <> WatcherChangeTy pes.Deleted Then

goFIFO.Enqueue( e)

End If

Catch ex As Exception

goEventlog.Writ eEntry(ex.Messa ge, EventLogEntryTy pe.Error)

End Try

End Sub

Private Sub moWatcher_Renam ed(ByVal sender As Object, ByVal e As System.IO.Renam edEventArgs) Handles moWatcher.Renam ed

goFIFO.Enqueue( CType(e, RenamedEventArg s))

End Sub

End Class

=============== =============== =============== ======

Sorry that the cut and paste lost the formatting. I hope this helps.

==

Al Reid

Feb 7 '06 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
9081
by: Josh Usovsky | last post by:
I'm setting up a watched folder using FileSystemWatcher. When I drop a small file into the watched folder, I can respond to a .Created event and process the file with other code. However, if I try copying a large file into the watched folder, the .Created event is fired off immediately and not when the file is finished copying. I have tried...
3
9451
by: Michael Stanbrook | last post by:
I'm using the FileSystemWatcher to looks for new files arriving in a directory. Files are coming in via FTP and tend to be larger is size (> 1MB at least). I would like to fire an event once the file is completely written. I understand there is nothing "exactly" like that due to how the OS filesystem works, so I am looking for a...
2
3300
by: Jack David | last post by:
Using the code below I am able to monitor a single directory for a new file and then kick-off a process to deal with the file. The question is??? How would I modify this code to be able to monitor a couple of different directories and based upon the directory where the new file is created kick-off a process Example: File A in Directory B...
3
4586
by: Stampede | last post by:
Hi, I write an application which waits for incomming files in a specified directory. I thought, that using the FileSystemWatcher would be the best, as it does exactly what I need. But now I have a problem: I wonna get shure, that really all files are processed and I need to be shure, that no file is processed twice. If my application is...
2
2146
by: Sacha Korell | last post by:
I need to set up a FileSystemWatcher in my web application (to automatically process uploaded files) and I'm trying to decide where to set it up. I would like to keep it within the web app, but it needs to watch a certain folder even when nobody is making a request. I could instatiate the FileSystemWatcher in the Application_Start event, but...
20
4430
by: J-T | last post by:
We are working on an asp.net application which is a 3-tier application.I was aksed to create a component which monitors a folder and gets the file and pass them to a class library in our business logic layer(so far so good and easy).I initialize my class which is using a FileSystemWatcher in my Global.asax and everything works fine.I have...
0
1597
by: mabra | last post by:
Hi All ! I am fighting with watching a large amount of files for their content and look for some keywords, which indicate dangerous error situations to me. This costs me currently hours each day. I know, how to use the FileSystemWatcher class. But in my case, I would like to open about 500 files and watch each of them for progress/enlarge...
5
5577
by: Lilith | last post by:
Would I be correct in assuming that the system file watcher is only valid for local drives and won't work for networked file system? -- Lilith
8
3353
by: Ollie Riches | last post by:
I'm looking into a production issue related to a windows service and System.Timers.Timer. The background is the windows service uses a System.Timers.Timer to periodically poll a directory location on a network for files and then copies these files to another location (on the network) AND then updates a record in the database. The file copying...
0
7434
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7692
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
7946
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7457
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
7791
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
6026
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
0
5078
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3491
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3470
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.