Counting file handles on a file

[Conrad F]

Hello all,

I am waiting for receipt of files in a directory. I use the
FileSystemWatcher to detect when files arrive in said folder. I need
to read the data from these files ASAP but the files are created and
detected before writing completes and so I cannot read them until the
file handle used in their creation has been released.

This would be easy to get around with file renaming or other file
locking mechanisms which I could use except that I have no control
over the file "drops" because they are from a 3rd party and not about
to change. I simply see the files arrive.

So, in brief, how can I determine the number of file handles on a file
so that when one arrives I can wait until it is released?

Thanks,
Conrad

[Willy Denoyette [MVP]]

"Conrad F" <co*********@hotmail.com> wrote in message
news:6a**************************@posting.google.c om...

Hello all,

I am waiting for receipt of files in a directory. I use the
FileSystemWatcher to detect when files arrive in said folder. I need
to read the data from these files ASAP but the files are created and
detected before writing completes and so I cannot read them until the
file handle used in their creation has been released.

This would be easy to get around with file renaming or other file
locking mechanisms which I could use except that I have no control
over the file "drops" because they are from a 3rd party and not about
to change. I simply see the files arrive.

So, in brief, how can I determine the number of file handles on a file
so that when one arrives I can wait until it is released?

Thanks,
Conrad

You can't touch the file as long as it isn't written and closed completely,
so you can't get at the file handles too, the only thing you can do is wait
in a loop that tries to open the file.

Willy.

[Wayne]

Not sure if this will work for your issue or even if it is the same in .net.
We have a process that does something similar to what you want, what we
noticed is that the file size remained 0 until the file had been completely
written. You may check to see if getting a list of the files shows this to
be true for you as well.

In our scenario the files are always less then 1k, always written by IIS and
we used the FindFirst/FindNext in win32 API to check for files.

--
Thanks
Wayne Sepega
Jacksonville, Fl
"When a man sits with a pretty girl for an hour, it seems like a minute. But
let him sit on a hot stove for a minute and it's longer than any hour.
That's relativity." - Albert Einstein

"Conrad F" <co*********@hotmail.com> wrote in message
news:6a**************************@posting.google.c om...

Hello all,

I am waiting for receipt of files in a directory. I use the
FileSystemWatcher to detect when files arrive in said folder. I need
to read the data from these files ASAP but the files are created and
detected before writing completes and so I cannot read them until the
file handle used in their creation has been released.

This would be easy to get around with file renaming or other file
locking mechanisms which I could use except that I have no control
over the file "drops" because they are from a 3rd party and not about
to change. I simply see the files arrive.

So, in brief, how can I determine the number of file handles on a file
so that when one arrives I can wait until it is released?

Thanks,
Conrad

[daiyue]

Hi,
I'm quite interested in your topic and do a little research,
I just found http://www.sysinternals.com/ntw2k/source/filemon.shtml
..That is a file moniotr application,at that page you can find
"On Windows NT the heart of FileMon is a file system driver that creates and
attaches filter device objects to target file system device objects so that
FileMon will see all IRPs and FastIO requests directed at drives. When
FileMon sees an open, create or close call, it updates an internal hash
table that serves as the mapping between internal file handles and file path
names. "

So I go on googling and find that maybe you need to do some kind of
driver development,which is "File System Filter Drivers".I think by
those way you can certainly get all the I/O operation you want to know,
which is out of my capability.

If anyone got a good idea,please let me know.
Good luck.
Daiyue

"Willy Denoyette [MVP]" <wi*************@pandora.be> дÈëÏûÏ¢ÐÂÎÅ:ex***************@TK2MSFTNGP15.phx.gb l...

"Conrad F" <co*********@hotmail.com> wrote in message
news:6a**************************@posting.google.c om...

Hello all,

I am waiting for receipt of files in a directory. I use the
FileSystemWatcher to detect when files arrive in said folder. I need
to read the data from these files ASAP but the files are created and
detected before writing completes and so I cannot read them until the
file handle used in their creation has been released.

This would be easy to get around with file renaming or other file
locking mechanisms which I could use except that I have no control
over the file "drops" because they are from a 3rd party and not about
to change. I simply see the files arrive.

So, in brief, how can I determine the number of file handles on a file
so that when one arrives I can wait until it is released?

Thanks,
Conrad

You can't touch the file as long as it isn't written and closed
completely, so you can't get at the file handles too, the only thing you
can do is wait in a loop that tries to open the file.

Willy.

[Willy Denoyette [MVP]]

"Wayne" <Me******@community.nospam> wrote in message
news:%2****************@tk2msftngp13.phx.gbl...

Not sure if this will work for your issue or even if it is the same in
.net.
We have a process that does something similar to what you want, what we
noticed is that the file size remained 0 until the file had been
completely
written. You may check to see if getting a list of the files shows this to
be true for you as well.

In our scenario the files are always less then 1k, always written by IIS
and
we used the FindFirst/FindNext in win32 API to check for files.

And what would be the advantage of this over simply trying to open the file
in a loop?

Willy.

[Willy Denoyette [MVP]]

"daiyue" <my******@gmail.com> wrote in message
news:eL**************@TK2MSFTNGP09.phx.gbl...

Hi,
I'm quite interested in your topic and do a little research,
I just found http://www.sysinternals.com/ntw2k/source/filemon.shtml
.That is a file moniotr application,at that page you can find
"On Windows NT the heart of FileMon is a file system driver that creates
and attaches filter device objects to target file system device objects so
that FileMon will see all IRPs and FastIO requests directed at drives.
When FileMon sees an open, create or close call, it updates an internal
hash table that serves as the mapping between internal file handles and
file path names. "

So I go on googling and find that maybe you need to do some kind of
driver development,which is "File System Filter Drivers".I think by
those way you can certainly get all the I/O operation you want to know,
which is out of my capability.

If anyone got a good idea,please let me know.
Good luck.
Daiyue

That won't help you either as part of this stuff is based on "non
documented" native OS calls.
Most sysinternals tools are great as a test/debug tools but I won't rely on
it for production code.

illy.

[Wayne]

Never said it would have an advantage over opening the file in a loop, just
a different approach.

--
Thanks
Wayne Sepega
Jacksonville, Fl
"When a man sits with a pretty girl for an hour, it seems like a minute. But
let him sit on a hot stove for a minute and it's longer than any hour.
That's relativity." - Albert Einstein

"Willy Denoyette [MVP]" <wi*************@pandora.be> wrote in message
news:uY**************@TK2MSFTNGP14.phx.gbl...

"Wayne" <Me******@community.nospam> wrote in message
news:%2****************@tk2msftngp13.phx.gbl...

Not sure if this will work for your issue or even if it is the same in
.net.
We have a process that does something similar to what you want, what we
noticed is that the file size remained 0 until the file had been
completely
written. You may check to see if getting a list of the files shows this to be true for you as well.

In our scenario the files are always less then 1k, always written by IIS
and
we used the FindFirst/FindNext in win32 API to check for files.

And what would be the advantage of this over simply trying to open the

file in a loop?

Willy.

[conradfripp]

Hi,

1) Hopefully, doing it some other way will allow asynchronous
monitoring and not put processor usage at 100% while it does nothing
but check a file thousands of times
2) I don't want to lock up the front-end GUI
3) If I tie the program up in a loop, it won't receive event
notifications of other file arrivals if it is stuck in a loop!

Conrad

[conradfripp]

I am at home now, but if this is true I could probably detect the
change of file size with the FileSystemWatcher class still, which is
ideal.

I will check it out tomorrow!

[conradfripp]

I am not sure that's true. I have (a few years ago) managed to read
large files while they were still being written by another process and
have consequently ended up reading past the end of a file before an EOF
marker is set - not with nice results either as it was in VB at the
time, threw a GPF and my IDE just disappeared!

If .NET does have to open the file and it is exclusively locked to
another process, then there will be a big delay in an order of seconds
the first time .NET throws an exception - it sounds a small amount of
time but it is large in terms of the amount of files being processed
and the speed requirments. The files are always 5k or less and are
purely XML.

I am fairly sure that a few years ago (when I still used the WIN32API
alot) that there was a call to enumerate the file handles on a
file!?!?!? How else can you tell if a file is in use by another process
without excessive process monitoring?

[Willy Denoyette [MVP]]

<co*********@hotmail.com> wrote in message
news:11**********************@c13g2000cwb.googlegr oups.com...

I am not sure that's true. I have (a few years ago) managed to read
large files while they were still being written by another process and
have consequently ended up reading past the end of a file before an EOF
marker is set - not with nice results either as it was in VB at the
time, threw a GPF and my IDE just disappeared!

If .NET does have to open the file and it is exclusively locked to
another process, then there will be a big delay in an order of seconds
the first time .NET throws an exception - it sounds a small amount of
time but it is large in terms of the amount of files being processed
and the speed requirments. The files are always 5k or less and are
purely XML.

I am fairly sure that a few years ago (when I still used the WIN32API
alot) that there was a call to enumerate the file handles on a
file!?!?!? How else can you tell if a file is in use by another process
without excessive process monitoring?

If the Win32 API existed years ago, it should still exist today, right?
The only way to get at the open (file) handles, is by using undocumented
native windows API's like NtQuerySystemInformation and NTQueryObject from
Ntdll.dll (this is b.t.w how it's done by sysinternals in it's tools like
ntfilemon and processexplorer) .
But this is expensive in terms of:
- memory consumption as you have to take a snapshot (at regular intervals
!!) of the open handles table , which can be several megabytes big, and
- in terms of CPU load, because you have to scan the table for the process
and the file in order to find (or not find) the corresponding handle.
And there is a security implication, you have to run with debug privileges.

You can tell if a file is in use by opening the file exclusive (that is non
shared), if it's still open by another process, an exception will be throw
on you, not after seconds but within miliseconds.
It take 20 lines of code to try it out yourself.

Willy.

[Willy Denoyette [MVP]]

What do you mean by checking a thousand times?
Just open the file in a loop and sleep - let's say a second or two for each
iteration, that won't consume that many cycles per iteration.

Willy.

<co*********@hotmail.com> wrote in message
news:11**********************@c13g2000cwb.googlegr oups.com...

Hi,

1) Hopefully, doing it some other way will allow asynchronous
monitoring and not put processor usage at 100% while it does nothing
but check a file thousands of times
2) I don't want to lock up the front-end GUI
3) If I tie the program up in a loop, it won't receive event
notifications of other file arrivals if it is stuck in a loop!

Conrad

[daiyue]

Willy,maybe you are right. I just don't like the try... catch
at here for it may be expensive in the clr.
"Willy Denoyette [MVP]" <wi*************@pandora.be> дÈëÏûÏ¢ÐÂÎÅ:ex***************@TK2MSFTNGP15.phx.gb l...

"Conrad F" <co*********@hotmail.com> wrote in message
news:6a**************************@posting.google.c om...

Hello all,

I am waiting for receipt of files in a directory. I use the
FileSystemWatcher to detect when files arrive in said folder. I need
to read the data from these files ASAP but the files are created and
detected before writing completes and so I cannot read them until the
file handle used in their creation has been released.

This would be easy to get around with file renaming or other file
locking mechanisms which I could use except that I have no control
over the file "drops" because they are from a 3rd party and not about
to change. I simply see the files arrive.

So, in brief, how can I determine the number of file handles on a file
so that when one arrives I can wait until it is released?

Thanks,
Conrad

You can't touch the file as long as it isn't written and closed
completely, so you can't get at the file handles too, the only thing you
can do is wait in a loop that tries to open the file.

Willy.

[Jon Skeet [C# MVP]]

daiyue <my******@gmail.com> wrote:

Willy,maybe you are right. I just don't like the try... catch
at here for it may be expensive in the clr.

I strongly suspect that the exception handling is actually less
expensive than going to the file system in the first place.

On my box, it takes about 3 seconds to successfully open a file 100,000
times. On the other hand, the same machine can throw and catch 100,000
exceptions in a second.

If you try, wait for (say) a tenth of a second, and then retry (however
many times you want to retry) you'll find that the performance penalty
for the exceptions is tiny.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Sean Hederman]

Instead of opening the file with .NET, why don't you use the OpenFile API
with the OF_SHARE_EXCLUSIVE flag set? Then you simply test the return
against IntPtr.Zero to see if it was opened successfully? If it was,
immediately call CloseHandle and open the file in the .NET way. Just make
sure you wrap that up in an exception block, just in case someone opens the
file between those two statements.

"daiyue" <my******@gmail.com> wrote in message
news:%2****************@TK2MSFTNGP14.phx.gbl...

Willy,maybe you are right. I just don't like the try... catch
at here for it may be expensive in the clr.
"Willy Denoyette [MVP]" <wi*************@pandora.be>
дÈëÏûÏ¢ÐÂÎÅ:ex***************@TK2MSFTNGP15.phx.gb l...

"Conrad F" <co*********@hotmail.com> wrote in message
news:6a**************************@posting.google.c om...

Hello all,

I am waiting for receipt of files in a directory. I use the
FileSystemWatcher to detect when files arrive in said folder. I need
to read the data from these files ASAP but the files are created and
detected before writing completes and so I cannot read them until the
file handle used in their creation has been released.

This would be easy to get around with file renaming or other file
locking mechanisms which I could use except that I have no control
over the file "drops" because they are from a 3rd party and not about
to change. I simply see the files arrive.

So, in brief, how can I determine the number of file handles on a file
so that when one arrives I can wait until it is released?

Thanks,
Conrad

You can't touch the file as long as it isn't written and closed
completely, so you can't get at the file handles too, the only thing you
can do is wait in a loop that tries to open the file.

Willy.

[Willy Denoyette [MVP]]

"Sean Hederman" <us***@blogentry.com> wrote in message
news:ct**********@ctb-nnrp2.saix.net...

Instead of opening the file with .NET, why don't you use the OpenFile API
with the OF_SHARE_EXCLUSIVE flag set? Then you simply test the return
against IntPtr.Zero to see if it was opened successfully? If it was,
immediately call CloseHandle and open the file in the .NET way. Just make
sure you wrap that up in an exception block, just in case someone opens
the file between those two statements.

And what would be the benefit, just saving the costs of a few exceptions
thrown at the expense of a extra PInvoke transitions each incurring a
security walk?
A PInvoke transition is more expensive than handling an exception!

Willy.

[conradfripp]

Probably mostly true, but whenever an exception is first thrown in .NET
you can already tell your about to get it because it sits there a few
seconds. The same code executing again later and throwing an exception
is, I agree, processed in milliseconds. A little JIT compiler thing I
suspect.

I was hoping to avoid tying up my main thread in a loop because the
file may not be free and other files will be arriving rapidly at peak
times and I need to handle them as well. I suppose I have to open the
file exclusively on a separate thread until successful and then call my
file processing code (main thread) once I get the file opened .

This sort of technique using exceptions, does not sit well with me
because at some point when turning on "Break on all exceptions" from
the VS Debug menu, the code will be impossible to debug because I will
always be throwing exceptions at a place I don't want to stop because
it's not for an error; I would would just be thwoing an exception as
part of a condition test which .NET doesn't allow for!

Would "int handles = File.GetFileHandleCount(string filename) "be so
hard? Why on a file is there not handle count when the handles are
conceptually similar to object handles on a COM object and you can get
a ref count on that?

I believe exceptions should only be thrown, caught and handled when all
other conditional checks have failed or invalid data causes failure - I
don't believe it good practice to throw them to determine a condition.

[hector]

Conrad,

No matter what, eventually you need to access the file.

So you might as well use the suggestion of trying to open the file in
exclusive mode. It works fine as long as you use the common method and 3rd
party applications do the same thing.

In a loop, attempt to open the file in exclusive mode which is:

access: GENERIC_READ | GENERIC_WRITE
share: 0 (No Sharing)

Since the file is still being written, you will be denied and get an access
error because you tried to open with GENERIC_WRITE. Once you have access,
the Zero share stops others from working on the file.

You need to explore this if the 3rd party application you are talking about
did not expect this exclusive holding.

You might be able to relax it by opening it with:

access: GENERIC_READ | GENERIC_WRITE
share: FILE_SHARE_READ

which will atleast allow access by the 3rd party system which may want to
make it available to other processes or clients.

But it really depends on the design framework and the possible I/O issues
with the files. You may still need to "copy" it as fast as possible to
minimize contention issues.

Even if you got the number of instances that have access to the file and
waited until this value went to zero, that will not help you because you
still need to open it yourself and you won't be able to stop others from
doing the same unless you open exclusive mode or FILE_SHARE_READ only.

So you might as well try to open it in a loop with some Sleep() value, maybe
50, 100, 250 or more. Here is an example C++ program that will copy a file
but wait if the file is currently open in write mode:

BOOL CopyLockFileWin(string strSource, string Destination)
{

ULONG nSizeWritten = 0;
ULONG numRead = 0;
int ReturnVal = 0;
char Buffer[16384];
ULONG AmountRead = 0;
unsigned long SizeOfFile = 0;

HANDLE hSource;
while (1) {
hSource = CreateFile(strSource.c_str(),
GENERIC_READ,
FILE_SHARE_READ,
NULL,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
NULL);
if (hSource != INVALID_HANDLE_VALUE) {
break;
}
int err = GetLastError();
if (err != 5 && err != 32) {
// unexpected error, get out
return FALSE;
}
// you can add some break out or sanity check logic here, like waiting
// no more than X seconds or minutes or whatever, also have some
// terminal process logic checking, etc.
Sleep(100);
}

HANDLE hTarget = CreateFile(Destination.c_str(),
GENERIC_WRITE,
0,
NULL ,
CREATE_ALWAYS,
FILE_ATTRIBUTE_NORMAL,
NULL);

if (hTarget != INVALID_HANDLE_VALUE) {
int err = GetLastError(); // preserve error
CloseHandle(hSource);
SetLastError(err);
return FALSE;
}

while (true) {
if (!ReadFile(hSource, Buffer, sizeof(Buffer), &AmountRead, NULL) ||
(AmountRead == 0)) {
break;
}
if (!WriteFile(hTarget, Buffer, AmountRead, &nSizeWritten, NULL)) {
break;
}
}
/* Clean UP */
CloseHandle(hTarget);
CloseHandle(hSource);
return TRUE;
}

---
Hector Santos
WINSERVER "Wildcat! Interactive Net Server"
http://www.winserver.com
http://www.santronics.com

"Conrad F" <co*********@hotmail.com> wrote in message
news:6a**************************@posting.google.c om...

Hello all,

I am waiting for receipt of files in a directory. I use the
FileSystemWatcher to detect when files arrive in said folder. I need
to read the data from these files ASAP but the files are created and
detected before writing completes and so I cannot read them until the
file handle used in their creation has been released.

This would be easy to get around with file renaming or other file
locking mechanisms which I could use except that I have no control
over the file "drops" because they are from a 3rd party and not about
to change. I simply see the files arrive.

So, in brief, how can I determine the number of file handles on a file
so that when one arrives I can wait until it is released?

Thanks,
Conrad

[Wayne]

Why not add the files that need to be processed to a queue, spin through the
queue trying to process each file, if you get an exception move to the next
file, if you are able to process the file then remove it from the Queue.
Safely lock the queue and thread the adding to it, or the processing of it.
This way your not just sitting on one file if you are unable to process it
and not taking up a new thread for each file you are unable to process due
to it being locked.

Wayne

<co*********@hotmail.com> wrote in message
news:11*********************@l41g2000cwc.googlegro ups.com...

I was hoping to avoid tying up my main thread in a loop because the
file may not be free and other files will be arriving rapidly at peak
times and I need to handle them as well. I suppose I have to open the
file exclusively on a separate thread until successful and then call my
file processing code (main thread) once I get the file opened .