By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
455,767 Members | 1,375 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 455,767 IT Pros & Developers. It's quick & easy.

WANTED: logging of all file operations on Windows

P: n/a

I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from within
a Python script I see a chance, that someone in this group was into it
already and is so kind to share here his experience.

I have put already much efforts into this subject googling around, but
up to now in vain. Best option I encountered yet is usage of
the Greyware 'System Change Log' service which monitors disks for
changes (http://www.greyware.com/software/sys...log/index.asp),
but in own tests it turned out, that the created log file does not cover
all file events as e.g. it is not possible to detect when a file is
moved to a new directory (creation of a new file is logged, but deletion
is not, not mentioning I would expect a file 'move' event).
The own Windows logging service rejected to start on my XP SP2 system
for unknown to me reasons - I don't know how to get it to work (yes, I
have used the administrator account).

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi
Jul 9 '06 #1
Share this Question
Share on Google+
9 Replies


P: n/a
you want a directory watching daemon. it isn't hard at all to build
from scratch.
first, determine which directories should be watched.
then, os.walk each directory, building a mapping from filename to mtime
[modified time; os.path.getmtime].
next is your main event loop. this while loop consists of os.walk-ing
each directory again, comparing the current mtime to the corresponding
entry in the mapping. if they differ, or if a filename isn't in the
mapping, something happened, at which point you can logick out whether
a file was moved, deleted, changed, or created.

so many folks have looked for this that i'll just write a generic one
and put it in the cheeseshop. look for "dirmon" in about a week.
Claudio Grondi wrote:
I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from within
a Python script I see a chance, that someone in this group was into it
already and is so kind to share here his experience.

I have put already much efforts into this subject googling around, but
up to now in vain. Best option I encountered yet is usage of
the Greyware 'System Change Log' service which monitors disks for
changes (http://www.greyware.com/software/sys...log/index.asp),
but in own tests it turned out, that the created log file does not cover
all file events as e.g. it is not possible to detect when a file is
moved to a new directory (creation of a new file is logged, but deletion
is not, not mentioning I would expect a file 'move' event).
The own Windows logging service rejected to start on my XP SP2 system
for unknown to me reasons - I don't know how to get it to work (yes, I
have used the administrator account).

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi
Jul 9 '06 #2

P: n/a
Claudio Grondi wrote:
I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from within
a Python script I see a chance, that someone in this group was into it
already and is so kind to share here his experience.

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi
On the offchance that you haven't seen it, you might
look at this:

http://timgolden.me.uk/python/win32_...rectorychanges

but since it doesn't fulfil your criterion of *not*
representing renames by a delete and an add, it may
well not be suitable. Apart from that, I think it does
what you want.

TJG
Jul 9 '06 #3

P: n/a
"faulkner" <fa*********@comcast.netwrote in message
news:11**********************@b28g2000cwb.googlegr oups.com...
you want a directory watching daemon. it isn't hard at all to build
from scratch.
first, determine which directories should be watched.
then, os.walk each directory, building a mapping from filename to mtime
[modified time; os.path.getmtime].
next is your main event loop. this while loop consists of os.walk-ing
each directory again, comparing the current mtime to the corresponding
entry in the mapping. if they differ, or if a filename isn't in the
mapping, something happened, at which point you can logick out whether
a file was moved, deleted, changed, or created.

so many folks have looked for this that i'll just write a generic one
and put it in the cheeseshop. look for "dirmon" in about a week.

While I am a fan of "brute force"
Jul 9 '06 #4

P: n/a
"faulkner" <fa*********@comcast.netwrote in message
news:11**********************@b28g2000cwb.googlegr oups.com...
you want a directory watching daemon. it isn't hard at all to build
from scratch.
first, determine which directories should be watched.
then, os.walk each directory, building a mapping from filename to mtime
[modified time; os.path.getmtime].
next is your main event loop. this while loop consists of os.walk-ing
each directory again, comparing the current mtime to the corresponding
entry in the mapping. if they differ, or if a filename isn't in the
mapping, something happened, at which point you can logick out whether
a file was moved, deleted, changed, or created.

so many folks have looked for this that i'll just write a generic one
and put it in the cheeseshop. look for "dirmon" in about a week.

Ahem... (sorry for premature usenet-post-ication...)

While I am a big fan of "brute force", there are OS services (at least on
Windows) for doing just this function, with asynchronous callbacks when
files are created, deleted, etc.

Here is a link that does a much better comparison of several options than I
could (including your brute force version):
http://tgolden.sc.sabren.com/python/...r_changes.html

Good luck!
-- Paul
Jul 9 '06 #5

P: n/a
faulkner wrote:
you want a directory watching daemon. it isn't hard at all to build
from scratch.
first, determine which directories should be watched.
then, os.walk each directory, building a mapping from filename to mtime
[modified time; os.path.getmtime].
next is your main event loop. this while loop consists of os.walk-ing
each directory again, comparing the current mtime to the corresponding
entry in the mapping. if they differ, or if a filename isn't in the
mapping, something happened, at which point you can logick out whether
a file was moved, deleted, changed, or created.

so many folks have looked for this that i'll just write a generic one
and put it in the cheeseshop. look for "dirmon" in about a week.
Yes, I _know_ about it and exactly this knowledge is the reason I am
looking for tracking single file system related _events_ as I expect a
professional operating system like Windows to provide such service. If
there is none, this will be sure a severe reason to go for Linux if it
has such one instead of going for a SVN server or special file systems
if there are any.

Has someone experience with SVN handling million(s) of files?

The problem is, that brute force applied to large amount of
files/directories is not a convenient way to backup/synchronize the few
new/changed/deleted/moved files/directories multiple times a day as the
brute force approach just makes the hard drive(s) unnecessary wasting
much energy and getting hot.

Claudio Grondi
>

Claudio Grondi wrote:
>>I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from within
a Python script I see a chance, that someone in this group was into it
already and is so kind to share here his experience.

I have put already much efforts into this subject googling around, but
up to now in vain. Best option I encountered yet is usage of
the Greyware 'System Change Log' service which monitors disks for
changes (http://www.greyware.com/software/sys...log/index.asp),
but in own tests it turned out, that the created log file does not cover
all file events as e.g. it is not possible to detect when a file is
moved to a new directory (creation of a new file is logged, but deletion
is not, not mentioning I would expect a file 'move' event).
The own Windows logging service rejected to start on my XP SP2 system
for unknown to me reasons - I don't know how to get it to work (yes, I
have used the administrator account).

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi

Jul 9 '06 #6

P: n/a
Tim Golden wrote:
Claudio Grondi wrote:
>I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from
within a Python script I see a chance, that someone in this group was
into it already and is so kind to share here his experience.

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi


On the offchance that you haven't seen it, you might
look at this:

http://timgolden.me.uk/python/win32_...rectorychanges
but since it doesn't fulfil your criterion of *not*
representing renames by a delete and an add, it may
well not be suitable. Apart from that, I think it does
what you want.

TJG
It seems, that it will be necessary to use some logic based on the
sequence of events to exactly detect rename and move changes done to
files/directories, but in principle it is the best approach I know about
yet.

Thank you!

By the way:
Is there something similar/same available for Linux?

Claudio Grondi
Jul 9 '06 #7

P: n/a
Tim Golden wrote:
Claudio Grondi wrote:
>I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from
within a Python script I see a chance, that someone in this group was
into it already and is so kind to share here his experience.

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi


On the offchance that you haven't seen it, you might
look at this:

http://timgolden.me.uk/python/win32_...rectorychanges
but since it doesn't fulfil your criterion of *not*
representing renames by a delete and an add, it may
well not be suitable. Apart from that, I think it does
what you want.

TJG
Here a small update to the code at
http://timgolden.me.uk/python/win32_...rectorychanges
:

ACTIONS = {
1 : "Created",
2 : "Deleted",
3 : "Updated",
4 : "Renamed from something"
5 : "Renamed to something",
}

The correction above is according to entries:
#define FILE_ACTION_ADDED 0x00000001
#define FILE_ACTION_REMOVED 0x00000002
#define FILE_ACTION_MODIFIED 0x00000003
#define FILE_ACTION_RENAMED_OLD_NAME 0x00000004
#define FILE_ACTION_RENAMED_NEW_NAME 0x00000005
in ..\PlatformSDK\Include\WinNT.h

Claudio Grondi
Jul 10 '06 #8

P: n/a
Claudio Grondi wrote:
Here a small update to the code at
http://timgolden.me.uk/python/win32_...rectorychanges
:

ACTIONS = {
1 : "Created",
2 : "Deleted",
3 : "Updated",
4 : "Renamed from something"
5 : "Renamed to something",
}

The correction above is according to entries:
#define FILE_ACTION_ADDED 0x00000001
#define FILE_ACTION_REMOVED 0x00000002
#define FILE_ACTION_MODIFIED 0x00000003
#define FILE_ACTION_RENAMED_OLD_NAME 0x00000004
#define FILE_ACTION_RENAMED_NEW_NAME 0x00000005
in ..\PlatformSDK\Include\WinNT.h

Claudio Grondi
Thanks. I've updated the site.

TJG
Jul 10 '06 #9

P: n/a
[Tim Golden]
>On the offchance that you haven't seen it, you might
look at this:

http://timgolden.me.uk/python/win32_...rectorychanges
[Claudio Grondi]
It seems, that it will be necessary to use some logic based on the
sequence of events to exactly detect rename and move changes done to
files/directories, but in principle it is the best approach I know about
yet.

By the way:
Is there something similar/same available for Linux?
I've never used them, but I seem to think there are a couple
of similar things for Linux, based on FAM or inotify:

(result of Googling)

http://python-fam.sourceforge.net/
http://www.gnome.org/~veillard/gamin/python.html
http://rudd-o.com/projects/python-inotify/

YMMV
TJG
Jul 10 '06 #10

This discussion thread is closed

Replies have been disabled for this discussion.