473,322 Members | 1,352 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

WANTED: logging of all file operations on Windows


I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from within
a Python script I see a chance, that someone in this group was into it
already and is so kind to share here his experience.

I have put already much efforts into this subject googling around, but
up to now in vain. Best option I encountered yet is usage of
the Greyware 'System Change Log' service which monitors disks for
changes (http://www.greyware.com/software/sys...log/index.asp),
but in own tests it turned out, that the created log file does not cover
all file events as e.g. it is not possible to detect when a file is
moved to a new directory (creation of a new file is logged, but deletion
is not, not mentioning I would expect a file 'move' event).
The own Windows logging service rejected to start on my XP SP2 system
for unknown to me reasons - I don't know how to get it to work (yes, I
have used the administrator account).

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi
Jul 9 '06 #1
9 2217
you want a directory watching daemon. it isn't hard at all to build
from scratch.
first, determine which directories should be watched.
then, os.walk each directory, building a mapping from filename to mtime
[modified time; os.path.getmtime].
next is your main event loop. this while loop consists of os.walk-ing
each directory again, comparing the current mtime to the corresponding
entry in the mapping. if they differ, or if a filename isn't in the
mapping, something happened, at which point you can logick out whether
a file was moved, deleted, changed, or created.

so many folks have looked for this that i'll just write a generic one
and put it in the cheeseshop. look for "dirmon" in about a week.
Claudio Grondi wrote:
I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from within
a Python script I see a chance, that someone in this group was into it
already and is so kind to share here his experience.

I have put already much efforts into this subject googling around, but
up to now in vain. Best option I encountered yet is usage of
the Greyware 'System Change Log' service which monitors disks for
changes (http://www.greyware.com/software/sys...log/index.asp),
but in own tests it turned out, that the created log file does not cover
all file events as e.g. it is not possible to detect when a file is
moved to a new directory (creation of a new file is logged, but deletion
is not, not mentioning I would expect a file 'move' event).
The own Windows logging service rejected to start on my XP SP2 system
for unknown to me reasons - I don't know how to get it to work (yes, I
have used the administrator account).

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi
Jul 9 '06 #2
Claudio Grondi wrote:
I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from within
a Python script I see a chance, that someone in this group was into it
already and is so kind to share here his experience.

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi
On the offchance that you haven't seen it, you might
look at this:

http://timgolden.me.uk/python/win32_...rectorychanges

but since it doesn't fulfil your criterion of *not*
representing renames by a delete and an add, it may
well not be suitable. Apart from that, I think it does
what you want.

TJG
Jul 9 '06 #3
"faulkner" <fa*********@comcast.netwrote in message
news:11**********************@b28g2000cwb.googlegr oups.com...
you want a directory watching daemon. it isn't hard at all to build
from scratch.
first, determine which directories should be watched.
then, os.walk each directory, building a mapping from filename to mtime
[modified time; os.path.getmtime].
next is your main event loop. this while loop consists of os.walk-ing
each directory again, comparing the current mtime to the corresponding
entry in the mapping. if they differ, or if a filename isn't in the
mapping, something happened, at which point you can logick out whether
a file was moved, deleted, changed, or created.

so many folks have looked for this that i'll just write a generic one
and put it in the cheeseshop. look for "dirmon" in about a week.

While I am a fan of "brute force"
Jul 9 '06 #4
"faulkner" <fa*********@comcast.netwrote in message
news:11**********************@b28g2000cwb.googlegr oups.com...
you want a directory watching daemon. it isn't hard at all to build
from scratch.
first, determine which directories should be watched.
then, os.walk each directory, building a mapping from filename to mtime
[modified time; os.path.getmtime].
next is your main event loop. this while loop consists of os.walk-ing
each directory again, comparing the current mtime to the corresponding
entry in the mapping. if they differ, or if a filename isn't in the
mapping, something happened, at which point you can logick out whether
a file was moved, deleted, changed, or created.

so many folks have looked for this that i'll just write a generic one
and put it in the cheeseshop. look for "dirmon" in about a week.

Ahem... (sorry for premature usenet-post-ication...)

While I am a big fan of "brute force", there are OS services (at least on
Windows) for doing just this function, with asynchronous callbacks when
files are created, deleted, etc.

Here is a link that does a much better comparison of several options than I
could (including your brute force version):
http://tgolden.sc.sabren.com/python/...r_changes.html

Good luck!
-- Paul
Jul 9 '06 #5
faulkner wrote:
you want a directory watching daemon. it isn't hard at all to build
from scratch.
first, determine which directories should be watched.
then, os.walk each directory, building a mapping from filename to mtime
[modified time; os.path.getmtime].
next is your main event loop. this while loop consists of os.walk-ing
each directory again, comparing the current mtime to the corresponding
entry in the mapping. if they differ, or if a filename isn't in the
mapping, something happened, at which point you can logick out whether
a file was moved, deleted, changed, or created.

so many folks have looked for this that i'll just write a generic one
and put it in the cheeseshop. look for "dirmon" in about a week.
Yes, I _know_ about it and exactly this knowledge is the reason I am
looking for tracking single file system related _events_ as I expect a
professional operating system like Windows to provide such service. If
there is none, this will be sure a severe reason to go for Linux if it
has such one instead of going for a SVN server or special file systems
if there are any.

Has someone experience with SVN handling million(s) of files?

The problem is, that brute force applied to large amount of
files/directories is not a convenient way to backup/synchronize the few
new/changed/deleted/moved files/directories multiple times a day as the
brute force approach just makes the hard drive(s) unnecessary wasting
much energy and getting hot.

Claudio Grondi
>

Claudio Grondi wrote:
>>I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from within
a Python script I see a chance, that someone in this group was into it
already and is so kind to share here his experience.

I have put already much efforts into this subject googling around, but
up to now in vain. Best option I encountered yet is usage of
the Greyware 'System Change Log' service which monitors disks for
changes (http://www.greyware.com/software/sys...log/index.asp),
but in own tests it turned out, that the created log file does not cover
all file events as e.g. it is not possible to detect when a file is
moved to a new directory (creation of a new file is logged, but deletion
is not, not mentioning I would expect a file 'move' event).
The own Windows logging service rejected to start on my XP SP2 system
for unknown to me reasons - I don't know how to get it to work (yes, I
have used the administrator account).

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi

Jul 9 '06 #6
Tim Golden wrote:
Claudio Grondi wrote:
>I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from
within a Python script I see a chance, that someone in this group was
into it already and is so kind to share here his experience.

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi


On the offchance that you haven't seen it, you might
look at this:

http://timgolden.me.uk/python/win32_...rectorychanges
but since it doesn't fulfil your criterion of *not*
representing renames by a delete and an add, it may
well not be suitable. Apart from that, I think it does
what you want.

TJG
It seems, that it will be necessary to use some logic based on the
sequence of events to exactly detect rename and move changes done to
files/directories, but in principle it is the best approach I know about
yet.

Thank you!

By the way:
Is there something similar/same available for Linux?

Claudio Grondi
Jul 9 '06 #7
Tim Golden wrote:
Claudio Grondi wrote:
>I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from
within a Python script I see a chance, that someone in this group was
into it already and is so kind to share here his experience.

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi


On the offchance that you haven't seen it, you might
look at this:

http://timgolden.me.uk/python/win32_...rectorychanges
but since it doesn't fulfil your criterion of *not*
representing renames by a delete and an add, it may
well not be suitable. Apart from that, I think it does
what you want.

TJG
Here a small update to the code at
http://timgolden.me.uk/python/win32_...rectorychanges
:

ACTIONS = {
1 : "Created",
2 : "Deleted",
3 : "Updated",
4 : "Renamed from something"
5 : "Renamed to something",
}

The correction above is according to entries:
#define FILE_ACTION_ADDED 0x00000001
#define FILE_ACTION_REMOVED 0x00000002
#define FILE_ACTION_MODIFIED 0x00000003
#define FILE_ACTION_RENAMED_OLD_NAME 0x00000004
#define FILE_ACTION_RENAMED_NEW_NAME 0x00000005
in ..\PlatformSDK\Include\WinNT.h

Claudio Grondi
Jul 10 '06 #8
Claudio Grondi wrote:
Here a small update to the code at
http://timgolden.me.uk/python/win32_...rectorychanges
:

ACTIONS = {
1 : "Created",
2 : "Deleted",
3 : "Updated",
4 : "Renamed from something"
5 : "Renamed to something",
}

The correction above is according to entries:
#define FILE_ACTION_ADDED 0x00000001
#define FILE_ACTION_REMOVED 0x00000002
#define FILE_ACTION_MODIFIED 0x00000003
#define FILE_ACTION_RENAMED_OLD_NAME 0x00000004
#define FILE_ACTION_RENAMED_NEW_NAME 0x00000005
in ..\PlatformSDK\Include\WinNT.h

Claudio Grondi
Thanks. I've updated the site.

TJG
Jul 10 '06 #9
[Tim Golden]
>On the offchance that you haven't seen it, you might
look at this:

http://timgolden.me.uk/python/win32_...rectorychanges
[Claudio Grondi]
It seems, that it will be necessary to use some logic based on the
sequence of events to exactly detect rename and move changes done to
files/directories, but in principle it is the best approach I know about
yet.

By the way:
Is there something similar/same available for Linux?
I've never used them, but I seem to think there are a couple
of similar things for Linux, based on FAM or inotify:

(result of Googling)

http://python-fam.sourceforge.net/
http://www.gnome.org/~veillard/gamin/python.html
http://rudd-o.com/projects/python-inotify/

YMMV
TJG
Jul 10 '06 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: j vickroy | last post by:
Hello, I'm trying to understand the behavior of the Python 2.3 logging module (MS Windows 2k) with regard to RotatingFileHandler. The following script illustrates a puzzling problem. What is...
1
by: j vickroy | last post by:
My system: MSW XP professional Python 2.3.3 logging package: 0.4.9.2 My problem: The log_test3.py script, provided with the logging package distribution, generates an unexpected message: No...
0
by: Neil Benn | last post by:
Hello, I'm running a test and having issues with logging, if I call logging.shutdown() and then want to start the logging going again then I get a problem as if I call shutdown, I can't get the...
8
by: qwweeeit | last post by:
Hi all, I wonder if it is possible to change (temporarily) a built-in function for logging purposes. Let me explain: I want to log all the 'open' operations, recording the file to be opened, the...
1
by: Oliver Eichler | last post by:
Hi, I experience several exceptions from python's logging system when using the rollover feature on Windows. Traceback (most recent call last): File "c:\Python24\lib\logging\handlers.py",...
1
by: timb | last post by:
Hi, does any have any sample code of the above? i have tried the example from the help but am unable to get my windows service to create a new logfile and start logging to it. I am able to...
16
by: Einar Høst | last post by:
Hi, I'm getting into the Trace-functionality in .NET, using it to provide some much-needed logging across dlls in the project we're working on. However, being a newbie, I'm wondering if some...
7
by: flupke | last post by:
Hi, i'm getting errors with the log module concerning RotatingFileHandler. I'm using Python 2.4.3 on Windows XP SP2. This used to work in previous python versions but since i upgraded to 2.4.3...
0
by: Chris Curvey | last post by:
Hi all, I just upgraded to 2.4.3 (from 2.4.1) on Windows. Now each time I run my unit tests, they always throw this error at the end of the test run: Error in atexit._run_exitfuncs:...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.