473,746 Members | 2,696 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

WANTED: logging of all file operations on Windows


I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from within
a Python script I see a chance, that someone in this group was into it
already and is so kind to share here his experience.

I have put already much efforts into this subject googling around, but
up to now in vain. Best option I encountered yet is usage of
the Greyware 'System Change Log' service which monitors disks for
changes (http://www.greyware.com/software/sys...log/index.asp),
but in own tests it turned out, that the created log file does not cover
all file events as e.g. it is not possible to detect when a file is
moved to a new directory (creation of a new file is logged, but deletion
is not, not mentioning I would expect a file 'move' event).
The own Windows logging service rejected to start on my XP SP2 system
for unknown to me reasons - I don't know how to get it to work (yes, I
have used the administrator account).

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi
Jul 9 '06 #1
9 2238
you want a directory watching daemon. it isn't hard at all to build
from scratch.
first, determine which directories should be watched.
then, os.walk each directory, building a mapping from filename to mtime
[modified time; os.path.getmtim e].
next is your main event loop. this while loop consists of os.walk-ing
each directory again, comparing the current mtime to the corresponding
entry in the mapping. if they differ, or if a filename isn't in the
mapping, something happened, at which point you can logick out whether
a file was moved, deleted, changed, or created.

so many folks have looked for this that i'll just write a generic one
and put it in the cheeseshop. look for "dirmon" in about a week.
Claudio Grondi wrote:
I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from within
a Python script I see a chance, that someone in this group was into it
already and is so kind to share here his experience.

I have put already much efforts into this subject googling around, but
up to now in vain. Best option I encountered yet is usage of
the Greyware 'System Change Log' service which monitors disks for
changes (http://www.greyware.com/software/sys...log/index.asp),
but in own tests it turned out, that the created log file does not cover
all file events as e.g. it is not possible to detect when a file is
moved to a new directory (creation of a new file is logged, but deletion
is not, not mentioning I would expect a file 'move' event).
The own Windows logging service rejected to start on my XP SP2 system
for unknown to me reasons - I don't know how to get it to work (yes, I
have used the administrator account).

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi
Jul 9 '06 #2
Claudio Grondi wrote:
I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from within
a Python script I see a chance, that someone in this group was into it
already and is so kind to share here his experience.

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi
On the offchance that you haven't seen it, you might
look at this:

http://timgolden.me.uk/python/win32_...rectorychanges

but since it doesn't fulfil your criterion of *not*
representing renames by a delete and an add, it may
well not be suitable. Apart from that, I think it does
what you want.

TJG
Jul 9 '06 #3
"faulkner" <fa*********@co mcast.netwrote in message
news:11******** **************@ b28g2000cwb.goo glegroups.com.. .
you want a directory watching daemon. it isn't hard at all to build
from scratch.
first, determine which directories should be watched.
then, os.walk each directory, building a mapping from filename to mtime
[modified time; os.path.getmtim e].
next is your main event loop. this while loop consists of os.walk-ing
each directory again, comparing the current mtime to the corresponding
entry in the mapping. if they differ, or if a filename isn't in the
mapping, something happened, at which point you can logick out whether
a file was moved, deleted, changed, or created.

so many folks have looked for this that i'll just write a generic one
and put it in the cheeseshop. look for "dirmon" in about a week.

While I am a fan of "brute force"
Jul 9 '06 #4
"faulkner" <fa*********@co mcast.netwrote in message
news:11******** **************@ b28g2000cwb.goo glegroups.com.. .
you want a directory watching daemon. it isn't hard at all to build
from scratch.
first, determine which directories should be watched.
then, os.walk each directory, building a mapping from filename to mtime
[modified time; os.path.getmtim e].
next is your main event loop. this while loop consists of os.walk-ing
each directory again, comparing the current mtime to the corresponding
entry in the mapping. if they differ, or if a filename isn't in the
mapping, something happened, at which point you can logick out whether
a file was moved, deleted, changed, or created.

so many folks have looked for this that i'll just write a generic one
and put it in the cheeseshop. look for "dirmon" in about a week.

Ahem... (sorry for premature usenet-post-ication...)

While I am a big fan of "brute force", there are OS services (at least on
Windows) for doing just this function, with asynchronous callbacks when
files are created, deleted, etc.

Here is a link that does a much better comparison of several options than I
could (including your brute force version):
http://tgolden.sc.sabren.com/python/...r_changes.html

Good luck!
-- Paul
Jul 9 '06 #5
faulkner wrote:
you want a directory watching daemon. it isn't hard at all to build
from scratch.
first, determine which directories should be watched.
then, os.walk each directory, building a mapping from filename to mtime
[modified time; os.path.getmtim e].
next is your main event loop. this while loop consists of os.walk-ing
each directory again, comparing the current mtime to the corresponding
entry in the mapping. if they differ, or if a filename isn't in the
mapping, something happened, at which point you can logick out whether
a file was moved, deleted, changed, or created.

so many folks have looked for this that i'll just write a generic one
and put it in the cheeseshop. look for "dirmon" in about a week.
Yes, I _know_ about it and exactly this knowledge is the reason I am
looking for tracking single file system related _events_ as I expect a
professional operating system like Windows to provide such service. If
there is none, this will be sure a severe reason to go for Linux if it
has such one instead of going for a SVN server or special file systems
if there are any.

Has someone experience with SVN handling million(s) of files?

The problem is, that brute force applied to large amount of
files/directories is not a convenient way to backup/synchronize the few
new/changed/deleted/moved files/directories multiple times a day as the
brute force approach just makes the hard drive(s) unnecessary wasting
much energy and getting hot.

Claudio Grondi
>

Claudio Grondi wrote:
>>I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from within
a Python script I see a chance, that someone in this group was into it
already and is so kind to share here his experience.

I have put already much efforts into this subject googling around, but
up to now in vain. Best option I encountered yet is usage of
the Greyware 'System Change Log' service which monitors disks for
changes (http://www.greyware.com/software/sys...log/index.asp),
but in own tests it turned out, that the created log file does not cover
all file events as e.g. it is not possible to detect when a file is
moved to a new directory (creation of a new file is logged, but deletion
is not, not mentioning I would expect a file 'move' event).
The own Windows logging service rejected to start on my XP SP2 system
for unknown to me reasons - I don't know how to get it to work (yes, I
have used the administrator account).

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi

Jul 9 '06 #6
Tim Golden wrote:
Claudio Grondi wrote:
>I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from
within a Python script I see a chance, that someone in this group was
into it already and is so kind to share here his experience.

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi


On the offchance that you haven't seen it, you might
look at this:

http://timgolden.me.uk/python/win32_...rectorychanges
but since it doesn't fulfil your criterion of *not*
representing renames by a delete and an add, it may
well not be suitable. Apart from that, I think it does
what you want.

TJG
It seems, that it will be necessary to use some logic based on the
sequence of events to exactly detect rename and move changes done to
files/directories, but in principle it is the best approach I know about
yet.

Thank you!

By the way:
Is there something similar/same available for Linux?

Claudio Grondi
Jul 9 '06 #7
Tim Golden wrote:
Claudio Grondi wrote:
>I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from
within a Python script I see a chance, that someone in this group was
into it already and is so kind to share here his experience.

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi


On the offchance that you haven't seen it, you might
look at this:

http://timgolden.me.uk/python/win32_...rectorychanges
but since it doesn't fulfil your criterion of *not*
representing renames by a delete and an add, it may
well not be suitable. Apart from that, I think it does
what you want.

TJG
Here a small update to the code at
http://timgolden.me.uk/python/win32_...rectorychanges
:

ACTIONS = {
1 : "Created",
2 : "Deleted",
3 : "Updated",
4 : "Renamed from something"
5 : "Renamed to something",
}

The correction above is according to entries:
#define FILE_ACTION_ADD ED 0x00000001
#define FILE_ACTION_REM OVED 0x00000002
#define FILE_ACTION_MOD IFIED 0x00000003
#define FILE_ACTION_REN AMED_OLD_NAME 0x00000004
#define FILE_ACTION_REN AMED_NEW_NAME 0x00000005
in ..\PlatformSDK\ Include\WinNT.h

Claudio Grondi
Jul 10 '06 #8
Claudio Grondi wrote:
Here a small update to the code at
http://timgolden.me.uk/python/win32_...rectorychanges
:

ACTIONS = {
1 : "Created",
2 : "Deleted",
3 : "Updated",
4 : "Renamed from something"
5 : "Renamed to something",
}

The correction above is according to entries:
#define FILE_ACTION_ADD ED 0x00000001
#define FILE_ACTION_REM OVED 0x00000002
#define FILE_ACTION_MOD IFIED 0x00000003
#define FILE_ACTION_REN AMED_OLD_NAME 0x00000004
#define FILE_ACTION_REN AMED_NEW_NAME 0x00000005
in ..\PlatformSDK\ Include\WinNT.h

Claudio Grondi
Thanks. I've updated the site.

TJG
Jul 10 '06 #9
[Tim Golden]
>On the offchance that you haven't seen it, you might
look at this:

http://timgolden.me.uk/python/win32_...rectorychanges
[Claudio Grondi]
It seems, that it will be necessary to use some logic based on the
sequence of events to exactly detect rename and move changes done to
files/directories, but in principle it is the best approach I know about
yet.

By the way:
Is there something similar/same available for Linux?
I've never used them, but I seem to think there are a couple
of similar things for Linux, based on FAM or inotify:

(result of Googling)

http://python-fam.sourceforge.net/
http://www.gnome.org/~veillard/gamin/python.html
http://rudd-o.com/projects/python-inotify/

YMMV
TJG
Jul 10 '06 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
11274
by: j vickroy | last post by:
Hello, I'm trying to understand the behavior of the Python 2.3 logging module (MS Windows 2k) with regard to RotatingFileHandler. The following script illustrates a puzzling problem. What is wrong with this script? Thanks, -- jv
1
3696
by: j vickroy | last post by:
My system: MSW XP professional Python 2.3.3 logging package: 0.4.9.2 My problem: The log_test3.py script, provided with the logging package distribution, generates an unexpected message: No handlers could be found for logger "root"
0
1679
by: Neil Benn | last post by:
Hello, I'm running a test and having issues with logging, if I call logging.shutdown() and then want to start the logging going again then I get a problem as if I call shutdown, I can't get the root logger again, such as : ..>>> import logging ..>>> objTestLogger = logging.getLogger() ..>>> objTestLogger.setLevel(logging.INFO)
8
1547
by: qwweeeit | last post by:
Hi all, I wonder if it is possible to change (temporarily) a built-in function for logging purposes. Let me explain: I want to log all the 'open' operations, recording the file to be opened, the "mode" (r/w/a...) and (possibly) the module which made the call. After that the 'open' can carry on following the standard built-in path. Possibly I don't want to have to modify the source of the application
1
2371
by: Oliver Eichler | last post by:
Hi, I experience several exceptions from python's logging system when using the rollover feature on Windows. Traceback (most recent call last): File "c:\Python24\lib\logging\handlers.py", line 62, in emit if self.shouldRollover(record): File "c:\Python24\lib\logging\handlers.py", line 132, in shouldRollover self.stream.seek(0, 2) #due to non-posix-compliant Windows feature
1
2273
by: timb | last post by:
Hi, does any have any sample code of the above? i have tried the example from the help but am unable to get my windows service to create a new logfile and start logging to it. I am able to create one however if my application is a windows form application. Thanks in advance
16
2174
by: Einar Høst | last post by:
Hi, I'm getting into the Trace-functionality in .NET, using it to provide some much-needed logging across dlls in the project we're working on. However, being a newbie, I'm wondering if some more experienced loggers can provide me with some ideas as to how to log in a simple yet flexible manner. For instance, I'd like the code to be as uncluttered as possible by Trace statements. As an example of basic logging functionality, I've come...
7
8584
by: flupke | last post by:
Hi, i'm getting errors with the log module concerning RotatingFileHandler. I'm using Python 2.4.3 on Windows XP SP2. This used to work in previous python versions but since i upgraded to 2.4.3 i get these errors: Traceback (most recent call last): File "C:\Python24\lib\logging\handlers.py", line 71, in emit if self.shouldRollover(record):
0
1237
by: Chris Curvey | last post by:
Hi all, I just upgraded to 2.4.3 (from 2.4.1) on Windows. Now each time I run my unit tests, they always throw this error at the end of the test run: Error in atexit._run_exitfuncs: Traceback (most recent call last): File "c:\python24\lib\atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "c:\python24\lib\logging\__init__.py", line 1333, in shutdown
0
8974
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8800
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9500
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9349
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9218
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
8227
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6772
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
2
2765
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2199
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.