wait until change

[Tom]

Hi,

I have the following problem. I analyse data from a file. When I am done
analysing all the data that was in that file, I want to let my program
wait until this specific file has changed (which indicates that new data
has been added to the file from a third party program that I don't
control). If that file has changed, I want my program to continue.

What would be the appropriate command to check if the file changed? And
how can I implement "wait until" into my program? I googled a lot to
find any hints, but couldn't find anything helpful. But I found
something about a python that was stabbed in NY in 2000. :-) Thought
that was funny. :-)

I would deeply appreciate, if you could maybe include some little
fragments of code to show me how to use your suggestions. I am
relatively new to python and I often have problems with Pythons Library
Reference, because it doesn't show examples and is very technically.
That makes it difficult for me to really understand that. It also would
be great, if you could tell me which modules I have to include. I hope
all that is not asked too much. I assume that both problems can be
solved with a few easy lines of code. I just don't know them. :-)

Thanks a lot for your help and suggestions.
Regards Tom

[Matthew Wilson]

I just took a midterm in my operating systems class, and a big topic was
"busy waiting" and semaphores.

And now, today, a real-world example! Anyway, the time.sleep() method
is probably good enough, but how would a semaphore solution work in
python? A semaphore solution is one where the process that uses the
updated file is asleep until another process says "Hey! Wake up! I
updated the file!". This avoids all those check_for_updat e() calls
which bog down the CPU.

Anyone?

[Peter Hansen]

Matthew Wilson wrote:

I just took a midterm in my operating systems class, and a big topic was
"busy waiting" and semaphores.

And now, today, a real-world example! Anyway, the time.sleep() method
is probably good enough, but how would a semaphore solution work in
python? A semaphore solution is one where the process that uses the
updated file is asleep until another process says "Hey! Wake up! I
updated the file!". This avoids all those check_for_updat e() calls
which bog down the CPU.

In this case, it appears the OP doesn't have the capability of changing
the other application, so what you suggest is out.

Dave's example with WMI uses the *operating system* to essentially
signal semaphore-like that an update has been made. As I suggested,
it's obviously much more complicated than the simple time.sleep()
plus os.stat() approach, but there are times when it's the right
way to approach this problem.

-Peter

[Thomas Heller]

Peter Hansen <pe***@engcorp. com> writes:

Matthew Wilson wrote:

I just took a midterm in my operating systems class, and a big topic was
"busy waiting" and semaphores.

And now, today, a real-world example! Anyway, the time.sleep() method
is probably good enough, but how would a semaphore solution work in
python? A semaphore solution is one where the process that uses the
updated file is asleep until another process says "Hey! Wake up! I
updated the file!". This avoids all those check_for_updat e() calls
which bog down the CPU.
In this case, it appears the OP doesn't have the capability of changing
the other application, so what you suggest is out.

Dave's example with WMI uses the *operating system* to essentially
signal semaphore-like that an update has been made. As I suggested,
it's obviously much more complicated than the simple time.sleep()
plus os.stat() approach, but there are times when it's the right
way to approach this problem.

Just for completeness, it should be mentioned that the win32 api
functions FindFirstChange Notification and FindNextChangeN otification
could also be used. They are wrapped in win32all, and probably could
also be made to work with ctypes. Platform dependend, sure, but maybe a
much more lightweight approach than using wmi (and, for me at least,
probably better understandable) .
-Peter

Thomas

[Michael Geary]

Thomas Heller wrote:

Just for completeness, it should be mentioned that the win32 api
functions FindFirstChange Notification and FindNextChangeN otification
could also be used. They are wrapped in win32all, and probably could
also be made to work with ctypes. Platform dependend, sure, but maybe a
much more lightweight approach than using wmi (and, for me at least,
probably better understandable) .

And to be *really* complete, Windows NT/2000/XP has the
ReadDirectoryCh angesW function, which has some improvements over the
Find...ChangeNo tification functions. Not available on Win95/98/Me, though.

-Mike

[Heiko Wundram]

Michael Geary wrote:

And to be *really* complete, Windows NT/2000/XP has the
ReadDirectoryC hangesW function, which has some improvements over the
Find...ChangeN otification functions. Not available on Win95/98/Me, though.

-Mike

And to be really, really complete, Unix (in my case Linux, afaik also
works under Solaris and HP-UX) has FAM (the file alteration monitor),
which works as a user-space daemon to which a program can attach to get
a file/directory change notification. The daemon itself uses a special
kernel module to get these notification right from the VFS (virtual file
system).

I recall that there was a Python-wrapper somewhere for the fam client
library.

HTH!

Heiko.

[Aahz]

In article <bm**********@n ews.uni-kl.de>, Tom <ll************ @gmx.net> wrote:

I have the following problem. I analyse data from a file. When I am done
analysing all the data that was in that file, I want to let my program
wait until this specific file has changed (which indicates that new data
has been added to the file from a third party program that I don't
control). If that file has changed, I want my program to continue.

What would be the appropriate command to check if the file changed? And
how can I implement "wait until" into my program? I googled a lot to
find any hints, but couldn't find anything helpful. But I found
something about a python that was stabbed in NY in 2000. :-) Thought
that was funny. :-)

os.stat() is the correct answer, but I question the other responses that
suggest using time.sleep(). Unless your application stores a lot of
data in memory, you might be better off using your OS to periodically
run your application; your application stores the result of os.stat() in
a private file. On Unix-like systems, the facility is called cron;
dunno what that would be on Windows. This would simplify your program;
it also means your program would automatically start when the system
gets rebooted.
--
Aahz (aa**@pythoncra ft.com) <*> http://www.pythoncraft.com/

"It is easier to optimize correct code than to correct optimized code."
--Bill Harlan

[Bryan]

Aahz wrote:

In article <bm**********@n ews.uni-kl.de>, Tom <ll************ @gmx.net> wrote:

I have the following problem. I analyse data from a file. When I am done
analysing all the data that was in that file, I want to let my program
wait until this specific file has changed (which indicates that new data
has been added to the file from a third party program that I don't
control). If that file has changed, I want my program to continue.

What would be the appropriate command to check if the file changed? And
how can I implement "wait until" into my program? I googled a lot to
find any hints, but couldn't find anything helpful. But I found
something about a python that was stabbed in NY in 2000. :-) Thought
that was funny. :-)

os.stat() is the correct answer, but I question the other responses that
suggest using time.sleep(). Unless your application stores a lot of
data in memory, you might be better off using your OS to periodically
run your application; your application stores the result of os.stat() in
a private file. On Unix-like systems, the facility is called cron;
dunno what that would be on Windows. This would simplify your program;
it also means your program would automatically start when the system
gets rebooted.

i'm currently running a python script that automatically runs when the system gets rebooted on windows. i use the
"scheduled tasks" control panel applet. you can have your program scheduled to run weekly, monthly, daily, once, at
system startup, at logon, or when idle. you can specify start/end times, durations, how to repeat the task, plus a
bunch more options.

bryan

[Mark Roach]

On Sat, 18 Oct 2003 03:29:33 +0200, Heiko Wundram wrote:

Michael Geary wrote:

And to be *really* complete, Windows NT/2000/XP has the
ReadDirectory ChangesW function, which has some improvements over the
Find...Change Notification functions. Not available on Win95/98/Me, though.

-Mike

And to be really, really complete, Unix (in my case Linux, afaik also
works under Solaris and HP-UX) has FAM (the file alteration monitor),
which works as a user-space daemon to which a program can attach to get
a file/directory change notification. The daemon itself uses a special
kernel module to get these notification right from the VFS (virtual file
system).

I recall that there was a Python-wrapper somewhere for the fam client
library.

There also appears to be a python wrapper for dnotify which is the
kernel-level file-change notification system so that you don't have to
have famd and portmap running just to get file notifications.. . It appears
to be apart of twisted according to this message:
http://twistedmatrix.com/pipermail/t...il/003873.html

-Mark

[Anand Pillai]

If you dont want to use sleep(), try this mechanism
based on 'Timer' class in the threading module.

1. Create a timer thread that calls a function
which checks for the time stamp using os.stat().
2. Schedule the timer to wake up after 'n' seconds
and call this function.
3. If the call succeeds all and well, otherwise in
the function, create a new timer and set it again
to wake itself up after another 'n' seconds.

A timer can be used only once so you need to make a new timer
for every wake-up call. Perhaps not very good on memory,
but it might save you the CPU cycles.

You can wrap the whole thing in a class with the timer
bound to a local class variable, and the function as
another class attribute.

HTH.

-Anand

Tom <ll************ @gmx.net> wrote in message news:<bm******* ***@news.uni-kl.de>...

Lane LiaBraaten wrote:

I would use os.stat() which returns all sorts of info about the file including
modification time.

More info at: http://web.pydoc.org/1.5.2/os.html

time=os.stat(f ile)[8]

# analyze data

while 1:
if os.stat(file)[8]>time:
#analyze data again

Hi Lane,

thanks for your answer it works great and gives me a lot of new ideas
that I can work with.
That "while 1" command looks interessting. I never used while that way.
The Library Reference doesn't explain it. But am I right if I assume
that this is a loop that continues forever? Despite using break of
course. I ask, because your suggestion works, but unfortunately uses
100% of my CPU power. :-( Well, it works, but is there maybe another way
to do the same without using so much of my resources? :-) That would be
great.

Thank's again.
Tom

[Anand Pillai]

I thought this was fairly straightforward , but when
I tried to write one I figured out that the standard
python Timers cannot be used.

Anyway here is how you would go about doing it.
This piece of code uses a time out as an exit condition,
whereas in the scenario given by the O.P it would be
the file's timestamp verification.

----------------------------------------------------
from threading import Thread
import time

class ModifiedTimer(T hread):
""" Modified timer with a maximum timeout
and repeated calls """

def __init__(self, interval, maxtime, target):
self.__interval = interval
self.__maxtime = maxtime
self.__tottime = 0.0
self.__target = target
self.__flag = 0
Thread.__init__ (self, None, 'mytimer', None)

def run(self):

while self.__flag==0:
time.sleep(self .__interval)
self.__target()
self.__tottime += self.__interval

if self.__tottime >= self.__maxtime:
print 'Timing out...'
self.end()

def end(self):
self.__flag=1

class MyTimer:

def __init__(self):
self.__interval = 5.0

def __timerfunc(sel f):
print 'Hello, this is the timer function!'

def make_timer(self , interval):
self.__interval = interval
self.__timer = ModifiedTimer(s elf.__interval, 60.0, self.__timerfun c)
self.__timer.st art()

if __name__=="__ma in__":
t=MyTimer()
t.make_timer(10 .0)

----------------------------------------------------------------

The code works as expected. I tested on Windows 98
using python 2.3.

-Anand Pillai


py*******@Hotpo p.com (Anand Pillai) wrote in message news:<84******* *************** ****@posting.go ogle.com>...

If you dont want to use sleep(), try this mechanism
based on 'Timer' class in the threading module.

1. Create a timer thread that calls a function
which checks for the time stamp using os.stat().
2. Schedule the timer to wake up after 'n' seconds
and call this function.
3. If the call succeeds all and well, otherwise in
the function, create a new timer and set it again
to wake itself up after another 'n' seconds.

A timer can be used only once so you need to make a new timer
for every wake-up call. Perhaps not very good on memory,
but it might save you the CPU cycles.

You can wrap the whole thing in a class with the timer
bound to a local class variable, and the function as
another class attribute.

HTH.

-Anand

Tom <ll************ @gmx.net> wrote in message news:<bm******* ***@news.uni-kl.de>...

Lane LiaBraaten wrote:

I would use os.stat() which returns all sorts of info about the file including
modification time.

More info at: http://web.pydoc.org/1.5.2/os.html

time=os.stat(f ile)[8]

# analyze data

while 1:
if os.stat(file)[8]>time:
#analyze data again

Hi Lane,

thanks for your answer it works great and gives me a lot of new ideas
that I can work with.
That "while 1" command looks interessting. I never used while that way.
The Library Reference doesn't explain it. But am I right if I assume
that this is a loop that continues forever? Despite using break of
course. I ask, because your suggestion works, but unfortunately uses
100% of my CPU power. :-( Well, it works, but is there maybe another way
to do the same without using so much of my resources? :-) That would be
great.

Thank's again.
Tom