By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
434,677 Members | 1,070 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 434,677 IT Pros & Developers. It's quick & easy.

how to know if folder contents have changed

P: n/a
hi
i am trying to create a cache of digitized values of around 100
image files in a folder..In my program i would like to know from time
to time if a new image has been added or removed from the folder..

one scheme suggested was to create a string from the names of sorted
image files and give it as the cache name..
ie ,if i have one.jpg,three.jpg,new.jpg ,
i will name the cache as 'newonethree.cache' and everytime i want to
check if new image added/removed i wd create a string from the
contents of folder and compare it with cachename.

this scheme is ok for a small number of files,..

can someone suggest a better way? i know it is a general programming
problem..but i wish to know if a python solution exists

Nov 12 '07 #1
Share this Question
Share on Google+
7 Replies


P: n/a
On Sun, 11 Nov 2007 21:03:33 -0800, de****@gmail.com wrote:
one scheme suggested was to create a string from the names of sorted
image files and give it as the cache name..
ie ,if i have one.jpg,three.jpg,new.jpg ,
i will name the cache as 'newonethree.cache' and everytime i want to
check if new image added/removed i wd create a string from the
contents of folder and compare it with cachename.

this scheme is ok for a small number of files,..
Not really.

`xxx.jpg` -`xxx.cache`

Now `xxx.jpg` is deleted and `x.jpg` and `xx.jpg` are created.

`x.jpg`, `xx.jpg` -`xxx.cache`
can someone suggest a better way? i know it is a general programming
problem..but i wish to know if a python solution exists
Don't store the names in the cache file name but in the cache file. Take
a look at the `set()` type for operations to easily find out the
differences between two set of names and the `pickle` module to store
Python objects in files.

Ciao,
Marc 'BlackJack' Rintsch
Nov 12 '07 #2

P: n/a
de****@gmail.com wrote:
can someone suggest a better way? i know it is a general programming
problem..but i wish to know if a python solution exists
Use pyfam. I believe all docs are in fam but it integrates with that.
Nov 12 '07 #3

P: n/a
On Nov 11, 11:03 pm, "dev...@gmail.com" <dev...@gmail.comwrote:
hi
i am trying to create a cache of digitized values of around 100
image files in a folder..In my program i would like to know from time
to time if a new image has been added or removed from the folder..
Why not use the file creation/modification timestamps?

Nov 12 '07 #4

P: n/a
2007/11/12, da*********@gmail.com <da*********@gmail.com>:
Why not use the file creation/modification timestamps?
because you'd have to

a) create a thread that pulls all the time for changes or
b) test everytime for changes

fam informs in a notification like way.

Personally I'd create a "hidden" cache file parsable by configparser
and have filename = $favorite_checksum_algo - key value pairs in it if
it's not a long running process.

Otherwise I'd probably go with fam (or hal i think that's the other
thing that does that)

hth
martin

--
http://noneisyours.marcher.name
http://feeds.feedburner.com/NoneIsYours
Nov 12 '07 #5

P: n/a
On Nov 12, 11:27 am, "Martin Marcher" <mar...@marcher.namewrote:
2007/11/12, davisn90...@gmail.com <davisn90...@gmail.com>:
Why not use the file creation/modification timestamps?

because you'd have to

a) create a thread that pulls all the time for changes or
Given that it would only involve a check of one timestamp (the
directory the files are located in), I don't think polling "from time
to time" would be unreasonable. The modification timestamp of the
directory should be sufficient given the use case. Even if it's not,
tracking modification times for the files in the directory would not
be unreasonable.
b) test everytime for changes
Checking a timestamp should be a very quick operation. Unless
"everytime" occurs *very* frequently, it's certainly not unreasonable.
fam informs in a notification like way.
FAM would work too. However,
1) According to http://oss.sgi.com/projects/fam/faq.html#what_os_fam,
FAM "should be fairly easy to port to ... Unix-like operating
systems ....". If the original poster is a user of a "Uniix-like
operating system" he/she may actually be able to use it. Regardless,
it seems to me that you would lose a great deal of portability (i.e.,
is there a Windows port?), which may or may not be important to the
poster.
2) FAM undoubtedly uses some system resources. Probably very little,
but it's still an overhead that must be taken into account.
3) You still need to use another method for maintaining state across
program invocations, do you not?

Using timestamps are:
1) Portable. Can you name one OS that does not provide timestamps?
Last I checked, even Windows does :-)
2) Storage efficient. I don't have to actually *store* the
timestamps. I can just check to see if a file/directory was modified
after the last time I checked.
3) Easy to maintain persistent state -- just store the timestamp!
Personally I'd create a "hidden" cache file parsable by configparser
and have filename = $favorite_checksum_algo - key value pairs in it if
it's not a long running process.
What is your reasoning for this? It seems to me that it is
inefficient and unreliable. First of all you have to compute the
checksum (which undoubtedly would involve reading every byte the file)
-- not just once, but "everytime" (or however often you perform the
check). Secondly, it is possible for the checksum to be the same even
if the file has changed. Unlikely? Perhaps (depends on checksum
algorithm used). Impossible? No. So, in effect, you are using a
"slow" algorithm that is known to give incorrect results in certain
cases -- all to replace something as basic as timestamps?
Otherwise I'd probably go with fam (or hal i think that's the other
thing that does that)

hth
martin

--http://noneisyours.marcher.namehttp://feeds.feedburner.com/NoneIsYours
Thanks for the critique -- feel free to punch holes.

--Nathan Davis

Nov 14 '07 #6

P: n/a
I think that without further information from the OP about the
requirements all we can do is guessing. So both of our solutions are
just theory after all (just my personal opinion)

2007/11/14, da*********@gmail.com <da*********@gmail.com>:
On Nov 12, 11:27 am, "Martin Marcher" <mar...@marcher.namewrote:
2007/11/12, davisn90...@gmail.com <davisn90...@gmail.com>:

a) create a thread that pulls all the time for changes or

Given that it would only involve a check of one timestamp (the
directory the files are located in), I don't think polling "from time
to time" would be unreasonable. The modification timestamp of the
directory should be sufficient given the use case. Even if it's not,
tracking modification times for the files in the directory would not
be unreasonable.
Not for the 400 Files but the OP asks about more files too. How about
40.000 files or 400.000 files? That could be a problem...
b) test everytime for changes

Checking a timestamp should be a very quick operation. Unless
"everytime" occurs *very* frequently, it's certainly not unreasonable.
See above I think it also depends on the number of files
fam informs in a notification like way.

FAM would work too. However,
1) According to http://oss.sgi.com/projects/fam/faq.html#what_os_fam,
FAM "should be fairly easy to port to ... Unix-like operating
systems ....". If the original poster is a user of a "Uniix-like
operating system" he/she may actually be able to use it. Regardless,
it seems to me that you would lose a great deal of portability (i.e.,
is there a Windows port?), which may or may not be important to the
poster.
I don't use windows so speaking about portability you are right. It
may be a personal thing but I stopped providing solution (or trying to
think about them) for windows (another discussion probably best placed
in a forum about social interests or something....)
2) FAM undoubtedly uses some system resources. Probably very little,
but it's still an overhead that must be taken into account.
Both is true but most Linux distributions do use FAM at some point
anyway so the overhead is actually very little. Also I think that on
most OSs there is a similiar thing like FAM that could be used...
3) You still need to use another method for maintaining state across
program invocations, do you not?
You need some method no matter wether your program is a long running
process or just invoked in irregular intervals.

After all I'm pretty sure that there is something FAM like that is
available on most OSs. FAM isn't probably available on OSX either but
I guess they provide some mechanism. If you want it really portable
I'd use an abstraction layer that tries to communicate with some
notification daemon which is probably available on the host os and if
all that fails provide a fallback implementation that does naive
tests. All accessible thru the same abstraction interface.
Using timestamps are:
1) Portable. Can you name one OS that does not provide timestamps?
Last I checked, even Windows does :-)
2) Storage efficient. I don't have to actually *store* the
timestamps. I can just check to see if a file/directory was modified
after the last time I checked.
read below, a changed timestamp isn't necessarily a sign that a file
has indeed changed (backups, ....)
3) Easy to maintain persistent state -- just store the timestamp!
Well >>>I don't have to actually *store* the timestamps.<<< and
>>>just store the timestamp!<<< are a bit confusing. I think you
absolutely need to store the timestamp since between runs you won't
know what to check for anyway (new files, deleted files, changed files
- if these cases are important to you)
Personally I'd create a "hidden" cache file parsable by configparser
and have filename = $favorite_checksum_algo - key value pairs in it if
it's not a long running process.

What is your reasoning for this?
because all I need to do to check for changes is getCache(configFile)
and compare the results to getActual(os.listdir) and those 2 methods
would give me the needed info (of course I'm just blindly guessing as
I don't know anything about the further requirements)

Of course with a lot of files this could be a problem. I wouldn't want
a configparser object with 40.000 (or even just a few thousand)
entries to be alive all the time. You'd probably have to create some
iterator for the file so that you can check thru the entries in a
memory efficient way...
It seems to me that it is
inefficient and unreliable. First of all you have to compute the
checksum (which undoubtedly would involve reading every byte the file)
-- not just once, but "everytime" (or however often you perform the
check). Secondly, it is possible for the checksum to be the same even
if the file has changed. Unlikely? Perhaps (depends on checksum
algorithm used). Impossible? No. So, in effect, you are using a
"slow" algorithm that is known to give incorrect results in certain
cases -- all to replace something as basic as timestamps?
It seems you are absolutely linking checksum with something like md5 or sha...

Maybe that was badly stated, depending on the use case of course a
timestamp could also be considered a valid checksum. However to be
safe some timestamp isn't really giving me the information. A lot of
backup tools do update the timestamp (atime in unix, dunno about
windows) and that could lead to even more wasting of resources.

Consider you are checking some CSV files with timestampt which upon
change initiate some real intensive number crunching. Now you do that
because you figured "Hey the timestamp has changed, I need to redo my
calculations..." while in fact just the backup programm was running.
But as I said it depends on the use case what you consider a valid to
know that a file changed...

So the checksum algo is something that should be chosen depending on

a) interval of checks (like you say)
b) need to be sure that 2 files don't actually have the same checksum

I guess a simple approach could be something like the Message-ID
header in emails, a bit adapted to local use cases.
Otherwise I'd probably go with fam (or hal i think that's the other
thing that does that)


--
http://noneisyours.marcher.name
http://feeds.feedburner.com/NoneIsYours
Nov 17 '07 #7

P: n/a
I just found this for win32 which seems to be the same as FAM provides:
http://tgolden.sc.sabren.com/python/...r_changes.html
So it's not about FAM as a definitive product to be used but more like
something nearer to the OS that is there anyway and will tell you
about it...

--
http://noneisyours.marcher.name
http://feeds.feedburner.com/NoneIsYours
Nov 17 '07 #8

This discussion thread is closed

Replies have been disabled for this discussion.