Determining when a file has finished copying

[writeson]

Hi all,

I'm writing some code that monitors a directory for the appearance of
files from a workflow. When those files appear I write a command file
to a device that tells the device how to process the file. The
appearance of the command file triggers the device to grab the
original file. My problem is I don't want to write the command file to
the device until the original file from the workflow has been copied
completely. Since these files are large, my program has a good chance
of scanning the directory while they are mid-copy, so I need to
determine which files are finished being copied and which are still
mid-copy.

I haven't seen anything on Google talking about this, and I don't see
an obvious way of doing this using the os.stat() method on the
filepath. Anyone have any ideas about how I might accomplish this?

Thanks in advance!
Doug

[Sean DiZazzo]

On Jul 9, 5:34*pm, keith <ke...@keithper kins.netwrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ethan Furman wrote:

writeson wrote:

Guys,

Thanks for your replies, they are helpful. I should have included in
my initial question that I don't have as much control over the program
that writes (pgm-W) as I'd like. Otherwise, the write to a different
filename and then rename solution would work great. There's no way to
tell from the os.stat() methods to tell when the file is finished
being copied? I ran some test programs, one of which continously
copies big files from one directory to another, and another that
continously does a glob.glob("*.pd f") on those files and looks at the
st_atime and st_mtime parts of the return value of os.stat(filenam e).
From that experiment it looks like st_atime and st_mtime equal each
other until the file has finished being copied. Nothing in the
documentation about st_atime or st_mtime leads me to think this is
true, it's just my observations about the two test programs I've
described.

Any thoughts? Thanks!
Doug

The solution my team has used is to monitor the file size. *If the file
has stopped growing for x amount of time (we use 45 seconds) the file is
done copying. *Not elegant, but it works.
--
Ethan

Also I think that matching the md5sums may work. *Just set up so that it
checks the copy's md5sum every couple of seconds (or whatever time
interval you want) and matches against the original's. *When they match
copying's done. I haven't actually tried this but think it may work.
Any more experienced programmers out there let me know if this is
unworkable please.
K
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla -http://enigmail.mozdev .org

iD8DBQFIdVkX8vm NfzrLpqoRAsJ2AK Cp8wMz93Vz8y9K+ MDSP33kH/WHngCgl/wM
qTFBfyIEGhu/dNSQzeRrwYQ=
=Xvjq
-----END PGP SIGNATURE-----

I use a combination of both the os.stat() on filesize, and md5.
Checking md5s works, but it can take a long time on big files. To fix
that, I wrote a simple sparse md5 sum generator. It takes a small
number bytes from various areas of the file, and creates an md5 by
combining all the sections. This is, in fact, the only solution I have
come up with for watching a folder for windows copys.

The filesize solution doesn't work when a user copies into the watch
folder using drag and drop on Windows because it allocates all the
attributes of the file before any data is written. The filesize will
always show the full size of the file.

~Sean

[Larry Bates]

Sean DiZazzo wrote:

On Jul 9, 5:34 pm, keith <ke...@keithper kins.netwrote:

>-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ethan Furman wrote:

>>writeson wrote:
Guys,
Thanks for your replies, they are helpful. I should have included in
my initial question that I don't have as much control over the program
that writes (pgm-W) as I'd like. Otherwise, the write to a different
filename and then rename solution would work great. There's no way to
tell from the os.stat() methods to tell when the file is finished
being copied? I ran some test programs, one of which continously
copies big files from one directory to another, and another that
continousl y does a glob.glob("*.pd f") on those files and looks at the
st_atime and st_mtime parts of the return value of os.stat(filenam e).
From that experiment it looks like st_atime and st_mtime equal each
other until the file has finished being copied. Nothing in the
documentatio n about st_atime or st_mtime leads me to think this is
true, it's just my observations about the two test programs I've
described.
Any thoughts? Thanks!
Doug
The solution my team has used is to monitor the file size. If the file
has stopped growing for x amount of time (we use 45 seconds) the file is
done copying. Not elegant, but it works.
--
Ethan

Also I think that matching the md5sums may work. Just set up so that it
checks the copy's md5sum every couple of seconds (or whatever time
interval you want) and matches against the original's. When they match
copying's done. I haven't actually tried this but think it may work.
Any more experienced programmers out there let me know if this is
unworkable please.
K
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla -http://enigmail.mozdev .org

iD8DBQFIdVkX8v mNfzrLpqoRAsJ2A KCp8wMz93Vz8y9K +MDSP33kH/WHngCgl/wM
qTFBfyIEGhu/dNSQzeRrwYQ=
=Xvjq
-----END PGP SIGNATURE-----

I use a combination of both the os.stat() on filesize, and md5.
Checking md5s works, but it can take a long time on big files. To fix
that, I wrote a simple sparse md5 sum generator. It takes a small
number bytes from various areas of the file, and creates an md5 by
combining all the sections. This is, in fact, the only solution I have
come up with for watching a folder for windows copys.

The filesize solution doesn't work when a user copies into the watch
folder using drag and drop on Windows because it allocates all the
attributes of the file before any data is written. The filesize will
always show the full size of the file.

~Sean

While a lot depends on HOW the copying program does its copy, I've recently been
able to get pyinotify to watch folders. By watching for IN_CLOSE_WRITE events I
can see when files are closed by the writer and then process them instantly
after they have been written. Now if the writer does something like:

open
write
close
open append
write
close
..
..
..

This won't work as well.

FYI,
Larry

[Ethan Furman]

Sean DiZazzo wrote:

On Jul 9, 5:34 pm, keith <ke...@keithper kins.netwrote:

>>-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ethan Furman wrote:

>>>writeson wrote:

Guys,

>>>>Thanks for your replies, they are helpful. I should have included in
my initial question that I don't have as much control over the program
that writes (pgm-W) as I'd like. Otherwise, the write to a different
filename and then rename solution would work great. There's no way to
tell from the os.stat() methods to tell when the file is finished
being copied? I ran some test programs, one of which continously
copies big files from one directory to another, and another that
continous ly does a glob.glob("*.pd f") on those files and looks at the
st_atime and st_mtime parts of the return value of os.stat(filenam e).

>From that experiment it looks like st_atime and st_mtime equal each

other until the file has finished being copied. Nothing in the
documentati on about st_atime or st_mtime leads me to think this is
true, it's just my observations about the two test programs I've
described .

>>>>Any thoughts? Thanks!
Doug

>>>The solution my team has used is to monitor the file size. If the file
has stopped growing for x amount of time (we use 45 seconds) the file is
done copying. Not elegant, but it works.
--
Ethan

Also I think that matching the md5sums may work. Just set up so that it
checks the copy's md5sum every couple of seconds (or whatever time
interval you want) and matches against the original's. When they match
copying's done. I haven't actually tried this but think it may work.
Any more experienced programmers out there let me know if this is
unworkable please.
K
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla -http://enigmail.mozdev .org

iD8DBQFIdVkX8 vmNfzrLpqoRAsJ2 AKCp8wMz93Vz8y9 K+MDSP33kH/WHngCgl/wM
qTFBfyIEGhu/dNSQzeRrwYQ=
=Xvjq
-----END PGP SIGNATURE-----

I use a combination of both the os.stat() on filesize, and md5.
Checking md5s works, but it can take a long time on big files. To fix
that, I wrote a simple sparse md5 sum generator. It takes a small
number bytes from various areas of the file, and creates an md5 by
combining all the sections. This is, in fact, the only solution I have
come up with for watching a folder for windows copys.

The filesize solution doesn't work when a user copies into the watch
folder using drag and drop on Windows because it allocates all the
attributes of the file before any data is written. The filesize will
always show the full size of the file.

~Sean

Good info, Sean, thanks. One more option may be to attempt to rename
the file -- if it's still open for copying, that will fail; success
indicates the copy is done. Of course, as Larry Bates pointed out, this
could fail if the copy is followed by a re-open and appending.
Hopefully that's not an issue for the OP.
--
Ethan

[Wilbert Berendsen]

You could also copy to a different name on the same disk, and when the copying
has been finished just 'move' (mv) the file to the filename the other
application expects. E.g. QMail works this way, writing incoming mails in
folders.

Met vriendelijke groet,
Wilbert Berendsen

--
http://www.wilbertberendsen.nl/
"You must be the change you wish to see in the world."
-- Mahatma Gandhi