How to find out if another process is using a file

[Tom Wright]

I'm writing a program which reads a series of data files as they are dumped
into a directory by another process. At the moment, it gets sporadic bugs
when it tries to read files which are only partially written.

I'm looking for a function which will tell me if a file is opened in
write-mode by another process - if it is, my program will ignore it for now
and come back to it later. This needs to work on linux and windows. Mac
OS would be a bonus too. An os-independent solution would be good, but I
could write os-specific options and have it pick the appropriate one.

Is there any way of doing this? I've drawn a blank with google so far.

A nasty hack would be to use the file modification time, and wait until
that's a few seconds in the past, but is there a nice solution?
--
I'm at CAMbridge, not SPAMbridge

[js]

How about using lock?
Let writing process locks the files before writing, and unlock after
the job's done.

I think it'd work file in most environment.

On 1/19/07, Tom Wright <te***@spam.ac.ukwrote:

I'm writing a program which reads a series of data files as they are dumped
into a directory by another process. At the moment, it gets sporadic bugs
when it tries to read files which are only partially written.

I'm looking for a function which will tell me if a file is opened in
write-mode by another process - if it is, my program will ignore it for now
and come back to it later. This needs to work on linux and windows. Mac
OS would be a bonus too. An os-independent solution would be good, but I
could write os-specific options and have it pick the appropriate one.

Is there any way of doing this? I've drawn a blank with google so far.

A nasty hack would be to use the file modification time, and wait until
that's a few seconds in the past, but is there a nice solution?
--
I'm at CAMbridge, not SPAMbridge
--
http://mail.python.org/mailman/listinfo/python-list

[Gabriel Genellina]

"Tom Wright" <te***@spam.ac.ukescribió en el mensaje
news:eo**********@gemini.csx.cam.ac.uk...

I'm writing a program which reads a series of data files as they are
dumped
into a directory by another process. At the moment, it gets sporadic bugs
when it tries to read files which are only partially written.

I'm looking for a function which will tell me if a file is opened in
write-mode by another process - if it is, my program will ignore it for
now
and come back to it later. This needs to work on linux and windows. Mac
OS would be a bonus too. An os-independent solution would be good, but I
could write os-specific options and have it pick the appropriate one.

Use os.open with the O_EXCL flag; will fail if the other process has the
file still open (and will fail if another process is reading the file, too,
not just if someone is writing).

--
Gabriel Genellina

[Donn Cave]

In article <ma***************************************@python. org>,
"Gabriel Genellina" <ga******@yahoo.com.arwrote:

"Tom Wright" <te***@spam.ac.ukescribió en el mensaje
news:eo**********@gemini.csx.cam.ac.uk...

I'm writing a program which reads a series of data files as they are
dumped
into a directory by another process. At the moment, it gets sporadic bugs
when it tries to read files which are only partially written.

I'm looking for a function which will tell me if a file is opened in
write-mode by another process - if it is, my program will ignore it for
now
and come back to it later. This needs to work on linux and windows. Mac
OS would be a bonus too. An os-independent solution would be good, but I
could write os-specific options and have it pick the appropriate one.

Use os.open with the O_EXCL flag; will fail if the other process has the
file still open (and will fail if another process is reading the file, too,
not just if someone is writing).

O_EXCL fails if the file exists at all - whether closed or open.

Donn Cave, do**@u.washington.edu

[Nick Maclaren]

In article <do************************@gnus01.u.washington.ed u>,
Donn Cave <do**@u.washington.eduwrites:
|In article <ma***************************************@python. org>,
| "Gabriel Genellina" <ga******@yahoo.com.arwrote:
| "Tom Wright" <te***@spam.ac.ukescribió en el mensaje
| news:eo**********@gemini.csx.cam.ac.uk...
|
| I'm writing a program which reads a series of data files as they are
| dumped
| into a directory by another process. At the moment, it gets sporadic bugs
| when it tries to read files which are only partially written.
| >
| I'm looking for a function which will tell me if a file is opened in
| write-mode by another process - if it is, my program will ignore it for
| now
| and come back to it later. This needs to work on linux and windows. Mac
| OS would be a bonus too. An os-independent solution would be good, but I
| could write os-specific options and have it pick the appropriate one.
|
| Use os.open with the O_EXCL flag; will fail if the other process has the
| file still open (and will fail if another process is reading the file, too,
| not just if someone is writing).
|>
|O_EXCL fails if the file exists at all - whether closed or open.

Yes. In theory. In practice, it usually works on normal files, provided
that all opens are local. Under some circumstances, it will even work
for NFS mounted files, as far as I recall.

MVS (now zOS) and other mainframe systems had what the poster wants, but
modern systems don't. I shall not recommend that the poster asks IBM
for a copy of zOS, for reasons that will be well-known to anyone who used
MVS (which are NOT the ones claimed by the Unix brigade, which are mostly
bogus) :-)

Under Linux, you can do something with fuser, and I am pretty certain that
modern Macintoshes (i.e. BSD) will have an equivalent. It can't be made
reliable (unlike under MVS), but might reduce the number of problems.
Regards,
Nick Maclaren.

[Donn Cave]

In article <eo**********@gemini.csx.cam.ac.uk>,
nm**@cus.cam.ac.uk (Nick Maclaren) wrote:

In article <do************************@gnus01.u.washington.ed u>,
Donn Cave <do**@u.washington.eduwrites:
|In article <ma***************************************@python. org>,
| "Gabriel Genellina" <ga******@yahoo.com.arwrote:
| "Tom Wright" <te***@spam.ac.ukescribió en el mensaje
| news:eo**********@gemini.csx.cam.ac.uk...

| Use os.open with the O_EXCL flag; will fail if the other process has the
| file still open (and will fail if another process is reading the file,
| too,
| not just if someone is writing).
|>
|O_EXCL fails if the file exists at all - whether closed or open.

Yes. In theory. In practice, it usually works on normal files, provided
that all opens are local. Under some circumstances, it will even work
for NFS mounted files, as far as I recall.

Mm, by "fail", I meant

An attempt to open with O_EXCL set will "fail" if the file exists at all,
i.e., the file will not be opened, a negative value will be returned,
and errno will be set to EEXIST.

What I neglected to mention is that this effect obtains when O_EXCL
is used in combination with O_CREAT. Without O_CREAT, O_EXCL doesn't
mean anything and is ignored.

If there is any significant difference between theory and practice
in this matter, it's news to me.

Donn

[Nick Maclaren]

In article <do************************@gnus01.u.washington.ed u>,
Donn Cave <do**@u.washington.eduwrites:
| |>
| |O_EXCL fails if the file exists at all - whether closed or open.
|
| Yes. In theory. In practice, it usually works on normal files, provided
| that all opens are local. Under some circumstances, it will even work
| for NFS mounted files, as far as I recall.
|>
|Mm, by "fail", I meant
|>
|An attempt to open with O_EXCL set will "fail" if the file exists at all,
|i.e., the file will not be opened, a negative value will be returned,
|and errno will be set to EEXIST.
|>
|What I neglected to mention is that this effect obtains when O_EXCL
|is used in combination with O_CREAT. Without O_CREAT, O_EXCL doesn't
|mean anything and is ignored.
|>
|If there is any significant difference between theory and practice
|in this matter, it's news to me.

Actually, it is undefined behaviour in POSIX, and I have used a Unix
where O_EXCL on its own failed if the file existed (of course, it also
failed if it DIDN'T exist, because of the lack of O_CREAT). I can't
remember which - perhaps AIX or OSF/1 (not DEC).

But I was referring to a different problem. Some remote and parallel
filing systems handle such things very badly, and it is often possible
to create a file even if it exists and O_CREAT|O_EXCL is set. A similar
remark applies to 'special' files, even on fairly mainstream, local
filing systems.
Regards,
Nick Maclaren.

[Tom Wright]

js wrote:

How about using lock?
Let writing process locks the files before writing, and unlock after
the job's done.

Is locking mandatory or co-operative? I don't have any control over the
process which is doing the writing, so if it's co-operative it's no good to
me.

If it's mandatory, then I can try to acquire a lock on the file - if this
fails or blocks, then the other process must have it open. Will this work?

[Diez B. Roggisch]

Tom Wright wrote:

js wrote:

>How about using lock?
Let writing process locks the files before writing, and unlock after
the job's done.

Is locking mandatory or co-operative? I don't have any control over the
process which is doing the writing, so if it's co-operative it's no good
to me.

If it's mandatory, then I can try to acquire a lock on the file - if this
fails or blocks, then the other process must have it open. Will this
work?

AFAIK it's cooperative.

Diez