Occasional OSError: [Errno 13] Permission denied on Windows

[Alec Wysoker]

Using Python 2.3.5 on Windows XP, I occasionally get OSError: [Errno
13] Permission denied when calling os.remove(). This can occur with a
file that is not used by any other process on the machine, and is
created by the python.exe invocation that is trying to delete it. It
can happen with various pieces of my system - almost anywhere I try to
delete a file.

I have assumed that the problem is that I was holding on to a handle to
the file that I was trying to remove. I have scoured my code and close
any handle to the file that I can find. The intermittent nature of
this problem leads me to believe that I'm not explicitly holding onto a
file object somewhere.

My next theory was that there was some object holding onto a file
handle for which there wasn't an extant reference, but which hadn't
been garbage-collected. So, I tried removing files like this:

try:
os.remove(strPath)
except OSError:
# Wild guess that garbage collection might clear errno 13
gc.collect()
os.remove(strPath)

This does indeed reduce the frequency of the problem, but it doesn't
make it go away completely. I really need to fix this problem, because
I'm developing an app that won't survive if it fails to delete its
files.

Thanks,
Alec Wysoker

[Patrick Maupin]

Alec Wysoker wrote:

Using Python 2.3.5 on Windows XP, I occasionally get OSError:
[Errno 13] Permission denied when calling os.remove(). This can
occur with a file that is not used by any other process on the
machine, and is created by the python.exe invocation that is
trying to delete it. It can happen with various pieces of my
system - almost anywhere I try to delete a file.

The previous company I worked for had a real-time virus scanner that
would cause problems like this. Drove me nuts because I never could
get them to shut the dang thing off.

Regards,
Pat

[Tim Peters]

[Alec Wysoker]

Using Python 2.3.5 on Windows XP, I occasionally get OSError: [Errno
13] Permission denied when calling os.remove(). This can occur with a
file that is not used by any other process on the machine,
How do you know that?
and is created by the python.exe invocation that is trying to delete it. It
can happen with various pieces of my system - almost anywhere I try to
delete a file.

I have assumed that the problem is that I was holding on to a handle to
the file that I was trying to remove. I have scoured my code and close
any handle to the file that I can find. The intermittent nature of
this problem leads me to believe that I'm not explicitly holding onto a
file object somewhere.

My next theory was that there was some object holding onto a file
handle for which there wasn't an extant reference, but which hadn't
been garbage-collected. So, I tried removing files like this:

try:
os.remove(strPath)
except OSError:
# Wild guess that garbage collection might clear errno 13
gc.collect()
os.remove(strPath)

This does indeed reduce the frequency of the problem, but it doesn't
make it go away completely.

Replace gc.collect() there with a short sleep (say, time.sleep(0.2)),
and see whether that does just as well at reducing the frequency of
the problem. My bet is that it will.

Best guess is that some process you haven't thought about yet _is_
opening the file. For example, what you're seeing is common if
Copernic Desktop Search is installed and its "Index new and modified
files on the fly" option is enabled. Other searching/indexing apps,
"file deletion recovery" services, and even some virus scanners can
have similar effects. That's why I asked at the start how you _know_
no other process is opening the file. The symtoms you describe are
consistent with some background-level utility app/service briefly
opening files for its own purposes.

In that case, anything that burns some time and tries again will work
better. Replacing gc.collect() with time.sleep() is an easy way to
test that hypothesis; because gc.collect() does an all-generations
collection, it can consume measurable time.

[Alec Wysoker]

Interesting theory. I do have a virus scanner, and also Google Desktop
Search. Sometimes I get this error when running a large suite of unit
tests. Each unit test starts off by cleaning the test output
directory, and failing if it can't do so. I will see many (hundreds?)
tests fail because the file can't be removed, and then eventually the
file will get removed. My feeling is that if a virus scanner or
desktop search were keeping the file open, that it would only do so
briefly. On the other hand, when I'm running unit tests I'm chewing up
a lot of CPU so perhaps it takes longer in those cases.

If we assume that this theory is correct, then what is the solution?
Sleep for a little while and try again?

[Alec Wysoker]

>> Using Python 2.3.5 on Windows XP, I occasionally get OSError: [Errno

13] Permission denied when calling os.remove(). This can occur with a
file that is not used by any other process on the machine,

How do you know that?

Yeah, good point. I don't really know. I should have said no process
that I am aware of is using the file. I'll give sleep a try instead of
gc. Thanks.

[Grant Edwards]

On 2006-01-05, Alec Wysoker <al***@pobox.com> wrote:

Using Python 2.3.5 on Windows XP, I occasionally get OSError: [Errno
13] Permission denied when calling os.remove(). This can occur with a
file that is not used by any other process on the machine,

How do you know that?

Yeah, good point. I don't really know. I should have said no process
that I am aware of is using the file.

<ObRant>
If you're running XP, there's probably a shocking amount of
stuff going on that you're not aware of. Even if your machine
isn't 0wned by some script kiddie or other bit of spy/adware
you've uknkowingly installed, it's still 0wned by Microsoft.
</ObRant>

--
Grant Edwards grante Yow! Make me look like
at LINDA RONSTADT again!!
visi.com

[Ernst Noch]

Alec Wysoker wrote:

Using Python 2.3.5 on Windows XP, I occasionally get OSError: [Errno
13] Permission denied when calling os.remove(). This can occur with a
file that is not used by any other process on the machine,

How do you know that?

Yeah, good point. I don't really know. I should have said no process
that I am aware of is using the file. I'll give sleep a try instead of
gc. Thanks.

FWIW, you could use Filemon from Sysinternals,
http://www.sysinternals.com/Utilities/Filemon.html

If you are there, be sure to also look the other stuff on that side,
Mark Russinovich a has written a lot of very nice Windows utilities.

[pyguy2]

File attributes may be an issue to. Take look at the recipe at:

http://aspn.activestate.com/ASPN/Coo.../Recipe/303343

which ensures the file attributes are normal before you delete it.

john

[Patrick Maupin]

Tim Peters wrote:

In that case, anything that burns some time and tries again
will work better. Replacing gc.collect() with time.sleep() is
an easy way to test that hypothesis; because gc.collect()
does an all-generations collection, it can consume measurable time.

An slight enhancement to this hypothesis-tester (which might even
approach being production-worthy) would be to maintain a list of items
which were not deleted on a particular pass. Sleep after each pass,
then try to kill all the items on the list again. Maintain a counter
of the number of passes which have been made since the last time the
undeleted item list shrank (via a completed deletion), and if the total
time exceeds 'x' or the number of passes since a completed deletion
exceeds 'y', then bail and ask the user to help you out.

Regards,
Pat

[Alec Wysoker]

> File attributes may be an issue to. Take look at the recipe at:

http://aspn.activestate.com/ASPN/Coo.../Recipe/303343 which ensures the file attributes are normal before you delete it.

I don't think file attributes are an issue, because generally I can
manually delete the file in question, and sometimes my app is
eventually able to delete the file. You're not suggesting that some
other process is temporarily changing the file attributes, are you?

[Alec Wysoker]

I tried something not exactly like this, but in the same spirit. I
don't generally have a list of files I want to delete - just one. I
try to delete it and if I get errno 13 I sleep for a little while (0.2)
and then try again. If the same problem then I add 1 sec to the sleep
time and try again. After sleeping about 10 sec total I give up.

Unfortunately, one of my testers still experiences the problem even
with this fix. I am surprised that a virus checker or file indexer
would hold onto a file for so long. Ugh.

Patrick Maupin wrote:

Tim Peters wrote:

In that case, anything that burns some time and tries again
will work better. Replacing gc.collect() with time.sleep() is
an easy way to test that hypothesis; because gc.collect()
does an all-generations collection, it can consume measurable time.

An slight enhancement to this hypothesis-tester (which might even
approach being production-worthy) would be to maintain a list of items
which were not deleted on a particular pass. Sleep after each pass,
then try to kill all the items on the list again. Maintain a counter
of the number of passes which have been made since the last time the
undeleted item list shrank (via a completed deletion), and if the total
time exceeds 'x' or the number of passes since a completed deletion
exceeds 'y', then bail and ask the user to help you out.

Regards,
Pat

[Alec Wysoker]

Well, I found a solution to some of these problems. Perhaps this new
solution, combined with the sleeping to allow
virus-checker,file-indexer,etc. will fix the problem.

I opened a file with shelve, and when adding a particular entry found
it to be corrupt. On Windows, shelve uses bsddb.hashopen. I figured I
would just delete the file, lose whatever data was in there, and start
over. It appears that once a bsddb hash format file is opened and
found to be corrupt, you can't delete the file until the process exits!

I tried opening the shelf with flag='n' in order to wipe the contents
of the file, but under the covers that just calls os.unlink, which
fails with the same errno 13.

Ultimately, I found I could clear out the file like this:

db = anydbm.open(strPath, 'c')
if hasattr(db, 'db') and hasattr(db.db, 'truncate'):
db.db.truncate()
db.close()
return True

Pretty ugly but it seems to work.

-Alec
Alec Wysoker wrote:

I tried something not exactly like this, but in the same spirit. I
don't generally have a list of files I want to delete - just one. I
try to delete it and if I get errno 13 I sleep for a little while (0.2)
and then try again. If the same problem then I add 1 sec to the sleep
time and try again. After sleeping about 10 sec total I give up.

Unfortunately, one of my testers still experiences the problem even
with this fix. I am surprised that a virus checker or file indexer
would hold onto a file for so long. Ugh.