By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
464,696 Members | 1,357 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 464,696 IT Pros & Developers. It's quick & easy.

Problems with file IO in a thread, and shutil

P: n/a
Hi,

I have a multithreaded application. There are two threads, T1 and T2.
Suppose that there are two folders A, B. Thread T1 fetches data from
network and creates files in folder A and after creating each file, it
moves the file to folder B, using shutil.move().

Thread T2, takes files from folder B and processes it.
Note: Only the thread T1 has access to the files in folder A.

psuedo code
=========
class T1(thread):
def run():
self.download()

def download():
data = from_network
filename = os.path.join ( "A", "file1")

f = open ( filename, "w")
for d in data:
f.write ( d + "\n" )
f.flush()
f.close()

shutil.move(os.path.join ( "A", "file1"), os.path.join("B",
"file1"))
class T2(thread):
run()
process(listdir(os.path.join("B")))

All the files has similar contents. But in some cases(very rare), when
thread T1 tries to move the newly created file from folder A to folder
B, an exception occurs (on shutil.move()).

The exception is WindowsError(32, 'The process cannot access the file
because it is being used by another process'). I looked inside the
shutil.move. What I found was due to this WindowsError, the usual
os.rename() inside shutil.move() fails and the file is copied using
copy2(src, dst). Finally os.unlink() also failed. (So though copying
occurred, deleting the source file failed)

I am not getting why this error comes. Has anyone faced similar
situation? Please help me.

Thanks and Regards
Roopesh
Jun 27 '08 #1
Share this Question
Share on Google+
4 Replies

P: n/a
On Jun 25, 4:11*pm, Roopesh <roopesh....@gmail.comwrote:
Hi,

I have a multithreaded application. There are two threads, T1 and T2.
Suppose that there are two folders A, B. *Thread T1 fetches data from
network and creates files in folder A and after creating each file, it
moves the file to folder B, using shutil.move().

Thread T2, takes files from folder B and processes it.
Note: Only the thread T1 has access to the files in folder A.

psuedo code
=========
class T1(thread):
* *def run():
* * *self.download()

* def download():
* * *data = from_network
* * *filename = os.path.join ( "A", "file1")

* * *f = open ( filename, "w")
* * *for d in data:
* * * * f.write ( d + "\n" )
* * *f.flush()
* * *f.close()

* * shutil.move(os.path.join ( "A", "file1"), *os.path.join("B",
"file1"))

class T2(thread):
* run()
* * * process(listdir(os.path.join("B")))

All the files has similar contents. But in some cases(very rare), when
thread T1 tries to move the newly created file from folder A to folder
B, an exception occurs (on shutil.move()).

The exception is WindowsError(32, 'The process cannot access the file
because it is being used by another process'). I looked inside the
shutil.move. What I found was due to this WindowsError, the usual
os.rename() inside shutil.move() fails and the file is copied using
copy2(src, dst). Finally os.unlink() also failed. (So though copying
occurred, deleting the source file failed)

I am not getting why this error comes. Has anyone faced similar
situation? Please help me.

Thanks and Regards
Roopesh
Do you have any anti-virus/anti-spyware software installed? (If not,
why not? :-))

It might be that your anti-virus/anti-spyware software is seeing the
new file appear and scanning it. If that happens just when
shutil.move() attempts to move it then the move will fail. You could
have your code wait for a while and then try again.
Jun 27 '08 #2

P: n/a
Thanks for the reply. I did testing in a clean system, were anti virus/
spyware is not installed. It still gave this problem, in say 1 out of
1000 cases.

By any chance would it be possible that the Windows OS has not
completed writing to the file even after file.flush() and file.close()
is called?

Thanks
Roopesh
Jun 27 '08 #3

P: n/a
On Jun 26, 8:06*am, Roopesh <roopesh....@gmail.comwrote:
Thanks for the reply. I did testing in a clean system, were anti virus/
spyware is not installed. It still gave this problem, in say 1 out of
1000 cases.

By any chance would it be possible that the Windows OS has not
completed writing to the file even after file.flush() and file.close()
is called?

Thanks
Roopesh
Maybe it's a Windows service, eg the indexing service or generating
thumbnails. Anyway, retrying after a delay would still be worth it.
Jun 27 '08 #4

P: n/a
Maybe it's a Windows service, eg the indexing service or generating
thumbnails. Anyway, retrying after a delay would still be worth it.
Yes, this thing works :-) Thanks a lot.

Thanks
Roopesh
Jun 28 '08 #5

This discussion thread is closed

Replies have been disabled for this discussion.