473,224 Members | 1,510 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,224 software developers and data experts.

Drop folder and race conditions

I have a need to implement a drop folder upload mechanism for secure uploading
of files to a server. At first glance this appears that it would be an easy
application to write. Then I begin to think about the race conditions that
exist between the process that will wake up to upload the files and the fact
that the user can add additional files to the drop folder at any point in time.
I would like to clear out files/folders after they have been processed but it is
possible that while the background process was uploading files, the user added
additional files/folders to the drop folder that make the deletion of folders
impossible. This project has become significantly more complex than it appeared
at first. Anyone out there have any "sage" advice on how to tackle this beast?

Thanks in advance for any pointers.

Regards,
Larry Bates
Oct 9 '07 #1
1 1506
On Tue, 09 Oct 2007 07:05:57 -0500, Larry Bates wrote:
I have a need to implement a drop folder upload mechanism for secure
uploading of files to a server. At first glance this appears that it
would be an easy application to write. Then I begin to think about the
race conditions that exist between the process that will wake up to
upload the files and the fact that the user can add additional files to
the drop folder at any point in time. I would like to clear out
files/folders after they have been processed but it is possible that
while the background process was uploading files, the user added
additional files/folders to the drop folder that make the deletion of
folders impossible. This project has become significantly more complex
than it appeared at first. Anyone out there have any "sage" advice on
how to tackle this beast?

Off the top of my head...

Recursively copy the contents of each folder to the server, deepest
first, deleting each file as it's copied. Then delete the folder as soon
as you've emptied it (but naturally not the top-level drop folder). If it
happens to *not* be empty (because the user has added additional files to
it), just catch the error and ignore it. In another few minutes, you'll
try again, copying the newly added files and then delete the folder.

Because each file is being deleted as soon as it is copied, you'll won't
build up an ever-increasing collection of files. At worst, you may have a
collection of folders -- but unless you're fighting a hostile process
that can add folders quicker than you can delete them, you'll eventually
win.

Maybe you need a heuristic that says "if the number of 'directory not
empty' errors keeps increasing, assume we're dealing with a hostile
process and take extra steps". Say, lock the drop folder for five minutes
while you flush it.

Another possibility is that before you start copying from the folders,
you change their permissions to prohibit the user adding extra files into
them. That way, nothing can be added to them once you start copying, and
you can be sure that you can delete them once empty.
--
Steven
Oct 9 '07 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: GIMME | last post by:
One of my coworkers insists that one should never use static methods because race conditions exist. Thinking along the lines that all variable assignments are assignments to static variables. ...
18
by: Urs Vogel | last post by:
Hi I wrote an application server (a remoting sinlgeton), where processes must be stopped in very rare cases, done thru a Thread.Abort(). Occasionally, and only after a Thread.Abort(), this...
6
by: Ernesto | last post by:
I couldn't find this with a search, but isn't there a way to overwrite a previous folder (or at least not perform osmkdir( ) if your program detects it already exists). Thanks !
2
by: manuelg | last post by:
Here is a code fragment, where I am trying to copy a file, avoiding overwrites and race conditions. The filename gets a '02','03','04' etc appended to the end if a file with that name already...
4
by: SStory | last post by:
I want to drag a message from Outlook to a richtextbox on a vb.net form. I don't get the message body. I have searched all over the place and found nothing. Does anyone know how to do this? I...
5
by: mars | last post by:
I use TurboGears to do some web service. TurboGears use cherrypy. When web browser access this site, the cherrypy will call my python program. So my program looks like a lib. When web browser...
3
by: penny111 | last post by:
Hi there, For my application, i need to have 3 drop down lists 1. drop down list of folder names 2. drop down list of documents in the folder selected 3. drop down list of instances of the...
9
by: Keith G Hicks | last post by:
I'm having a lot of trouble with "file in use" errors in my "folder watcher" project. Starting and stopping the watcher and reading my XML file work fine. Once the watcher is started, I'm reading...
0
by: moltendorf | last post by:
I've been trying to find a suitable method for preventing race conditions in my own code. Currently I'm using a file and the flock function to prevent code in other threads from executing at the...
1
isladogs
by: isladogs | last post by:
The next online meeting of the Access Europe User Group will be on Wednesday 6 Dec 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, Mike...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
0
by: mar23 | last post by:
Here's the situation. I have a form called frmDiceInventory with subform called subfrmDice. The subform's control source is linked to a query called qryDiceInventory. I've been trying to pick up the...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
by: jimatqsi | last post by:
The boss wants the word "CONFIDENTIAL" overlaying certain reports. He wants it large, slanted across the page, on every page, very light gray, outlined letters, not block letters. I thought Word Art...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.