crash during file writing, how to recover ?

[Joseph]

Hi
I'm writing a commercial program which must be reliable. It has to do
some basic reading and writing to and from files on the hard disk,
and also to a floppy.
I have foreseen a potential problem. The program may crash
unexpectedly while writing to the file. If so, my program should
detect this during startup, and then (during startup) probably delete the
data added to the file and redo the writing operation.

Are file writing operations atomic ? ie when you write to a file,
will it either do it succesfully, OR say half fail (eg write a few letters
and not finish), OR not commit any changes to the file if a crash at
this point occurs?

My next question is how is this handled in commercial programming? I
plan on writing a flag (say, a simple char) to another file (this
would signal that a file write is about to begin), and then
removing this char after the file writing operation is completed.
Then on startup i just check the flags. if flag hasn't been removed a
crash occurred, so have to open file and get rid of any garbage.

Has anyone done anything similar b4? if so how did you handle this
crash scenario. My application could totally stuff up if i don't
handle this right.

by the way, i'm using the java language and api. this might effect
how files are written to, so i thought i should mention this.
MANY THANKS
Joseph

[Roedy Green]

On Fri, 30 Apr 2004 01:36:56 GMT, Joseph <ka****@bigpond .com> wrote or
quoted :

Are file writing operations atomic ? ie when you write to a file,
will it either do it succesfully, OR say half fail (eg write a few letters
and not finish), OR not commit any changes to the file if a crash at
this point occurs?

Imagine a floppy being written. The power fails half way through
writing one of your tracks. You then have gibberish on your floppy.
Further the fat and directory are likely out of sync. When you use
CHKDSK /F it tries to fix this. The file contains gibberish. You
probably should bulk erase and reformat such a floppy.

A very easy way to detect the problem is to write small file before
you start. You do your thing, and on exit you erase the file. You
can do this to any app by doing the creating testing and deleting in
bat language.

When you start, you check for the presence of the file. If you see it
you assume the worst, and demand a restore from backup or whatever you
need to do to get going again safely.
--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.

[Liz]

"Roedy Green" <se*@mindprod.c om.invalid> wrote in message
news:45******** *************** *********@4ax.c om...

On Fri, 30 Apr 2004 01:36:56 GMT, Joseph <ka****@bigpond .com> wrote or
quoted :

Are file writing operations atomic ? ie when you write to a file,
will it either do it succesfully, OR say half fail (eg write a few lettersand not finish), OR not commit any changes to the file if a crash at
this point occurs?

Imagine a floppy being written. The power fails half way through
writing one of your tracks. You then have gibberish on your floppy.
Further the fat and directory are likely out of sync. When you use
CHKDSK /F it tries to fix this. The file contains gibberish. You
probably should bulk erase and reformat such a floppy.

A very easy way to detect the problem is to write small file before
you start. You do your thing, and on exit you erase the file. You
can do this to any app by doing the creating testing and deleting in
bat language.

When you start, you check for the presence of the file. If you see it
you assume the worst, and demand a restore from backup or whatever you
need to do to get going again safely.
--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.

I had an interesting situation a few years ago. I was using stacker
on my hard disk since it was only 20 megabytes. And I happened to
be doing a squeeze type operation when my battery died. At the next
powerup, stacker gave a message on the screen that it discovered that
it crashed during a squeeze and proceeded to automatically clean up.
I was impressed.

[Calum]

Joseph wrote:

Hi
I'm writing a commercial program which must be reliable. It has to do
some basic reading and writing to and from files on the hard disk,
and also to a floppy.
I have foreseen a potential problem. The program may crash
unexpectedly while writing to the file. If so, my program should
detect this during startup, and then (during startup) probably delete the
data added to the file and redo the writing operation.

Are file writing operations atomic ? ie when you write to a file,
will it either do it succesfully, OR say half fail (eg write a few letters
and not finish), OR not commit any changes to the file if a crash at
this point occurs?

My next question is how is this handled in commercial programming? I
plan on writing a flag (say, a simple char) to another file (this
would signal that a file write is about to begin), and then
removing this char after the file writing operation is completed.
Then on startup i just check the flags. if flag hasn't been removed a
crash occurred, so have to open file and get rid of any garbage.

Has anyone done anything similar b4? if so how did you handle this
crash scenario. My application could totally stuff up if i don't
handle this right.

by the way, i'm using the java language and api. this might effect
how files are written to, so i thought i should mention this.

One approach is to write to a temporary file, then when writing has
completed successfully, and the file has been closed, rename the
temporary file to the target filename. That way you won't run out of
disk space either. If you need to overwrite an old file, delete it just
before renaming. If your program crashes during temporary file
creation, you'll be left with a damaged temp file that is never used
again - no big deal.

Calum

[Norm Dresner]

Always write to a new file and then delete the old one and rename the new.
Also, consider writing a journal file before writing to the new file so you
can at least "recover" what's missing.

Norm,

"Joseph" <ka****@bigpond .com> wrote in message
news:Y4******** *******@news-server.bigpond. net.au...

Hi
I'm writing a commercial program which must be reliable. It has to do
some basic reading and writing to and from files on the hard disk,
and also to a floppy.
I have foreseen a potential problem. The program may crash
unexpectedly while writing to the file. If so, my program should
detect this during startup, and then (during startup) probably delete the
data added to the file and redo the writing operation.

Are file writing operations atomic ? ie when you write to a file,
will it either do it succesfully, OR say half fail (eg write a few letters
and not finish), OR not commit any changes to the file if a crash at
this point occurs?

My next question is how is this handled in commercial programming? I
plan on writing a flag (say, a simple char) to another file (this
would signal that a file write is about to begin), and then
removing this char after the file writing operation is completed.
Then on startup i just check the flags. if flag hasn't been removed a
crash occurred, so have to open file and get rid of any garbage.

Has anyone done anything similar b4? if so how did you handle this
crash scenario. My application could totally stuff up if i don't
handle this right.

by the way, i'm using the java language and api. this might effect
how files are written to, so i thought i should mention this.
MANY THANKS
Joseph

[Kasper Dupont]

Norm Dresner wrote:

Always write to a new file and then delete the old one and rename the new.

No need to delete the old one first. Renaming will
automatically delete the old file.

--
Kasper Dupont -- der bruger for meget tid paa usenet.
For sending spam use ab***@mk.lir.dk and ka*****@mk.lir. dk
/* Would you like fries with that? */

[Nate Smith]

Kasper Dupont wrote:

Norm Dresner wrote:

Always write to a new file and then delete the old one and rename the new.

No need to delete the old one first. Renaming will
automatically delete the old file.

often that will lead to a "cant rename, file already exists"
sort of error....
- nate

[Kasper Dupont]

Nate Smith wrote:

Kasper Dupont wrote:

Norm Dresner wrote:

Always write to a new file and then delete the old one and rename the new.

No need to delete the old one first. Renaming will
automatically delete the old file.

often that will lead to a "cant rename, file already exists"
sort of error....

I'm pretty sure the posix standard requires rename to
atomically remove and replace the target if it already
exists. But I don't have access to the standard, so
somebody else will have to check.

And using rename to delete the file is the correct way
to do because of the atomic behavioure. Deleting the old
file before renaming would introduce a race condition.

--
Kasper Dupont -- der bruger for meget tid paa usenet.
For sending spam use ab***@mk.lir.dk and ka*****@mk.lir. dk
/* Would you like fries with that? */

[Roedy Green]

On Fri, 30 Apr 2004 18:25:02 +0200, Kasper Dupont
<ka*****@daimi. au.dk> wrote or quoted :

No need to delete the old one first. Renaming will
automaticall y delete the old file.

Here is the sort of code I use to rewrite the contents of a file.
// create a tempfile in the same directory as
// the input file we have just processed.
File tempfile = HunkIO.createTe mpFile ("temp", ".tmp",
fileBeingProces sed );
FileWriter emit = new FileWriter( tempfile );
emit.write( result );
emit.close();
// successfully created output in same directory as input,
// Now make it replace the input file.
fileBeingProces sed.delete();
tempfile.rename To( fileBeingProces sed );

This effectively makes the tempfile disappear, but without the delete
it would not make the old version disappear. Or would it?

It would be nice to have the delete/rename atomic.
--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.

[Kasper Dupont]

Roedy Green wrote:

On Fri, 30 Apr 2004 18:25:02 +0200, Kasper Dupont
<ka*****@daimi. au.dk> wrote or quoted :

No need to delete the old one first. Renaming will
automaticall y delete the old file.
Here is the sort of code I use to rewrite the contents of a file.

// create a tempfile in the same directory as
// the input file we have just processed.
File tempfile = HunkIO.createTe mpFile ("temp", ".tmp",
fileBeingProces sed );
FileWriter emit = new FileWriter( tempfile );
emit.write( result );
emit.close();
// successfully created output in same directory as input,
// Now make it replace the input file.
fileBeingProces sed.delete();
tempfile.rename To( fileBeingProces sed );

Well, I don't write java code I usually use C, so I don't
know exactly how those methods are implemented. But I
know it is impossible to delete a file using any kind of
handle, you need to use the name. So exactly what is the
meaning of `fileBeingProce ssed.delete();' ? Does it delete
whatever file has the name originally used to open
fileBeingProces sed?

In the next line it looks like fileBeingProces sed is a
string, but then you wouldn't be able to delete the file
the way it is done in the code.

This effectively makes the tempfile disappear, but without the delete
it would not make the old version disappear. Or would it?
If the renameTo method calls the rename system call, it
will make the old file disappear.

It would be nice to have the delete/rename atomic.

You have it. At least on any posix compliant system you do.

--
Kasper Dupont -- der bruger for meget tid paa usenet.
For sending spam use ab***@mk.lir.dk and ka*****@mk.lir. dk
/* Would you like fries with that? */