473,836 Members | 2,187 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

crash during file writing, how to recover ?

Hi
I'm writing a commercial program which must be reliable. It has to do
some basic reading and writing to and from files on the hard disk,
and also to a floppy.
I have foreseen a potential problem. The program may crash
unexpectedly while writing to the file. If so, my program should
detect this during startup, and then (during startup) probably delete the
data added to the file and redo the writing operation.

Are file writing operations atomic ? ie when you write to a file,
will it either do it succesfully, OR say half fail (eg write a few letters
and not finish), OR not commit any changes to the file if a crash at
this point occurs?

My next question is how is this handled in commercial programming? I
plan on writing a flag (say, a simple char) to another file (this
would signal that a file write is about to begin), and then
removing this char after the file writing operation is completed.
Then on startup i just check the flags. if flag hasn't been removed a
crash occurred, so have to open file and get rid of any garbage.

Has anyone done anything similar b4? if so how did you handle this
crash scenario. My application could totally stuff up if i don't
handle this right.

by the way, i'm using the java language and api. this might effect
how files are written to, so i thought i should mention this.
MANY THANKS
Joseph

Jul 17 '05
48 8522
On Fri, 30 Apr 2004 21:12:36 +0200, Kasper Dupont
<ka*****@daimi. au.dk> wrote or quoted :
`fileBeingProc essed.delete(); '? Does it delete
whatever file has the name originally used to open
fileBeingProce ssed?


fileBeingProces sed is of type File. See
http://mindprod.com/jgloss/file.html

It contains a partly parsed filename. It is NOT a handle to an open
file as in C.
--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
Jul 17 '05 #11
Joseph wrote:

<snipped>
My next question is how is this handled in commercial programming? I
plan on writing a flag (say, a simple char) to another file (this
would signal that a file write is about to begin), and then
removing this char after the file writing operation is completed.
Then on startup i just check the flags. if flag hasn't been removed a
crash occurred, so have to open file and get rid of any garbage.


Why not just write your 'dirty flag' in the same file?

<snipped>

goose
Jul 17 '05 #12
Joseph wrote:
Hi
I'm writing a commercial program which must be reliable. It has to do
some basic reading and writing to and from files on the hard disk,
and also to a floppy.
I have foreseen a potential problem. The program may crash
unexpectedly while writing to the file. If so, my program should
detect this during startup, and then (during startup) probably delete the
data added to the file and redo the writing operation.

Are file writing operations atomic ? ie when you write to a file,
will it either do it succesfully, OR say half fail (eg write a few letters
and not finish), OR not commit any changes to the file if a crash at
this point occurs?

My next question is how is this handled in commercial programming? I
plan on writing a flag (say, a simple char) to another file (this
would signal that a file write is about to begin), and then
removing this char after the file writing operation is completed.
Then on startup i just check the flags. if flag hasn't been removed a
crash occurred, so have to open file and get rid of any garbage.

Has anyone done anything similar b4? if so how did you handle this
crash scenario. My application could totally stuff up if i don't
handle this right.

by the way, i'm using the java language and api. this might effect
how files are written to, so i thought i should mention this.
MANY THANKS
Joseph


Just another followup, possibly about a condition that
you have not considered.

Do you need to guard against a hard-disk crash while
writing? If your program does not, by some definitions
this is "not reliable." (Is restoring from "last
week's backup" OK with the customer?)

You can only guard against a single hardware failure
at a time. As I mentioned elsethread, DBMS's use
a log file to log the changes. This log file must
be on a separate hardware device to guard against
a single hardware failure. Thus, either the logfile
or the data file survives. If the logfile is on the
device that fails, then, no problem. If the data-file
is on the device that fails, it may be reconstructed
from the last backup of the data files and applying
all the log-files since the backup.

I am not sure if the "rename" strategy mentioned by
other posters will be atomic over multiple physical
devices nor do I know about what size files you
are talking about. If it's several GB, then the
copy from one disk to another will take considerable
time. Then again, you may not need to be at this
level of paranoia. :)

--
"It is impossible to make anything foolproof
because fools are so ingenious"
- A. Bloch
Jul 17 '05 #13
Nick Landsberg wrote:

You can only guard against a single hardware failure
at a time.
Actually you can guard against multiple hardware
failures, but it will get expensive.

I am not sure if the "rename" strategy mentioned by
other posters will be atomic over multiple physical
devices


That depends. If you use a filesystem on a raid it
should be atomic. But raid is not 100% safe. With
an unfortunate sequence of events even a raid will
lose data. If we are talking independent filesystems
the rename will just report an error.

--
Kasper Dupont -- der bruger for meget tid paa usenet.
For sending spam use ab***@mk.lir.dk and ka*****@mk.lir. dk
/* Would you like fries with that? */
Jul 17 '05 #14
Kasper Dupont wrote:
Nick Landsberg wrote:
You can only guard against a single hardware failure
at a time.

Actually you can guard against multiple hardware
failures, but it will get expensive.


True. I inadvertantly left the word
"economical ly".
I am not sure if the "rename" strategy mentioned by
other posters will be atomic over multiple physical
devices

That depends. If you use a filesystem on a raid it
should be atomic. But raid is not 100% safe. With
an unfortunate sequence of events even a raid will
lose data. If we are talking independent filesystems
the rename will just report an error.

--
"It is impossible to make anything foolproof
because fools are so ingenious"
- A. Bloch
Jul 17 '05 #15
In article <40************ ***@daimi.au.dk >, Kasper Dupont <ka*****@daimi. au.dk> wrote:
I'm pretty sure the posix standard requires rename to
atomically remove and replace the target if it already
exists. But I don't have access to the standard, so
somebody else will have to check.

And using rename to delete the file is the correct way
to do because of the atomic behavioure. Deleting the old
file before renaming would introduce a race condition.


That sounds like a dangerous approach to me.

Why not rename the old file *first*, before writing the new one. Then
if the program starts and finds the most recently written file is
corrupt due to a crash, the last good file remains as a backup.
Renaming an existing file should be quick, and you can wait until it's
done before starting to write.

- Gerry Quinn

Jul 17 '05 #16
For the most part, I think this thread demonstrates confusion caused by
cross-posting. We've got answers from people in
comp.lang.java. programmer answering as if this were entirely a Java
question and people from comp.os.linux.d evelopment.apps answering as if
it's a Linux question... and we don't know who's right!

Nevertheless, some clarification about Java:

Kasper Dupont wrote:
Well, I don't write java code I usually use C, so I don't
know exactly how those methods are implemented. But I
know it is impossible to delete a file using any kind of
handle, you need to use the name. So exactly what is the
meaning of `fileBeingProce ssed.delete();' ? Does it delete
whatever file has the name originally used to open
fileBeingProces sed?

In the next line it looks like fileBeingProces sed is a
string, but then you wouldn't be able to delete the file
the way it is done in the code.
Java's standard API class java.io.File is confusingly named. It
represents an abstract path name, not a file. Having a java.io.File
object doesn't even imply the existence of a file in the filesystem,
though File does expose an API method called exists() that tells you
whether this file exists in the filesystem or not. Certain operations
that deal with directory management (rename, delete, etc.) are
implemented for objects of the File class. So Roedy's calls make
perfect sense because they don't operate on a file descriptor, but
rather on a file name.
If the renameTo method calls the rename system call, it
will make the old file disappear.


Not surprisingly, it's not specified whether File.renameTo results in a
call to the rename system call. More surprisingly, it's not specified
whether File.renameTo will succeed if a file already exists by the
target name. That said, renameTo returns a success flag (which is ugly
in Java, but nevertheless happens). So it's entirely possible to write:

if (!newFile.renam eTo(fileBeingPr ocessed))
{
fileBeingProces sed.delete();
newFile.renameT o(fileBeingProc essed);
}

Additional error checking would be nice in case the rename fails, for
example, on a Windows machine because of open file descriptors to the
file. Windows file handling doesn't separate the existence of a file
from its directory entry in the way POSIX does.

--
www.designacourse.com
The Easiest Way to Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation
Jul 17 '05 #17
Gerry Quinn wrote:

In article <40************ ***@daimi.au.dk >, Kasper Dupont <ka*****@daimi. au.dk> wrote:
I'm pretty sure the posix standard requires rename to
atomically remove and replace the target if it already
exists. But I don't have access to the standard, so
somebody else will have to check.
I just checked susv3 does require rename to make
sure at any point in time, the target name will
refere to either the old or the new file. And if
rename fails the target must be unaffected.

And using rename to delete the file is the correct way
to do because of the atomic behavioure. Deleting the old
file before renaming would introduce a race condition.


That sounds like a dangerous approach to me.

Why not rename the old file *first*, before writing the new one. Then
if the program starts and finds the most recently written file is
corrupt due to a crash, the last good file remains as a backup.
Renaming an existing file should be quick, and you can wait until it's
done before starting to write.


But then you'd have a window where no file exist with
the given name. The approach I suggested is safe. When
creating the new file first create it with a different
name. And when you have finished writing you rename it
such that it atomically replaces the old file.

There is nothing dangerous to it.

--
Kasper Dupont -- der bruger for meget tid paa usenet.
For sending spam use ab***@mk.lir.dk and ka*****@mk.lir. dk
/* Would you like fries with that? */
Jul 17 '05 #18
Chris Smith wrote:

Not surprisingly, it's not specified whether File.renameTo results in a
call to the rename system call. More surprisingly, it's not specified
whether File.renameTo will succeed if a file already exists by the
target name.


Well I don't know anything about Java. But I know that
the right way to do this requires use of the rename
system call to atomically remove the old file and
replace it with the new. If that is not what happens it
means either the Java program or the Java VM is broken.

--
Kasper Dupont -- der bruger for meget tid paa usenet.
For sending spam use ab***@mk.lir.dk and ka*****@mk.lir. dk
/* Would you like fries with that? */
Jul 17 '05 #19
Gerry Quinn wrote:
Kasper Dupont <ka*****@daimi. au.dk> wrote:
I'm pretty sure the posix standard requires rename to
atomically remove and replace the target if it already
exists. But I don't have access to the standard, so
somebody else will have to check.

And using rename to delete the file is the correct way
to do because of the atomic behavioure. Deleting the old
file before renaming would introduce a race condition.


That sounds like a dangerous approach to me.

Why not rename the old file *first*, before writing the new one.
Then if the program starts and finds the most recently written
file is corrupt due to a crash, the last good file remains as a
backup. Renaming an existing file should be quick, and you can
wait until it's done before starting to write.


That is not his point. If a rename can fail because the target
file pre-exists, the delete/rename sequence has a hole between
delete and rename in which some other process can create that file
name, and cause a failure. This is a race condition. It is
especially likely to occur with database systems which inherently
tend to service multiple processes from the same database, and
have to 'take steps' to ensure the self-consistency of that
database.

One cure is to provide atomic operations, often by the use of
critical sections or other synchronization primitives. Another is
the concept of 'transactions'.

--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

Jul 17 '05 #20

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

110
10654
by: alf | last post by:
Hi, is it possible that due to OS crash or mysql itself crash or some e.g. SCSI failure to lose all the data stored in the table (let's say million of 1KB rows). In other words what is the worst case scenario for MyISAM backend? Also is it possible to not to lose data but get them corrupted?
0
9811
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
10822
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
10577
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10241
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9359
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7774
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6975
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5812
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
4003
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.