473,854 Members | 1,822 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

crash during file writing, how to recover ?

Hi
I'm writing a commercial program which must be reliable. It has to do
some basic reading and writing to and from files on the hard disk,
and also to a floppy.
I have foreseen a potential problem. The program may crash
unexpectedly while writing to the file. If so, my program should
detect this during startup, and then (during startup) probably delete the
data added to the file and redo the writing operation.

Are file writing operations atomic ? ie when you write to a file,
will it either do it succesfully, OR say half fail (eg write a few letters
and not finish), OR not commit any changes to the file if a crash at
this point occurs?

My next question is how is this handled in commercial programming? I
plan on writing a flag (say, a simple char) to another file (this
would signal that a file write is about to begin), and then
removing this char after the file writing operation is completed.
Then on startup i just check the flags. if flag hasn't been removed a
crash occurred, so have to open file and get rid of any garbage.

Has anyone done anything similar b4? if so how did you handle this
crash scenario. My application could totally stuff up if i don't
handle this right.

by the way, i'm using the java language and api. this might effect
how files are written to, so i thought i should mention this.
MANY THANKS
Joseph

Jul 17 '05
48 8531
what about using copy first?
and what about exclusive access?

outline:
---------------------------------------------------------

actually first, grab the oldFile exclusively
if successful
then
open oldFile for Read
open saveFile for Write

copy oldFile to saveFile

close oldFile
close saveFile

if copy was successful
then
open oldFile for Write
open saveFile for Read

do the cloudy processing thing that rewrites oldFile
from the saveFile stuff & the "new stuff", whatever
the new stuff is (the changes, updates, etc.)

close oldFile
close saveFile

in case of crash in middle, saveFile should have the recovery
or if the copy step failed, oldFile is still unchanged,
and politely notify any interested parties

now release the exclusive hold on oldFile & let the race resume

else
wait until can grab oldFile exclusively in time,
where some waiting period has been established
to quit trying, and try again a "second" later
or
in case of time out, notify interested parties and
maybe quit thread with flag (duck out of the race)

NOTE:
presumably there are already other things in place elsewhere
that preserve the "new stuff" (changes, etc.) through crashes

-------------------------------------------------------------------

- nate, trying to keep from spilling his white russian

Jul 17 '05 #21
Chris Smith wrote:
Not surprisingly, it's not specified whether File.renameTo results in a
call to the rename system call. More surprisingly, it's not specified
whether File.renameTo will succeed if a file already exists by the
target name. That said, renameTo returns a success flag (which is ugly
in Java, but nevertheless happens). So it's entirely possible to write:

if (!newFile.renam eTo(fileBeingPr ocessed))
{
fileBeingProces sed.delete();
newFile.renameT o(fileBeingProc essed);
}


Just to be inconsistent, the Microsoft "rename" function specifically
requires "The new name must not be the name of an existing file or
directory". It makes some sense - I wouldn't expect a function called
"rename" to delete a file, but I can see situations where either
behaviour would be desired.

So it's possible Java will have different behaviour on Windows and
POSIX, though I can't be bothered to check this.

I guess for ultra-safety, the old file could be renamed to something
else, before renaming the new file to the target filename. Of course
then you'd just clutter up the directory with old files, but there are
circumstances where you want to be able to roll back.

Calum
Jul 17 '05 #22
On Sat, 01 May 2004 23:59:52 +0100, Calum <ca********@ntl world.com>
wrote or quoted :

I guess for ultra-safety, the old file could be renamed to something
else, before renaming the new file to the target filename. Of course
then you'd just clutter up the directory with old files, but there are
circumstance s where you want to be able to roll back.

The problem is a lack of naming convention for temporary file.

If at least Java could name all temps in a standard way, some tool
such as batik could periodically cleanup the trash left over from
crashes.

--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
Jul 17 '05 #23
"goose" <ru**@webmail.c o.za> wrote in message
news:40******** ******@webmail. co.za...
Joseph wrote:

<snipped>
My next question is how is this handled in commercial programming? I
plan on writing a flag (say, a simple char) to another file (this
would signal that a file write is about to begin), and then
removing this char after the file writing operation is completed.
Then on startup i just check the flags. if flag hasn't been removed a
crash occurred, so have to open file and get rid of any garbage.


Why not just write your 'dirty flag' in the same file?

Because if the power fails during this write-op, the entire track of the
media may be lost.

The real question is, what else do you need to hold up your pants
besides a belt and suspenders?

Norm

Jul 17 '05 #24
Norm Dresner wrote:
"goose" <ru**@webmail.c o.za> wrote in message
news:40******** ******@webmail. co.za...
Joseph wrote:

<snipped>
My next question is how is this handled in commercial programming? I
plan on writing a flag (say, a simple char) to another file (this
would signal that a file write is about to begin), and then
removing this char after the file writing operation is completed.
Then on startup i just check the flags. if flag hasn't been removed a
crash occurred, so have to open file and get rid of any garbage.


Why not just write your 'dirty flag' in the same file?


Because if the power fails during this write-op, the entire track of the
media may be lost.

The real question is, what else do you need to hold up your pants
besides a belt and suspenders?

Norm


Well, that depends Norm:)

How critical is that data? How much is the customer
willing to pay to ensure data integrity and consistency?

How does the customer define "reliabilty " for the OP's
application? Have they quantified it? ("Never losing
any data" is not a reasonable requirement. "Never"
is not a number.) If it's something like "no more than 3
records per 1,000,000 are allowed to be lost", how
will this be validated/tested? At what cost?
What is the test plan? What is the probability
of a hardware outage (on this specific hardware)?
What is the probability of a software bug scribbling
all over the data? (Suggestions about keeping a
previous copy of "good" data address a part of this
problem.) How do you validate any particular
data file at startup? How long will it take?
How long will it take to bring it up to date?
(Is it necessary to bring it up to date, or is
last night's copy ok?)

All of these questions (and more) may have
to be answered before the final
solution is decided upon. Everytime you kick
up the requirement a notch, the development (not just
coding) cost usually goes up proportionately ,
which brings us back to "how much is the
customer willing to pay?"

<War Story>

I have heard of (but not personally worked on) a system
where the customer *insisted* on triple-mirroring
(primary and 3 copies). Each string was on a separate
controller and on a separate UPS. Each "CPU-box" was also
duplicated and on a separate UPS and could take over the
processing in case of a single CPU failure. All data was also
replicated to a remote site which could pick up the
load in case of a real disaster (e.g. tornado or
earthquake ... yep, one of the data centers was sitting
right smack dab on the Hayward Fault, the other was
in Kansas... go figure).

(As you may have intuited, this was a *financial* application
with some ungodly number of transactions per hour.
The bean counters get apoplectic when they lose track
of a few cents here of there. To me, this was
an extreme case of overkill, but the customer
was willing to pay for it. This may also give you
some clue as to why your credit card interest
rates are so high :)

</War Story>

It is highly doubtful that the OP needs this kind
of fault tolerance or fault recovery, but we
really don't know the customer's requirements.

--
"It is impossible to make anything foolproof
because fools are so ingenious"
- A. Bloch
Jul 17 '05 #25
Calum wrote:
Chris Smith wrote:
Not surprisingly, it's not specified whether File.renameTo results in
a call to the rename system call. More surprisingly, it's not
specified whether File.renameTo will succeed if a file already exists
by the target name. That said, renameTo returns a success flag (which
is ugly in Java, but nevertheless happens). So it's entirely possible
to write:

if (!newFile.renam eTo(fileBeingPr ocessed))
{
fileBeingProces sed.delete();
newFile.renameT o(fileBeingProc essed);
}


Just to be inconsistent, the Microsoft "rename" function specifically
requires "The new name must not be the name of an existing file or
directory". It makes some sense - I wouldn't expect a function called
"rename" to delete a file, but I can see situations where either
behaviour would be desired.


The Win32 API supplies MoveFile() and MoveFileEx().

MoveFile() will fail if the destination exists, or if you're trying to
move a directory to another device.

MoveFileEx() allows you to specify a number of options, including an
option to replace an existing file.

Of course MoveFileEx() isn't available on Win9x/Me...

Another option is to copy oldfile to backup, open oldfile for writing,
replace its contents with newfile, close everything and delete backup
and newfile. While far from being atomic, it is recoverable. If the
operation fails at any point you still have the files (oldfile and
newfile) in one form or another.

Of course if you're talking about a multi-gigabyte data store, this is
gonna take a while :>

--
Corey Murtagh
The Electric Monk
"Quidquid latine dictum sit, altum viditur!"
Jul 17 '05 #26
your talking about a forward error recovery pattern.
check out the design philosophy of POET, the object database engine that
used a technique that's known as Jounalizing under Linux today.
Essentially, they maintained a separate "Ledger" that tracked all
transactions to the objectdatabase and if for any reason on a restart
that that ledger was found out of sync or not properly closed, they knew
there was unfinished work with the object file.

- perry
Corey Murtagh wrote:
Calum wrote:
Chris Smith wrote:
Not surprisingly, it's not specified whether File.renameTo results in
a call to the rename system call. More surprisingly, it's not
specified whether File.renameTo will succeed if a file already exists
by the target name. That said, renameTo returns a success flag
(which is ugly in Java, but nevertheless happens). So it's entirely
possible to write:

if (!newFile.renam eTo(fileBeingPr ocessed))
{
fileBeingProces sed.delete();
newFile.renameT o(fileBeingProc essed);
}

Just to be inconsistent, the Microsoft "rename" function specifically
requires "The new name must not be the name of an existing file or
directory". It makes some sense - I wouldn't expect a function called
"rename" to delete a file, but I can see situations where either
behaviour would be desired.

The Win32 API supplies MoveFile() and MoveFileEx().

MoveFile() will fail if the destination exists, or if you're trying to
move a directory to another device.

MoveFileEx() allows you to specify a number of options, including an
option to replace an existing file.

Of course MoveFileEx() isn't available on Win9x/Me...

Another option is to copy oldfile to backup, open oldfile for writing,
replace its contents with newfile, close everything and delete backup
and newfile. While far from being atomic, it is recoverable. If the
operation fails at any point you still have the files (oldfile and
newfile) in one form or another.

Of course if you're talking about a multi-gigabyte data store, this is
gonna take a while :>


Jul 17 '05 #27
In article <40************ ***@daimi.au.dk >, Kasper Dupont <ka*****@daimi. au.dk> wrote:
Gerry Quinn wrote:

In article <40************ ***@daimi.au.dk >, Kasper Dupont

<ka*****@daimi. au.dk> wrote:
>I'm pretty sure the posix standard requires rename to
>atomically remove and replace the target if it already
>exists. But I don't have access to the standard, so
>somebody else will have to check.
I just checked susv3 does require rename to make
sure at any point in time, the target name will
refere to either the old or the new file. And if
rename fails the target must be unaffected.


Well, that justifies your approach, so.
Why not rename the old file *first*, before writing the new one. Then
if the program starts and finds the most recently written file is
corrupt due to a crash, the last good file remains as a backup.
Renaming an existing file should be quick, and you can wait until it's
done before starting to write.


But then you'd have a window where no file exist with
the given name. The approach I suggested is safe. When
creating the new file first create it with a different
name. And when you have finished writing you rename it
such that it atomically replaces the old file.

There is nothing dangerous to it.


Given the above guarantee (which probably means the OS does something
like I was suggesting!)

The window where no file exists is not a problem because software knows
what the alternative name for the original file is, if the proper file
is corrupt or non-existent.

- Gerry Quinn



Jul 17 '05 #28

Hi all

I'm thankful to everyone for sharing their opinion. I read everybody's post
and learned allot. It turns out that my algorithm will be simple, maybe
something like this:
Make a copy of existing data_file in the same directory

operate on this new copy (data_file2) to add data

close data_file2 when done

delete data_file (original)

rename data_file2 to data_file


on startup the sw will be able to detect if a crash occurred and respond
appropriately. The code on startup should be pretty straight forward

By the way, the platform is java on linux (probably red hat)

Joseph


Jul 17 '05 #29
Calum wrote:
Just to be inconsistent, the Microsoft "rename" function
specifically requires "The new name must not be the name
of an existing file or directory". It makes some sense - I
wouldn't expect a function called "rename" to delete a file,
but I can see situations where either behaviour would be desired.


I can't. You should *NOT* be able to rename/mv a file ONTOP of
an existing file (IMO, obviously).

I've always considered the unice ability to do so one of those
razor-sharp pointy bits you need to be very, very careful about.

--
|_ CJSonnack <Ch***@Sonnack. com> _____________| How's my programming? |
|_ http://www.Sonnack.com/ _______________ ____| Call: 1-800-DEV-NULL |
|______________ _______________ _______________ _|_____________ __________|
Jul 17 '05 #30

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

110
10666
by: alf | last post by:
Hi, is it possible that due to OS crash or mysql itself crash or some e.g. SCSI failure to lose all the data stored in the table (let's say million of 1KB rows). In other words what is the worst case scenario for MyISAM backend? Also is it possible to not to lose data but get them corrupted?
0
9903
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, weíll explore What is ONU, What Is Router, ONU & Routerís main usage, and What is the difference between ONU and Router. Letís take a closer look ! Part I. Meaning of...
0
9752
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
11041
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10692
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10766
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
5753
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5946
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4565
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
3193
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.