What could cause XML files to not be completely written?

[darrel]

We've had a month long headache with one of our functions in the CMS we've
built.

Whenever a page is updated in our CMS, we do a recursive call to the DB to
get all the pages (and child pages, and child child pages, etc.) and then
create an XML file, and save that to the filesystem. The front end then uses
that XML file along with some XSL to create the site menus, site maps, etc.

We're having a problem where 99 out of 100 times, this works fine. But that
one time, we'll get an incomplete (and therefore invalid) XML file written
to the filesystem. No error. Nothing. Just a bad file.

What could be the possible causes of this? I've included the function that
writes the XML at the bottom of this message if that helps.

At this point, we're looking into two solutions which just seem a bit hacky:

1) write the file, open the file, validate it, if invalid, write again,
repeat a few times and then report an error if you have to.

2) forget the filesystem. Write the XML file as a VARCHAR back into a DB
table and just grab it from there.

Any other thoughts? Things to look out for? Solutions? Theories?

Thanks!

------------------------------------------------------------------

Imports System.IO
Public Class menuWriter
Shared fs As System.IO.FileStream
'shared objXMLWriter As system.xml.XmlTextWriter

Shared Sub createXMLfileWithXmlWriter(ByVal parentNode As Integer, ByVal
intLevel As Integer, ByVal rowCount As Integer, ByVal createParentNode As
Boolean, ByVal siteID As Integer)
Try
' Create the parent node
'dim objXMLWriter As system.xml.XmlTextWriter
' xml writing adapted from
' http://www.developerfusion.com/show/2646/
If
System.IO.Directory.Exists(System.Web.HttpContext. Current.Server.MapPath("/mjb05/xml/"
& siteID)) Then
'directory exists. Do nothing
Else
'directory doesn't exist. Creat directory
System.IO.Directory.CreateDirectory(System.Web.Htt pContext.Current.Server.MapPath("/mjb05/xml/"
& siteID))
End If

Dim objXMLWriter As System.Xml.XmlTextWriter
Dim path As String =
System.Web.HttpContext.Current.Server.MapPath("/mjb05/xml/" & siteID &
"/siteMenu.xml").ToString
fs = New FileStream(path, FileMode.Create, FileAccess.Write, FileShare.None)
SyncLock fs
objXMLWriter = New System.xml.XmlTextWriter(fs,
System.Text.Encoding.Default)
objXMLWriter.Formatting = objXMLWriter.Formatting.Indented
objXMLWriter.Indentation = 3
objXMLWriter.WriteStartDocument()
objXMLWriter.WriteComment("Created on " & Now())
objXMLWriter.WriteStartElement("menuItems")
WriteXmlData(objXMLWriter, parentNode, siteID)
objXMLWriter.WriteEndElement() 'close menuItems node
objXMLWriter.WriteEndDocument() 'close docment
objXMLWriter.Flush()
objXMLWriter.Close()
End SyncLock
Catch ex As Exception
System.Web.HttpContext.Current.Response.Write("Err or writing XML file<br
/>")
System.Web.HttpContext.Current.Response.Write(ex.M essage.ToString)
End Try
End Sub

Shared Sub WriteXmlData(ByVal objXMLWriter As System.Xml.XmlTextWriter,
ByVal parentNode As Integer, ByVal siteID As Integer, Optional ByVal
intLevel As Int32 = 0)
'retrieve all children of parentNode
' grab the data from the db
Dim DS As New DataSet
Dim strConnect As String
strConnect =
System.Configuration.ConfigurationSettings.AppSett ings("DBConn")
Dim strChk As String
strChk = "SELECT pageID, siteID, parentID, linkText, browserTitle,
contentTitle, hidden, sort, customContent, customContentVariables,
menuCategory, isCategory, homePage " & _
"FROM WeSiteMenus " & _
"WHERE ParentID = " & parentNode & " AND siteID = " & siteID & " " & _
"ORDER BY homePage DESC, sort ASC"
Dim objConnect As New System.Data.OleDb.OleDbConnection(strConnect)
objConnect.Open()
Dim objCommand As New System.Data.OleDb.OleDbCommand(strChk, objConnect)
Dim objOleDbAdapter As New System.Data.OleDb.OleDbDataAdapter(strChk,
strConnect)
objOleDbAdapter.Fill(DS, "webPages")

' Walk through results whether it's a list or a single contact.
' based off of this tutorial:
' http://www.wwwcoder.com/main/parenti...8/default.aspx
Dim rowCount As Int32 = 0
While rowCount < DS.Tables(0).Rows.Count 'ie, if there IS data, then...
objXMLWriter.WriteStartElement("menuItem")

objXMLWriter.WriteElementString("pageID",
Trim(DS.Tables(0).Rows(rowCount)("pageID").ToStrin g))
objXMLWriter.WriteElementString("parentID",
Trim(DS.Tables(0).Rows(rowCount)("parentID").ToStr ing))
'linkURL...need to actuall add this to the DB!
objXMLWriter.WriteElementString("linkURL", "")
objXMLWriter.WriteElementString("linkText",
Trim(DS.Tables(0).Rows(rowCount)("linkText").ToStr ing))
objXMLWriter.WriteElementString("browserTitle",
Trim(DS.Tables(0).Rows(rowCount)("browserTitle").T oString))
objXMLWriter.WriteElementString("contentTitle",
Trim(DS.Tables(0).Rows(rowCount)("contentTitle").T oString))
objXMLWriter.WriteElementString("hidden",
Trim(DS.Tables(0).Rows(rowCount)("hidden").ToStrin g))
objXMLWriter.WriteElementString("customContent",
Trim(DS.Tables(0).Rows(rowCount)("customContent"). ToString))
objXMLWriter.WriteElementString("customContentVari ables",
Trim(DS.Tables(0).Rows(rowCount)("customContentVar iables").ToString))
objXMLWriter.WriteElementString("menuCategory",
Trim(DS.Tables(0).Rows(rowCount)("menuCategory").T oString))
objXMLWriter.WriteElementString("isCategory",
Trim(DS.Tables(0).Rows(rowCount)("isCategory").ToS tring))
objXMLWriter.WriteElementString("level", intLevel.ToString)
'now call the subroutine we're in to see if this value has
'any children and increase the indent, and so on...
WriteXmlData(objXMLWriter,
Convert.ToInt32(DS.Tables(0).Rows(rowCount).Item(0 )), siteID, intLevel + 1)
rowCount = rowCount + 1
objXMLWriter.WriteEndElement() 'close menuItem node
End While
' clean up
objConnect.Close()
objOleDbAdapter.Dispose()
objCommand.Dispose()
End Sub
End Class

[Peter Flynn]

darrel wrote:

We've had a month long headache with one of our functions in the CMS we've
built.

Whenever a page is updated in our CMS, we do a recursive call to the DB to
get all the pages (and child pages, and child child pages, etc.) and then
create an XML file, and save that to the filesystem. The front end then uses
that XML file along with some XSL to create the site menus, site maps, etc.

We're having a problem where 99 out of 100 times, this works fine. But that
one time, we'll get an incomplete (and therefore invalid) XML file written
to the filesystem. No error. Nothing. Just a bad file.
You mean the file just ends in mid-element or something?
The only obvious things that come to mind are a buggy serializer
or a timing problem between the DB and the OS or disk subsystem.
What could be the possible causes of this? I've included the function that
writes the XML at the bottom of this message if that helps.

At this point, we're looking into two solutions which just seem a bit hacky:

1) write the file, open the file, validate it, if invalid, write again,
repeat a few times and then report an error if you have to.

2) forget the filesystem. Write the XML file as a VARCHAR back into a DB
table and just grab it from there.

Easier would be to run an external standalone validator (ie unconnected
to and independent of any other part of your system) which will pass
control onwards if the file is OK, but otherwise terminate the process
and restart it at the file export step.

///Peter
--
XML FAQ: http://xml.silmaril.ie/

[darrel]

> You mean the file just ends in mid-element or something?

Yep. It'll just cut of mid-element. Randomly anywhere in the document.

The only obvious things that come to mind are a buggy serializer
I'm not sure if I'd know what to look for regarding that. What would that
actually be?
or a timing problem between the DB and the OS or disk subsystem.
Well, I'm grabbing all the data from the DB at once, and then writing it to
the XML file in one swoop, so that probably isn't it. No doubt it could be
the OS or disk subsystem, though. NO idea how to even begin to figure out
what specifically to look at in that context, though.
Easier would be to run an external standalone validator (ie unconnected
to and independent of any other part of your system) which will pass
control onwards if the file is OK, but otherwise terminate the process
and restart it at the file export step.

Whoa. So, I'd be passing something to another app and back again? I'm not
sure if that sounds easier to me ;o)

What kind of external validator are we talking about? Just another web app
that I'd whip up? What would be the advantage of externalizing the validator
(vs just validating what was written in the app itself?)

-Darrel

[Peter Flynn]

darrel wrote:

What kind of external validator are we talking about? Just another web app
that I'd whip up? What would be the advantage of externalizing the validator
(vs just validating what was written in the app itself?)

Any of the standalone parsers will do: it doesn't actually have to
validate if all you want is to test the file's completeness. If we
assume that the file is either complete (and correct) or incomplete
(and that no other mid-way corruption takes place) then a simple
well-formedness check will show if the file is complete or not.

My reason for suggesting an external process was simply to guarantee
that the parse was not being affected by anything else that might have
gone on inside your app.

///Peter
--
XML FAQ: http://xml.silmaril.ie/