Validating UpdateGram

[Iain]

I've got a system which takes an XML file, translates it into an update gram
and then loads it into my database with SQLXML3 (all in dot net).

But it's fragile. And the SQLXML 3 error reporting is not absolutely
wonderful.

So what I want to do is to validate it before I upload it.

When I try and do this I get MILLIONS of errors (well lots) complaining
about elements like ROOT which are part of the
urn:schemas-microsoft-com:mapping-schema namespace.

SO I guess I need an xsd file which describes this.

There must be one somewhere? Surely? But I can find no trace in MSDN, on
the web or in the groups.

Can anyone point me to such animals?

Iain

P.S. If they do not exist as such it would seem to me they should do. Both
for formal documentation and to help people like me.

THanks

[Christoph Schittko [MVP]]

At first I thought they would be quite helpful myself. If you think about it
though, you will not get very far with validating updategrams with schemas
since the schema to be very loose. The schema can only define the <before>
and <after> elements. It cannot define their content because the content is
specific to yor application. Furthermore, updategrams expect annotations to
the application-specific content which cannot expressed in a generic schema
either. If you really want to thoroghly want to validate your updategrams,
you have to define a comprehensive schema yourself.

However, most errors you get back using updategrams are not based on invalid
XML. They are usually caused by limitations of
SQLXML. Some of the error information comes back in the SqlXmlException [0].
If the error information is not sufficient, you can only trace the SQL
SQLXML sends to SQL Server using Sql Profiler, then execute the SQL that
causes errors using Query Analyzer and determine the source of the error.
That technique is not necesserily for the faint of heart ;)

--
HTH
Christoph Schittko [MVP, XmlInsider]
Software Architect, .NET Mentor

[0] http://www.topxml.com/sqlxml/sqlxmlexception_class.asp

"Iain" <id********@dircon.co.uk> wrote in message
news:u7*************@tk2msftngp13.phx.gbl...

I've got a system which takes an XML file, translates it into an update gram and then loads it into my database with SQLXML3 (all in dot net).

But it's fragile. And the SQLXML 3 error reporting is not absolutely
wonderful.

So what I want to do is to validate it before I upload it.

When I try and do this I get MILLIONS of errors (well lots) complaining
about elements like ROOT which are part of the
urn:schemas-microsoft-com:mapping-schema namespace.

SO I guess I need an xsd file which describes this.

There must be one somewhere? Surely? But I can find no trace in MSDN, on
the web or in the groups.

Can anyone point me to such animals?

Iain

P.S. If they do not exist as such it would seem to me they should do. Both for formal documentation and to help people like me.

THanks

[Iain]

Thanks for your reply Christoph.

Well. An updategram scheme is just a schema - and it can be pretty
comprehensive, of course.

What is missing from it are simply the definitions in the namespace
'schemas-microsoft-com:mapping-schema'. (or rather the urn which it refers
to). So definitions of at-identity and <ROOT> and stuff should fix this.

Were this available, I believe that (after adding it to the
SchemaCollection) you should be able to validate the XML input before
passing it through to SQLXML for processing.

At the moment, I check the raw input file against an XSD, then translate it
to an updategram (with quite a bit of mangling to make nice XML fit into the
more restricted format which SQLXML accepts). Then apply the updategram.
Any errors in my XSLT are therefore hard to debug (especially give that the
level of reporting from SQLXML amounts to 'oops' as you suggested).

Of course there are still subtle issues with the SQL data bits and I have NO
desire for the SQL trace learning curve.

So MS. PLEASE can we have an XSD for this?

Iain

PS. Christoph, you'll know this... When I do my Xslt I get an output which
is all on one line. I've tried puting <output indent="yes" ...> in the xslt
which is ignored. What do I need to do to get nicely formated text?
"Christoph Schittko [MVP]" <ch********************@austin.rr.com> wrote in
message news:#R**************@TK2MSFTNGP10.phx.gbl...

At first I thought they would be quite helpful myself. If you think about it though, you will not get very far with validating updategrams with schemas
since the schema to be very loose. The schema can only define the <before>
and <after> elements. It cannot define their content because the content is specific to yor application. Furthermore, updategrams expect annotations to the application-specific content which cannot expressed in a generic schema either. If you really want to thoroghly want to validate your updategrams,
you have to define a comprehensive schema yourself.

However, most errors you get back using updategrams are not based on invalid XML. They are usually caused by limitations of
SQLXML. Some of the error information comes back in the SqlXmlException [0]. If the error information is not sufficient, you can only trace the SQL
SQLXML sends to SQL Server using Sql Profiler, then execute the SQL that
causes errors using Query Analyzer and determine the source of the error.
That technique is not necesserily for the faint of heart ;)

--
HTH
Christoph Schittko [MVP, XmlInsider]
Software Architect, .NET Mentor

[Christoph Schittko [MVP]]

Iain,

The first problem defining schema for an updategram is that the root node of
an updategram is not part of the urn:schemas-microsoft-com:xml-updategram
namespace. In fact it's not part of any namespace, which makes things
already very complicated, esp. if you tried to validate content inside
<before> or <after> that wasn't part of any namespace either.

Microsoft could publish a schema that defines schema types for <before> and
<after>, but those schemas would be extremely lose. At best it could be
something like:

<complexType name="updgType">
<sequence minOccurs="0" maxOccurs="unbounded">
<element name="before" type="beforeType"/>
<element name="after" type="afterType" />
<sequence>
</complexType>

<complexType name="beforeType">
<sequence>
<any />
</sequence>
</complexType>

<complexType name="afterType">
<sequence>
<any />
</sequence>
<attribute name="returnid" type="string" />
</complexType>

Now you still don't have a schema that you can validate against. You could
define a schema that defines the ROOT element (and possible all other
content that's not part of the namespacem but that is still not solving your
problem.

Another issue you have now are the id, at-identity and guid attributes from
the updategram namespace. If your <before> and <after> elements contain
content that is bound to a namespace then you need to extend that schema to
allow all element content to have these attributes, otherwise you could not
validate the content of the updategram against the schemas that define the
content.

Does this help clearing things up a little bit?
--
HTH
Christoph Schittko [MVP, XmlInsider]
Software Architect, .NET Mentor
"Iain" <id********@dircon.co.uk> wrote in message
news:uH**************@TK2MSFTNGP12.phx.gbl...

Thanks for your reply Christoph.

Well. An updategram scheme is just a schema - and it can be pretty
comprehensive, of course.

What is missing from it are simply the definitions in the namespace
'schemas-microsoft-com:mapping-schema'. (or rather the urn which it refers
to). So definitions of at-identity and <ROOT> and stuff should fix this.

Were this available, I believe that (after adding it to the
SchemaCollection) you should be able to validate the XML input before
passing it through to SQLXML for processing.

At the moment, I check the raw input file against an XSD, then translate it to an updategram (with quite a bit of mangling to make nice XML fit into the more restricted format which SQLXML accepts). Then apply the updategram.
Any errors in my XSLT are therefore hard to debug (especially give that the level of reporting from SQLXML amounts to 'oops' as you suggested).

Of course there are still subtle issues with the SQL data bits and I have NO desire for the SQL trace learning curve.

So MS. PLEASE can we have an XSD for this?

Iain

PS. Christoph, you'll know this... When I do my Xslt I get an output which is all on one line. I've tried puting <output indent="yes" ...> in the xslt which is ignored. What do I need to do to get nicely formated text?
"Christoph Schittko [MVP]" <ch********************@austin.rr.com> wrote in
message news:#R**************@TK2MSFTNGP10.phx.gbl...

At first I thought they would be quite helpful myself. If you think about

it

though, you will not get very far with validating updategrams with

schemas since the schema to be very loose. The schema can only define the <before> and <after> elements. It cannot define their content because the content

is

specific to yor application. Furthermore, updategrams expect annotations

to

the application-specific content which cannot expressed in a generic

schema

either. If you really want to thoroghly want to validate your updategrams, you have to define a comprehensive schema yourself.

However, most errors you get back using updategrams are not based on

invalid

XML. They are usually caused by limitations of
SQLXML. Some of the error information comes back in the SqlXmlException

[0].

If the error information is not sufficient, you can only trace the SQL
SQLXML sends to SQL Server using Sql Profiler, then execute the SQL that
causes errors using Query Analyzer and determine the source of the error. That technique is not necesserily for the faint of heart ;)

--
HTH
Christoph Schittko [MVP, XmlInsider]
Software Architect, .NET Mentor

[Iain]

Thanks for your feedback.

I think I'd have to try and do this to really understand it.

What I have, of course is an XSD that defines the datastructure of the tables I'm writing into and how to map to those tables from the xml file I put in. Snippet might look like below.

--------------------------

<xsd:schema xmlns:disk="http://DOD.COM/Disks"
targetNamespace="http://DOD.COM/Disks"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:sql="urn:schemas-microsoft-com:mapping-schema">
<xsd:annotation>
<xsd:appinfo>
<sql:relationship name="OrdersInBatch" parent="OrderBatches" parent-key="OrderBatchID" child="Orders"
child-key="OrderBatchID" />
</xsd:appinfo>
</xsd:annotation>
<xsd:complexType name="OrderType"></xsd:complexType>
<xsd:element name="OrderBatch" sql:relation="OrderBatches">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="Order" maxOccurs="unbounded" sql:relationship="OrdersInBatch" sql:relation="Orders">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="ShipName" type="disk:String50" minOccurs="1" maxOccurs="1" />
</xsd:sequence>
<xsd:attribute name="FileName" type="xsd:string" use="required" />
</xsd:complexType>
</xsd:element>
</xsd:schema>
-------------------------

Now surely if I changed this (maybe with an xslt to make it automatic) so that the "ROOT" element was insterted, with an import with the sqlxml namespace defined, this would allow validation?

Something like what you describe below but with the

<sequence>
<any />
</sequence>

replaced by my xsd contents?

I may give this a go, especially if you think it makes sense. As I am only uploading data then I only need concern myself with the <after> element.

Iain
"Christoph Schittko [MVP]" <ch********************@austin.rr.com> wrote in message news:OS**************@TK2MSFTNGP12.phx.gbl...

Iain,

The first problem defining schema for an updategram is that the root node of
an updategram is not part of the urn:schemas-microsoft-com:xml-updategram
namespace. In fact it's not part of any namespace, which makes things
already very complicated, esp. if you tried to validate content inside
<before> or <after> that wasn't part of any namespace either.

Microsoft could publish a schema that defines schema types for <before> and
<after>, but those schemas would be extremely lose. At best it could be
something like:

<complexType name="updgType">
<sequence minOccurs="0" maxOccurs="unbounded">
<element name="before" type="beforeType"/>
<element name="after" type="afterType" />
<sequence>
</complexType>

<complexType name="beforeType">
<sequence>
<any />
</sequence>
</complexType>

<complexType name="afterType">
<sequence>
<any />
</sequence>
<attribute name="returnid" type="string" />
</complexType>

Now you still don't have a schema that you can validate against. You could
define a schema that defines the ROOT element (and possible all other
content that's not part of the namespacem but that is still not solving your
problem.

Another issue you have now are the id, at-identity and guid attributes from
the updategram namespace. If your <before> and <after> elements contain
content that is bound to a namespace then you need to extend that schema to
allow all element content to have these attributes, otherwise you could not
validate the content of the updategram against the schemas that define the
content.

Does this help clearing things up a little bit?
--
HTH
Christoph Schittko [MVP, XmlInsider]
Software Architect, .NET Mentor

[Christoph Schittko [MVP]]

If you were to add a definition for a ROOT element to the schema then that ROOT element would belong to the http://DOD.COM/Disks namespace. I seem to remember that updategrams work regardless what namespace the ROOT element belongs to because an updategram is a specialized case of a SQLXML template. I am not sure about this though. It's for you to try that out. ;)

You you need to define the root type to have zero or more <before><after> groups :
<schema targetNamespace="urn:schemas-microsoft-com:mapping-schema"
xmlns="http://www.w3.org/2001/XMLSchema">
<complexType name="updgType">
<sequence minOccurs="0" maxOccurs="unbounded">
<element name="before" type="beforeType" minOccurs="0" maxOccurs="1"/>
<element name="after" type="afterType" minOccurs="0" maxOccurs="1"/>
<sequence>
</complexType>

<complexType name="beforeType">
<sequence>
<any /> <!-- no validation possible -->
</sequence>
</complexType>

<complexType name="afterType">
<sequence>
<any /> <!-- no validation possible -->
</sequence>
<attribute name="returnid" type="string" />
</complexType>

</schema>

<schema xmlns:disk="http://DOD.COM/Disks"
targetNamespace="http://DOD.COM/Disks"
xmlns="http://www.w3.org/2001/XMLSchema"
xmlns:sql="urn:schemas-microsoft-com:mapping-schema">

<element name="ROOT" type="sql:updgType" />

That's the first problem solved, but now it gets harder. You need to define the content the ROOT element. You could go with the types I dummied up in my previous post but the benefits of validating against that schema are marginal. You ensure that the ROOT element has groups of <before><after> elements, where either one of them can be optional. You are not able to validate the content inside the <before> or <after> elements because the urn:schemas-microsoft-com:mapping-schema namespace cannot know anything about any application specific schemas and has to define both types as completely open:

<element name="before">
<complexType>
<sequence>
<any namespace="##any" minOccurs="0" maxOccurs="unbounded"/>
</sequence>
</complexType>
</element>

You can't "forward declare" elements and types like you can in programming languages like C++. Even of you tried to define your own custom schema for the urn:schemas-microsoft-com:mapping-schema namespace that only your application uses to validate updategrams like this:

<element name="before">
<complexType>
<sequence>
<any namespace="http://DOD.COM/Disks" minOccurs="0" maxOccurs="unbounded"/>
</sequence>
</complexType>
</element>

I don't believe you couldn't solve that problem. You would introduce a circular dependency and if I read the XML Schema spec [0] right then this is not allowed.

The id, at-identity and guid attributes pose another probem. Updategrams define the to propagating automatically generated @@IDENTITY values You can define them in an attributeGroup in the schema for the mapping schema namespace:

<attributeGroup>
<attribute name="id" type="int" />
<attribute name="at-identity" type="string" />
<attribute name="guid" type="string" />
</attributeGroup>

However, these attributes need to occur on the elements in the XML that you are inserting. Therefore you need to CHANGE(!) your application's schema to make these attributes schema legal on every element where you need expect them to let SQLXML know about @@IDENTITY values. Remember though, that this is probably pointless, since you probably can't build a schema that will validate the types from your schema because of the circular reference problem.

Regardless how you order the two schemas, you will always have a circular reference problem. The mapping-schema schema needs to know about your application's schema's namespace to enable validation of the content of the <before> and <after> element and the application's schema needs to know about the attributeGroup for inserting into tables with @@IDENTITY columns. Avoiding the circular reference with wildcard content renders a schema will not allow reasonable validation of updategrams.

I hope this post makes sense.
--
HTH
Christoph Schittko [MVP, XmlInsider]
Software Architect, .NET Mentor
"Iain" <id********@dircon.co.uk> wrote in message news:#2**************@TK2MSFTNGP12.phx.gbl...
Thanks for your feedback.

I think I'd have to try and do this to really understand it.

What I have, of course is an XSD that defines the datastructure of the tables I'm writing into and how to map to those tables from the xml file I put in. Snippet might look like below.

--------------------------

<xsd:schema xmlns:disk="http://DOD.COM/Disks"
targetNamespace="http://DOD.COM/Disks"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:sql="urn:schemas-microsoft-com:mapping-schema">
<xsd:annotation>
<xsd:appinfo>
<sql:relationship name="OrdersInBatch" parent="OrderBatches" parent-key="OrderBatchID" child="Orders"
child-key="OrderBatchID" />
</xsd:appinfo>
</xsd:annotation>
<xsd:complexType name="OrderType"></xsd:complexType>
<xsd:element name="OrderBatch" sql:relation="OrderBatches">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="Order" maxOccurs="unbounded" sql:relationship="OrdersInBatch" sql:relation="Orders">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="ShipName" type="disk:String50" minOccurs="1" maxOccurs="1" />
</xsd:sequence>
<xsd:attribute name="FileName" type="xsd:string" use="required" />
</xsd:complexType>
</xsd:element>
</xsd:schema>
-------------------------

Now surely if I changed this (maybe with an xslt to make it automatic) so that the "ROOT" element was insterted, with an import with the sqlxml namespace defined, this would allow validation?

Something like what you describe below but with the

<sequence>
<any />
</sequence>

replaced by my xsd contents?

I may give this a go, especially if you think it makes sense. As I am only uploading data then I only need concern myself with the <after> element.

Iain
"Christoph Schittko [MVP]" <ch********************@austin.rr.com> wrote in message news:OS**************@TK2MSFTNGP12.phx.gbl...

Iain,

The first problem defining schema for an updategram is that the root node of
an updategram is not part of the urn:schemas-microsoft-com:xml-updategram
namespace. In fact it's not part of any namespace, which makes things
already very complicated, esp. if you tried to validate content inside
<before> or <after> that wasn't part of any namespace either.

Microsoft could publish a schema that defines schema types for <before> and
<after>, but those schemas would be extremely lose. At best it could be
something like:

<complexType name="updgType">
<sequence minOccurs="0" maxOccurs="unbounded">
<element name="before" type="beforeType"/>
<element name="after" type="afterType" />
<sequence>
</complexType>

<complexType name="beforeType">
<sequence>
<any />
</sequence>
</complexType>

<complexType name="afterType">
<sequence>
<any />
</sequence>
<attribute name="returnid" type="string" />
</complexType>

Now you still don't have a schema that you can validate against. You could
define a schema that defines the ROOT element (and possible all other
content that's not part of the namespacem but that is still not solving your
problem.

Another issue you have now are the id, at-identity and guid attributes from
the updategram namespace. If your <before> and <after> elements contain
content that is bound to a namespace then you need to extend that schema to
allow all element content to have these attributes, otherwise you could not
validate the content of the updategram against the schemas that define the
content.

Does this help clearing things up a little bit?
--
HTH
Christoph Schittko [MVP, XmlInsider]
Software Architect, .NET Mentor

[Iain]

I'm gonna have to try it.

Bear in mind that I am NOT looking for a general solution: I KNOW that I
have no before content and I can control my namespace if need be - and I've
already transformed my input from some more general XML to the subset of the
dialect which SQLXML likes.

I'll let you know as and when and if I get anywhere!

Thanks!

Iain
"Christoph Schittko [MVP]" <ch********************@austin.rr.com> wrote in
message news:eJ**************@TK2MSFTNGP10.phx.gbl...
If you were to add a definition for a ROOT element to the schema then that
ROOT element would belong to the http://DOD.COM/Disks namespace. I seem to
remember that updategrams work regardless what namespace the ROOT element
belongs to because an updategram is a specialized case of a SQLXML template.
I am not sure about this though. It's for you to try that out. ;)

You you need to define the root type to have zero or more <before><after>
groups :
<schema targetNamespace="urn:schemas-microsoft-com:mapping-schema"
xmlns="http://www.w3.org/2001/XMLSchema">
<complexType name="updgType">
<sequence minOccurs="0" maxOccurs="unbounded">
<element name="before" type="beforeType" minOccurs="0" maxOccurs="1"/>
<element name="after" type="afterType" minOccurs="0" maxOccurs="1"/>
<sequence>
</complexType>

<complexType name="beforeType">
<sequence>
<any /> <!-- no validation possible -->
</sequence>
</complexType>

<complexType name="afterType">
<sequence>
<any /> <!-- no validation possible -->
</sequence>
<attribute name="returnid" type="string" />
</complexType>

</schema>

<schema xmlns:disk="http://DOD.COM/Disks"
targetNamespace="http://DOD.COM/Disks"
xmlns="http://www.w3.org/2001/XMLSchema"
xmlns:sql="urn:schemas-microsoft-com:mapping-schema">

<element name="ROOT" type="sql:updgType" />

That's the first problem solved, but now it gets harder. You need to define
the content the ROOT element. You could go with the types I dummied up in my
previous post but the benefits of validating against that schema are
marginal. You ensure that the ROOT element has groups of <before><after>
elements, where either one of them can be optional. You are not able to
validate the content inside the <before> or <after> elements because the
urn:schemas-microsoft-com:mapping-schema namespace cannot know anything
about any application specific schemas and has to define both types as
completely open:

<element name="before">
<complexType>
<sequence>
<any namespace="##any" minOccurs="0" maxOccurs="unbounded"/>
</sequence>
</complexType>
</element>

You can't "forward declare" elements and types like you can in programming
languages like C++. Even of you tried to define your own custom schema for
the urn:schemas-microsoft-com:mapping-schema namespace that only your
application uses to validate updategrams like this:

<element name="before">
<complexType>
<sequence>
<any namespace="http://DOD.COM/Disks" minOccurs="0" maxOccurs="unbounded"/>
</sequence>
</complexType>
</element>

I don't believe you couldn't solve that problem. You would introduce a
circular dependency and if I read the XML Schema spec [0] right then this is
not allowed.

The id, at-identity and guid attributes pose another probem. Updategrams
define the to propagating automatically generated @@IDENTITY values You can
define them in an attributeGroup in the schema for the mapping schema
namespace:

<attributeGroup>
<attribute name="id" type="int" />
<attribute name="at-identity" type="string" />
<attribute name="guid" type="string" />
</attributeGroup>

However, these attributes need to occur on the elements in the XML that you
are inserting. Therefore you need to CHANGE(!) your application's schema to
make these attributes schema legal on every element where you need expect
them to let SQLXML know about @@IDENTITY values. Remember though, that this
is probably pointless, since you probably can't build a schema that will
validate the types from your schema because of the circular reference
problem.

Regardless how you order the two schemas, you will always have a circular
reference problem. The mapping-schema schema needs to know about your
application's schema's namespace to enable validation of the content of the
<before> and <after> element and the application's schema needs to know
about the attributeGroup for inserting into tables with @@IDENTITY columns.
Avoiding the circular reference with wildcard content renders a schema will
not allow reasonable validation of updategrams.

I hope this post makes sense.
--
HTH
Christoph Schittko [MVP, XmlInsider]
Software Architect, .NET Mentor