huge xml

[joe]

hi,

I have to import a huge xml file into our system. the problem is that
the xml source can be larger than 2GB. I wanted to use the
XmlTextReader class for this purpose but it's limited to 2GB (which
makes sense to me, why would people make so big xml files?).
The xml file basically contains a big list of products, which I want
to read one by one. The xml itself is referring to a dtd. Now I have 2
problems. First, of course that the validation will take ages (it
needs to run later on a server enviroment) so I need a way to tell the
XmlTextReader that it should ignore the dtd. second, I don't want to
validate that the xml is well formed (also that requires that the
whole document is processed), I only need to know that each 'product'
node is well formed. now my question. which class should I use to
accomplish this? any ideas?

thank you,
Sergio

[Bruce Wood]

First, where did you read that XmlTextReader is limited to 2GB? I can't
imagine why this would be so, because XmlTextReader does not read the
entire document in order to check for well-formedness or validity. Your
statement that the reader has to read the entire document to check for
these things is false: so far as I know, XmlTextReader reads the
document line-by-line, or even character-by-character, remembering only
enough state to perform validation. The size of the file should not
matter.

(If that didn't make sense, think of it this way: XmlTextReader reads
the document bit by bit, and only knows that the document is
well-formed and valid _up to the point it's read so far_.)

I can think of only one situation in which a parser would have to
remember a lot of past history in order to validate, and that is if
you're using keys and references in XML, and the parser wants to
guarantee that the keys are unique and that the references match up.
Since DTDs can't express keys and references, you're safe there.

Besides, you can always instruct the reader not to validate, although
you can't tell it not to check for well-formedness (which definitely
_doesn't_ require having the entire document in hand).

I wrote something I call an XmlFragmentReader on top of XmlTextReader
that does what you want: it reads an XML document in chunks, returning
each chunk (in your case each product) as an XmlDocument (a DOM tree).
I designed it precisely for parsing documents like yours.

However, I'm still wondering where you read that XmlTextReader can
handle only up to 2GB of XML. I'm always open to being proven wrong. :)

[Richard L Rosenheim]

Bruce,

See
http://msdn.microsoft.com/library/de...hXmlReader.asp

After the class table, there the following paragraph:
Note The XmlTextReader and XmlValidatingReader are constrained on the
size of files they can read. They cannot read files larger than 2 gigabytes.
If it is possible, split the source file into smaller, multiple files.
Like you, I'm baffled why there is such a limitation. Does anyone know if
this limitation will still be true with 2.0?

Richard Rosenheim
"Bruce Wood" <br*******@canada.com> wrote in message
news:11**********************@o13g2000cwo.googlegr oups.com...

First, where did you read that XmlTextReader is limited to 2GB? I can't
imagine why this would be so, because XmlTextReader does not read the
entire document in order to check for well-formedness or validity. Your
statement that the reader has to read the entire document to check for
these things is false: so far as I know, XmlTextReader reads the
document line-by-line, or even character-by-character, remembering only
enough state to perform validation. The size of the file should not
matter.

(If that didn't make sense, think of it this way: XmlTextReader reads
the document bit by bit, and only knows that the document is
well-formed and valid _up to the point it's read so far_.)

I can think of only one situation in which a parser would have to
remember a lot of past history in order to validate, and that is if
you're using keys and references in XML, and the parser wants to
guarantee that the keys are unique and that the references match up.
Since DTDs can't express keys and references, you're safe there.

Besides, you can always instruct the reader not to validate, although
you can't tell it not to check for well-formedness (which definitely
_doesn't_ require having the entire document in hand).

I wrote something I call an XmlFragmentReader on top of XmlTextReader
that does what you want: it reads an XML document in chunks, returning
each chunk (in your case each product) as an XmlDocument (a DOM tree).
I designed it precisely for parsing documents like yours.

However, I'm still wondering where you read that XmlTextReader can
handle only up to 2GB of XML. I'm always open to being proven wrong. :)

[Bruce Wood]

How, exactly, does Microsoft expect one to "split the source files into
smaller, multiple files"?

Using msxsl, which is probably based on XmlTextReader in the first
place? That would be rich: "Please split the file up so that
XmlTextReader can read it. By the way, the only way to split the file
up is with a tool that uses XmlTextReader...."

I would love to know what that limitation is all about, since all
documentation I've read and everything I know about XmlTextReader
indicates that it does _not_ read the entire file at once.

Sergio,

It shouldn't be too hard to write a cheap-n-nasty little application
that just reads the XML as a text stream, recognizes your (unique) XML
format, and breaks the file into manageable chunks (say every n
products) that you could read using XmlTextReader.

[Oleg Tkachenko [MVP]]

Bruce Wood wrote:

First, where did you read that XmlTextReader is limited to 2GB? I can't
imagine why this would be so, because XmlTextReader does not read the
entire document in order to check for well-formedness or validity.

It happens to be Int32.MaxValue value :)

--
Oleg Tkachenko [XML MVP, MCP]
http://blog.tkachenko.com

[Oleg Tkachenko [MVP]]

joe wrote:

I have to import a huge xml file into our system. the problem is that
the xml source can be larger than 2GB. I wanted to use the
XmlTextReader class for this purpose but it's limited to 2GB (which
makes sense to me, why would people make so big xml files?).
The xml file basically contains a big list of products, which I want
to read one by one. The xml itself is referring to a dtd. Now I have 2
problems. First, of course that the validation will take ages (it
needs to run later on a server enviroment) so I need a way to tell the
XmlTextReader that it should ignore the dtd. second, I don't want to
validate that the xml is well formed (also that requires that the
whole document is processed), I only need to know that each 'product'
node is well formed. now my question. which class should I use to
accomplish this? any ideas?

First of all XmlTextReader never validates. It also can't resolve
external entities. It only checks if referred DTD exists. And it doesn't
read the whole document unless you call Read() method.
2Gb limit is apparently int number limit for some internal counters. I'm
not sure about .NET 2.0, but I will test this case for sure.
I believe the bets solution would be to split document at the time it
gets created. How do you get the document?

--
Oleg Tkachenko [XML MVP, MCP]
http://blog.tkachenko.com

[Richard L Rosenheim]

I thought of the same question. The best I could figure was that the author
of that recommendation assumed that it would be possible for the source
program to split the XML data into multiple files. Of course, that is also
assuming that you have any say/control/input over the program that
originally generated the XML file.

Richard Rosenheim

"Bruce Wood" <br*******@canada.com> wrote in message
news:11*********************@g14g2000cwa.googlegro ups.com...

How, exactly, does Microsoft expect one to "split the source files into
smaller, multiple files"?

Using msxsl, which is probably based on XmlTextReader in the first
place? That would be rich: "Please split the file up so that
XmlTextReader can read it. By the way, the only way to split the file
up is with a tool that uses XmlTextReader...."

I would love to know what that limitation is all about, since all
documentation I've read and everything I know about XmlTextReader
indicates that it does _not_ read the entire file at once.

Sergio,

It shouldn't be too hard to write a cheap-n-nasty little application
that just reads the XML as a text stream, recognizes your (unique) XML
format, and breaks the file into manageable chunks (say every n
products) that you could read using XmlTextReader.

[Bruce Wood]

Ahh, I hadn't thought of that. Yes, you're right.

Of course, I still wonder why XmlTextReader is keeping internal state
of the kind that would care how long the document is...?

[Jay B. Harlow [MVP - Outlook]]

Joe,
I'm not sure how real the 2GB limit is, as a FileStream has a Int64.MaxValue
file limit! In other words the size of the file should really be limited by
the amount of disk space you have!

I just created a 2.2GB XML file via XmlTextWriter and was able to read the
entire file via XmlTextReader.

I suspect the 2GB limit has more to do with XmlTextReader.LineNumber and
XmlTextReader.LinePosition, both which are Int32 values (As Oleg suggests
2GB is Int32.MaxValue).

Unfortunately it took 1/2 hour to create the file & 1 hour to read the file
on my desktop, I may set up a couple other test cases on my test server
(faster CPU, more disk space) with larger files...

Hope this helps
Jay

"joe" <ru*@on-consult.ch> wrote in message
news:b3**************************@posting.google.c om...

hi,

I have to import a huge xml file into our system. the problem is that
the xml source can be larger than 2GB. I wanted to use the
XmlTextReader class for this purpose but it's limited to 2GB (which
makes sense to me, why would people make so big xml files?).
The xml file basically contains a big list of products, which I want
to read one by one. The xml itself is referring to a dtd. Now I have 2
problems. First, of course that the validation will take ages (it
needs to run later on a server enviroment) so I need a way to tell the
XmlTextReader that it should ignore the dtd. second, I don't want to
validate that the xml is well formed (also that requires that the
whole document is processed), I only need to know that each 'product'
node is well formed. now my question. which class should I use to
accomplish this? any ideas?

thank you,
Sergio

[Jay B. Harlow [MVP - Outlook]]

Interesting. In case anyone is interested.

The 2GB limit appears to be XmlTextReader.LinePosition only.

If I create an XML file (3.9G in my test case) with out formatting, so
basically the file is a single line longer then 2G I get an IndexOutOfRange
exception, as the LinePosition reaches 2G.

However if I create an XML file where each node is on its own line (8.5G in
my test case), plus bunches of blank lines (to get > 2G lines) then the
XmlTextReader.LineNumber property starts returning negative numbers. However
no exception...

So it appears to me that the 2G limit is really the length of one line
within the document...

Jay

"Jay B. Harlow [MVP - Outlook]" <Ja************@msn.com> wrote in message
news:ed**************@TK2MSFTNGP10.phx.gbl...

Joe,
I'm not sure how real the 2GB limit is, as a FileStream has a
Int64.MaxValue file limit! In other words the size of the file should
really be limited by the amount of disk space you have!

I just created a 2.2GB XML file via XmlTextWriter and was able to read the
entire file via XmlTextReader.

I suspect the 2GB limit has more to do with XmlTextReader.LineNumber and
XmlTextReader.LinePosition, both which are Int32 values (As Oleg suggests
2GB is Int32.MaxValue).

Unfortunately it took 1/2 hour to create the file & 1 hour to read the
file on my desktop, I may set up a couple other test cases on my test
server (faster CPU, more disk space) with larger files...

Hope this helps
Jay

"joe" <ru*@on-consult.ch> wrote in message
news:b3**************************@posting.google.c om...

hi,

I have to import a huge xml file into our system. the problem is that
the xml source can be larger than 2GB. I wanted to use the
XmlTextReader class for this purpose but it's limited to 2GB (which
makes sense to me, why would people make so big xml files?).
The xml file basically contains a big list of products, which I want
to read one by one. The xml itself is referring to a dtd. Now I have 2
problems. First, of course that the validation will take ages (it
needs to run later on a server enviroment) so I need a way to tell the
XmlTextReader that it should ignore the dtd. second, I don't want to
validate that the xml is well formed (also that requires that the
whole document is processed), I only need to know that each 'product'
node is well formed. now my question. which class should I use to
accomplish this? any ideas?

thank you,
Sergio

[Bruce Wood]

Thanks, Jay. That's very useful information.