10GB XML Blows out Memory, Suggestions?

[axwack]

I wrote a program that takes an XML file into memory using Minidom. I
found out that the XML document is 10gb.

I clearly need SAX or something else?

Any suggestions on what that something else is? Is it hard to convert
the code from DOM to SAX?

[Rene Pijlman]

ax****@gmail.co m:

I wrote a program that takes an XML file into memory using Minidom. I
found out that the XML document is 10gb.

I clearly need SAX or something else?

Any suggestions on what that something else is?

PullDOM.
http://www-128.ibm.com/developerwork...tipulldom.html
http://www.prescod.net/python/pulldom.html
http://docs.python.org/lib/module-xml.dom.pulldom.html (not much)

--
René Pijlman

[Mathias Waack]

ax****@gmail.co m wrote:

I wrote a program that takes an XML file into memory using Minidom. I
found out that the XML document is 10gb.

I clearly need SAX or something else?
More memory;)
Maybe you should have a look at pulldom, a combination of sax and dom: it
reads your document in a sax-like manner and expands only selected
sub-trees.
Any suggestions on what that something else is? Is it hard to convert
the code from DOM to SAX?

Assuming a good design of course not. Esp. if you only need some selected
parts of the document SAX should be your choice.

Mathias

[Diez B. Roggisch]

ax****@gmail.co m schrieb:

I wrote a program that takes an XML file into memory using Minidom. I
found out that the XML document is 10gb.

I clearly need SAX or something else?

Any suggestions on what that something else is? Is it hard to convert
the code from DOM to SAX?

Yes.

You could used elementtree iterparse - that should be the easiest solution.

http://effbot.org/zone/element-iterparse.htm

Diez

[K.S.Sreeram]

ax****@gmail.co m wrote:

I wrote a program that takes an XML file into memory using Minidom. I
found out that the XML document is 10gb.

With a 10gb file, you're best bet might be to juse use Expat and C!!

Regards
Sreeram

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEhXVqrgn 0plK5qqURArsdAK CyjsORjKDZlZNhw R82C9bMDKtGtgCf VeCz
mgU+25qIR6eiyLV c/QOPn+U=
=Zv2q
-----END PGP SIGNATURE-----

[Nicola Musatti]

ax****@gmail.co m wrote:

I wrote a program that takes an XML file into memory using Minidom. I
found out that the XML document is 10gb.

I clearly need SAX or something else?

What you clearly need is a better suited file format, but I suspect
you're not in a position to change it, are you?

Cheers,
Nicola Musatti

[Diez B. Roggisch]

K.S.Sreeram schrieb:

ax****@gmail.co m wrote:

I wrote a program that takes an XML file into memory using Minidom. I
found out that the XML document is 10gb.

With a 10gb file, you're best bet might be to juse use Expat and C!!

No what exactly makes C grok a 10Gb file where python will fail to do so?

What the OP needs is a different approach to XML-documents that won't
parse the whole file into one giant tree - but I'm pretty sure that
(c)ElementTree will do the job as well as expat. And I don't recall the
OP musing about performances woes, btw.

Diez

[Paul McGuire]

<ax****@gmail.c om> wrote in message
news:11******** ************@u7 2g2000cwu.googl egroups.com...

I wrote a program that takes an XML file into memory using Minidom. I
found out that the XML document is 10gb.

I clearly need SAX or something else?

You clearly need something instead of XML.

This sounds like a case where a prototype, which worked for the developer's
simple test data set, blows up in the face of real user/production data.
XML adds lots of overhead for nested structures, when in fact, the actual
meat of the data can be relatively small. Note also that this XML overhead
is directly related to the verbosity of the XML designer's choice of tag
names, and whether the designer was predisposed to using XML elements over
attributes. Imagine a record structure for a 3D coordinate point (described
here in no particular coding language):

struct ThreeDimPoint:
xValue : integer,
yValue : integer,
zValue : integer

Directly translated to XML gives:

<ThreeDimPoin t>
<xValue>4</xValue>
<yValue>5</yValue>
<zValue>6</zValue>
</ThreeDimPoint>

This expands 3 integers to a whopping 101 characters. Throw in namespaces
for good measure, and you inflate the data even more.

Many Java folks treat XML attributes as anathema, but look how this cuts
down the data inflation:

<ThreeDimPoin t xValue="4" yValue="5" zValue="6"/>

This is only 50 characters, or *only* 4 times the size of the contained data
(assuming 4-byte integers).

Try zipping your 10Gb file, and see what kind of compression you get - I'll
bet it's close to 30:1. If so, convert the data to a real data storage
medium. Even a SQLite database table should do better, and you can ship it
around just like a file (just can't open it up like a text file).

-- Paul

[skip]

Paul> You clearly need something instead of XML.

Amen, brother...

+1 QOTW.

Skip

[Kay Schluehr]

ax****@gmail.co m wrote:

I wrote a program that takes an XML file into memory using Minidom. I
found out that the XML document is 10gb.

I clearly need SAX or something else?

Any suggestions on what that something else is? Is it hard to convert
the code from DOM to SAX?

If your XML files grow so large you might rethink the representation
model. Maybe you give eXist a try?

http://exist.sourceforge.net/

Regards,
Kay