XML SAX parser bug?

[mitsura]

Hi,

I think I ran into a bug in the XML SAX parser.

part of my program consist of reading a rather large XML file (about
10Mb) containing a few thousand elements.
I have the following problem. Sometimes that SAX parses misreads a
line.
Let me explain: the XML file contains a few thousand lines like this:
"
<TargetRef>WINO SSPI:Storage@@n 91c90a.cmc.com</TargetRef>
"
where 'n91c90a.cmc.co m' is the name of a system and thus changes per
system.
I a few cases, the SAX parser misreads the line. The parser sometimes
plits characters the line in:
"WINOSSPI:Stora ge@@n" and "91c90a.cmc.com ".
I put a 'print characters' line in the 'characters' method of the
parser that is how I found out.
It only happens for a few of the thousand lines but you can imagine
that is very annoying.

I checked for errors in the XML file but the file seems ok.

Is this a bug or am I doing something wrong?
I am new to Python.

I am using Python 2.4.1, pyWin32 extension 2.4 and PyXML 0.8.4

Any help very much appreciated.

Kris

[Fredrik Lundh]

mi*****@skynet. be wrote:

I think I ran into a bug in the XML SAX parser.

part of my program consist of reading a rather large XML file (about
10Mb) containing a few thousand elements.
I have the following problem. Sometimes that SAX parses misreads a
line.
Let me explain: the XML file contains a few thousand lines like this:
"
<TargetRef>WINO SSPI:Storage@@n 91c90a.cmc.com</TargetRef>
"
where 'n91c90a.cmc.co m' is the name of a system and thus changes per
system.
I a few cases, the SAX parser misreads the line. The parser sometimes
plits characters the line in:
"WINOSSPI:Stora ge@@n" and "91c90a.cmc.com ".
I put a 'print characters' line in the 'characters' method of the
parser that is how I found out.
It only happens for a few of the thousand lines but you can imagine
that is very annoying.

I checked for errors in the XML file but the file seems ok.

Is this a bug or am I doing something wrong?

it's not a bug; the parser is free to split up character runs (due to buffering,
entities or character references, etc). it's up to you to merge character runs
into strings.

</F>

[mitsura]

Fredrik Lundh schreef:

mi*****@skynet. be wrote:

I think I ran into a bug in the XML SAX parser.

part of my program consist of reading a rather large XML file (about
10Mb) containing a few thousand elements.
I have the following problem. Sometimes that SAX parses misreads a
line.
Let me explain: the XML file contains a few thousand lines like this:
"
<TargetRef>WINO SSPI:Storage@@n 91c90a.cmc.com</TargetRef>
"
where 'n91c90a.cmc.co m' is the name of a system and thus changes per
system.
I a few cases, the SAX parser misreads the line. The parser sometimes
plits characters the line in:
"WINOSSPI:Stora ge@@n" and "91c90a.cmc.com ".
I put a 'print characters' line in the 'characters' method of the
parser that is how I found out.
It only happens for a few of the thousand lines but you can imagine
that is very annoying.

I checked for errors in the XML file but the file seems ok.

Is this a bug or am I doing something wrong?

it's not a bug; the parser is free to split up character runs (due to buffering,
entities or character references, etc). it's up to you to merge character runs
into strings.

</F>

Thanks for the feedback,

but how do I detect that the parser has split up the characters? I gues
I need to detect it in order to reconstruct the complete string

[Martin v. Löwis]

mi*****@skynet. be wrote:

but how do I detect that the parser has split up the characters? I gues
I need to detect it in order to reconstruct the complete string

Don't try to detect it. Instead, assume it always happens, and collect
the strings in characters(), rather than processing them. Do something
like this

def startElement(se lf, ...):
self.chardata = ""

def characters(self , data):
self.chardata += data

def endElement(self , ...):
process(self.ch ardata)

This is simplified - you might have to deal with nested elements,
somehow.

Regards,
Martin

[uche.ogbuji]

mi*****@skynet. be wrote:

Fredrik Lundh schreef:

mi*****@skynet. be wrote:

I think I ran into a bug in the XML SAX parser.

part of my program consist of reading a rather large XML file (about
10Mb) containing a few thousand elements.
I have the following problem. Sometimes that SAX parses misreads a
line.

it's not a bug; the parser is free to split up character runs (due to buffering,
entities or character references, etc). it's up to you to merge character runs
into strings.

but how do I detect that the parser has split up the characters? I gues
I need to detect it in order to reconstruct the complete string

Here's a recipe:

http://aspn.activestate.com/ASPN/Coo.../Recipe/265881

Using this filter you can then write SAX code that assumes normalized
text events. Also, 4Suite's SAX implementation, Saxlette,
automatically does this text event merging for you at C speed:

http://4suite.org/docs/CoreManual.xml#saxlette

--
Uche Ogbuji Fourthought, Inc.
http://uche.ogbuji.net http://fourthought.com
http://copia.ogbuji.net http://4Suite.org
Articles: http://uche.ogbuji.net/tech/publications/