By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,071 Members | 1,260 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,071 IT Pros & Developers. It's quick & easy.

Problem with minidom and special chars in HTML

P: n/a
Hi :-)
I currently have quite a big problem with minidom and special chars (for
example ü) in HTML.

Let's say I have following input file:
--------------------------------------------------
<?xml version="1.0"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
<body>
&uuml;
</body>
</html>
--------------------------------------------------

And following python script:
--------------------------------------------------
from xml.dom import minidom
if __name__ == '__main__':
doc = minidom.parse('test2.html')
f = open('test3.html','w+')
f.write(doc.toxml())
f.close()
--------------------------------------------------

test3.html only has a blank line where should be the &uuml; It is simply
removed.

Any idea how I could solve this problem?

MfG, Horst
Jul 18 '05 #1
Share this Question
Share on Google+
6 Replies


P: n/a
Horst Gutmann wrote:
I currently have quite a big problem with minidom and special chars (for example &uuml;) in HTML.

Let's say I have following input file:
--------------------------------------------------
<?xml version="1.0"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
<body>
&uuml;
</body>
</html>
-------------------------------------------------- test3.html only has a blank line where should be the &uuml; It is simply
removed.

Any idea how I could solve this problem?


umm. doesn't that doctype point to an SGML DTD? even if minidom did fetch
external DTD's (I don't think it does), it would probably choke on that DTD.

running your documents through "tidy -asxml -numeric" before parsing them as
XML might be a good idea...

http://tidy.sourceforge.net/ (command-line binaries, library)
http://utidylib.berlios.de/ (python bindings)

</F>

Jul 18 '05 #2

P: n/a
Fredrik Lundh wrote:
umm. doesn't that doctype point to an SGML DTD? even if minidom did fetch
external DTD's (I don't think it does), it would probably choke on that DTD.

running your documents through "tidy -asxml -numeric" before parsing them as
XML might be a good idea...

http://tidy.sourceforge.net/ (command-line binaries, library)
http://utidylib.berlios.de/ (python bindings)

</F>


Thanks, but the problem is, that I can't use the numeric representations
of these special chars. I will probably simply play find&replace before
feeding the document into minidom and change the output back afterwards :-)

MfG, Horst
Jul 18 '05 #3

P: n/a
Horst Gutmann napisał(a):
I currently have quite a big problem with minidom and special chars (for
example &uuml;) in HTML.

Let's say I have following input file:
--------------------------------------------------
<?xml version="1.0"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
HTML4 is not an XML application. Even if minidom will fetch this DTD and
be able to parse character entities, it may not be able to parse the
document.
Any idea how I could solve this problem?


Don't use minidom or convert HTML4 to XHTML and change declaration of
doctype.

--
Jarek Zgoda
http://jpa.berlios.de/ | http://www.zgodowie.org/
Jul 18 '05 #4

P: n/a
Jarek Zgoda wrote:
Horst Gutmann napisał(a):
I currently have quite a big problem with minidom and special chars
(for example &uuml;) in HTML.

Let's say I have following input file:
--------------------------------------------------
<?xml version="1.0"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">

HTML4 is not an XML application. Even if minidom will fetch this DTD and
be able to parse character entities, it may not be able to parse the
document.
Any idea how I could solve this problem?

Don't use minidom or convert HTML4 to XHTML and change declaration of
doctype.

This was just a bad example :-) I get the same problem with XHTML in the
doctype. The funny thing here IMO is, that the special chars are simply
removed. No warning, no nothing :-(

MfG, Horst
Jul 18 '05 #5

P: n/a
Horst Gutmann napisał(a):
Don't use minidom or convert HTML4 to XHTML and change declaration of
doctype.

This was just a bad example :-) I get the same problem with XHTML in the
doctype. The funny thing here IMO is, that the special chars are simply
removed. No warning, no nothing :-(


As Fredrik pointed out, it's minidom that cann't fetch DTD from remote
location. Download this DTD file to your local machine (it lies at
exactly this URI), try changing PUBLIC identifier to SYSTEM and give
local path to this file.

--
Jarek Zgoda
http://jpa.berlios.de/ | http://www.zgodowie.org/
Jul 18 '05 #6

P: n/a
Horst Gutmann wrote:
I currently have quite a big problem with minidom and special chars
(for example &uuml;) in HTML.


Yes. Ignoring the issue of the wrong doctype, minidom is a pure XML
parser and knows nothing of XHTML and its doctype's entities 'uuml' and
the like. Only the built-in entities (&amp; etc.) will work.

Unfortunately the parser minidom uses won't read external entities -
including the external subset of the DTD (which is where all the stuff
about what "&uuml;" means is stored). And because minidom does not
support EntityReference nodes, the information that there was an entity
reference there at all gets thrown away as it is replaced with the
empty string. Which is kind of bad.

Possible workarounds:

1. pass minidom a different parser to use, one which supports external
entities and which will parse all the DTD stuff. I don't know if there
is anything suitable available, though...

2. use a DOM implementation with the option to support external
entities. For example, with pxdom, one can use DOM Level 3 LS methods,
or pxdom.parse(f, {'pxdom-external-entities': True}).

However note that reading and parsing an external entity will introduce
significant slowdown, especially in the case of the rather complex
multi-file XHTML DTD. Other possibilities:

3. hack the content on the way into the parser to replace the DOCTYPE
declaration with one including entity definitions in the internal
subset:

<!DOCTYPE html PUBLIC "..." "..." [
<!ENTITY uuml "ü">
...
]>
<html>...

4. hack the content on the way into the parser to replace entity
references with character references, eg. &uuml; -> ü. This is
'safe' for simple documents without an internal subset; charrefs and
entrefs can be used in the same places with the same meaning, except
for some issues in the internal subset.

5. use a DOM implementation that supports EntityReference nodes, such
as pxdom. Entity references with no replacement text (or all entity
references if the DOM Level 3 LS parameter 'entities' is set) will
exist as EntityReference DOM objects instead of being flattened to
text. They can safely be reserialized as &uuml; without the
implementation having to know what text they represent.

Entities are a big source of complication and confusion, which I wish
had not made it into XML!

--
Andrew Clover
mailto:an*@doxdesk.com
http://www.doxdesk.com/

Jul 18 '05 #7

This discussion thread is closed

Replies have been disabled for this discussion.