473,394 Members | 1,693 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,394 software developers and data experts.

How *extract* data from XHTML Transitional web pages? got xml.dom.minidom troubles..

I'm trying to extract some data from an XHTML Transitional web page.

What is best way to do this?

xml.dom.minidom.parseString("text of web page") gives errors about it
not being well formed XML.

Do I just need to add something like <?xml ...?or what?

Chris

Mar 2 '07 #1
4 3840
se******@spawar.navy.mil wrote:
I'm trying to extract some data from an XHTML Transitional web page.

What is best way to do this?
An XML parser should be sufficient. However...
xml.dom.minidom.parseString("text of web page") gives errors about it
not being well formed XML.

Do I just need to add something like <?xml ...?or what?
If the page isn't well-formed then it isn't proper XHTML since the
XHTML specification [1] says...

4.1. Documents must be well-formed

Yes, it's a heading, albeit in an "informative" section describing how
XHTML differs from HTML 4. See "3.2. User Agent Conformance" for a
"normative" mention of well-formedness.

You could try libxml2dom (or other libxml2-based solutions) for some
fairly effective HTML parsing:

libxml2dom.parseString("text of document here", html=1)

See http://www.python.org/pypi/libxml2dom for more details.

Paul

[1] http://www.w3.org/TR/xhtml1/

Mar 3 '07 #2
Den Fri, 02 Mar 2007 15:32:58 -0800 skrev se******@spawar.navy.mil:
I'm trying to extract some data from an XHTML Transitional web page.
xml.dom.minidom.parseString("text of web page") gives errors about it
not being well formed XML.
Do I just need to add something like <?xml ...?or what?
As many HTML Transitional pages are very bad formed, you can't really
create a dom of them.

I've written multiple grabbers, which grab tv data from html pages, and
parses it into xml.

Basicly there are three ways to get the info:

# Use find(): If you are only searching for a few data pieces, you
might be able to find some html code always appearing before the data you
need.

# Use regular expressions: This can very quickly get all data from a
table or so into a nice list. Only problem is regular expressions having
a little steep learing curve.

# Use a SAX parser: This will iterate through all html items, not
carring if they validate or not. You will define a method to be called
each time it finds a tag, a piece of text etc.
What is best way to do this?
In the beginning I mostly did the SAX way, but it really generates a lot
of code, which is not necessaryly more readable than the regular
expressions.
Mar 3 '07 #3
se******@spawar.navy.mil wrote:
I'm trying to extract some data from an XHTML Transitional web page.

What is best way to do this?
May I suggest html5lib [1]? It's based on the parsing section of the
WHATWG "HTML5" spec [2] which is in turn based on the behavior of major
web browsers so it should parse more or less* any invalid markup you
throw at it. Despite the name "html5lib" it works with any (X)HTML
document. By default, you have the option of producing a minidom tree,
an ElementTree, or a "simpletree" - a lightweight DOM-like
html5lib-specific tree.

If you are happy to pull from SVN I recommend that version; it has a few
bug fixes over the 0.2 release as well as improved features including
better error reporting and detection of encoding from <metaelements
(the next release is imminent).

[1] http://code.google.com/p/html5lib/
[2] http://whatwg.org/specs/web-apps/current-work/#parsing

* There might be a problem if e.g. the document uses a character
encoding that python does not support, otherwise it should parse anything.
Mar 3 '07 #4
se******@spawar.navy.mil a écrit :
I'm trying to extract some data from an XHTML Transitional web page.

What is best way to do this?

xml.dom.minidom.
As a side note, cElementTree is probably a better choice. Or even a
simple SAX parser.
>parseString("text of web page") gives errors about it
not being well formed XML.
If it's not well-formed XML, most - if not all - XML parsers will shoke
on it.
Do I just need to add something like <?xml ...?or what?
How could we say without looking at the XML ?

But anyway, even if the XHTML is crappy, BeautifulSoup may do the job...

HTH
Mar 3 '07 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

24
by: Nobody | last post by:
Okay, you are all so smart in here. Answer me this: IE6 in standards mode doesn't seem to hide scrollbars on the body element (overflow:hide) Ain't this a quandary. I have it in my head that I...
17
by: Colin Cogle | last post by:
------- Line 47, column 8: there is no attribute "id" <DIV id="LeftNavigation" style="position:absolute; left:8px; top:6px; width:200p ------- Line 47, column 31: there is no attribute "style"...
35
by: The Bicycling Guitarist | last post by:
My web site has not been spidered by Googlebot since April 2003. The site in question is at www.TheBicyclingGuitarist.net/ I received much help from this NG and the stylesheets NG when updating the...
4
by: glaros | last post by:
hi, I am trying to validate 2 ASP pages(1 posting and 1 validating) and I cannot get validation because I get in the validation site: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0...
8
by: Anthony Williams | last post by:
Morning all, I'm having a wee problem with a project I'm working on at the moment. I'm leading my company into producing a website, based upon Web Standards, which will be created using XHTML...
6
by: Rolf Welskes | last post by:
Hello, if I have for example: <table style="width: 100%; height: 100%;" border="1"> <tr> <td style="width: 100px">k </td> <td style="width: 100px">k </td> </tr>
2
by: Radu | last post by:
Hi. I have been working at home on a web project (VSNET 2005 SP1). Now I have brought the project at work, and I suddenly have plenty of warnings like: Validation (XHTML 1.0 Transitional) -...
7
by: DesertPrince | last post by:
I like making pages with the XHTML strict doctype. So I was building a page where I have two images that need to be next to each other, one on top of the other, so the colors match up. In Firefox...
21
by: =?iso-8859-2?Q?K=F8i=B9tof_=AEelechovski?= | last post by:
It is common knowledge that XHTML is better HTML and you can serve XHTML content as HTML. However, the second statement is incorrect, for various reasons; it is enough to say that the HTML...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.