I've been using HTMLParser to scrape Web sites. The trouble with this
is, there's a lot of malformed HTML out there. Real browsers have to be
written to cope gracefully with this, but HTMLParser does not. Not only
does it raise an exception, but the parser object then gets into a
confused state after that so you cannot continue using it.
The way I'm currently working around this is to do a dummy pre-parsing
run with a dummy (non-subclassed) HTMLParser object. Every time I hit
HTMLParseError, I note the line number in a set of lines to skip, then
create a new HTMLParser object and restart the scan from the beginning,
skipping all the lines I've noted so far. Only when I get to the end
without further errors do I do the proper parse with all my appropriate
actions.
8  2209 
Lawrence D'Oliveiro: I've been using HTMLParser to scrape Web sites. The trouble with this
is, there's a lot of malformed HTML out there. Real browsers have to be
written to cope gracefully with this, but HTMLParser does not.
There are two solutions to this:
1. Tidy the source before parsing it. http://www.egenix.com/files/python/mxTidy.html
2. Use something more foregiving, like BeautifulSoup. http://www.crummy.com/software/BeautifulSoup/
--
René Pijlman
Lawrence D'Oliveiro wrote: I've been using HTMLParser to scrape Web sites. The trouble with this
is, there's a lot of malformed HTML out there. Real browsers have to be
written to cope gracefully with this, but HTMLParser does not. Not only
does it raise an exception, but the parser object then gets into a
confused state after that so you cannot continue using it.
The way I'm currently working around this is to do a dummy pre-parsing
run with a dummy (non-subclassed) HTMLParser object. Every time I hit
HTMLParseError, I note the line number in a set of lines to skip, then
create a new HTMLParser object and restart the scan from the beginning,
skipping all the lines I've noted so far. Only when I get to the end
without further errors do I do the proper parse with all my appropriate
actions.
You could try HTMLTidy ( http://www.egenix.com/files/python/mxTidy.html)
as a first step to get well formed HTML.
Daniel
[Daniel] You could try HTMLTidy (http://www.egenix.com/files/python/mxTidy.html)
as a first step to get well formed HTML.
But Tidy fails on huge numbers of real-world HTML pages. Simple things like
misspelled tags make it fail: from mx.Tidy import tidy
results = tidy("<html><bo dy><pree>Hello world!</pre></body></html>")
print results[3]
line 1 column 7 - Warning: inserting missing 'title' element
line 1 column 13 - Error: <pree> is not recognized!
line 1 column 13 - Warning: discarding unexpected <pree>
line 1 column 31 - Warning: discarding unexpected </pre>
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.
Is there a Python HTML tidier which will do as good a job as a browser?
--
Richie
Rene Pijlman wrote: Lawrence D'Oliveiro: I've been using HTMLParser to scrape Web sites. The trouble with this
is, there's a lot of malformed HTML out there. Real browsers have to be
written to cope gracefully with this, but HTMLParser does not.
There are two solutions to this:
1. Tidy the source before parsing it.
http://www.egenix.com/files/python/mxTidy.html
2. Use something more foregiving, like BeautifulSoup.
http://www.crummy.com/software/BeautifulSoup/
You can also use the HTML parser from libxml2 or any of the available
wrappers for it.
Bye,
Walter Dörwald
Richie Hindle wrote:
But Tidy fails on huge numbers of real-world HTML pages. Simple things like
misspelled tags make it fail:
from mx.Tidy import tidy
results = tidy("<html><bo dy><pree>Hello world!</pre></body></html>")
[Various error messages]
Is there a Python HTML tidier which will do as good a job as a browser?
As pointed out elsewhere, libxml2 will attempt to parse HTML if asked
to:
import libxml2dom
d = libxml2dom.pars eString("<html> <body><pree>Hel lo world!</pre></body></html>", html=1)
print d.toString()
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
"http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body><pr ee>Hello world!</pree></body></html>
See how it fixes up the mismatching tags. The libxml2dom package is
available in the usual place: http://www.python.org/pypi/libxml2dom
Paul
In article <fr************ *************** *****@4ax.com>,
Rene Pijlman <re************ ********@my.add ress.is.invalid > wrote: 2. Use something more foregiving, like BeautifulSoup.
http://www.crummy.com/software/BeautifulSoup/
That sounds like what I'm after!
[Richie] But Tidy fails on huge numbers of real-world HTML pages. [...]
Is there a Python HTML tidier which will do as good a job as a browser?
[Walter] You can also use the HTML parser from libxml2
[Paul] libxml2 will attempt to parse HTML if asked to [...] See how it fixes
up the mismatching tags.
Great! Many thanks.
--
Richie Hindle ri****@entrian. com
"Lawrence D'Oliveiro" <ld*@geek-central.gen.new _zealand> writes: I've been using HTMLParser to scrape Web sites. The trouble with this
is, there's a lot of malformed HTML out there. Real browsers have to be
written to cope gracefully with this, but HTMLParser does not. Not only
does it raise an exception, but the parser object then gets into a
confused state after that so you cannot continue using it.
[...]
sgmllib.SGMLPar ser (or htmllib.HTMLPar ser) is more tolerant than
HTMLParser.HTML Parser.
BeautifulSoup derives from sgmllib.SGMLPar ser, and introduces extra
robustness, of a sort.
John This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics | |
by: Adonis |
last post by:
When parsing my html files, I use handle_pi to capture some embedded python
code, but I have noticed that in the embedded python code if it contains
html, HTMLParser will parse it as well, and thus causes an error when I exec
the code, raises an EOL error. I have a work around for this as I use
different set of characters rather that <tag> use something like (tag) then
revert it back to <tag> via another function, I was wondering if there...
|
by: Sean Cody |
last post by:
I'm trying to take a webpage that has a nxn table of entries (bus times) and
convert it to a 2D array (list of lists). Initially this was simple but I
need to be able to access whole 'columns' of data so the 2D array cannot be
sparse but in the HTML file I'm parsing there can be sparse entries which
are repsented in the table as   entities. The sparse output breaks my
ability to use entire columns and have entries correspond properly....
|
by: Kevin T. Ryan |
last post by:
Hi all -
I'm somewhat new to python (about 1 year), and I'm trying to write a program
that opens a file like object w/ urllib.urlopen, and then parse the data by
passing it to a class that subclasses HTMLParser.HTMLParser. On the web
page, however, there is javascript - and I think that is causing an error
in parsing the data. Here's the error:
Traceback (most recent call last):
File "<stdin>", line 1, in ?
|
by: florent |
last post by:
I'm trying to parse html documents from the web, using the HTMLParser
class of the HTMLParser module (python 2.3), but some web documents are
not fully valids. When the parser finds an invalid tag, he raises an
exception. Then it seems impossible to resume the parsing just after
where the exception was raised. I'd like to continue parsing an html
document even if an invalid tag was found. Is it possible to do this ?
Here is a little non...
|
by: Rene Pijlman |
last post by:
The code below results in an exception (Python 2.4.2):
HTMLParser.HTMLParseError: bad end tag: "</foo' + 'bar>", at line 4,
column 6
Should it? The end tag it chokes on is in comment, isn't it?
import HTMLParser
HTMLParser.HTMLParser().feed("""
<html><head><title></title></head><body><script>
| | |
by: Kenneth McDonald |
last post by:
I'm writing a program that will parse HTML and (mostly) convert it to
MediaWiki format. The two Python modules I'm aware of to do this are
HTMLParser and htmllib. However, I'm currently experiencing either real
or conceptual difficulty with both, and was wondering if I could get
some advice.
The problem I'm having with HTMLParser is simple; I don't seem to be
getting the actual text in the HTML document. I've implemented the
do_data...
|
by: ychaouche |
last post by:
Hi, python experts.
<console trace>
chaouche@CAY:~/TEST$ python nettoyageHTML.py
chaouche@CAY:~/TEST$
</console trace>
This is the nettoyageHTML.py python script
<code>
|
by: jonbutler88 |
last post by:
Just writing a simple website spider in python, keep getting these
errors, not sure what to do. The problem seems to be in the feed()
function of htmlparser.
Traceback (most recent call last):
File "spider.py", line 38, in <module>
s.crawl(site)
File "spider.py", line 30, in crawl
self.parse(url)
File "spider.py", line 21, in parse
|
by: globalrev |
last post by:
tried all kinds of combos to get this to work.
http://docs.python.org/lib/module-HTMLParser.html
from HTMLParser import HTMLParser
class MyHTMLParser(HTMLParser):
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
| | |
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| | |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |
