473,377 Members | 1,119 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,377 software developers and data experts.

HTMLParser question

Hi,
I have some HTML that looks essentially consists of a series of <div>'s
and each <div> having one of two classes (tnt-question or tnt-answer).
I'm using HTMLParser to handle the tags as:

class MyHTMLParser(HTMLParser.HTMLParser):

def handle_starttag(self, tag, attrs):
if len(attrs) == 1:
cls,whichcls = attrs[0]
if whichcls == 'tnt-question':
print self.get_starttag_text(), self.getpos()
def handle_endtag(self, tag):
pass
def handle_data(self, data):
print data

if __name__ == '__main__':

htmldata = string.join(open('tt.html','r').readlines())
parser = MyHTMLParser()
parser.feed( htmldata )

However what I would like is that when the parser reaches some HTML like
this:

<div class="tnt-question">
How do I add a user to a MySQL system?
</div>

I should get back the data between the open and close tags. However the
above code prints the text contained between all tags, not just the <div>
tags with the class='tnt-question'.

Is there a way to call handle_data() when a specific tag is being handled?
Placing a call to handle_data() in handle_starttag seems to be the way -
but I';m not sure how to actually do it - what data should I pass to the
call?

Any pointers would be appreciated
Thanks,
Rajarshi
Jul 18 '05 #1
1 2733
Rajarshi Guha wrote:
Hi,
I have some HTML that looks essentially consists of a series of <div>'s
and each <div> having one of two classes (tnt-question or tnt-answer).
I'm using HTMLParser to handle the tags as:

class MyHTMLParser(HTMLParser.HTMLParser):

def handle_starttag(self, tag, attrs):
if len(attrs) == 1:
cls,whichcls = attrs[0]
if whichcls == 'tnt-question':
print self.get_starttag_text(), self.getpos()
def handle_endtag(self, tag):
pass
def handle_data(self, data):
print data

if __name__ == '__main__':

htmldata = string.join(open('tt.html','r').readlines())
parser = MyHTMLParser()
parser.feed( htmldata )

However what I would like is that when the parser reaches some HTML like
this:

<div class="tnt-question">
How do I add a user to a MySQL system?
</div>

I should get back the data between the open and close tags. However the
above code prints the text contained between all tags, not just the <div>
tags with the class='tnt-question'.

Is there a way to call handle_data() when a specific tag is being handled?
Placing a call to handle_data() in handle_starttag seems to be the way -
but I';m not sure how to actually do it - what data should I pass to the
call?

Set a flag, when you the parser calls handle_starttag() and the tag
matches your criteria, unset it, when the corresponding endtag is found
(you'll probably have to count the nesting depth, so for
<div class="printme">Yo <div>man</div>!</div>
the flag is unset on the second </div>). Then in handle_data() only
print it, when the flag is set.
Jul 18 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Adonis | last post by:
When parsing my html files, I use handle_pi to capture some embedded python code, but I have noticed that in the embedded python code if it contains html, HTMLParser will parse it as well, and thus...
11
by: Sean Cody | last post by:
I'm trying to take a webpage that has a nxn table of entries (bus times) and convert it to a 2D array (list of lists). Initially this was simple but I need to be able to access whole 'columns' of...
4
by: Kevin T. Ryan | last post by:
Hi all - I'm somewhat new to python (about 1 year), and I'm trying to write a program that opens a file like object w/ urllib.urlopen, and then parse the data by passing it to a class that...
9
by: florent | last post by:
I'm trying to parse html documents from the web, using the HTMLParser class of the HTMLParser module (python 2.3), but some web documents are not fully valids. When the parser finds an invalid tag,...
8
by: Lawrence D'Oliveiro | last post by:
I've been using HTMLParser to scrape Web sites. The trouble with this is, there's a lot of malformed HTML out there. Real browsers have to be written to cope gracefully with this, but HTMLParser...
6
by: Rene Pijlman | last post by:
The code below results in an exception (Python 2.4.2): HTMLParser.HTMLParseError: bad end tag: "</foo' + 'bar>", at line 4, column 6 Should it? The end tag it chokes on is in comment, isn't...
1
by: Kenneth McDonald | last post by:
I'm writing a program that will parse HTML and (mostly) convert it to MediaWiki format. The two Python modules I'm aware of to do this are HTMLParser and htmllib. However, I'm currently...
8
by: jonbutler88 | last post by:
Just writing a simple website spider in python, keep getting these errors, not sure what to do. The problem seems to be in the feed() function of htmlparser. Traceback (most recent call last):...
3
by: globalrev | last post by:
tried all kinds of combos to get this to work. http://docs.python.org/lib/module-HTMLParser.html from HTMLParser import HTMLParser class MyHTMLParser(HTMLParser):
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.