473,721 Members | 2,205 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Help Parsing an HTML File

Hello Python Community,

It'd be great if someone could provide guidance or sample code for
accomplishing the following:

I have a single unicode file that has descriptions of hundreds of
objects. The file fairly resembles HTML-EXAMPLE pasted below.

I need to parse the file in such a way to extract data out of the html
and to come up with a tab separated file that would look like OUTPUT-
FILE below.

Any tips, advice and guidance is greatly appreciated.

Thanks,

Egon


=====OUTPUT-FILE=====
/please note that the first line of the file contains column headers/
------Tab Separated Output File Begin------
H1 H2 DIV Segment1 Segment2 Segment3
RoséH1-1 RoséH2-1 RoséDIV-1 RoséSegmentDIV1-1 RoséSegmentDIV2-1
RoséSegmentDIV3-1
PinkH1-2 PinkH2-2 PinkDIV2-2 PinkSegmentDIV1-2 No-Value No-Value
BlackH1-3 BlackH2-3 BlackDIV2-3 BlackSegmentDIV 1-3 No-Value No-Value
YellowH1-4 YellowH2-4 YellowDIV2-4 YellowSegmentDI V1-4
YellowSegmentDI V2-4 No-Value
------Tab Separated Output File End------

=====HTML-EXAMPLE=====
------HTML Example Begin------
<html>

<h1>RoséH1-1</h1>
<h2>RoséH2-1</h2>
<div>RoséDIV-1</div>
<div "segment1">Rosé SegmentDIV1-1</div><br>
<div "segment2">Rosé SegmentDIV2-1</div><br>
<div "segment3">Rosé SegmentDIV3-1</div><br>
<br>
<br>

<h1>PinkH1-2</h1>
<h2>PinkH2-2</h2>
<div>PinkDIV2-2</div>
<div "segment1">Pink SegmentDIV1-2</div><br>
<br>
<comment></comment>

<h1>BlackH1-3</h1>
<h2>BlackH2-3</h2>
<div>BlackDIV 2-3</div>
<div "segment1">Blac kSegmentDIV1-3</div><br>

<h1>YellowH1-4</h1>
<h2>YellowH2-4</h2>
<div>YellowDI V2-4</div>
<div "segment1">Yell owSegmentDIV1-4</div><br>
<div "segment2">Yell owSegmentDIV2-4</div><br>

</html>
------HTML Example End------
Feb 15 '08 #1
4 1538
I need to parse the file in such a way to extract data out of the html
and to come up with a tab separated file that would look like OUTPUT-
FILE below.

BeautifulSoup[1]. Your one-stop-shop for all your HTML parsing
needs.

What you do with the parsed data, is an exercise left to the
reader, but it's navigable.

-tkc

[1] http://www.crummy.com/software/BeautifulSoup/


Feb 15 '08 #2
On Feb 15, 3:28 pm, egonslo...@gmai l.com wrote:
Hello Python Community,

It'd be great if someone could provide guidance or sample code for
accomplishing the following:

I have a single unicode file that has descriptions of hundreds of
objects. The file fairly resembles HTML-EXAMPLE pasted below.

I need to parse the file in such a way to extract data out of the html
and to come up with a tab separated file that would look like OUTPUT-
FILE below.

Any tips, advice and guidance is greatly appreciated.

Thanks,

Egon

=====OUTPUT-FILE=====
/please note that the first line of the file contains column headers/
------Tab Separated Output File Begin------
H1 H2 DIV Segment1 Segment2 Segment3
RoséH1-1 RoséH2-1 RoséDIV-1 RoséSegmentDIV1-1 RoséSegmentDIV2-1
RoséSegmentDIV3-1
PinkH1-2 PinkH2-2 PinkDIV2-2 PinkSegmentDIV1-2 No-Value No-Value
BlackH1-3 BlackH2-3 BlackDIV2-3 BlackSegmentDIV 1-3 No-Value No-Value
YellowH1-4 YellowH2-4 YellowDIV2-4 YellowSegmentDI V1-4
YellowSegmentDI V2-4 No-Value
------Tab Separated Output File End------

=====HTML-EXAMPLE=====
------HTML Example Begin------
<html>

<h1>RoséH1-1</h1>
<h2>RoséH2-1</h2>
<div>RoséDIV-1</div>
<div "segment1">Rosé SegmentDIV1-1</div><br>
<div "segment2">Rosé SegmentDIV2-1</div><br>
<div "segment3">Rosé SegmentDIV3-1</div><br>
<br>
<br>

<h1>PinkH1-2</h1>
<h2>PinkH2-2</h2>
<div>PinkDIV2-2</div>
<div "segment1">Pink SegmentDIV1-2</div><br>
<br>
<comment></comment>

<h1>BlackH1-3</h1>
<h2>BlackH2-3</h2>
<div>BlackDIV 2-3</div>
<div "segment1">Blac kSegmentDIV1-3</div><br>

<h1>YellowH1-4</h1>
<h2>YellowH2-4</h2>
<div>YellowDI V2-4</div>
<div "segment1">Yell owSegmentDIV1-4</div><br>
<div "segment2">Yell owSegmentDIV2-4</div><br>

</html>
------HTML Example End------
Pyparsing, ElementTree and lxml are all good candidates as well.
BeautifulSoup takes care of malformed html though.

http://pyparsing.wikispaces.com/
http://effbot.org/zone/element-index.htm
http://codespeak.net/lxml/

Mike
Feb 15 '08 #3
eg********@gmai l.com wrote:
I have a single unicode file that has descriptions of hundreds of
objects. The file fairly resembles HTML-EXAMPLE pasted below.

I need to parse the file in such a way to extract data out of the html
and to come up with a tab separated file that would look like OUTPUT-
FILE below.

=====OUTPUT-FILE=====
/please note that the first line of the file contains column headers/
------Tab Separated Output File Begin------
H1 H2 DIV Segment1 Segment2 Segment3
RoséH1-1 RoséH2-1 RoséDIV-1 RoséSegmentDIV1-1 RoséSegmentDIV2-1
------Tab Separated Output File End------

=====HTML-EXAMPLE=====
------HTML Example Begin------
<html>

<h1>RoséH1-1</h1>
<h2>RoséH2-1</h2>
<div>RoséDIV-1</div>
<div "segment1">Rosé SegmentDIV1-1</div><br>
<div "segment2">Rosé SegmentDIV2-1</div><br>
<div "segment3">Rosé SegmentDIV3-1</div><br>
<br>
<br>

</html>
------HTML Example End------
Now, what ugly markup is that? You will never manage to get any HTML compliant
parser return the "segmentX" stuff in there. I think your best bet is really
going for pyparsing or regular expressions (and I actually recommend pyparsing
here).

Stefan
Feb 16 '08 #4
Stefan Behnel wrote:
eg********@gmai l.com wrote:
>I have a single unicode file that has descriptions of hundreds of
objects. The file fairly resembles HTML-EXAMPLE pasted below.

I need to parse the file in such a way to extract data out of the html
and to come up with a tab separated file that would look like OUTPUT-
FILE below.

=====OUTPUT-FILE=====
/please note that the first line of the file contains column headers/
------Tab Separated Output File Begin------
H1 H2 DIV Segment1 Segment2 Segment3
RoséH1-1 RoséH2-1 RoséDIV-1 RoséSegmentDIV 1-1 RoséSegmentDIV 2-1
------Tab Separated Output File End------

=====HTML-EXAMPLE=====
------HTML Example Begin------
<html>

<h1>RoséH1-1</h1>
<h2>RoséH2-1</h2>
<div>RoséDI V-1</div>
<div "segment1">Rosà ©SegmentDIV1-1</div><br>
<div "segment2">Rosà ©SegmentDIV2-1</div><br>
<div "segment3">Rosà ©SegmentDIV3-1</div><br>
<br>
<br>

</html>
------HTML Example End------

Now, what ugly markup is that? You will never manage to get any HTML
compliant parser return the "segmentX" stuff in there. I think your best
bet is really going for pyparsing or regular expressions (and I actually
recommend pyparsing here).

Stefan
In practice the following might be sufficient:

from BeautifulSoup import BeautifulSoup

def chunks(bs):
chunk = []
for tag in bs.findAll(["h1", "h2", "div"]):
if tag.name == "h1":
if chunk:
yield chunk
chunk = []
chunk.append(ta g)
if chunk:
yield chunk

def process(filenam e):
bs = BeautifulSoup(o pen(filename))
for chunk in chunks(bs):
columns = [tag.string for tag in chunk]
columns += ["No Value"] * (6 - len(columns))
print "\t".join(colum ns)

if __name__ == "__main__":
process("exampl e.html")

The biggest caveat is that only columns at the end of a row may be left out.

Peter
Feb 16 '08 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
3658
by: Willem Ligtenberg | last post by:
I decided to use SAX to parse my xml file. But the parser crashes on: File "/usr/lib/python2.3/site-packages/_xmlplus/sax/handler.py", line 38, in fatalError raise exception xml.sax._exceptions.SAXParseException: NCBI_Entrezgene.dtd:8:0: error in processing external entity reference This is caused by: <!DOCTYPE Entrezgene-Set PUBLIC "-//NCBI//NCBI Entrezgene/EN" "NCBI_Entrezgene.dtd">
16
2901
by: Terry | last post by:
Hi, This is a newbie's question. I want to preload 4 images and only when all 4 images has been loaded into browser's cache, I want to start a slideshow() function. If images are not completed loaded into cache, the slideshow doesn't look very nice. I am not sure how/when to call the slideshow() function to make sure it starts after the preload has been completed.
8
5477
by: baustin75 | last post by:
Posted: Mon Oct 03, 2005 1:41 pm Post subject: cannot mail() in ie only when debugging in php designer 2005 -------------------------------------------------------------------------------- Hello, I have a very simple problem but cannot seem to figure it out. I have a very simple php script that sends a test email to myself. When I debug it in PHP designer, it works with no problems, I get the test email. If
3
3504
by: Pir8 | last post by:
I have a complex xml file, which contains stories within a magazine. The structure of the xml file is as follows: <?xml version="1.0" encoding="ISO-8859-1" ?> <magazine> <story> <story_id>112233</story_id> <pub_name>Puleen's Publication</pub_name> <pub_code>PP</pub_code> <edition_date>20031201</edition_date>
4
4860
by: Rick Walsh | last post by:
I have an HTML table in the following format: <table> <tr><td>Header 1</td><td>Header 2</td></tr> <tr><td>1</td><td>2</td></tr> <tr><td>3</td><td>4</td></tr> <tr><td>5</td><td>6</td></tr> </table> With an XSLT styles sheet, I can use for-each to grab the values in
4
6834
by: Neil.Smith | last post by:
I can't seem to find any references to this, but here goes: In there anyway to parse an html/aspx file within an asp.net application to gather a collection of controls in the file. For instance what I'm trying to do is upload a html file onto the web server, convert it to aspx file and then parse it for input tags/controls, which in turn will become fields in a newly created database table. Clearly when the aspx file is called the...
0
5571
by: gunimpi | last post by:
http://www.vbforums.com/showthread.php?p=2745431#post2745431 ******************************************************** VB6 OR VBA & Webbrowser DOM Tiny $50 Mini Project Programmer help wanted ******************************************************** For this teeny job, please refer to: http://feeds.reddit.com/feed/8fu/?o=25
2
2102
by: hzgt9b | last post by:
I've written a simple javascript page that parses an XML file... (Actually I just modified the "Parsing an XML File" sample from http://www.w3schools.com/dom/dom_parser.asp) The page works great standalone... but when I try to make this work under frames I get "Error: Object required" when the following line executes: xmlDoc.getElementsByTagName("to"); The standalone file is named treeView.htm (attached). You should be
2
3612
by: Felipe De Bene | last post by:
I'm having problems parsing an HTML file with the following syntax : <TABLE cellspacing=0 cellpadding=0 ALIGN=CENTER BORDER=1 width='100%'> <TH BGCOLOR='#c0c0c0' Width='3%'>User ID</TH> <TH Width='10%' BGCOLOR='#c0c0c0'>Name</TH><TH width='7%' BGCOLOR='#c0c0c0'>Date</TH> and so on.... whenever I feed the parser with such file I get the error :
2
6356
by: embz | last post by:
this post concerns three pages. 1. this page: http://www.katherine-designs.com/sendemail.php i get the following errors: a lot of it seems to deal with the PHP code i inserted to the page. as my PHP skills are close to nil, i'm wary about fiddling with it myself. =\ 2. now this page: http://www.katherine-designs.com/contact.php
0
9376
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9234
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9086
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
8028
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
5994
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4501
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
3207
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2599
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2146
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.