473,289 Members | 1,839 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,289 software developers and data experts.

BeautifulSoup and Problem Tables

Hi

I would appreciate some help. I am trying to learn Python and want to
use BeautifulSoup to pull some data from tables. I was really psyched
earlier tonight when I discovered that I could do this

from BeautifulSoup import BeautifulSoup
bst=file(r"c:\bstest.htm").read()
soup=BeautifulSoup(bst)
rows=soup.findAll('tr')
len(rows)
a=len(rows[0].findAll('td'))
b=len(rows[1].findAll('td'))
c=len(rows[2].findAll('td'))
d=len(rows[3].findAll('td'))
e=len(rows[4].findAll('td'))
f=len(rows[5].findAll('td'))
g=len(rows[6].findAll('td'))
h=len(rows[8].findAll('td'))
i=len(rows[9].findAll('td'))
j=len(rows[10].findAll('td'))
k=rows[1].findAll('td')[1].contents[0]
So here I am chortling to myself thinking this is too easy. I know
that the data columns are in rows[0] and so I can learn some more
python to figure out how to create tuples so I can lable each data
item using the row and column headings plucked from the contents.

However, I discovered that my tables have inconsistent numbers of
rows. Even though the tables look pretty. It might be that the
column heading for the third column is "Apples" but the value for
"Apples" in the fourth row is not in the third position in the row but
the fourth.

Now I am reluctant to make any assumptions because the tables were
created inconsistently. What I mean is that in some tables if there is
no value for a row/column intersection then there is a blank line, in
other tables if there is no value for a row/column intersection then
the length of k (as above) is 0.

I have been Googling for some insight into this and I have not been
successful finding anything. I would really appreciate any suggestions
or some direction about how to better describe the problem.
Sep 21 '08 #1
2 2596
On Sat, 20 Sep 2008 20:51:52 -0700 (PDT), ac***********@gmail.com wrote:
[snip]
from BeautifulSoup import BeautifulSoup
bst=file(r"c:\bstest.htm").read()
soup=BeautifulSoup(bst)
rows=soup.findAll('tr')
len(rows)
a=len(rows[0].findAll('td'))
b=len(rows[1].findAll('td'))
c=len(rows[2].findAll('td'))
d=len(rows[3].findAll('td'))
e=len(rows[4].findAll('td'))
f=len(rows[5].findAll('td'))
g=len(rows[6].findAll('td'))
h=len(rows[8].findAll('td'))
i=len(rows[9].findAll('td'))
j=len(rows[10].findAll('td'))
k=rows[1].findAll('td')[1].contents[0]
[snip]
However, I discovered that my tables have inconsistent numbers of
rows.
[snip]
I have been Googling for some insight into this and I have not been
successful finding anything. I would really appreciate any suggestions
or some direction about how to better describe the problem.
Would it be accurate to describe the problem as wanting to
extract the contents of the cth column of the rth row of a
table in spite of various pathologies in the construction of
the table?

If so, maybe it would help to post sample HTML (trimmed to a
minimum) of the pathologies that must be handled. I gotta
confess, though, that it doesn't take many rowspans or colspans
to put this problem beyond my reach.

--
To email me, substitute nowhere->spamcop, invalid->net.
Sep 21 '08 #2
ac***********@gmail.com wrote:
Hi

I would appreciate some help. I am trying to learn Python and want to
use BeautifulSoup to pull some data from tables. I was really psyched
earlier tonight when I discovered that I could do this

from BeautifulSoup import BeautifulSoup
bst=file(r"c:\bstest.htm").read()
soup=BeautifulSoup(bst)
rows=soup.findAll('tr')
len(rows)
a=len(rows[0].findAll('td'))
b=len(rows[1].findAll('td'))
c=len(rows[2].findAll('td'))
d=len(rows[3].findAll('td'))
e=len(rows[4].findAll('td'))
f=len(rows[5].findAll('td'))
g=len(rows[6].findAll('td'))
h=len(rows[8].findAll('td'))
i=len(rows[9].findAll('td'))
j=len(rows[10].findAll('td'))
k=rows[1].findAll('td')[1].contents[0]
So here I am chortling to myself thinking this is too easy. I know
that the data columns are in rows[0] and so I can learn some more
python to figure out how to create tuples so I can lable each data
item using the row and column headings plucked from the contents.

However, I discovered that my tables have inconsistent numbers of
rows. Even though the tables look pretty. It might be that the
column heading for the third column is "Apples" but the value for
"Apples" in the fourth row is not in the third position in the row but
the fourth.

Now I am reluctant to make any assumptions because the tables were
created inconsistently. What I mean is that in some tables if there is
no value for a row/column intersection then there is a blank line, in
other tables if there is no value for a row/column intersection then
the length of k (as above) is 0.

I have been Googling for some insight into this and I have not been
successful finding anything. I would really appreciate any suggestions
or some direction about how to better describe the problem.
This may help, and it may not.

One of the most useful features of BeautifulSoup is the .prettify()
function. Use it, and you can examine your xml with a browser or
decent editor. You will be able, hopefully, to see attributes that
may explain your difficulties. The one that snuck up and bit me was
table:number-columns-repeated -- I don't know what is biting you,
but it is weird the way known cells are completely skipped with this
attribute -- you have to know about it and check for it.

If there is a way to deal with that attribute using BeautifulSoup I
was unable to find it given the documentation and examples available
to me. I took a step back and tried xml.sax -- it is straighforward
and easy to use -- you build an xml.sax.handler for your xml, and it
deals with it.

It worked for me, but, as always, YMMV

sc
Sep 22 '08 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: ted | last post by:
Hi, I'm using the BeautifulSoup module and having some trouble processing a file. It's not printing what I'm expecting. In the code below, I'm expecting cells with only "bgcolor" attributes to...
7
by: Gonzillaaa | last post by:
I'm trying to get the data on the "Central London Property Price Guide" box at the left hand side of this page http://www.findaproperty.com/regi0018.html I have managed to get the data :) but...
7
by: John Nagle | last post by:
I've been parsing existing HTML with BeautifulSoup, and occasionally hit content which has something like "Design & Advertising", that is, an "&" instead of an "&". Is there some way I can get...
5
by: John Nagle | last post by:
This, which is from a real web site, went into BeautifulSoup: <param name="movie" value="/images/offersBanners/sw04.swf?binfot=We offer fantastic rates for selected weeks or days!!&blinkt=Click...
9
by: Mizipzor | last post by:
Is there a way to "subscribe" to individual topics? im currently getting bombarded with daily digests and i wish to only receive a mail when there is activity in a topic that interests me. Can this...
11
by: John Nagle | last post by:
The syntax that browsers understand as HTML comments is much less restrictive than what BeautifulSoup understands. I keep running into sites with formally incorrect HTML comments which are parsed...
2
by: Frank Stutzman | last post by:
I've got a simple script that looks like (watch the wrap): --------------------------------------------------- import BeautifulSoup,urllib ifile =...
5
by: Larry Bates | last post by:
Info: Python version: ActivePython 2.5.1.1 Platform: Windows I wanted to install BeautifulSoup today for a small project and decided to use easy_install. I can install other packages just...
3
by: Magnus.Moraberg | last post by:
Hi, I wish to extract all the words on a set of webpages and store them in a large dictionary. I then wish to procuce a list with the most common words for the language under consideration. So,...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.