Parsing Html code in Python

[Patrick C]

Hello world, i'm a big-time rookie using 2.5 and windows...

I have a block of HTML to parse that has 2 damn near identical lines. The difference is the numbers in them and they are always changing. The two nearly identical lines are:



<tr><td bgcolor="EEEEEE"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000"><b>Total</b></font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">3,421</font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">1,221</font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">3,189</font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">1,775</font></td></tr>



And



<tr><td bgcolor="EEEEEE"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000"><b>Total</b></font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">1,478,819,000</font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">42,765,000</font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">2,023,516,000</font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">3,448,129,075</font></td></tr>



I believe the only difference is the numbers in them, which, again, are constantly changing.



I tried this :



>>> def scrape():

... for line in urllib2.urlopen('http://bigcharts.marketwatch.com/markets/'):

... if '<tr><td bgcolor="EEEEEE"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000"><b>Total</b></font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">' in line:

... return line

...

>>> data = r'>([0-9,]+)<'

>>> dataList = re.findall(data, scrape())

>>> print dataList

['3,421', '1,221', '3,189', '1,775']



In my If statement I used the code that goes all the way up to the numbers. But as you see, the code is only getting the 1st line with numbers, not the 2nd.



Any guess why that might be happening? And how to fix this?

Thanks

Pat

[bartonc]

I think that you'll want to look into the re module. Regular Expressions are the way to go for this kind of task.

[bvdet]

Hello world, i'm a big-time rookie using 2.5 and windows...

I have a block of HTML to parse that has 2 damn near identical lines. The difference is the numbers in them and they are always changing. The two nearly identical lines are:



<tr><td bgcolor="EEEEEE"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000"><b>Total</b></font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">3,421</font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">1,221</font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">3,189</font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">1,775</font></td></tr>



And



<tr><td bgcolor="EEEEEE"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000"><b>Total</b></font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">1,478,819,000</font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">42,765,000</font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">2,023,516,000</font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">3,448,129,075</font></td></tr>



I believe the only difference is the numbers in them, which, again, are constantly changing.



I tried this :



>>> def scrape():

... for line in urllib2.urlopen('http://bigcharts.marketwatch.com/markets/'):

... if '<tr><td bgcolor="EEEEEE"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000"><b>Total</b></font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">' in line:

... return line

...

>>> data = r'>([0-9,]+)<'

>>> dataList = re.findall(data, scrape())

>>> print dataList

['3,421', '1,221', '3,189', '1,775']



In my If statement I used the code that goes all the way up to the numbers. But as you see, the code is only getting the 1st line with numbers, not the 2nd.



Any guess why that might be happening? And how to fix this?

Thanks

Pat

Pat,
Your scrape() only returns one string. You need to do something like this (untested):

import re
 
def scrape():
    s = 'part of the string'
    # you have to create the file object
    return [line for line in file_object if s in line]
 
data = r'>([0-9,]+)<'
dataList = []
for item in scrape():
    dataList.append(re.findall(data, item)

[bvdet]

Patrick,

urllib.urlopen(url) creates a file-like object, suitable for iteration.

[ghostdog74]

only tested with provided data:

for line in open("file"):
    line = line.strip().split()
    for item in line:
        if "COLOR" in item and "</font>" in item \
           and not "Total" in item:            
            stindex=item.index('">')
            endindex=item.index("</font>")
            print item[stindex+2:endindex]
 

output:

3,421
1,221
3,189
1,775
1,478,819,000
42,765,000
2,023,516,000
3,448,129,075
 

[Patrick C]

Patrick,

urllib.urlopen(url) creates a file-like object, suitable for iteration.

Ok...that i think/thought I could do.

So i tried this

 import urllib2, urllib, re
 def test():
    s = '<tr><td bgcolor="EEEEEE"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000"><b>Total</b></font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">'
    def scrape():
        for line in urllib.urlopen('http://bigcharts.marketwatch.com/markets/'):
             return [line for line in scrape() if s in line]
 
data = r'>([0-9,]+)<'
datalist = []
for item in test():
    datalist.append(re.findall(data, item)
 
 

There are a few things that's funky with this i'm sure. Here are the problem i'm having...

1. after the last line it doesn't get me out of that function (maybe that's the wrong word, either way i don't get the >>> prompt)
2. you had suggested i use the line return [line for line in #file_object if s in line]
... what would the variable name for my file_object be in this case...I used test() becuase it calls the urllib.open which you said acts like a file_object


thanks
pc

[Patrick C]

Ok I tried your method and had more success than the other way suggested, but still ran into some deadends....

here's what I did:

>>> import urllib2
>>> import re
>>> 
>>> def scrape():
...     for line in urllib2.urlopen('http://bigcharts.marketwatch.com/markets/'):
...         line = line.strip().split()
...         for item in line:
...             if "COLOR" in item and "</font>" in item and not "Total" in item:
...                 stindex=item.index('">')
...                 endindex=item.index("</font>")
...                 print item[stindex+2:endindex]
...                 
>>> print scrape()
<b>Issues:</b>
<b>NYSE</b>
<b>AMEX</b>
<b>Nasdaq</b>
<b>Advancing</b>
613
309
761
529
<b>Declining</b>
2,689
792
2,278
729
<b>Unchanged</b>
110
99
139
415
3,412
1,200
3,178
1,673
26
40
28
134
15
15
33
169
<b>Volume:</b>
<b>Advancing</b>
322,581,000
12,414,000
336,729,000
370,865,638
<b>Declining</b>
1,437,309,000
31,021,000
2,031,162,000
1,712,486,962
<b>Unchanged</b>
11,209,000
2,062,000
-2,345,316,000
508,293,225
1,771,099,000
45,497,000
22,575,000
2,591,645,825
None
>>> 
 

Now that works GREAT, but, how can I call just one of the numubers, say for example 45,497,000, or whatever number would fall into that place when it gets updated.

If I do return item[stindex+2:endindex]
Then do print scrape()
I just get the top line.

Being as I want to scrape the data and put it into a file, i think i'll need to return at some point.

How do i return w getting mor than just 1 line.

thanks
pc

only tested with provided data:

for line in open("file"):
    line = line.strip().split()
    for item in line:
        if "COLOR" in item and "</font>" in item \
           and not "Total" in item:            
            stindex=item.index('">')
            endindex=item.index("</font>")
            print item[stindex+2:endindex]
 

output:

3,421
1,221
3,189
1,775
1,478,819,000
42,765,000
2,023,516,000
3,448,129,075
 

[ghostdog74]

Now that works GREAT, but, how can I call just one of the numbers, say for example 45,497,000, or whatever number would fall into that place when it gets updated.

don't understand. does it mean that you want to only get the numbers that are updated on the website, and you only want to see those updated from your previous values? then one way is to store the values somewhere, like a file, database or pickle objects, then on the next run, grab the numbers and compare to saved data. however, the code will be longer that what i have provided, also you will need to redefine how you are actually going to search for specific patterns..

[bvdet]

Patrick,

Here's a little test script similar to what you are doing:

def scrape():
    patt = re.compile(r'>([0-9,]+)<')
    s = '<tr><td bgcolor="EEEEEE"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000"><b>Total</b></font></td><td bgcolor="EEEEEE" align="right"><FONT FACE="Arial,Helvetica,sans-serif" SIZE="2" COLOR="#000000">'
    data = [line for line in urllib.urlopen('http://bigcharts.marketwatch.com/markets/') if s in line]
    dataList = []
    for item in data:
        dataList += patt.findall(item)
    return dataList
 
numList = scrape()    
print numList    
 
'''
>>> ['3,412', '1,200', '3,178', '1,673', '1,771,099,000', '45,497,000', '22,575,000', '2,591,645,825']
'''

Now that you have the data stored in a list, you can do a number of things. I don't like testing for 'in' with a string that long though.

[Patrick C]

What i'm shooting for is to make a scrape that when it runs it gets the numbers that are being constantly updated from the site. The example you gave does, but I can't seem to figure out how to just get one of the numbers.

I'd then store them in a file and build a database from it.

In the other example that BVDET gave it prints the numers out in a string, which is what I need because I only want to reference some of the numbers scraped. Now i'll add a few lines so it writes to a file.

Does that explain what I'm aiming to do better? Sorry for the confusion.
And thanks a lot for your help, it's very much appreciated.
pc

don't understand. does it mean that you want to only get the numbers that are updated on the website, and you only want to see those updated from your previous values? then one way is to store the values somewhere, like a file, database or pickle objects, then on the next run, grab the numbers and compare to saved data. however, the code will be longer that what i have provided, also you will need to redefine how you are actually going to search for specific patterns..