Example Script to parse web page links and extract data?

[livin]

I'm hoping someone knows of an example script I can see to help me build
mine.

I'm looking for an easy way to automate the below web site browsing and pull
the data I'm searching for.
Here's steps it needs to accomplish...

1) login to the site (windows dialog when hitting web page) *optional*

2) Choose menu link from ASP page (script shows/hides menu items depending
on mouseover) *optional*

3) Basic Search Form and enter zip code or city to pull all the data.

4) After search, table shows many links (hundreds sometimes) to the actual
data I need.
Links are this format... <a href="javascrip t:GetAgent('AA0 59')">

5) Each link opens new window with table providing required data.
The URLs that each href opens is this...
http://armls.marketlinx.com/Roster/S...sp?PubID=AA059 where the
PubID is record I need.

Table format looks like this:

<tr>

<td bgcolor="#C0C0C 0" align="center">

<a href="javascrip t:GetAgent('MA1 42')">

<font face="Arial" size="2">6</font></a></td>

<td><font face="Arial" size="2">

<a href="javascrip t:GetAgent('MA1 42')">Alaze</a><br></font></td>

<td><font face="Arial" size="2">Mark <br></font></td>
<td><font face="Arial" size="2">MA142</font><br>

</td>

<td><font face="Arial" size="2">

<a href="javascrip t:GetBroker('CO LD56')">Banker Success
Realty</a><br></font></td>

<td>COLD56</td>

<td><font face="Arial" size="2"><scrip t LANGUAGE="javas cript">

<!--

writePhoneNumbe r('480-999-9999');

//--></script></td>

</tr>

[Steven Bethard]

livin wrote:

I'm looking for an easy way to automate the below web site browsing and pull
the data I'm searching for.
This is a task that BeautifulSoup[1] is usually good for.
4) After search, table shows many links (hundreds sometimes) to the actual
data I need.
Links are this format... <a href="javascrip t:GetAgent('AA0 59')">

5) Each link opens new window with table providing required data.
The URLs that each href opens is this...
http://armls.marketlinx.com/Roster/S...sp?PubID=AA059 where the
PubID is record I need.

I'm not entirely sure I got your problem description right, but I think
points 4 and 5 would look something like:

base_url = 'http://armls.marketlin x.com/.../Member.asp?PubI D=AA059'
html = urllib.urlopen( base_url).read( )
soup = BeautifulSoup.B eautifulSoup(ht ml)

link_matcher = re.compile(r'ja vascript:GetAge nt('[^']*')
for link_elem in soup('a', {'href': link_matcher}):
...

HTH,

STeVe

[1] http://www.crummy.com/software/BeautifulSoup/