By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,908 Members | 1,892 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,908 IT Pros & Developers. It's quick & easy.

webspider, regexp not working, why?

P: n/a
url = re.compile(r"^((ht|f)tp(s?)\:\/\/|~/|/)?([\w]+:\w+@)?([a-zA-Z]
{1}
([\w\-]+\.)+
([\w]{2,5}))(:[\d]{1,5})?((/?\w+/)+|/?)(\w+\.[\w]{3,4})?((\?\w+=\w+)?
(&
\w+=\w+)*)?")

why isnt this url catching something like:

<link rel="alternate" type="application/rss+xml" title="Python
Screencasts"
href="http://www.showmedo.com/latestVideoFeed/rss2.0?
tag=python" />

site = urllib.urlopen("http://www.python.org")
for row in site:
obj = url.search(row)
if obj != None:
print "url: ", obj.group()

i know it works because it can catch
www.hello.com in a txt-file and i can catch emails of websites with
another regexp.

search and match yields the same results.

but when you put something like href= in front of it it doesnt work.

i see now that it has to match the beginning of the row or something,
because:
hi www.google.com
doesnt match but
www.google.com hi
matches.
i though a regexp would search a row/file and when it finds an
occurence report it, so a regexp of "lo" would match in lopez.
Jun 27 '08 #1
Share this Question
Share on Google+
2 Replies


P: n/a

-----Original Message-----
From: py********************************@python.org [mailto:python-
li*************************@python.org] On Behalf Of
no**********@yahoo.se
Sent: Friday, May 23, 2008 12:43 PM
To: py*********@python.org
Subject: webspider, regexp not working, why?

url = re.compile(r"^((ht|f)tp(s?)\:\/\/|~/|/)?([\w]+:\w+@)?([a-zA-Z]

search and match yields the same results.

but when you put something like href= in front of it it doesnt work.

a) '^' matches at the beginning of a line. So if 'href=' is at the
beginning of the line...

b) Regexes are hard enough to read as is. (http|ftp|https) is more
readable than ((ht|f)tp(s?).

c) If you're going to parse html/xml then bite the bullet and learn one
of the libraries specifically designed to parse html/xml. Many other
regex gurus have learned this lesson. Myself included. =)

*****

The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from all computers. GA621
Jun 27 '08 #2

P: n/a
On May 24, 3:26 am, "Reedick, Andrew" <jr9...@ATT.COMwrote:
c) If you're going to parse html/xml then bite the bullet and learn one
of the libraries specifically designed to parse html/xml. Many other
regex gurus have learned this lesson. Myself included. =)
Agreed. The BeautifulSoup approach is particularly nice (although not
part of stdlib):
>>import urllib
from BeautifulSoup import BeautifulSoup
html = urllib.urlopen('http://www.python.org/').read()
soup = BeautifulSoup(html)
links = [link['href'] for link in soup('link')]
links[0]
u'http://www.python.org/channews.rdf'

- alex23

Jun 27 '08 #3

This discussion thread is closed

Replies have been disabled for this discussion.