web crawler error: connection timed out

[rhitam30111985]

hi all,,, i am testing a web crawler on a site passsed as a command line argument.. it works fine until it finds a server which is down or some other error ... here is my code:

#! /usr/bin/python
import urllib
import re
import sys
 
 
def crawl(urllist,done):
 
    curl=urllist[0].upper()
 
    f = urllib.urlopen(curl)
    rx=re.compile("href=\"(http://[a-zA-Z0-9_\./\?&%=#\-]+)[\s\"]")    
 
    src=f.read()
    src.replace('\n',' ')
 
    ma =rx.findall(src)
 
    for i in range(0,len(ma)):
        ma[i]=ma[i].upper()
 
    urllist=urllist+ma
 
    done.append(curl.upper())        
 
    print "**Done**"+curl
 
 
    for i in range(0,len(done)):        
        while urllist.count(done[i]):
            urllist.pop(urllist.index(done[i]))
 
 
    if len(urllist)>0:
        crawl(urllist,done)
 
url=sys.argv[1]
url=url.upper()
 
print "Seed="+url
 
urllist=[url]
done=[]
crawl(urllist,done)
 

after a certain amount of crawling... the program crashes givig the following error:

File "./crawler.py", line 35, in crawl
crawl(urllist,done)
File "./crawler.py", line 11, in crawl
f = urllib.urlopen(curl)
File "/usr/lib/python2.4/urllib.py", line 82, in urlopen
return opener.open(url)
File "/usr/lib/python2.4/urllib.py", line 190, in open
return getattr(self, name)(url)
File "/usr/lib/python2.4/urllib.py", line 313, in open_http
h.endheaders()
File "/usr/lib/python2.4/httplib.py", line 798, in endheaders
self._send_output()
File "/usr/lib/python2.4/httplib.py", line 679, in _send_output
self.send(msg)
File "/usr/lib/python2.4/httplib.py", line 646, in send
self.connect()
File "/usr/lib/python2.4/httplib.py", line 630, in connect
raise socket.error, msg
IOError: [Errno socket error] (110, 'Connection timed out')


is there a way around this problem?

[ghostdog74]

you can use a try:except clause around the part where you open the url for reading.

[rhitam30111985]

welll i tried the following modification using try except:

try:    
        f = urllib.urlopen(curl)
        rx=re.compile("href=\"(http://[a-zA-Z0-9_\./\?&%=#\-]+)[\s\"]")    
 
        src=f.read()
        src.replace('\n',' ')
except IOError:
        pass
 
 

it gives following error after reaching http://freenode.net (seed url being wikipedia)

File "./crawler.py", line 19, in crawl
ma =rx.findall(src)
UnboundLocalError: local variable 'rx' referenced before assignment

[ghostdog74]

welll i tried the following modification using try except:

try:    
        f = urllib.urlopen(curl)
        rx=re.compile("href=\"(http://[a-zA-Z0-9_\./\?&%=#\-]+)[\s\"]")    
 
        src=f.read()
        src.replace('\n',' ')
except IOError:
        pass
 
 

it gives following error after reaching http://freenode.net (seed url being wikipedia)

File "./crawler.py", line 19, in crawl
ma =rx.findall(src)
UnboundLocalError: local variable 'rx' referenced before assignment

check to see if you have a correct regular expression.
Also you can use

try:
....
except Exception,e:
    print e
 

so that try:except catches all errors and you can print the errors out.