Hello
I'm a newcomer to the world of Python trying to write a web spider. I
downloaded the skeleton from
http://starship.python.net/crew/aahz...dPoolSpider.py
Some of the source shown below.
A couple of questions:
1) Why use the
if __name__ == '__main__':
construct?
2) In Retrievepool.__ init__ the Retriever.__ini t__ is called with
self.inputQueue and self.outputQueu e as arguments. Does this mean that
each Retriever thread has a reference to Retrievepool.in putQueue and
Retrievepool.ou tputQueue (ie. there is only one input and output queue and
the threads all share, pushing and popping whenever they want (which is
safe due to the synchronized nature of Queue)?
3) How many threads will be running? Spider.run initializes the
Retrievepool and this will consist of MAX_THREADS threads, so once the
crawler is running there will be the main thread (caught in the while loop
in Spider.run) and MAX_THREADS Retriever threads running, right?
Hmm... I think that's about it for now.
---------------------------------------------------------------------
MAX_THREADS = 3
....
class Retriever(threa ding.Thread):
def __init__(self, inputQueue, outputQueue):
threading.Threa d.__init__(self )
self.inputQueue = inputQueue
self.outputQueu e = outputQueue
def run(self):
while 1:
self.URL = self.inputQueue .get()
self.getPage()
self.outputQueu e.put(self.getL inks())
...
class RetrievePool:
def __init__(self, numThreads):
self.retrievePo ol = []
self.inputQueue = Queue.Queue()
self.outputQueu e = Queue.Queue()
for i in range(numThread s):
retriever = Retriever(self. inputQueue, self.outputQueu e)
retriever.start ()
self.retrievePo ol.append(retri ever)
...
class Spider:
def __init__(self, startURL, maxThreads):
self.URLs = []
self.queue = [startURL]
self.URLdict = {startURL: 1}
self.include = startURL
self.numPagesQu eued = 0
self.retriever = RetrievePool(ma xThreads)
def run(self):
self.startPages ()
while self.numPagesQu eued > 0:
self.queueLinks ()
self.startPages ()
self.retriever. shutdown()
self.URLs = self.URLdict.ke ys()
self.URLs.sort( )
...
if __name__ == '__main__':
startURL = sys.argv[1]
spider = Spider(startURL , MAX_THREADS)
spider.run()
print
for URL in spider.URLs:
print URL
--
Regards
/Thomas
