Hello all
I am trying to write a reliable web-crawler. I tried to write my own
using recursion and found I quickly hit the "too many sockets" open
problem. So I looked for a threaded version that I could easily extend.
The simplest/most reliable I found was called Spider.py (see attached).
At this stage I want a spider that I can point at a site, let it do
it's thing, and reliable get a callback of sorts... including the html
(for me to parse), the url of the page in question (so I can log it)
and the urls-found-on-that-page (so I can strip out any ones I really
don't want and add them to the "seen-list".
Now, this is my question.
The code above ALMOST works fine. The crawler crawls, I get the data I
need BUT... every now and again the code just pauses, I hit control-C
and it reports an error as if it has hit an exception and then carries
on!!! I like the fact that my spider_usage.py file has the minimum
amount of spider stuff in it... really just a main() and handle()
handler.
How does this happen... is a thread being killed and then a new one is
made or what? I suspect it may have something to do with sockets timing
out, but I have no idea...
By the way on small sites (100s of pages) it never gets to the stall,
it's on larger sites such as Amazon that it "fails"
This is my other question
It would be great to know, when the code is stalled, if it is doing
anything... is there any way to even print a full stop to screen?
This is my last question
Given python's suitability for this sort of thing (isn't google written
in it?) I can't believe that that there isn't a kick ass crawler
already out there...
regards
tom
http://www.theotherblog.com/Articles...rawler-spider/
