By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
431,990 Members | 1,741 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 431,990 IT Pros & Developers. It's quick & easy.

crawler pool

P: n/a
Hi,

I've coded a basic crawler where by you enter the URL and it will then
crawl the said URL. What I would like to do now is to take it one
step further and do the following:

1. pick up the url's I would like to crawl from a database and pass
them to the crawler. Once the crawler has crawled the website I would
then like to put a flag against it so that the url is not processed
for a certain period of time.

2. create a pool of crawler's that can that can individually be
invoked to process a given url running on separate threads.

Also what is the best way to make sure that the crawler is not
hammering a website.

Sorry about all of the questions I am a newbie. If anyone can point
me in the right direction it would be gratefully appreciated.

Steve
Nov 15 '05 #1
Share this Question
Share on Google+
1 Reply


P: n/a
Sounds like you are ready to start working with threads:
http://msdn.microsoft.com/library/de...sthreading.asp

To avoid hammering a site, be sure to look into Thread.Sleep

"Steve Ocsic" <st********@hotmail.com> wrote in message
news:d7**************************@posting.google.c om...
Hi,

I've coded a basic crawler where by you enter the URL and it will then
crawl the said URL. What I would like to do now is to take it one
step further and do the following:

1. pick up the url's I would like to crawl from a database and pass
them to the crawler. Once the crawler has crawled the website I would
then like to put a flag against it so that the url is not processed
for a certain period of time.

2. create a pool of crawler's that can that can individually be
invoked to process a given url running on separate threads.

Also what is the best way to make sure that the crawler is not
hammering a website.

Sorry about all of the questions I am a newbie. If anyone can point
me in the right direction it would be gratefully appreciated.

Steve

Nov 15 '05 #2

This discussion thread is closed

Replies have been disabled for this discussion.