By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
446,392 Members | 1,561 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 446,392 IT Pros & Developers. It's quick & easy.

Need to crawl website

P: n/a
I am trying to crawl my site to get a list of all the links. I am using the
regular
expressions to get the href tags from the pages and reading the source pages
using
xmlhttp module.

Is there an efficient way to loop through the links?
I am looping through the links and avoiding the duplicate links, but it is
taking over 2 hours to crawl my site!!
What am i doing wrong? What is making it sooooo slow.

Thanks again
Nov 13 '05 #1
Share this Question
Share on Google+
1 Reply


P: n/a
there is a great program out there called Xenu.exe
http://home.snafu.de/tilman/xenulink.html

I have tried it and really liked it. One of the best programs out there.

Does that help???
---
Please immediately let us know (by phone or return email) if (a) this email
contains a virus
(b) you are not the intended recipient
(c) you consider this email to be spam.
We have done our utmost to make sure that
none of the above are applicable. THANK YOU
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.691 / Virus Database: 452 - Release Date: 26/05/2004
---
Please immediately let us know (by phone or return email) if (a) this email
contains a virus
(b) you are not the intended recipient
(c) you consider this email to be spam.
We have done our utmost to make sure that
none of the above are applicable. THANK YOU
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.691 / Virus Database: 452 - Release Date: 26/05/2004
Nov 13 '05 #2

This discussion thread is closed

Replies have been disabled for this discussion.