473,412 Members | 5,714 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,412 software developers and data experts.

Crawl a site

Is it possible to crawl a site using ASP & XML HTTP?

I know you can hit one link, but how can you go through each link in a
page and validate that it returns 200?
Jul 19 '05 #1
1 1576
Dave wrote:
Is it possible to crawl a site using ASP & XML HTTP?

I know you can hit one link, but how can you go through each link in a
page and validate that it returns 200?


Using PerlScript and the Robot module.

http://search.cpan.org/~awrigley/WWW...b/WWW/Robot.pm

Eric
Jul 19 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: Steve Mauldin | last post by:
Can anyone recommend a good tool to crawl asp pages and make note of which Pages, Images, Includes, Etc. are used on a web site and which pages are not used? I have inherited a website that is...
1
by: Danny | last post by:
I am trying to crawl my site to get a list of links. I am using the regular expressions to get the href tags from the pages and reading the links using xmlhttp module. is there an efficient way...
0
by: Marlo Brandon | last post by:
Hiya , I want to create /find an ASP robot that will look at all the pages on a site (my site that is), and return every word, along with the URL, so I can put them all in my database. OK maybe...
1
by: Danny | last post by:
I am trying to crawl my site to get a list of all the links. I am using the regular expressions to get the href tags from the pages and reading the source pages using xmlhttp module. Is there...
2
by: mahesr | last post by:
Frds, how to crawl a specific word in whole website using php.need which page it has Example: want to crawl a 144 items found,on somepage.html. Pls if u have code pls mail email address...
2
by: John | last post by:
Hello everybody, I'm working on a Web site that reads a database file and diplays the contents in a datagrid my question is how can google index my page since in order to do that it needs to...
12
by: Snoopy33 | last post by:
I have a FE / BE setup on my database. One of the main reasons that I did this was so that I could edit the front end and upload it without causing problems with day to day activities. I have...
7
by: =?Utf-8?B?SklNLkgu?= | last post by:
How to get search engines crawl data I have a web application that uses user controls and pulls data directly from database and shows it to users in the internet. So there is not html that has the...
0
by: vinothPHP | last post by:
Hi Everyone, I want to make a crawler in php that will crawl site search results by images, links. when i am running the sample script in my localhost i am getting this error,...
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.