Web Crawling Spidering Question

Rusty Hill

My boss just gave me a project to create a spider that crawls a series of
web pages and harvests certain pieces of data they contain and store that
data in a database. Of course he needs all this done sooner than may be
realistic.

Can anyone point me to some samples, links or articles that might help me
significantly jump start my efforts on this? Thanks.

Jun 1 '07 #1

Subscribe Post Reply

1320

Nicholas Paldino [.NET/C# MVP]

Rusty,

Take a look at the HttpWebRequest and HttpWebResponse classes. These
will help you get the contents of the page.

As for placing the data in a database, if you are using Sql Server, take
a look at the System.Data.SqlClient namespace, as they will help you with
getting the data into the database.
--
- Nicholas Paldino [.NET/C# MVP]
- mv*@spam.guard.caspershouse.com

"Rusty Hill" <Ru********@hotmail.comwrote in message
news:Oh**************@TK2MSFTNGP03.phx.gbl...

My boss just gave me a project to create a spider that crawls a series of
web pages and harvests certain pieces of data they contain and store that
data in a database. Of course he needs all this done sooner than may be
realistic.

Can anyone point me to some samples, links or articles that might help me
significantly jump start my efforts on this? Thanks.

Jun 1 '07 #2

David

Have a look for searcharoo on codeproject.com.

--
Best regards,
Dave Colliver.
http://www.AshfieldFOCUS.com
~~
http://www.FOCUSPortals.com - Local franchises available
"Rusty Hill" <Ru********@hotmail.comwrote in message
news:Oh**************@TK2MSFTNGP03.phx.gbl...

My boss just gave me a project to create a spider that crawls a series of
web pages and harvests certain pieces of data they contain and store that
data in a database. Of course he needs all this done sooner than may be
realistic.

Can anyone point me to some samples, links or articles that might help me
significantly jump start my efforts on this? Thanks.

Jun 1 '07 #3

Hakan Fatih YILDIRIM

u will need httpwebrequest and hhtpwebresponse class in system.net
namespace and also an html parser to parse what you want.i have an
project about this.

Jun 3 '07 #4

Similar topics

Spidering the web to find RDF

by: Mark Watson | last post by:

Last year, I did an experiment of allowing a very polite web spider run for a few days trying to find RDF markup embedded in web pages. I found close to zero RDF - not encouraging! I a recent...

.NET Framework

Simple Java/XML question

by: Stevey | last post by:

I have the following XML file... <?xml version="1.0"?> <animals> <animal> <name>Tiger</name> <questions> <question index="0">true</question> <question index="1">true</question> </questions>

.NET Framework

C# Crawler and performance (speed of crawling)

by: Benjamin Lefevre | last post by:

I am currently developping a web crawler, mainly crawling mobile page (wml, mobile xhtml) but not only (also html/xml/...), and I ask myself which speed I can reach. This crawler is developped in...

.NET Framework

Filling question ID automatically by using query or VBA

by: Ekqvist Marko | last post by:

Hi, I have one Access database table including questions and answers. Now I need to give answer id automatically to questionID column. But I don't know how it is best (fastest) to do? table...

Microsoft Access / VBA

web crawling.

by: S Borg | last post by:

Hello, I have been writing very simple Python programs that parse HTML and such, mainly just to get a better feel for the language. Here is my question: If I parsed an HTML page into all of the...

Python

Web Crawling/Threading and Things That Go Bump in the Night

by: Remarkable | last post by:

Hello all I am trying to write a reliable web-crawler. I tried to write my own using recursion and found I quickly hit the "too many sockets" open problem. So I looked for a threaded version...

Python

spidering script

by: David Waizer | last post by:

Hello.. I'm looking for a script (perl, python, sh...)or program (such as wget) that will help me get a list of ALL the links on a website. For example ./magicscript.pl www.yahoo.com and...

Python

Javascript Collection, Obfuscation, Crawling?

by: Steve H. | last post by:

Hello all, I am a visiting researcher at a laboratory this summer and my current task is investigating javascript obfuscation techniques. I am trying to get a relatively large sample of website...

Javascript

Former Perl users ? : Spidering/Scraping in PHP

by: George Orwell | last post by:

Would I be missing much if I stopped trying to learn Perl well enough to use for spidering, screen scraping etc. and converted over to PHP ? I am looking to do all, or at least most of the hacks...

PHP

How to prevent crawling the whole website url within javascript/Dynamically load js

by: teressa | last post by:

Hi Everyone, I was given a task to fix our printer friendly pages: Best practice recommendation was to dynamically load a JavaScript page. I have an asp page which is a printer-friendly page...

Javascript

Easy Steps to Fix "Canon Printer Won't Connect to WiFi Network"

by: taylorcarr | last post by:

A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...

General

How to turn on java script in a villaon keypad mobile phone

by: Charles Arthur | last post by:

How do i turn on java script on a villaon, callus and itel keypad mobile phone

Java

Batch import of multiple excel files into the database

by: ryjfgjl | last post by:

If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...

Data Management

Merging data from multiple Excel files

by: ryjfgjl | last post by:

In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...

Data Management

Navigating the Data Structures and Algorithms (DSA)

by: BarryA | last post by:

What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...

Algorithms / Advanced Math

Is that possible of reading the .csv file in column wise and the column have different lengths ?

by: Sonnysonu | last post by:

This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...

C / C++

What is ONU?

by: marktang | last post by:

ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...

General

Changing the language in Windows 10

by: Hystou | last post by:

Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...

Windows Server

Maximizing Business Potential: The Nexus of Website Design and Digital Marketing

by: jinu1996 | last post by:

In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...

Online Marketing