Web Spider and Scraper

[UJ]

Does anybody have a code to 'scrape' the text off of a web page? I've got to
write a program that will go through some web pages and get the text to put
in a database.

TIA - Jeff.

[Stefan Z Camilleri]

On Sun, 21 Jan 2007 19:14:45 +0100, UJ <fr**@nowhere.comwrote:

Does anybody have a code to 'scrape' the text off of a web page? I've
got to
write a program that will go through some web pages and get the text to
put
in a database.

TIA - Jeff.

No, but it shouldn't be too complicated.

Simple open an imput stream and fetch the url, then just use RegEx to
extract the text you need.

--
- Stefan Z Camilleri
- www.szc001.com

[deciacco]

Also, if you have the cash take a look at webzinc.com.

"UJ" <fr**@nowhere.comwrote in message
news:On**************@TK2MSFTNGP04.phx.gbl...

Does anybody have a code to 'scrape' the text off of a web page? I've got
to write a program that will go through some web pages and get the text to
put in a database.

TIA - Jeff.

[=?Utf-8?B?UGV0ZXIgQnJvbWJlcmcgW0MjIE1WUF0=?=]

Before you start spending money on solutions, I'd suggest you download and
try out Simon Mourier's HtmlAgilityPack. Among other nice things, this will
turn any downloaded web page into an XPath - compliant DOM "HtmlDocument"
just like the .NET XmlDocument object, so that it is easy to XPath out
everything you need from the page.
Peter

--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"UJ" wrote:

Does anybody have a code to 'scrape' the text off of a web page? I've got to
write a program that will go through some web pages and get the text to put
in a database.

TIA - Jeff.

[jubil]

For simple websites, regex is a good solution. But for complicated
websites, manual web extraction is a nightmare to maintain! Especially
if you need to click through several websites before you arrive at the
website that contains the data that you want to extract. There are
several tools for this: Kapow is good but very expensive, iMacros can
do the same, is easier to use and costs much less. Both, kapow and
imacros offer good support. We use imacros and are very happy with it.

Jon

[Cor Ligthert [MVP]]

UJ,

MSHTML is made for this, it can give you from every detailed part of a page
the content or whatever. It follows completely the HTML DOM model.

Cor

[alex_f_il]

You can also try SWExplorerAutomation SWEA (http://webiussoft.com).
SWEA supports frames, popups, html and windows dialogs (alerts), AJAX.
With SWEA you can visually record script and generate C#/VB.NET code.
UJ wrote:

Does anybody have a code to 'scrape' the text off of a web page? I've got to
write a program that will go through some web pages and get the text to put
in a database.

TIA - Jeff.