By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
431,871 Members | 2,418 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 431,871 IT Pros & Developers. It's quick & easy.

.Net Web Spiders

P: n/a
et al,

Does anyone have any good links as to where to start in writing a
program like this?

I want to make something custom so that I can pull certain websites,
extract certain data from them, store in a SQL server to search
against.

I'm sure there's lots out there, I just can't seem to find it.

Albeit there might be programs out there to do this, but none that I
have seen to a decent job. And besides, I want to get some expirience
in VB.Net programming.

The first version can be just a simple spider that scans 1 website (I
have a test website w/ 6 pages on it). I figure this is small enough
for a start.
Any links anyone can provide would be greatly appreciated.

thx.

Nov 21 '05 #1
Share this Question
Share on Google+
2 Replies


P: n/a
Hi,

You are probably not the first one, you can go at least two directions

Use the axwebbrowser or use let say HttpWebRequest.

To search a document the best way is in my opinion MSHTML
The axwebbrowser is easy, mshtml is really very hard stuff to go for a
newbie.

HttpWebRequest
http://msdn.microsoft.com/library/de...classtopic.asp

webbrowser
http://support.microsoft.com/?kbid=311303

some faqs
http://support.microsoft.com/default...b;EN-US;311284

mshtml
http://msdn.microsoft.com/library/de...ng/hosting.asp

I hope this helps a little bit?

Cor
<no**************@hotmail.com>
et al,

Does anyone have any good links as to where to start in writing a
program like this?

I want to make something custom so that I can pull certain websites,
extract certain data from them, store in a SQL server to search
against.

I'm sure there's lots out there, I just can't seem to find it.

Albeit there might be programs out there to do this, but none that I
have seen to a decent job. And besides, I want to get some expirience
in VB.Net programming.

The first version can be just a simple spider that scans 1 website (I
have a test website w/ 6 pages on it). I figure this is small enough
for a start.
Any links anyone can provide would be greatly appreciated.

thx.

Nov 21 '05 #2

P: n/a
Thanks. I'll do some reading and give one, if not all of them, a shot.
thx.

Nov 21 '05 #3

This discussion thread is closed

Replies have been disabled for this discussion.