Hello clintonG,
I've never had to use one, so I cant make a recommendation. However, a google
search for "c# web spider code" yields an article [1] at codeproject.com
as the first hit which looks promising.
[1]
http://www.codeproject.com/csharp/DavWebSpider.asp
--
Matt Berther
http://www.mattberther.com
[color=blue]
> Thanks for comments Matt. I've needed search functionality for a web
> services project and Lucene is looking good but the project also
> requires spidering remote pages. I'll be looking around of course but
> have you any referrals to spiders written in .NET?
>
> <%= Clinton
>
> "Matt Berther" <mberther@hotmail.com> wrote in message
> news:221a09c7959e078c714d937694c49@news.microsoft. com...
>[color=green]
>> Hello clintonG,
>>
>> I've used Lucene.NET to spider a stack of xml for my company's
>> document library creating an index with relevant searchable fields.
>> The only difference between what I did with Lucene and what would
>> need to be done here is to create the code to spider the site. Surely
>> theres an abundance of this type of code on the web though. Once
>> you've done this, its just a matter of populating the index.
>>
>> One really nice thing about Lucene.NET is that the indexes generated
>> are completely compatible with the Jakarta Lucene project (for Java).
>> Also, there is a very nice API used to create/populate/search the
>> indexes.
>>
>> Let me know if there are other questions that you have.
>>
>> --
>> Matt Berther
>>
http://www.mattberther.com[color=darkred]
>>> Matt, I'm assuming your reference is based on having actually used
>>> Lucene.NET?
>>> Would you talk to me --- us --- more about it? Have you used it to
>>> spider
>>> sites?
>>> <%= Clinton Gallagher
>>> "Matt Berther" <mberther@hotmail.com> wrote in message
>>> news:221a09c79583e38c7149d089631ac@news.microsoft. com...
>>>
>>>> Hello AGB,
>>>>
>>>> Lucene.NET [1] is a great search engine and can be customized to do
>>>> almost anything you need. You will need to populate its index
>>>> though.
>>>>
>>>> [1]
http://searchblackbox.com/lucene/index.html?ldn2
>>>>
>>>> --
>>>> Matt Berther
>>>>
http://www.mattberther.com
>>>>> Hi all,
>>>>>
>>>>> I know this is a little off topic but I have been tasked with
>>>>> implementing a search engine in my website. Does anyone have any
>>>>> suggestions...I have looked at Google and Verity.
>>>>>
>>>>> Thanks
>>>>>[/color][/color][/color]
