Why crawler don't get to my aspx page unless the Work Process is started?

[Nicolas]

I need HELP!!!!!
The crawler (Google or other) don't index my web site unless the web site is
currently visited
If there is nobody visiting those .aspx page therefor activating the aspnet
no crawler is going throught the site

I play with the robots file the meta tag etc.
Also played with the crawler class but no success

Sub Application_BeginRequest(ByVal sender As Object, ByVal e As EventArgs)
Try
If Request.Browser.Crawler Then
Response.Redirect("index.htm")
End If
Catch ex As Exception
End Try
End Sub

The problem with the above it always goes to the same page therefor not very
efficient

I'm running W2003 with II6 and Framework 1.1 sp1

Thanks for your help please
http://www.novadata.com

Nicolas