By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
432,569 Members | 1,358 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 432,569 IT Pros & Developers. It's quick & easy.

User vs. Crawler

P: n/a
Hi,
Is there a way to know if a session on my web server is from an actual user or an automated crawler.

please advise.
G
Jul 19 '05 #1
Share this Question
Share on Google+
2 Replies


P: n/a
Perhaps by looking at the user-agent that is making the request, although I
can't say for certain that crawling software doesn't send browser-looking
user-agent info, or that this isn't possible if the crawler software writer
wants it to. That would be pretty snakey though, IMO.

Load a page with this code. Note the http_user_agent variable,
specifically.

<%
For each x in Request.serverVariables
Response.Write x & " = " & Request.Servervariables(x)
Response.Write "<hr>"
Next
%>

Ray at work

"Gomez" <an*******@discussions.microsoft.com> wrote in message
news:68**********************************@microsof t.com...
Hi,
Is there a way to know if a session on my web server is from an actual user or an automated crawler.
please advise.
G

Jul 19 '05 #2

P: n/a
"Gomez" <an*******@discussions.microsoft.com> wrote in message
news:68**********************************@microsof t.com...
Hi, Is there a way to know if a session on my web server
is from an actual user or an automated crawler.
Ray at <%=sLocation%> [MVP] wrote on 18 feb 2004 in
microsoft.public.inetserver.asp.general: Perhaps by looking at the user-agent that is making the request,
although I can't say for certain that crawling software doesn't send
browser-looking user-agent info, or that this isn't possible if the
crawler software writer wants it to.


Yes

Google puts "googlebot in Request.ServerVariables("HTTP_USER_AGENT")

More:

1
using
r = Request.ServerVariables("HTTP_REFERER")

you can check for:

"64.68." crawler googlebot.com
"216.239." crawler googlebot.com
"209.237.237." Alexa
"209.237.238." Alexa
"62.69.162." ingrid.ilse.nl
"216.39." AltaVista
etc., etc.

2
In hindsight, if a Request.ServerVariables("HTTP_REFERER") single IP
looks at many pages in a timeframe surpassing the maximum
vingerleftmousclick and if every page is a seperate session, it is
probably eiter a crawlerbot or [if there is also a strange supect
querystring] a hacker.
--
Evertjan.
The Netherlands.
(Please change the x'es to dots in my emailaddress)
Jul 19 '05 #3

This discussion thread is closed

Replies have been disabled for this discussion.