By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
431,967 Members | 2,124 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 431,967 IT Pros & Developers. It's quick & easy.

How can I tell if a visitor is a search engine bot?

P: n/a
Hi, I am writing some code to log vistors to my website.

I notice in some statistics pacakages that they are able to differentiate
between normal visitors and search bots. Does anyone know how that is done?

I thought that they might be looking at the IP address of the remote visitor
and then if the IP range is associated with Google or whatever than that
"person" is a bot...

Any ideas welcome
thanks
Mark
Nov 19 '05 #1
Share this Question
Share on Google+
4 Replies


P: n/a
You'll want to look at the Agent header coming in with the request.

Here are some reference sources:
http://www.google.com/search?hl=en&q...+agent+strings

HTH!

--
Scott
http://www.OdeToCode.com/blogs/scott/
Hi, I am writing some code to log vistors to my website.

I notice in some statistics pacakages that they are able to
differentiate between normal visitors and search bots. Does anyone
know how that is done?

I thought that they might be looking at the IP address of the remote
visitor and then if the IP range is associated with Google or whatever
than that "person" is a bot...

Any ideas welcome
thanks
Mark

Nov 19 '05 #2

P: n/a
You can check the Request.UserAgent property. Each major search bot
uses a different user agent, so you'd need to check against a list. The
biggest ones are googlebot, msnbot, and slurp (Yahoo).

Here's a partial list:
http://webmaster-resources.web-maste...ification.html

If you just want to prevent search engines from spidering all or
portions of your site, you can do this with a robots.txt file at the
root of your site:
http://www.searchengineworld.com/rob...s_tutorial.htm

Keep in mind that it is possible for users can change their user agent
(for instance via the Firefox User Agent Switcher extension), so if you
have pages of your site that you want to allow only subscribers or
search engines, you'd need to check against an IP whitelist to keep
people out.

- Jon
http://weblogs.asp.net/jgalloway

Nov 19 '05 #3

P: n/a
Hi, thanks for your help
Regards
Mark

"Mark" <ma**@Z-Zvolution.nZt> wrote in message
news:%2******************@TK2MSFTNGP11.phx.gbl...
Hi, I am writing some code to log vistors to my website.

I notice in some statistics pacakages that they are able to differentiate
between normal visitors and search bots. Does anyone know how that is done?
I thought that they might be looking at the IP address of the remote visitor and then if the IP range is associated with Google or whatever than that
"person" is a bot...

Any ideas welcome
thanks
Mark

Nov 19 '05 #4

P: n/a
note: following these bot rules is strictly up to the bot. the major engines
are polite and do, but not all.

-- bruce (sqlwork.com)

"jongalloway" <jo*********@gmail.com> wrote in message
news:11**********************@f14g2000cwb.googlegr oups.com...
| You can check the Request.UserAgent property. Each major search bot
| uses a different user agent, so you'd need to check against a list. The
| biggest ones are googlebot, msnbot, and slurp (Yahoo).
|
| Here's a partial list:
| http://webmaster-resources.web-maste...ification.html
|
| If you just want to prevent search engines from spidering all or
| portions of your site, you can do this with a robots.txt file at the
| root of your site:
| http://www.searchengineworld.com/rob...s_tutorial.htm
|
| Keep in mind that it is possible for users can change their user agent
| (for instance via the Firefox User Agent Switcher extension), so if you
| have pages of your site that you want to allow only subscribers or
| search engines, you'd need to check against an IP whitelist to keep
| people out.
|
| - Jon
| http://weblogs.asp.net/jgalloway
|
Nov 19 '05 #5

This discussion thread is closed

Replies have been disabled for this discussion.