473,327 Members | 2,069 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,327 software developers and data experts.

Distinguishing Crawlers

Hello,

Does anyone happen to have a Browsercaps update that properly sets the
Crawler attribute? I am seeing that google and others are being recognized as
browsers and not crawlers.

Thank you.
Nov 19 '05 #1
5 1185
Ian Lane .enizin.net> <ian@ wrote:
Hello,

Does anyone happen to have a Browsercaps update that properly sets the
Crawler attribute? I am seeing that google and others are being recognized as
browsers and not crawlers.

Thank you.


recognized by what?
The browscap is an outdated tech.... may want to look elsewhere

--
Curt Christianson
site: http://www.darkfalz.com
blog: http://blog.darkfalz.com
Nov 19 '05 #2
The machine.config browsercaps section sets the HttpBrowserCapabilities
object properties correct?

I need to know when a search engine is crawling my site so I know not to
prompt it to select a country for pricing. If there is something other than
using the Browser.Crawler property I would love to hear about it.

Thanks!

"Curt_C [MVP]" wrote:
Ian Lane .enizin.net> <ian@ wrote:
Hello,

Does anyone happen to have a Browsercaps update that properly sets the
Crawler attribute? I am seeing that google and others are being recognized as
browsers and not crawlers.

Thank you.


recognized by what?
The browscap is an outdated tech.... may want to look elsewhere

--
Curt Christianson
site: http://www.darkfalz.com
blog: http://blog.darkfalz.com

Nov 19 '05 #3
You might want to try adding this meta tag
to the pages you don't want crawlers/robots to index :

<META NAME="ROBOTS" CONTENT="NOINDEX">


Juan T. Llibre
ASP.NET MVP
http://asp.net.do/foros/
Foros de ASP.NET en Español
Ven, y hablemos de ASP.NET...
======================

"Ian Lane .enizin.net>" <ian@<nospam> wrote in message
news:FC**********************************@microsof t.com...
The machine.config browsercaps section sets the HttpBrowserCapabilities
object properties correct?

I need to know when a search engine is crawling my site so I know not to
prompt it to select a country for pricing. If there is something other than
using the Browser.Crawler property I would love to hear about it.

Thanks!

"Curt_C [MVP]" wrote:
Ian Lane .enizin.net> <ian@ wrote:
> Hello,
>
> Does anyone happen to have a Browsercaps update that properly sets the
> Crawler attribute? I am seeing that google and others are being recognized as
> browsers and not crawlers.
>
> Thank you.


recognized by what?
The browscap is an outdated tech.... may want to look elsewhere

--
Curt Christianson
site: http://www.darkfalz.com
blog: http://blog.darkfalz.com

Nov 19 '05 #4
> Hello,

Does anyone happen to have a Browsercaps update that properly sets the
Crawler attribute? I am seeing that google and others are being
recognized as browsers and not crawlers.

Thank you.


try here:

http://slingfive.com/pages/code/browserCaps/

Hans Kesting
Nov 19 '05 #5
Crawlers have more legs. If they have MANY more legs, they are called
"millipedes."

--
;-),

Kevin Spencer
Microsoft MVP
..Net Developer
Ambiguity has a certain quality to it.

"Ian Lane .enizin.net>" <ian@<nospam> wrote in message
news:D4**********************************@microsof t.com...
Hello,

Does anyone happen to have a Browsercaps update that properly sets the
Crawler attribute? I am seeing that google and others are being recognized
as
browsers and not crawlers.

Thank you.

Nov 19 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Araxes Tharsis | last post by:
Hi, This must be a very old and well studied question... I created a site using JSP, that permits the viewing of articles that are fully stored in a database. The url for the articles is something...
0
by: TomislaW | last post by:
I try to trace users on my web page In global.asax.cs on Application_BeginRequest I check if user has my cookie, if not I give him new cookie (integer identity number from database). When...
0
by: Stefano | last post by:
Hi all, I'm trying to create a browser definition file (.browser) that matches crawlers user agents. I don't want modify browser files in the Config system folder. I'd like to use App_Browsers...
3
by: rooznamechi.h | last post by:
Hi, I use Url rewriting on my website and my website works normally , but I don't know why Google crawlers can not read my pages . For example look at address below :...
26
by: tjhnson | last post by:
Hi, With properties, attributes and methods seem very similar. I was wondering what techniques people use to give clues to end users as to which 'things' are methods and which are attributes. ...
2
by: disappearedng | last post by:
Hi Does anyone here have a good recommendation for an open source crawler that I could get my hands on? It doesn't have to be python based. I am interested in learning how crawling works. I think...
2
by: disappearedng | last post by:
Hi Does anyone here have a good recommendation for an open source crawler that I could get my hands on? It doesn't have to be C++ based. I am interested in learning how crawling works. I think...
4
by: =?Utf-8?B?Wm9sdA==?= | last post by:
Hi, Would someone know where I could get a list of the supported crawlers for the HttpBrowserCapabilities? Is there a way to add new ones/modify the list? I have a web site for which I want to...
1
by: amanagarwal89 | last post by:
can ne body give me an insight on java cralers and what it takes to code web crawlers???
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.