standard for robot behaviour

[Biggie]

Hi,
is there any standard (RFC or other...) that specifies how to write rules
for robots / how robots should implement these rules?

There is a document called "Standard for Robot Exclusion" at this address:
http://www.robotstxt.org/wc/norobots.html
but it says it is unofficial. Is this "unofficial standard" widely followed?

Thanks.

[David Dorward]

Biggie wrote:

There is a document called "Standard for Robot Exclusion" at this address:
http://www.robotstxt.org/wc/norobots.html
but it says it is unofficial. Is this "unofficial standard" widely
followed?

Very. Bots which ignore it are rare (and tend to find themselves getting
unauthorised response headers for all requests). You shouldn't find it too
difficult to get your hands on a bot framework for whatever language you
are working in which already implements it. Perl's LWP is rather good, IMO.

--
David Dorward <http://blog.dorward.me.uk/> <http://dorward.me.uk/>
Home is where the ~/.bashrc is

[Big Bill]

On Thu, 17 Feb 2005 23:56:39 +0100, Biggie <pa*@de.spam.svp> wrote:

Hi,
is there any standard (RFC or other...) that specifies how to write rules
for robots / how robots should implement these rules?

There is a document called "Standard for Robot Exclusion" at this address:
http://www.robotstxt.org/wc/norobots.html
but it says it is unofficial. Is this "unofficial standard" widely followed?

Thanks.

Point is, if you want to exclude or limit the entry of robots,
realistically you can't. You can have a stab with robots.txt, but if
they want to ignore it, they will.

BB
--
www.kruse.co.uk SE*@kruse.demon.co.uk
home of SEO that's shiny!
--

[Biggie]

Thank you guys.