471,870 Members | 1,432 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,870 software developers and data experts.

Robots.txt issue

My robots.txt is as simple as below:

User-agent: *

Disallow: /auth

Is there something wrong with it? Yahoo Slurp still indexes content in
the auth folder...

Advice appreciated.

Jul 23 '05 #1
3 1388
"Spondishy" <sp*******@tiscali.co.uk> wrote:
My robots.txt is as simple as below:

User-agent: *

Disallow: /auth

Is there something wrong with it? Yahoo Slurp still indexes content in
the auth folder...


Try:
Disallow: /auth/*

Note that not all bots respect robots.txt rules.

--
Spartanicus
Jul 23 '05 #2
"Spondishy" ,comp.infosystems.www.authoring.html:
My robots.txt is as simple as below:

User-agent: *

Disallow: /auth


There should not be any blank line between User-Agent: and Disallow:
lines. cf <URL: http://www.robotstxt.org/wc/norobots.html >.
Jul 23 '05 #3
Spartanicus wrote:
Is there something wrong with it? Yahoo Slurp still indexes content in
the auth folder...

Try:
Disallow: /auth/*


The Robots Exclusion Protocol does not specify wildcards, some robots
support this as a proprietary extension.

The problem in the original post was probably the blank line between the
User-agent and the Disallow lines.

--
Klaus Johannes Rusch
Kl********@atmedia.net
http://www.atmedia.net/KlausRusch/
Jul 23 '05 #4

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

8 posts views Thread by Craig Cockburn | last post: by
56 posts views Thread by Anonymous, quoting Philip Ronan | last post: by
4 posts views Thread by Misfit | last post: by
4 posts views Thread by Les Caudle | last post: by
reply views Thread by YellowAndGreen | last post: by
aboka
reply views Thread by aboka | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.