469,356 Members | 2,055 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,356 developers. It's quick & easy.

Robots.txt issue

My robots.txt is as simple as below:

User-agent: *

Disallow: /auth

Is there something wrong with it? Yahoo Slurp still indexes content in
the auth folder...

Advice appreciated.

Jul 23 '05 #1
3 1365
"Spondishy" <sp*******@tiscali.co.uk> wrote:
My robots.txt is as simple as below:

User-agent: *

Disallow: /auth

Is there something wrong with it? Yahoo Slurp still indexes content in
the auth folder...


Try:
Disallow: /auth/*

Note that not all bots respect robots.txt rules.

--
Spartanicus
Jul 23 '05 #2
"Spondishy" ,comp.infosystems.www.authoring.html:
My robots.txt is as simple as below:

User-agent: *

Disallow: /auth


There should not be any blank line between User-Agent: and Disallow:
lines. cf <URL: http://www.robotstxt.org/wc/norobots.html >.
Jul 23 '05 #3
Spartanicus wrote:
Is there something wrong with it? Yahoo Slurp still indexes content in
the auth folder...

Try:
Disallow: /auth/*


The Robots Exclusion Protocol does not specify wildcards, some robots
support this as a proprietary extension.

The problem in the original post was probably the blank line between the
User-agent and the Disallow lines.

--
Klaus Johannes Rusch
Kl********@atmedia.net
http://www.atmedia.net/KlausRusch/
Jul 23 '05 #4

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

8 posts views Thread by Craig Cockburn | last post: by
56 posts views Thread by Anonymous, quoting Philip Ronan | last post: by
4 posts views Thread by Misfit | last post: by
4 posts views Thread by Les Caudle | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by suresh191 | last post: by
1 post views Thread by Marylou17 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.