Robots.txt issue

[Spondishy]

My robots.txt is as simple as below:

User-agent: *

Disallow: /auth

Is there something wrong with it? Yahoo Slurp still indexes content in
the auth folder...

Advice appreciated.

[Spartanicus]

"Spondishy" <sp*******@tiscali.co.uk> wrote:

My robots.txt is as simple as below:

User-agent: *

Disallow: /auth

Is there something wrong with it? Yahoo Slurp still indexes content in
the auth folder...

Try:
Disallow: /auth/*

Note that not all bots respect robots.txt rules.

--
Spartanicus

[Pierre Senellart]

"Spondishy" ,comp.infosystems.www.authoring.html:

My robots.txt is as simple as below:

User-agent: *

Disallow: /auth

There should not be any blank line between User-Agent: and Disallow:
lines. cf <URL: http://www.robotstxt.org/wc/norobots.html >.

[Klaus Johannes Rusch]

Spartanicus wrote:

Is there something wrong with it? Yahoo Slurp still indexes content in
the auth folder...

Try:
Disallow: /auth/*

The Robots Exclusion Protocol does not specify wildcards, some robots
support this as a proprietary extension.

The problem in the original post was probably the blank line between the
User-agent and the Disallow lines.

--
Klaus Johannes Rusch
Kl********@atmedia.net
http://www.atmedia.net/KlausRusch/