473,321 Members | 1,667 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,321 software developers and data experts.

Robots.txt issue

My robots.txt is as simple as below:

User-agent: *

Disallow: /auth

Is there something wrong with it? Yahoo Slurp still indexes content in
the auth folder...

Advice appreciated.

Jul 23 '05 #1
3 1416
"Spondishy" <sp*******@tiscali.co.uk> wrote:
My robots.txt is as simple as below:

User-agent: *

Disallow: /auth

Is there something wrong with it? Yahoo Slurp still indexes content in
the auth folder...


Try:
Disallow: /auth/*

Note that not all bots respect robots.txt rules.

--
Spartanicus
Jul 23 '05 #2
"Spondishy" ,comp.infosystems.www.authoring.html:
My robots.txt is as simple as below:

User-agent: *

Disallow: /auth


There should not be any blank line between User-Agent: and Disallow:
lines. cf <URL: http://www.robotstxt.org/wc/norobots.html >.
Jul 23 '05 #3
Spartanicus wrote:
Is there something wrong with it? Yahoo Slurp still indexes content in
the auth folder...

Try:
Disallow: /auth/*


The Robots Exclusion Protocol does not specify wildcards, some robots
support this as a proprietary extension.

The problem in the original post was probably the blank line between the
User-agent and the Disallow lines.

--
Klaus Johannes Rusch
Kl********@atmedia.net
http://www.atmedia.net/KlausRusch/
Jul 23 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: Craig Cockburn | last post by:
Hi I'm aware of the use of robots.txt and the use of <META NAME="ROBOTS" CONTENT="index,follow"> However, what would be more useful is to be able to control within a page which elements of...
56
by: Anonymous, quoting Philip Ronan | last post by:
Subject: Warning: robots.txt unreliable in Apache servers From: Philip Ronan <invalid@invalid.invalid> Newsgroups: alt.internet.search-engines Message-ID: <BF89BF33.39FDF%invalid@invalid.invalid>...
2
by: Janus Knudsen | last post by:
Hello Im collecting ideas for a piece of software I've in mind. I need to create an application which can be started with some parameters, the application have to be started in many instances...
4
by: Misfit | last post by:
I have wondered, and I've tried this on a few random sites. I type the name of a site. www.somesite.com and follow it with /robots.txt. This can tell the robots not to bother indexing the...
5
by: John Nagle | last post by:
Python's "robots.txt" file parser may be misinterpreting a special case. Given a robots.txt file like this: User-agent: * Disallow: // Disallow: /account/registration Disallow: /account/mypro...
4
by: Les Caudle | last post by:
I'm noticing that web requrests are coming in with /robots.txt appended at the end: http://www.domain.com/ProductDetails.aspx?productID=527/robots.txt I can correct these, one by one for each...
2
by: John Nagle | last post by:
For some reason, Python's parser for "robots.txt" files doesn't like Wikipedia's "robots.txt" file: False The Wikipedia robots.txt file passes robots.txt validation, and it doesn't disallow...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.