How to keep a site out of the search engines?

[Steve]

Hi;

I have a site that I do not want the search engines to pick up
on.....attracts people and problems I do not want.

Is there a tag ( or some other means ) of preventing this?

Thanks

Steve

[David Dorward]

Steve wrote:

I have a site that I do not want the search engines to pick up
on
http://www.robotstxt.org/wc/exclusion-admin.html
.....attracts people and problems I do not want.

If you want a private website, then use some form of password
protection on it.

[ironcorona]

David Dorward wrote:

Steve wrote:

I have a site that I do not want the search engines to pick up
on

http://www.robotstxt.org/wc/exclusion-admin.html

Does this actually work? Or is it like <meta name="robots"
content="noinde x,nofollow"> which google still crawls but doesn't index.
Though I guess that's the same difference.

--
Brian O'Connor (ironcorona)

[David Dorward]

ironcorona wrote:

David Dorward wrote:

Steve wrote:

I have a site that I do not want the search engines to pick up
on

http://www.robotstxt.org/wc/exclusion-admin.html

Does this actually work?

Reputable bots obey it.
--
David Dorward <http://blog.dorward.me .uk/> <http://dorward.me.uk/>
Home is where the ~/.bashrc is

[Si Ballenger]

On 24 May 2006 06:52:41 -0700, "Steve" <st**********@y ahoo.com>
wrote:

Hi;

I have a site that I do not want the search engines to pick up
on.....attract s people and problems I do not want.

Is there a tag ( or some other means ) of preventing this?

Don't put clickable links to it in a pubically accessable web
page.

[dk_sz]

> Is there a tag ( or some other means ) of preventing this?

You could use robots.txt... However, a better yet solution
may be to use .htaccess... Perhaps a user/password system?

Do have access to the webserver? Apache?
What about PHP (or ASP if Windows) ?
--
best regards
Thomas Schulz
http://www.micro-sys.dk/products/sitemap-generator/
http://www.micro-sys.dk/products/website-analyzer/

[Andy Dingley]

Si Ballenger wrote:

Don't put clickable links to it in a pubically accessable web
page.

Not reliable. Google Toolbar (for just one) is a backchannel that
feeds the URLs of "hidden" web sites back to Google, where they then
get spidered.

You also have no control over other people linking to your site.
If you want to "avoid indexing", then just use robots.txt (Maybe your
site isn't released yet).

If you want to keep your content hidden, then disallow anyone and
everyone from accessing it (by web server config, such as .htacccess).
Then specifically _allow_ content to be visible to a small set of
permitted users, such as by password access.

There is no practical way to identify "a spider" as distinct from "a
user". So any attempt to make content generally available and
_disallow_ spiders is always doomed to be unreliable and susceptible to
some level of leakage.

[Jukka K. Korpela]

di*****@codesmi ths.com <di*****@codesm iths.com> scripsit:

There is no practical way to identify "a spider" as distinct from "a
user".
There is. At least search engine spiders obey the Robots Exclusion Standard
in practice. Occasionally, there might be a misbehaving spider, but such
spiders are rare and they serve odd purposes. You can't beat them, or
separate them from users in any reasonable way, but there's no need to do
that either. They don't make your page findable using commonly used search
engines.
So any attempt to make content generally available and
_disallow_ spiders is always doomed to be unreliable and susceptible
to some level of leakage.

That's correct if you mean complete control. Generally, complete control
does not work on the WWW.

--
Yucca, http://www.cs.tut.fi/~jkorpela/

[Andy Dingley]

Jukka K. Korpela wrote:

di*****@codesmi ths.com <di*****@codesm iths.com> scripsit:

There is no practical way to identify "a spider" as distinct from "a
user".
There is. At least search engine spiders obey the Robots Exclusion Standard
in practice.

That is requesting a behaviour from a spider, not identifying it.

Occasionally, there might be a misbehaving spider,
Such as Google. Google's spiders have Been Evil of late, hammering on
some sites excessively and also ignoring the robot exclusion protocol
in the case of deep URLs obtained through the Google Toolbar.
Generally, complete control does not work on the WWW.

It does, provided you begin by forbidding _everything_ to _everyone_,
then only relaxing this rule for the very small domain where you can
control things (generally a crypto or password based subset of accepted
user agents). The "lack of control on the web" is a result of the web
being broadly accessible to a broad range of agents (and generally a
good thing too).

[Alan Silver]

In article <11************ **********@i40g 2000cwc.googleg roups.com>,
"Andy Dingley <di*****@codesm iths.com>" <di*****@codesm iths.com> writes

Google Toolbar (for just one) is a backchannel that feeds the URLs of
"hidden" web sites back to Google, where they then get spidered.

Also, Google has taken to looking at newly registered domain names to
see if there is a web site there. This means that even if your site
doesn't have any links to it and you don't use the Google toolbar,
Google could still find it!!

--
Alan Silver
(anything added below this line is nothing to do with me)