----- Original Message -----
From: "Neal" <>
Newsgroups: comp.infosystems.
www.authoring.html
Sent: Tuesday, June 15, 2004 9:43 AM
Subject: Re: html source to prevent web bot searching
On Tue, 15 Jun 2004 13:25:52 GMT, lostinspace
<> wrote:
If you are as knowledgable in regards to these as issues as
you
believe you are than WHY didn't you advise the op before I?
Posts propogate, and people check in, differently. Possible he never saw
the post before you did.
It's quite easy for you to sit on your backside and tear apart emails
"after the fact."
You really have less of a clue than you understand. I've been using
these methods and htaccess methods for nearly six years on my websites and
today
your attempting to convey that in that period I learned nothing of
traffic
patterne :-))
I think he'd like some evidence, beyond "I heard this works". I would as
well. Google's methods are well known. Where did you learn of the other
methods?
Hello Neal,
When I began with my websites, I also started following
alt.html and alt.
www.webmaster. From following threads of interest, I began
doing internet searches on lead words which had been supplied in
conversation. Later I participated in a Webmaster World forum surrounding
identifying robots (that forum has since been non-activated.)
There is so much more to this than was previously conveyed, however I felt
no reason to overwhelm the original inquiry.
Brian's concerns and interest are of no relevance to me. I provided the OP
with some lines as he requested which will possibly lead him to some
expanded insights, provided he learns how to use SE's :-)))
htaccess? Just do a google.
Proxies? Google or anybody else will be no help here. Most of the
proxy-server cache bots don't even identify themselves when spidering. AOL
is easy, they are using a UA ( "Mozilla/3.01 (compatible;)" ) .
This very extensive thread will provide you with a wealth of information:
http://www.webmasterworld.com/forum1...ht=perfect+ban
I'm not sure if the search capability still exists in that forum:
http://www.webmasterworld.com/forum11/index.htm
The entire defunct forum surrounded what has been touched on here.
In the end, each webmaster does what he/she determines to best enhance their
websites. Personally, I've in effect created an intranet on the open
internet by denying countries and regions. To take the time to explain and
"debate" over issues that some folks believe they understand to be so and
what I actually see occur is IMO not worth the any time spent convincing
them otherwise. Nor is it my desire to chase URL's that I long ago chased to
solve issues ONLY to support an effective solution which I implemented long
ago, merely to support a mail submission I provided an insight to.
I rarely post in this forum and this required detail for assisting somebody
explains why. :-(((
I provided the simple solution that the OP was asking for. In his original
mail, he inquired about something to include in the <head></head> although
he didn't realize that. He made NO inquiry about robots.txt, htaccess,
proxies, cache or all this other nonsense (at least in regard to his
inquiry.) In effect, I answered his question and I'm required to defend
myself. BS!