473,326 Members | 2,095 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

Robots.txt ?

How can I regulate an agent or bot to my site monthly ?
they are visiting to often
I'd like to allow them monthly only

any ideas ?

tia

Jul 17 '05 #1
4 1480
Maverick wrote :
How can I regulate an agent or bot to my site monthly ?
they are visiting to often
I'd like to allow them monthly only

Try:
<meta name="revisit-after" content="30 days">

HTH
Martin

--
Epur Si Muove (Gallileo Gallilei)
Jul 17 '05 #2
Maverick wrote:
How can I regulate an agent or bot to my site monthly ?
they are visiting to often
I'd like to allow them monthly only

any ideas ?


You could try generateing robots.txt dynamically. Based on the user
agent, just decide whether you wish to let it in today or not, and
based on that decision, output the user agent deny for it or don't.

It's good to know however that not all robots follow the robots.txt
(though the well behaving ones do). Another thing to note, is that
Google (for example) drops the pages it can't access from it's
database.

--
Markku Uttula

Jul 17 '05 #3
Where does that go;
does that have to be in the head section of every page on the whole web
?

tia

Jul 17 '05 #4
Maverick wrote :
Where does that go;
does that have to be in the head section of every page on the whole web
?


you might want to read this:
http://www.robotstxt.org/wc/exclusion.html
Or just use Google with search terms like:
robot.txt
meta robot
and so on.

HTH
Martin

--
Epur Si Muove (Gallileo Gallilei)
Jul 17 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: Craig Cockburn | last post by:
Hi I'm aware of the use of robots.txt and the use of <META NAME="ROBOTS" CONTENT="index,follow"> However, what would be more useful is to be able to control within a page which elements of...
56
by: Anonymous, quoting Philip Ronan | last post by:
Subject: Warning: robots.txt unreliable in Apache servers From: Philip Ronan <invalid@invalid.invalid> Newsgroups: alt.internet.search-engines Message-ID: <BF89BF33.39FDF%invalid@invalid.invalid>...
2
by: Janus Knudsen | last post by:
Hello Im collecting ideas for a piece of software I've in mind. I need to create an application which can be started with some parameters, the application have to be started in many instances...
4
by: Misfit | last post by:
I have wondered, and I've tried this on a few random sites. I type the name of a site. www.somesite.com and follow it with /robots.txt. This can tell the robots not to bother indexing the...
5
by: John Nagle | last post by:
This bug, " robotparser interactively prompts for username and password", has been open since 2003. It killed a big batch job of ours last night. Module "robotparser" naively uses "urlopen" to...
5
by: John Nagle | last post by:
Python's "robots.txt" file parser may be misinterpreting a special case. Given a robots.txt file like this: User-agent: * Disallow: // Disallow: /account/registration Disallow: /account/mypro...
4
by: Les Caudle | last post by:
I'm noticing that web requrests are coming in with /robots.txt appended at the end: http://www.domain.com/ProductDetails.aspx?productID=527/robots.txt I can correct these, one by one for each...
2
by: John Nagle | last post by:
For some reason, Python's parser for "robots.txt" files doesn't like Wikipedia's "robots.txt" file: False The Wikipedia robots.txt file passes robots.txt validation, and it doesn't disallow...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.