By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
455,444 Members | 1,634 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 455,444 IT Pros & Developers. It's quick & easy.

"The Python Robot" is going mad ?

P: n/a
I have python related stuff on some of my web pages.

This month "The Python Robot" is going for over 400 MB / 4000 accesses
downloads in my stats ! increasing frequency the last months!? thats 25x
more than Google Bot & Inktomi Slurp - the second loquacious.

These numbers a going to cause substantial server cost. Why does "The Python
Robot" need to scan my pages 20 times a day? How can I cool down this
special bot a little bit ( robots.txt ? )

Jul 18 '05 #1
Share this question for a faster answer!
Share on Google+

This discussion thread is closed

Replies have been disabled for this discussion.