469,923 Members | 1,348 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,923 developers. It's quick & easy.

"The Python Robot" is going mad ?

I have python related stuff on some of my web pages.

This month "The Python Robot" is going for over 400 MB / 4000 accesses
downloads in my stats ! increasing frequency the last months!? thats 25x
more than Google Bot & Inktomi Slurp - the second loquacious.

These numbers a going to cause substantial server cost. Why does "The Python
Robot" need to scan my pages 20 times a day? How can I cool down this
special bot a little bit ( robots.txt ? )

Robert
Jul 18 '05 #1
0 1223

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

reply views Thread by python-help-bounces | last post: by
12 posts views Thread by beliavsky | last post: by
6 posts views Thread by Kalle Anke | last post: by
3 posts views Thread by brianlum | last post: by
14 posts views Thread by Paddy3118 | last post: by
92 posts views Thread by ureuffyrtu955 | last post: by
3 posts views Thread by kyosohma | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.