473,289 Members | 1,963 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,289 software developers and data experts.

robotparser behavior on 403 (Forbidden) robot.txt files

I just discovered that the "robotparser" module interprets
a 403 ("Forbidden") status on a "robots.txt" file as meaning
"all access disallowed". That's unexpected behavior.

A major site ("http://www.aplus.net/robot.txt") has their
"robots.txt" file set up that way.

There's no real "robots.txt" standard, unfortunately.
So it's not definitively a bug.

John Nagle
SiteTruth
Jun 27 '08 #1
1 2072
I just discovered that the "robotparser" module interprets
a 403 ("Forbidden") status on a "robots.txt" file as meaning
"all access disallowed". That's unexpected behavior.
That's specified in the "norobots RFC":

http://www.robotstxt.org/norobots-rfc.txt

- On server response indicating access restrictions (HTTP Status
Code 401 or 403) a robot should regard access to the site
completely restricted.

So if a site returns 403, we should assume that it did so
deliberately, and doesn't want to be indexed.
A major site ("http://www.aplus.net/robot.txt") has their
"robots.txt" file set up that way.
You should try "http://www.aplus.net/robots.txt" instead,
which can be accessed just fine.

Regards,
Martin
Jun 27 '08 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Dave Opstad | last post by:
Take a look at this snippet: >>> class L(list): .... def __init__(self, v): .... super(L, self).__init__(v) .... def __setitem__(self, key, value): .... print...
4
by: DaKoadMunky | last post by:
<CODE> #include <iostream> using namespace std; int Foo(int x,int y) { int result = x; result*=y; result+=y;
0
by: Layne L via DotNetMonster.com | last post by:
Trying to install a .NET application on a Windows 2003 Server ( .NET v1.1) and for any . aspx file get the following error: 403.1 Forbidden: Execute Access Forbidden The application has...
1
by: Graham | last post by:
Hi, One of our clients sites has a files directory that stores different types of files: doc, pdf, xls. Every now and again we get an error email with the following info: PATH:...
6
by: Axel Dahmen | last post by:
Hi, in one of my pages I want to return "403 Forbidden" to the client depending on some logic in my .aspx code-behind. So I set "Response.Status = 403". But apparently IE seems to ignore the...
1
by: PK9 | last post by:
I am receiving the following error in Visual Studio 2003: "the web server reported the following error when attempting to create or open the web project HTTP/1.1 403 Access Forbidden" I am...
12
by: Rajesh S R | last post by:
Can anyone tell me what is the difference between undefined behavior and unspecified behavior? Though I've read what is given about them, in ISO standards, I'm still not able to get the...
5
by: John Nagle | last post by:
Python's "robots.txt" file parser may be misinterpreting a special case. Given a robots.txt file like this: User-agent: * Disallow: // Disallow: /account/registration Disallow: /account/mypro...
2
by: John Nagle | last post by:
For some reason, Python's parser for "robots.txt" files doesn't like Wikipedia's "robots.txt" file: False The Wikipedia robots.txt file passes robots.txt validation, and it doesn't disallow...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.