Socket exceptions aren't in the standard exception hierarchy

[John Nagle]

Here are three network-related exceptions. These
were caught by "except" with no exception type, because
none of the more specific exceptions matched. This
is what a traceback produced:
1. File "D:\Python24\lib\socket.py", line 295, in read
data = self._sock.recv(recv_size)
timeout: timed out

2. File "D:\Python24\lib\socket.py", line 295, in read
data = self._sock.recv(recv_size)
error: (10054, 'Connection reset by peer')

3. File "D:\Python24\lib\socket.py", line 317, in readline
data = recv(1)
IOError: [Errno socket error] timed out

For 1 and 2, those are errors that aren't in the
exception hierarchy. Looking at the C code for "socketmodule.c",
it's clear that "socket.error" doesn't inherit from any standard
exception class. See, in "init_socket()":

socket_error = PyErr_NewException("socket.error", NULL, NULL);

That first NULL should be some parent exception, maybe "IOError".
As it is, "socket.error" is outside the standard exception hierarchy.
That's not too good.

Case #3, IOError, should have been caught by this:

except IOError, message: # I/O error

But it wasn't. The "IOError" fell through, was caught by the
next outer exception block, and was logged as a generic
error.

I can't find where in the Python socket module an "IOError"
could be raised. I would have expected "socket.timeout".

Anyway, I need to know the full set of exceptions that can
be raised by sockets. Thanks.

John Nagle

[John Nagle]

Dennis Lee Bieber wrote:

On Sun, 22 Apr 2007 23:20:25 -0700, John Nagle <na***@animats.com>
declaimed the following in comp.lang.python:

>>2. File "D:\Python24\lib\socket.py", line 295, in read
data = self._sock.recv(recv_size)
error: (10054, 'Connection reset by peer')

That looks like M$ Windows version of UNIX/Linux error number 54
(pretty much all Windows socket errors are UNIX number+10000)

Errors coming from Windows may not be mapped to specific Python
exceptions, but rather to some general error conditions. {hypothesis} As
such, the Windows errors may not match what UNIX/Linux report.

Actually, that's not what's happening. The socket module is
explicitly raising "socket.error" in C code. It's not an OSError or
a WindowsError, although the text makes it look like one.

The problem is not that socket errors aren't entirely portable.
It's that they're not even in the Python exception hierarchy.
See "http://docs.python.org/lib/module-exceptions.html".
They have their own hierarchy, which starts at "socket.error".
All built-in exceptions were supposed to be converted to the
standard exception hierarchy back before Python 2.0, but these
weren't.

Either they should be under IOError, or there should be
"NetworkError" under EnvironmentError, and they should be under
that. "NetworkError", alongside IOError in the hierarchy,
would be useful. All the things that go wrong in networking
belong under there. Socket-level errors, SSL/TLS-level errors,
and HTTP/FTP/etc. level errors all belong under NetworkError.

This has to be fixed before PEP 352, when the exception
hierarchy is enforced, or the socket exceptions won't even work
right.

John Nagle

[Steve Holden]

John Nagle wrote:

Dennis Lee Bieber wrote:

>On Sun, 22 Apr 2007 23:20:25 -0700, John Nagle <na***@animats.com>
declaimed the following in comp.lang.python:

>>2. File "D:\Python24\lib\socket.py", line 295, in read
data = self._sock.recv(recv_size)
error: (10054, 'Connection reset by peer')

That looks like M$ Windows version of UNIX/Linux error number 54
(pretty much all Windows socket errors are UNIX number+10000)

Errors coming from Windows may not be mapped to specific Python
exceptions, but rather to some general error conditions. {hypothesis} As
such, the Windows errors may not match what UNIX/Linux report.

Actually, that's not what's happening. The socket module is
explicitly raising "socket.error" in C code. It's not an OSError or
a WindowsError, although the text makes it look like one.

The problem is not that socket errors aren't entirely portable.
It's that they're not even in the Python exception hierarchy.
See "http://docs.python.org/lib/module-exceptions.html".
They have their own hierarchy, which starts at "socket.error".
All built-in exceptions were supposed to be converted to the
standard exception hierarchy back before Python 2.0, but these
weren't.

Either they should be under IOError, or there should be
"NetworkError" under EnvironmentError, and they should be under
that. "NetworkError", alongside IOError in the hierarchy,
would be useful. All the things that go wrong in networking
belong under there. Socket-level errors, SSL/TLS-level errors,
and HTTP/FTP/etc. level errors all belong under NetworkError.

This has to be fixed before PEP 352, when the exception
hierarchy is enforced, or the socket exceptions won't even work
right.

John:

Where did you get this information? If true it would certainly need to
be logged as a bug, but under Windows on 2,4 I see

>>issubclass(socket.gaierror, Exception)

True

>>>

and the same under Cygwin 2.5. I am presuming most other users will see
the same thing.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Recent Ramblings http://holdenweb.blogspot.com

[John Nagle]

Steve Holden wrote:

John Nagle wrote:

>Dennis Lee Bieber wrote:

>>On Sun, 22 Apr 2007 23:20:25 -0700, John Nagle <na***@animats.com>
declaimed the following in comp.lang.python:
2. File "D:\Python24\lib\socket.py", line 295, in read
data = self._sock.recv(recv_size)
error: (10054, 'Connection reset by peer')

That looks like M$ Windows version of UNIX/Linux error number 54
(pretty much all Windows socket errors are UNIX number+10000)

Errors coming from Windows may not be mapped to specific Python
exceptions, but rather to some general error conditions. {hypothesis} As
such, the Windows errors may not match what UNIX/Linux report.

Actually, that's not what's happening. The socket module is
explicitly raising "socket.error" in C code. It's not an OSError or
a WindowsError, although the text makes it look like one.

The problem is not that socket errors aren't entirely portable.
It's that they're not even in the Python exception hierarchy.
See "http://docs.python.org/lib/module-exceptions.html".
They have their own hierarchy, which starts at "socket.error".
All built-in exceptions were supposed to be converted to the
standard exception hierarchy back before Python 2.0, but these
weren't.

Either they should be under IOError, or there should be
"NetworkError" under EnvironmentError, and they should be under
that. "NetworkError", alongside IOError in the hierarchy,
would be useful. All the things that go wrong in networking
belong under there. Socket-level errors, SSL/TLS-level errors,
and HTTP/FTP/etc. level errors all belong under NetworkError.

This has to be fixed before PEP 352, when the exception
hierarchy is enforced, or the socket exceptions won't even work
right.

John:

Where did you get this information? If true it would certainly need to
be logged as a bug, but under Windows on 2,4 I see

>>issubclass(socket.gaierror, Exception)

True

>>>

and the same under Cygwin 2.5. I am presuming most other users will see
the same thing.

regards
Steve

Ah. "socket.error" is a subclass of "Exception", but not
of "StandardError".

issubclass(socket.error,StandardError)

is False.
John Nagle

[Daniel Nogradi]

>>2. File "D:\Python24\lib\socket.py", line 295, in read

>>data = self._sock.recv(recv_size)
error: (10054, 'Connection reset by peer')

That looks like M$ Windows version of UNIX/Linux error number 54
(pretty much all Windows socket errors are UNIX number+10000)

Errors coming from Windows may not be mapped to specific Python
exceptions, but rather to some general error conditions. {hypothesis} As
such, the Windows errors may not match what UNIX/Linux report.
Actually, that's not what's happening. The socket module is
explicitly raising "socket.error" in C code. It's not an OSError or
a WindowsError, although the text makes it look like one.

The problem is not that socket errors aren't entirely portable.
It's that they're not even in the Python exception hierarchy.
See "http://docs.python.org/lib/module-exceptions.html".
They have their own hierarchy, which starts at "socket.error".
All built-in exceptions were supposed to be converted to the
standard exception hierarchy back before Python 2.0, but these
weren't.

Either they should be under IOError, or there should be
"NetworkError" under EnvironmentError, and they should be under
that. "NetworkError", alongside IOError in the hierarchy,
would be useful. All the things that go wrong in networking
belong under there. Socket-level errors, SSL/TLS-level errors,
and HTTP/FTP/etc. level errors all belong under NetworkError.

This has to be fixed before PEP 352, when the exception
hierarchy is enforced, or the socket exceptions won't even work
right.

John:

Where did you get this information? If true it would certainly need to
be logged as a bug, but under Windows on 2,4 I see

>>issubclass(socket.gaierror, Exception)

True

>>>

and the same under Cygwin 2.5. I am presuming most other users will see
the same thing.

regards
Steve

Ah. "socket.error" is a subclass of "Exception", but not
of "StandardError".

issubclass(socket.error,StandardError)

is False.

On linux, python 2.5:

>>import socket
issubclass(socket.error,Exception)

True

[Steve Holden]

John Nagle wrote:
[socket.error bug report]

>>
Where did you get this information? If true it would certainly need to
be logged as a bug, but under Windows on 2,4 I see

> >>issubclass(socket.gaierror, Exception)

True

> >>>

and the same under Cygwin 2.5. I am presuming most other users will see
the same thing.

regards
Steve

Ah. "socket.error" is a subclass of "Exception", but not
of "StandardError".

issubclass(socket.error,StandardError)

is False.

Right, so this *is* a big, as long as Brett Cannon's upcoming (further)
reorganization of the standard exception hierarchy doesn't stamp on it.
It probably *was* overlooked in the reorganization of the hierarchy, and
this implies there may be other extensions that also make the same error.

It should be logged as a bug in the tracker - the fix is probably pretty
simple, but it'll need some consideration of the forward compatibility
considerations.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Recent Ramblings http://holdenweb.blogspot.com

[John Nagle]

Steve Holden wrote:

John Nagle wrote:
[socket.error bug report]

>>>
Where did you get this information? If true it would certainly need
to be logged as a bug, but under Windows on 2,4 I see

>>issubclass(socket.gaierror, Exception)
True
>>>

and the same under Cygwin 2.5. I am presuming most other users will
see the same thing.

regards
Steve

Ah. "socket.error" is a subclass of "Exception", but not
of "StandardError".

issubclass(socket.error,StandardError)

is False.

Right, so this *is* a bug, as long as Brett Cannon's upcoming (further)
reorganization of the standard exception hierarchy doesn't stamp on it.
It probably *was* overlooked in the reorganization of the hierarchy, and
this implies there may be other extensions that also make the same error.

It should be logged as a bug in the tracker - the fix is probably pretty
simple, but it'll need some consideration of the forward compatibility
considerations.

regards
Steve

The bug tracker shows quite a number of problems related to
socket exceptions. Searching for "socket AND exception" returns
45 results. These seem to be the most relevant ones:

[ 805194 ] Inappropriate error received using socket timeout
[ 1019808 ] wrong socket error returned
[ 1571878 ] Improvements to socket module exceptions
[ 708927 ] socket timeouts produce wrong errors in win32

None of them quite cover this new issue, but it's clear
that the area needs some work. Should this new issue
be added as a new bug or as an added comment to one of the
above?

Just figuring out what exceptions can be raised from
the socket module is tough. I've seen exceptions
derived from "socket.error", exceptions from IOError,
and exceptions from the SSL layer, which patches the
sockets module when loaded. These are non-bug
exceptions; that is, the problem is out in the network,
external to the program. I'm still not sure I have all
the possibilities covered.

Retrying on unknown exceptions isn't the answer;
that leads to exception loops if there's a program bug.
The most important distinction with sockets is "external
network problem" vs. "local program program". I'd like
to see a "NetworkException" in the exception hierarchy,
with all the things that can go wrong due to conditions
external to the local machine under that exception.

I'd suggest the following:

1. Add "NetworkError" under "IOError" in
the exception hierarchy.

2. Put the existing "socket.error" under "NetworkError".
Since "socket.error" needs to be reparented anyway (it's currently
a direct descendant of "Exception") this provides a good place for it.

3. Find any places where the socket module can raise IOError
or OSError due to an external network condition, and make them raise
something under NetworkError instead. Code that catches IOError
will still work.

4. Put all errors in the various SSL modules (SSLError, etc.)
which can be raised due to external network conditions under "NetworkError".

5. Move "urllib2.URLError", which is currently under IOError, to be
under NetworkError.

6. Move the misc. errors from "urllib", like "ContentTooShortError",
which are currently under IOError, down a level under NetworkError.

Then, programs that catch NetworkError could be sure of catching
all network trouble conditions, but not local code bugs.

John Nagle

[Steve Holden]

John Nagle wrote:

Steve Holden wrote:

>John Nagle wrote:
[socket.error bug report]

>>>Where did you get this information? If true it would certainly need
to be logged as a bug, but under Windows on 2,4 I see

>>issubclass(socket.gaierror, Exception)
True
>>>

and the same under Cygwin 2.5. I am presuming most other users will
see the same thing.

regards
Steve

Ah. "socket.error" is a subclass of "Exception", but not
of "StandardError".

issubclass(socket.error,StandardError)

is False.

Right, so this *is* a bug, as long as Brett Cannon's upcoming (further)
reorganization of the standard exception hierarchy doesn't stamp on it.
It probably *was* overlooked in the reorganization of the hierarchy, and
this implies there may be other extensions that also make the same error.

It should be logged as a bug in the tracker - the fix is probably pretty
simple, but it'll need some consideration of the forward compatibility
considerations.

regards
Steve

The bug tracker shows quite a number of problems related to
socket exceptions. Searching for "socket AND exception" returns
45 results. These seem to be the most relevant ones:

[ 805194 ] Inappropriate error received using socket timeout
[ 1019808 ] wrong socket error returned
[ 1571878 ] Improvements to socket module exceptions
[ 708927 ] socket timeouts produce wrong errors in win32

None of them quite cover this new issue, but it's clear
that the area needs some work. Should this new issue
be added as a new bug or as an added comment to one of the
above?

Just figuring out what exceptions can be raised from
the socket module is tough. I've seen exceptions
derived from "socket.error", exceptions from IOError,
and exceptions from the SSL layer, which patches the
sockets module when loaded. These are non-bug
exceptions; that is, the problem is out in the network,
external to the program. I'm still not sure I have all
the possibilities covered.

Retrying on unknown exceptions isn't the answer;
that leads to exception loops if there's a program bug.
The most important distinction with sockets is "external
network problem" vs. "local program program". I'd like
to see a "NetworkException" in the exception hierarchy,
with all the things that can go wrong due to conditions
external to the local machine under that exception.

I'd suggest the following:

1. Add "NetworkError" under "IOError" in
the exception hierarchy.

2. Put the existing "socket.error" under "NetworkError".
Since "socket.error" needs to be reparented anyway (it's currently
a direct descendant of "Exception") this provides a good place for it.

3. Find any places where the socket module can raise IOError
or OSError due to an external network condition, and make them raise
something under NetworkError instead. Code that catches IOError
will still work.

4. Put all errors in the various SSL modules (SSLError, etc.)
which can be raised due to external network conditions under "NetworkError".

5. Move "urllib2.URLError", which is currently under IOError, to be
under NetworkError.

6. Move the misc. errors from "urllib", like "ContentTooShortError",
which are currently under IOError, down a level under NetworkError.

Then, programs that catch NetworkError could be sure of catching
all network trouble conditions, but not local code bugs.

John Nagle

All these notes should be included in the bug report, as I suspect the
module would benefit from additional clarity. I believe socket just grew
up as a clever way of making as much use of the platform socket library
as possible, and that the approach's deficiencies have become more
obvious as time has gone by and Python has matured.

The work you describe is not insignificant, so it should be fully
described if it's to stand any chance of being implemented. I *do* think
it would rationalize things considerably, but that's only me ...

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Recent Ramblings http://holdenweb.blogspot.com

[John Nagle]

Steve Holden wrote:

John Nagle wrote:

>Steve Holden wrote:

>>John Nagle wrote:
[socket.error bug report]

All these notes should be included in the bug report, as I suspect the
module would benefit from additional clarity.

Done. See

[ 1706815 ] socket.error exceptions not subclass of StandardError

Also see

[ 805194 ] Inappropriate error received using socket timeout
[ 1019808 ] wrong socket error returned
[ 1571878 ] Improvements to socket module exceptions
[ 708927 ] socket timeouts produce wrong errors in win32

for related but not identical problems in that area.

John Nagle

[Steve Holden]

John Nagle wrote:

Steve Holden wrote:

>John Nagle wrote:

>>Steve Holden wrote:

John Nagle wrote:
[socket.error bug report]

>All these notes should be included in the bug report, as I suspect the
module would benefit from additional clarity.

Done. See

[ 1706815 ] socket.error exceptions not subclass of StandardError

Also see

[ 805194 ] Inappropriate error received using socket timeout
[ 1019808 ] wrong socket error returned
[ 1571878 ] Improvements to socket module exceptions
[ 708927 ] socket timeouts produce wrong errors in win32

for related but not identical problems in that area.

Thanks. At least this is less likely to be overlooked now.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Recent Ramblings http://holdenweb.blogspot.com

[John Nagle]

On a related note, M2Crypto's exceptions are direct children
of Exception. If we add NetworkError, there will be a better
place to put them.

Currently, you have to catch, at least,

M2Crypto.SSL.Checker.SSLVerificationError
M2Crypto.SSL.SSLError

both of which can be raised by socket operations if
M2Crypto is installed. I'm not sure what errors the
stock SSL module raises.

(I'm running stress tests on a dedicated machine in a colocation
facility. It's examining 11,000 known spam and malware sites right
now. This exercises the error handling, forcing many unusual cases
and logging the problems. That's why I'm discovering all these library
issues.)

John Nagle

Steve Holden wrote:

John Nagle wrote:

>Steve Holden wrote:

>>John Nagle wrote:

Steve Holden wrote:

John Nagle wrote:
[socket.error bug report]

>>All these notes should be included in the bug report, as I suspect
the module would benefit from additional clarity.

Done. See

[ 1706815 ] socket.error exceptions not subclass of StandardError

Also see

[ 805194 ] Inappropriate error received using socket timeout
[ 1019808 ] wrong socket error returned
[ 1571878 ] Improvements to socket module exceptions
[ 708927 ] socket timeouts produce wrong errors in win32

for related but not identical problems in that area.

Thanks. At least this is less likely to be overlooked now.

regards
Steve