closing a server socket

[simon place]

Spent some very frustrating hours recoding to find a way of closing a server
socket, i'd not thought it would be any problem,

however, after complete failure and as a last resort, i looked at the python
wrapper module for sockets, and found that the close command doesn't actually
call the underlying close! this didn't seem right, so i added it, and my code
now works simply and as expected.
def close(self):
self._sock.close() # added 2003-oct-13
self._sock = _closedsocket()
self.send = self.recv = self.sendto = self.recvfrom = self._sock._dummy
close.__doc__ = _realsocket.close.__doc__

Probably only on win32, the comments in the socket module seem to indicate
different codings on different platforms.

PythonWin 2.3.2 (#49, Oct 2 2003, 20:02:00) [MSC v.1200 32 bit (Intel)] on win32.

[Peter Hansen]

simon place wrote:

Spent some very frustrating hours recoding to find a way of closing a server
socket, i'd not thought it would be any problem,

however, after complete failure and as a last resort, i looked at the python
wrapper module for sockets, and found that the close command doesn't actually
call the underlying close! this didn't seem right, so i added it, and my code
now works simply and as expected.

def close(self):
self._sock.close() # added 2003-oct-13
self._sock = _closedsocket()
self.send = self.recv = self.sendto = self.recvfrom = self._sock._dummy
close.__doc__ = _realsocket.close.__doc__

Probably only on win32, the comments in the socket module seem to indicate
different codings on different platforms.

None of this should be necessary!

What problems were you having that led you to try the above abomination in
your effort to resolve them?

-Peter

[simon place]

Peter Hansen wrote:

simon place wrote:

Spent some very frustrating hours recoding to find a way of closing a server
socket, i'd not thought it would be any problem,

however, after complete failure and as a last resort, i looked at the python
wrapper module for sockets, and found that the close command doesn't actually
call the underlying close! this didn't seem right, so i added it, and my code
now works simply and as expected.

def close(self):
self._sock.close() # added 2003-oct-13
self._sock = _closedsocket()
self.send = self.recv = self.sendto = self.recvfrom = self._sock._dummy
close.__doc__ = _realsocket.close.__doc__

Probably only on win32, the comments in the socket module seem to indicate
different codings on different platforms.

None of this should be necessary!

What problems were you having that led you to try the above abomination in
your effort to resolve them?

-Peter

The problem is i couldn't find anyway, in 'normal' code, to close a server
socket, ie unblock an accept.

tried to use non-blocking mode, but not allowed on a server socket, which
makes sense, but isn't documented.

tried to use time-outs, which was close, but seems to close the accepted
connection sockets on a server socket time-out.

tried to fire a dummy connection to unblock the accept, which works, but if
your ip changes, i.e. a dialup, you can't do that.

i only hacked the module because of frustration, but it took about a minute to
spot, and now every thing works the original way i tried, and i've been doing
some thrashing and had no problems.

[Peter Hansen]

simon place wrote:

Peter Hansen wrote:

None of this should be necessary!

What problems were you having that led you to try the above abomination in
your effort to resolve them?

The problem is i couldn't find anyway, in 'normal' code, to close a server
socket, ie unblock an accept.

tried to use non-blocking mode, but not allowed on a server socket, which
makes sense, but isn't documented.

It does work, which is why it isn't documented that it doesn't. <wink>

What did you try, that non-blocking server sockets don't work for you?

(I assume you are talking about using select.select() to detect when the
listening socket has a client connection waiting to be accepted.)

-Peter

[simon place]

i'm trying to see if anyone else thinks this is a bug,

when you call socket.socket.close() the wrapper only marks the socket as
closed, so no exception to unblock socket.socket.accept(), and you can
actually still get connections on this socket?

( this may only be for windows. )

simon.

[Peter Hansen]

simon place wrote:

i'm trying to see if anyone else thinks this is a bug,

when you call socket.socket.close() the wrapper only marks the socket as
closed, so no exception to unblock socket.socket.accept(), and you can
actually still get connections on this socket?

( this may only be for windows. )

A repost? What didn't you like about my previous replies?

I guess you are basically saying that you want to use a blocking
socket (because you can't get non-blocking ones to work, though they
do work in general?) but you still want to be able to terminate
your call from another thread by forcibly closing the socket...

If that's so, give it up. You can't do that because the
accept() call is stuck down in C code which can't be
interrupted from another Python thread. Your only hope would be
to have the server call be in a separate process, which you
could "kill", but then you'd of course have to talk to this
other process with sockets, and be back where you started.

Use non-blocking sockets. They do work.

-Peter

[Alex Martelli]

Peter Hansen wrote:
...

I guess you are basically saying that you want to use a blocking
socket (because you can't get non-blocking ones to work, though they
do work in general?) but you still want to be able to terminate
your call from another thread by forcibly closing the socket...

If that's so, give it up. You can't do that because the
VERY good point -- and since by a weird coincidence I've had to deal
with EXACTLY this problem TWICE over the last few days for two
different clients (!), both times in the context of xml-rpc serving,
let me emphasize this once again: that's not the way to do it.

Use non-blocking sockets. They do work.

Or architect a solution where the blocked thread will get out
from its stuck state in the blocking socket "naturally". That's
quite easy in XML-RPC, e.g. as follows...:
_baseclass = SimpleXMLRPCServer.SimpleXMLRPCServer
class TerminatableServer(_baseclass):
allow_reuse_address = True

def __init__(self, addr, *args, **kwds):
self.myhost, self.myport = addr
_baseclass.__init__(self, addr, *args, **kwds)
self.register_function(self.terminate_server)

quit = False
def serve_forever(self):
while not self.quit:
self.handle_request()
self.server_close()

def terminate_server(self, authentication_token):
if not check_as_valid(authentication_token):
return 1, "Invalid or expired authentication token"
self.quit = True
return 0, "Server terminated on host %r, port %r" % (
self.myhost, self.myport)

(you'll want some way to verify authentication, of course, but
that's probably something you already have somewhere in such an
architecture, so I'm simplistically representing it by the
authentication_token parameter and a hypothetical check_as_valid
function that's able to check it).

The point is that in this way server termination is architected
as just one transaction -- the self.handle_request() in the body
of the while will eventually hand off control to terminate_server,
which may set self.quit, and that in turn will terminate the
while loop, since as soon as handle_request is done the loop's
condition is checked again. There are other ways, of course, but
I've found this one clean and reliable for these specific needs.
Alex

[simon place]

Alex Martelli wrote:

Peter Hansen wrote:
...

I guess you are basically saying that you want to use a blocking
socket (because you can't get non-blocking ones to work, though they
do work in general?) but you still want to be able to terminate
your call from another thread by forcibly closing the socket...

If that's so, give it up. You can't do that because the

VERY good point -- and since by a weird coincidence I've had to deal
with EXACTLY this problem TWICE over the last few days for two
different clients (!), both times in the context of xml-rpc serving,
let me emphasize this once again: that's not the way to do it.

Use non-blocking sockets. They do work.

Or architect a solution where the blocked thread will get out
from its stuck state in the blocking socket "naturally". That's
quite easy in XML-RPC, e.g. as follows...:
_baseclass = SimpleXMLRPCServer.SimpleXMLRPCServer
class TerminatableServer(_baseclass):
allow_reuse_address = True

def __init__(self, addr, *args, **kwds):
self.myhost, self.myport = addr
_baseclass.__init__(self, addr, *args, **kwds)
self.register_function(self.terminate_server)

quit = False
def serve_forever(self):
while not self.quit:
self.handle_request()
self.server_close()

def terminate_server(self, authentication_token):
if not check_as_valid(authentication_token):
return 1, "Invalid or expired authentication token"
self.quit = True
return 0, "Server terminated on host %r, port %r" % (
self.myhost, self.myport)

(you'll want some way to verify authentication, of course, but
that's probably something you already have somewhere in such an
architecture, so I'm simplistically representing it by the
authentication_token parameter and a hypothetical check_as_valid
function that's able to check it).

The point is that in this way server termination is architected
as just one transaction -- the self.handle_request() in the body
of the while will eventually hand off control to terminate_server,
which may set self.quit, and that in turn will terminate the
while loop, since as soon as handle_request is done the loop's
condition is checked again. There are other ways, of course, but
I've found this one clean and reliable for these specific needs.
Alex

Yes, no problem closing a server before it returns to accept another
connection, but when the handler is threaded,

see below, current situation

import socket

s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.bind(('127.0.0.1',80))
s.listen(1)
import threading
t=threading.Thread(target=s.accept)
t.start()
s.close()
import inspect
inspect.getsource(socket.socket.close) ' def close(self):\n self._sock = _closedsocket()\n self.send
= self.recv = self.sendto = self.recvfrom = self._sock._dummy\n' t.isAlive() True s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect(('127.0.0.1',80))
t.isAlive() False
i.e. closing server socket does not unblock accept and you can still get
connections on it!
( because it has not actually been closed!)
how is this acceptable?

with socket modified to actually close the socket,
import socket
import inspect
inspect.getsource(socket.socket.close) ' def close(self):\n self._sock.close()\n self._sock = _closeds
ocket()\n self.send = self.recv = self.sendto = self.recvfrom = self._soc
k._dummy\n' import threading
s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.bind(('127.0.0.1',80))
s.listen(1)
t=threading.Thread(target=s.accept)
t.start()
t.isAlive() True s.close()
Exception in thread Thread-1: Traceback (most recent call last):
File "C:\PYTHON23\lib\threading.py", line 436, in __bootstrap
self.run()
File "C:\PYTHON23\lib\threading.py", line 416, in run
self.__target(*self.__args, **self.__kwargs)
File "socket.py", line 168, in accept
sock, addr = self._sock.accept()
error: (10038, 'Socket operation on non-socket')

t.isAlive() False

i.e. close socket generates an exception on accept and no more connections.

If there's some underlying difficulty that causes this to come back and bite
the testing i've done shows this to take a very long time.

simon.

[Alex Martelli]

simon place wrote:
...

Yes, no problem closing a server before it returns to accept another
connection, but when the handler is threaded,
When the server is threaded, it should be designed accordingly; see
Python's standard library, module SocketServer etc, for one right
way to design a threaded server.
i.e. closing server socket does not unblock accept and you can still get
connections on it!
( because it has not actually been closed!)
how is this acceptable?
It's not: in general, two threads should NEVER access the same
resource simultaneously, unless the resource is specifically
known to be thread safe. sockets aren't. It's not acceptable,
and the bug is in the program which has one of its threads
make a sock.close() call while another is waiting on a
sock.accept() on the same socket sock.

with socket modified to actually close the socket, ... i.e. close socket generates an exception on accept and no more
connections.
On _your_ platform, no doubt. I'd bet substantial money that,
without even going to especially _exotic_ platforms, you'll find
terrible misbehavior with such mods on at least one of the
platforms I routinely program for.

If there's some underlying difficulty that causes this to come back and
bite the testing i've done shows this to take a very long time.

Just as much time as is necessary to go and purchase some
other platform to try it on, I suspect.
Alex