Select weirdness

[Ron Garret]

Here's my code. It's a teeny weeny little HTTP server. (I'm not really
trying to reinvent the wheel here. What I'm really doing is writing a
dispatching proxy server, but this is the shortest way to illustrate the
problem I'm having):

from SocketServer import *
from socket import *
from select import select

class myHandler(Strea mRequestHandler ):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[])[0]
if sl:
l = self.rfile.read line()
if len(l)<3: break
print l,
pass
pass
print>>self.wfi le, 'HTTP/1.0 200 OK'
print>>self.wfi le, 'Content-type: text/plain'
print>>self.wfi le
print>>self.wfi le, 'foo'
self.rfile.clos e()
self.wfile.clos e()
print '<<<<<<<<<<<<'
pass
pass

def main():
server = TCPServer(('',8 080), myHandler)
server.serve_fo rever()
pass

if __name__ == '__main__': main()
If I telnet into this server and type in an HTTP request manually it
works fine. But when I try to access this with a browser (I've tried
Firefox and Safari -- both do the same thing) it hangs immediately after
reading the first line in the request (i.e. before reading the first
header).

When I click the "stop" button in the browser it breaks the logjam and
the server reads the headers (but then of course it dies trying to write
the response to a now-closed socket).

The only difference I can discern is that the browser send \r\n for
end-of-line while telnet just sends \n. But I don't see why that should
make any difference. So I'm stumped. Any clues would be much
appreciated.

Thanks,
rg

[Bjoern Schliessmann]

Ron Garret wrote:

print>>self.wfi le, 'HTTP/1.0 200 OK'
print>>self.wfi le, 'Content-type: text/plain'
print>>self.wfi le
print>>self.wfi le, 'foo'
[...]
If I telnet into this server and type in an HTTP request manually
it works fine. But when I try to access this with a browser (I've
tried Firefox and Safari -- both do the same thing) it hangs
immediately after reading the first line in the request (i.e.
before reading the first header).

Could the problem be that print ends lines with "\r\n" not on all
platforms? Try doing it explicitly.

The only difference I can discern is that the browser send \r\n
for end-of-line while telnet just sends \n.

Then, I think, your telnet client is broken. I've seem multiple
applications relying on telnet or similar protocols which
required "\r\n" as end-of-line. E. g. I needed to hack
twisted.protoco ls.basic.LineRe ceiverFactory since it showed the
exact behaviour you described for browsers. It showed up because
busybox' telnet only sends \n as EOL.

But I don't see why that should make any difference.

Easy. If you only accept "\r\n" as EOL, you'll wait forever unless
you really receive it.

Regards,
Björn

--
BOFH excuse #81:

Please excuse me, I have to circuit an AC line through my head to
get this database working.

[Irmen de Jong]

Ron Garret wrote:

Here's my code. It's a teeny weeny little HTTP server. (I'm not really
trying to reinvent the wheel here. What I'm really doing is writing a
dispatching proxy server, but this is the shortest way to illustrate the
problem I'm having):

from SocketServer import *
from socket import *
from select import select

class myHandler(Strea mRequestHandler ):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[])[0]
if sl:
l = self.rfile.read line()
if len(l)<3: break
print l,
pass
pass
print>>self.wfi le, 'HTTP/1.0 200 OK'
print>>self.wfi le, 'Content-type: text/plain'
print>>self.wfi le
print>>self.wfi le, 'foo'
self.rfile.clos e()
self.wfile.clos e()
print '<<<<<<<<<<<<'
pass
pass

You shouldn't use a select() of your own inside the handle method.
The socketserver takes care of that for you.

What you need to do is read the input from the socket until you've reached
the end-of-input marker. That marker is an empty line containing just "\r\n"
when dealing with HTTP GET requests.

Any reason don't want to use the BasicHTTPServer that comes with Python?

Anyway, try the following instead:

from SocketServer import *
from socket import *
from select import select

class myHandler(Strea mRequestHandler ):
def handle(self):
print '>>>>>>>>>>>'
while True:
l = self.rfile.read line()
print repr(l)
if not l or l=='\r\n':
break
print>>self.wfi le, 'HTTP/1.0 200 OK'
print>>self.wfi le, 'Content-type: text/plain'
print>>self.wfi le
print>>self.wfi le, 'foo'
self.rfile.clos e()
self.wfile.clos e()
print '<<<<<<<<<<<<'

def main():
server = TCPServer(('',8 080), myHandler)
server.serve_fo rever()

if __name__ == '__main__': main()

--Irmen

[Ron Garret]

In article <46************ *********@news. xs4all.nl>,
Irmen de Jong <ir**********@x s4all.nlwrote:

Ron Garret wrote:

Here's my code. It's a teeny weeny little HTTP server. (I'm not really
trying to reinvent the wheel here. What I'm really doing is writing a
dispatching proxy server, but this is the shortest way to illustrate the
problem I'm having):

from SocketServer import *
from socket import *
from select import select

class myHandler(Strea mRequestHandler ):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[])[0]
if sl:
l = self.rfile.read line()
if len(l)<3: break
print l,
pass
pass
print>>self.wfi le, 'HTTP/1.0 200 OK'
print>>self.wfi le, 'Content-type: text/plain'
print>>self.wfi le
print>>self.wfi le, 'foo'
self.rfile.clos e()
self.wfile.clos e()
print '<<<<<<<<<<<<'
pass
pass

You shouldn't use a select() of your own inside the handle method.
The socketserver takes care of that for you.

I don't understand why socketserver calling select should matter. (And
BTW, there are no calls to select in SocketServer.py . I'm using
Python2.5.)

What you need to do is read the input from the socket until you've reached
the end-of-input marker. That marker is an empty line containing just "\r\n"
when dealing with HTTP GET requests.

Any reason don't want to use the BasicHTTPServer that comes with Python?

Yes, but it's a long story. What I'm really doing is writing a
dispatching proxy. It reads the HTTP request and then redirects it to a
number of different servers depending on the URL. The servers are all
local and served off of unix sockets, not TCP sockets (which is why I
can't just configure Apache to do this).

The reason I'm running this weird configuration (in case you're
wondering) is because this is a development machine and multiple copies
of the same website run on it simultaneously. Each copy of the site
runs a customized server to handle AJAX requests efficiently.

>
Anyway, try the following instead:

That won't work for POST requests.

rg

[Ron Garret]

In article <59************ *@mid.individua l.net>,
Bjoern Schliessmann <us************ **************@ spamgourmet.com >
wrote:

The only difference I can discern is that the browser send \r\n
for end-of-line while telnet just sends \n.

....

But I don't see why that should make any difference.

Easy. If you only accept "\r\n" as EOL, you'll wait forever unless
you really receive it.

But it's SELECT that is hanging. Select knows (or ought to know)
nothing of end-of-line markers. Readline (when it is called) is doing
the Right Thing, and my end-of-request test is (currently) reading a
line less than 3 characters long. All that is working. The problem is
that select is saying there is no input (and therefore hanging) when in
fact there is input.

rg

[Ron Garret]

In article <rN************ *************** **@news.gha.cha rtermi.net>,
Ron Garret <rN*******@flow net.comwrote:

Here's my code. It's a teeny weeny little HTTP server. (I'm not really
trying to reinvent the wheel here. What I'm really doing is writing a
dispatching proxy server, but this is the shortest way to illustrate the
problem I'm having):

from SocketServer import *
from socket import *
from select import select

class myHandler(Strea mRequestHandler ):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[])[0]
if sl:
l = self.rfile.read line()
if len(l)<3: break
print l,
pass
pass
print>>self.wfi le, 'HTTP/1.0 200 OK'
print>>self.wfi le, 'Content-type: text/plain'
print>>self.wfi le
print>>self.wfi le, 'foo'
self.rfile.clos e()
self.wfile.clos e()
print '<<<<<<<<<<<<'
pass
pass

def main():
server = TCPServer(('',8 080), myHandler)
server.serve_fo rever()
pass

if __name__ == '__main__': main()
If I telnet into this server and type in an HTTP request manually it
works fine. But when I try to access this with a browser (I've tried
Firefox and Safari -- both do the same thing) it hangs immediately after
reading the first line in the request (i.e. before reading the first
header).

When I click the "stop" button in the browser it breaks the logjam and
the server reads the headers (but then of course it dies trying to write
the response to a now-closed socket).

The only difference I can discern is that the browser send \r\n for
end-of-line while telnet just sends \n. But I don't see why that should
make any difference. So I'm stumped. Any clues would be much
appreciated.

I have reproduced the problem using Telnet, so that proves it's not an
EOL issue. The problem seems to be timing-related. If I type the
request in manually it works. If I paste it in all at once, it hangs.
It's actually even weirder than that: if I pipe the request into a
telnet client then it hangs after the request line, just with a browser
(or wget). But if I literally paste the request into a window running a
telnet client, it gets past the request line and four out of seven
headers before it hangs.

rg

[Ron Garret]

I think I've figured out what's going on.

First, here's the smoking gun: I changed the code as follows:

class myHandler(Strea mRequestHandler ):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[],1)[0]
print sl
l = self.rfile.read line()
if len(l)<3: break
print l,
pass

(Rest of code is unchanged.)

In other words, I select on the rfile object and added a timeout.

Here's the result when I cut-and-paste an HTTP request:

>>>>>>>>>>>

[<socket._fileob ject object at 0x6b110>]
GET /foo/baz HTTP/1.1
[<socket._fileob ject object at 0x6b110>]
Accept: */*
[<socket._fileob ject object at 0x6b110>]
Accept-Language: en
[<socket._fileob ject object at 0x6b110>]
Accept-Encoding: gzip, deflate
[<socket._fileob ject object at 0x6b110>]
Cookie: c1=18:19:55.042 196; c2=18:19:55.042 508
[]
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en)
AppleWebKit/419 (KHTM
[]
L, like Gecko) Safari/419.3
[]
Connection: keep-alive
[]
Host: localhost:8081
[]
<<<<<<<<<<<<

As you can see, the select call shows input available for a while (five
lines) and then shows no input available despite the fact that there is
manifestly still input available.

The answer is obvious: select is looking only at the underlying socket,
and not at the rfile buffers.

So... is this a bug in select? Or a bug in my code?

rg

[Ron Garret]

In article <rN************ *************** **@news.gha.cha rtermi.net>,
Ron Garret <rN*******@flow net.comwrote:

The answer is obvious: select is looking only at the underlying socket,
and not at the rfile buffers.

Here is conclusive proof that there's a bug in select:

from socket import *
from select import select
s=socket(AF_INE T, SOCK_STREAM)
s.bind(('',8080 ))
s.listen(5)
f = s.accept()[0].makefile()
# Now telnet to port 8080 and enter two lines of random text
f.readline()
select([f],[],[],1)
f.readline()

Here's the sample input:

[ron@mickey:~/devel/sockets]$ telnet localhost 8081
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
123
321
And this is the result:

>>f.readline( )

'123\r\n'

>>select([f],[],[],1)

# After one second...
([], [], [])

>>f.readline( )

'321\r\n'

>>>

So this is clearly a bug, but surely I'm not the first person to have
encountered this? Is there a known workaround?

rg

[Erik Max Francis]

Ron Garret wrote:

So this is clearly a bug, but surely I'm not the first person to have
encountered this? Is there a known workaround?

It's hard to see how this demonstrates a bug in anything, since you're
telnetting to the wrong port in your example.

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM, Y!M erikmaxfrancis
Being in love for real / It ain't like a movie screen
-- India Arie

[Ron Garret]

In article <1a************ *************** ***@speakeasy.n et>,
Erik Max Francis <ma*@alcyone.co mwrote:

Ron Garret wrote:

So this is clearly a bug, but surely I'm not the first person to have
encountered this? Is there a known workaround?

It's hard to see how this demonstrates a bug in anything, since you're
telnetting to the wrong port in your example.

Geez you people are picky. Since I ran this several times I ran into
the TIM_WAIT problem. Here's the actual transcript:

Python 2.5 (r25:51908, Mar 1 2007, 10:09:05)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright" , "credits" or "license" for more information.

>>from socket import *
from select import select
s=socket(AF_I NET, SOCK_STREAM)
s.bind(('',80 80))

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1, in bind
socket.error: (48, 'Address already in use')

>>s.bind(('',80 81))
s.listen(5)
f = s.accept()[0].makefile()
f.readline( )

'123\r\n'

>>select([f],[],[],1)

([], [], [])

>>f.readline( )

'321\r\n'