socket.makefile() buggy?

[ahlongxp]

socket.makefile() may lose data when "connection reset by peer".
and socket.recv() will never lose the data.

change the "1" to "0" in the client code to see the difference.

confirmed on both windows and linux.

so I guess there is a problem with makefile().

# Echo server program
import socket

HOST = '' # Symbolic name meaning the local host
PORT = 50007 # Arbitrary non-privileged port
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
while(1):
conn, addr = s.accept()
print 'Connected by', addr
conn.settimeout(1)
toread = 99
retrytime = 0
reads = 0
while reads < toread and retrytime < 10:
try:
data = conn.recv(min(32,toread-reads))
if not data: continue
print data
reads += len(data)
except:
retrytime += 1
print "timeout %d" % retrytime
continue
#conn.shutdown(socket.SHUT_RD)#no more read
if reads == toread:
conn.send("OK")
else:
conn.send("NOT OK")
conn.close()

# Echo client program
import socket

HOST = 'localhost' # The remote host
PORT = 50007 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
for i in range(12):
print "time %d" % i
try:
s.send('0123456789')
except socket.error, e:
print "socket error:", e
break
#data = s.recv(1024)
#print "data %d" %i, data
#s.shutdown(socket.SHUT_WR)#no more write

'''
try changing 1 to 0.
'''
if 1:
data=s.recv(1024)
else:
rf = s.makefile("rb")
data = rf.read()
rf.close()
s.close()
print 'Received', repr(data)

[Steve Holden]

That's a pretty pejorative subject line for someone who's been
programming Python [guessing by the date of your first post] for about a
month.

Perhaps "Incomprehensible behavior from socket.makefile()", or "I have
written a buggy network application"? That would at least show that you
are considering the possibility you yourself are the cause of the
problem ;-)

Python has been around for a long time, so you should ask yourself how
likely it is that you would be the first to discover such a fundamental
flaw? I'd be very surprised if someone doesn't point you at an article
on "how to ask smart questions", but I will content myself with that
oblique reference.

ahlongxp wrote:

socket.makefile() may lose data when "connection reset by peer".
and socket.recv() will never lose the data.

change the "1" to "0" in the client code to see the difference.

confirmed on both windows and linux.

so I guess there is a problem with makefile().

# Echo server program
import socket

HOST = '' # Symbolic name meaning the local host
PORT = 50007 # Arbitrary non-privileged port
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
while(1):
conn, addr = s.accept()
print 'Connected by', addr
conn.settimeout(1)
toread = 99
retrytime = 0
reads = 0
while reads < toread and retrytime < 10:
try:
data = conn.recv(min(32,toread-reads))
if not data: continue
print data
reads += len(data)
except:
retrytime += 1
print "timeout %d" % retrytime
continue
#conn.shutdown(socket.SHUT_RD)#no more read
if reads == toread:
conn.send("OK")
else:
conn.send("NOT OK")
conn.close()

# Echo client program
import socket

HOST = 'localhost' # The remote host
PORT = 50007 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
for i in range(12):
print "time %d" % i
try:
s.send('0123456789')
except socket.error, e:
print "socket error:", e
break
#data = s.recv(1024)
#print "data %d" %i, data
#s.shutdown(socket.SHUT_WR)#no more write

'''
try changing 1 to 0.
'''
if 1:
data=s.recv(1024)
else:
rf = s.makefile("rb")
data = rf.read()
rf.close()
s.close()
print 'Received', repr(data)

The big problem here seems to be that your server is closing the socket
after reading 99 bytes, but the client is actually sending 120. If you
change the "99" in the server to "120" you will see that the client
works when it uses the makefile().read(). I can't be bothered to debug
the exact reason why the 21 bytes of buffered data doesn't cause a
problem with the recv() version, but I wouldn't regard the error you are
getting as a bug in Python, rather as a bug in your application.

The underlying cause of all this appears to be your failure to
understand that network protocols should ideally allow the endpoints to
determine when a complete message has been transmitted, and to consume
all transmitted data.

You can't just ignore some or all of a client request and expect it not
to cause problems. If you look at some of the better-known TCP-based
application protocols you will see they take pains to ensure that
clients and servers can deterministically parse each others' messages.

In summary, a better protocol design will cause you rather less grief
and a little more humility will result in less acidity from crabby old
geeks like me.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

[Paul McGuire]

On Jul 8, 8:54 am, Steve Holden <[e-mail]:

That's a pretty pejorative subject line for someone who's been
programming Python [guessing by the date of your first post] for about a
month.

Perhaps "Incomprehensible behavior from socket.makefile()", or "I have
written a buggy network application"? That would at least show that you
are considering the possibility you yourself are the cause of the
problem ;-)

Python has been around for a long time, so you should ask yourself how
likely it is that you would be the first to discover such a fundamental
flaw? I'd be very surprised if someone doesn't point you at an article
on "how to ask smart questions", but I will content myself with that
oblique reference.

ahlongxp wrote:

socket.makefile() may lose data when "connection reset by peer".
and socket.recv() will never lose the data.

change the "1" to "0" in the client code to see the difference.

confirmed on both windows and linux.

so I guess there is a problem with makefile().

# Echo server program
import socket

HOST = '' # Symbolic name meaning the local host
PORT = 50007 # Arbitrary non-privileged port
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
while(1):
conn, addr = s.accept()
print 'Connected by', addr
conn.settimeout(1)
toread = 99
retrytime = 0
reads = 0
while reads < toread and retrytime < 10:
try:
data = conn.recv(min(32,toread-reads))
if not data: continue
print data
reads += len(data)
except:
retrytime += 1
print "timeout %d" % retrytime
continue
#conn.shutdown(socket.SHUT_RD)#no more read
if reads == toread:
conn.send("OK")
else:
conn.send("NOT OK")
conn.close()

# Echo client program
import socket

HOST = 'localhost' # The remote host
PORT = 50007 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
for i in range(12):
print "time %d" % i
try:
s.send('0123456789')
except socket.error, e:
print "socket error:", e
break
#data = s.recv(1024)
#print "data %d" %i, data
#s.shutdown(socket.SHUT_WR)#no more write

'''
try changing 1 to 0.
'''
if 1:
data=s.recv(1024)
else:
rf = s.makefile("rb")
data = rf.read()
rf.close()
s.close()
print 'Received', repr(data)

The big problem here seems to be that your server is closing the socket
after reading 99 bytes, but the client is actually sending 120. If you
change the "99" in the server to "120" you will see that the client
works when it uses the makefile().read(). I can't be bothered to debug
the exact reason why the 21 bytes of buffered data doesn't cause a
problem with the recv() version, but I wouldn't regard the error you are
getting as a bug in Python, rather as a bug in your application.

The underlying cause of all this appears to be your failure to
understand that network protocols should ideally allow the endpoints to
determine when a complete message has been transmitted, and to consume
all transmitted data.

You can't just ignore some or all of a client request and expect it not
to cause problems. If you look at some of the better-known TCP-based
application protocols you will see they take pains to ensure that
clients and servers can deterministically parse each others' messages.

In summary, a better protocol design will cause you rather less grief
and a little more humility will result in less acidity from crabby old
geeks like me.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------- Hide quoted text -

- Show quoted text -

I don't think you're crabby, Steve.

-- Paul

[ahlongxp]

On Jul 8, 9:54 pm, Steve Holden <[e-mail]:

That's a pretty pejorative subject line for someone who's been
programming Python [guessing by the date of your first post] for about a
month.

I have to admit it that I'm quite a newbie programmer.

Perhaps "Incomprehensible behavior from socket.makefile()", or "I have
written a buggy network application"? That would at least show that you
are considering the possibility you yourself are the cause of the
problem ;-)

Thanks. I'll be more careful about my words.
but I did show some possibility myself is the cause of the problem.
Is there any chance you missed the "?" and "guess" at the same time.

Python has been around for a long time, so you should ask yourself how
likely it is that you would be the first to discover such a fundamental
flaw?

very low but not zero.
Miracle happens.

>I'd be very surprised if someone doesn't point you at an article
on "how to ask smart questions", but I will content myself with that
oblique reference.

The big problem is that, I didn't know such a person like you before.
It's my misfortune.

The big problem here seems to be that your server is closing the socket
after reading 99 bytes, but the client is actually sending 120. If you
change the "99" in the server to "120" you will see that the client
works when it uses the makefile().read(). I can't be bothered to debug
the exact reason why the 21 bytes of buffered data doesn't cause a
problem with the recv() version, but I wouldn't regard the error you are
getting as a bug in Python, rather as a bug in your application.

I don't know the underlying details of networking.
But with recv() version I can get things done as expected while
makefile()
version can't.
I guess it's not that unreasonable to make a guess like I did.

The underlying cause of all this appears to be your failure to
understand that network protocols should ideally allow the endpoints to
determine when a complete message has been transmitted, and to consume
all transmitted data.

You can't just ignore some or all of a client request and expect it not
to cause problems.

Back to the problem, I make it working by adding a sleep(0.5) just
before the
server closes the connection.
Actually this is Hendrik van Rooyen's idea.
And luckily I got a another lesson
http://groups.google.com/group/comp....02772dfe27d8f1

>If you look at some of the better-known TCP-based
application protocols you will see they take pains to ensure that
clients and servers can deterministically parse each others' messages.

I found something useful in RFC2616--Hypertext Transfer Protocol --
HTTP/1.1.
[PAGE49]
- If an origin server receives a request that does not include
an
Expect request-header field with the "100-continue"
expectation,
the request includes a request body, and the server responds
with a final status code before reading the entire request
body
from the transport connection, then the server SHOULD NOT
close
the transport connection until it has read the entire request,
or until the client closes the connection. Otherwise, the
client
might not reliably receive the response message. However, this
requirement is not be construed as preventing a server from
defending itself against denial-of-service attacks, or from
badly broken client implementations.

"Otherwise, the client might not reliably receive the response
message".

My problem seems fixed. But it's not "relialbe" though it's working
pretty
good under windows 2k, windowsxp, and linux.
I may reconsider my design and willprobably try dividing request with
body into two steps like HTTP does.

In summary, a better protocol design will cause you rather less grief
and a little more humility will result in less acidity from crabby old
geeks like me.

I do appreciate your response.
I mean it.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

And last but not least, I' here to be helped and help as long as I
can.
But as you noticed I'm not very good at expressing myself in English.
I didn't mean to offense anyone but I might seem to be rude or
offensive.
I hope you can understand.
--
ahlongxp

Software College,Northeastern University,China
ah******@gmail.com
http://www.herofit.cn

[Steve Holden]

ahlongxp wrote:

On Jul 8, 9:54 pm, Steve Holden <[e-mail]:

>That's a pretty pejorative subject line for someone who's been
programming Python [guessing by the date of your first post] for about a
month.

[...]

And last but not least, I' here to be helped and help as long as I
can.
But as you noticed I'm not very good at expressing myself in English.
I didn't mean to offense anyone but I might seem to be rude or
offensive.
I hope you can understand.

Sure. I felt it necessary to explain my remarks as my own remarks seemed
potentially rude or offensive. Naturally not everyone has English as a
first language, but your first post was good enough that it wasn't
immediately obvious in your case.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

[John J. Lee]

Steve Holden <st***@holdenweb.comwrites:

ahlongxp wrote:

>On Jul 8, 9:54 pm, Steve Holden <[e-mail]:

>>That's a pretty pejorative subject line for someone who's been
programming Python [guessing by the date of your first post] for about a
month.

[...]

>And last but not least, I' here to be helped and help as long as I
can.
But as you noticed I'm not very good at expressing myself in English.
I didn't mean to offense anyone but I might seem to be rude or
offensive.
I hope you can understand.

Sure. I felt it necessary to explain my remarks as my own remarks
seemed potentially rude or offensive. Naturally not everyone has
English as a first language, but your first post was good enough that
it wasn't immediately obvious in your case.

This is the danger in getting too good at another language :-)
ahlongxp, maybe you should drop some deliberate mistakes in your
too-good English ;-)

And FWLIW, I'd be less surprised than Steve at your finding a real bug
in Python's socket code...
John

[ahlongxp]

On Jul 11, 7:51 am, [e-mail] (John J. Lee) wrote:

Steve Holden <[e-mail]:

ahlongxp wrote:

On Jul 8, 9:54 pm, Steve Holden <[e-mail]:
That's a pretty pejorative subject line for someone who's been
programming Python [guessing by the date of your first post] for about a
month.

[...]

And last but not least, I' here to be helped and help as long as I
can.
But as you noticed I'm not very good at expressing myself in English.
I didn't mean to offense anyone but I might seem to be rude or
offensive.
I hope you can understand.

Sure. I felt it necessary to explain my remarks as my own remarks
seemed potentially rude or offensive. Naturally not everyone has
English as a first language, but your first post was good enough that
it wasn't immediately obvious in your case.

This is the danger in getting too good at another language :-)
ahlongxp, maybe you should drop some deliberate mistakes in your
too-good English ;-)

And FWLIW, I'd be less surprised than Steve at your finding a real bug
in Python's socket code...

John

Sorry, I don't think I understand what you are trying to say.
But thanks for your response anyway.

Frank Swarbrick, thank your for your advice. I'll think about it.

--
ahlongxp

Software College,Northeastern University,China
ah******@gmail.com
http://www.herofit.cn