Problems Returning an HTTP 200 Ok Message

[Guy Davidson]

Hi Folks,

I'm having some issues with an small socket based server I'm writing,
and I was hoping I could get some help.

My code (attached below) us supposed to read an HTTP Post message
coming from a power meter, parse it, and return a proper HTTP 200 Ok
message. The problem is that the socket fails to send the entire
message as one message, creating a fragmented message which the power
meter then fails to read and accept.

Is there any way to force the socket to send the entire message at
once? Am I doing anything wrong? Is there an easier way to implement
this functionality?

Thanks,

Guy Davidson

Code:

# server application to read data from power meter
# listens on port 80
# receives HTTP POST messages

# author: Maria Kazandjieva <ma******@cs.st anford.edu>, Guy Davidson
<gu********@yah oo.com>
# updated: July 7th, 2008: staring to add database writing
functionality

import time
import datetime #datetime for response message
from socket import * #socket module
import DataPacket #local file with data packet class
import library
import MySQLdb

#formats a proper response message

myHost = ""
myPort = 80

#response = "HTTP/1.1 200 OK\r\nDate: Fri, 20 June 2008 20:40:34 GMT\r
\nServer:SING\r \nX-Powered-By: maria\r\nConten t-Length: 7\r
\nConnection:cl ose\r\nContent-Type: text/html\r\n\r\n[0!:20]\n"

#initializing the socket:
s = socket(AF_INET, SOCK_STREAM) # create a TCP socket
s.setsockopt(SO L_SOCKET, SO_REUSEADDR, 1) # allow port reuse? I hope
s.bind((myHost, myPort)) # bind it to the server port
s.listen(50) # allow 50 simultaneous
pending connections
print('Socket initialized')

#intializing the database
db = MySQLdb.connect (host='localhos t', user='powernet' ,
passwd='jplicks ', db='test')
cursor = db.cursor()
print('Database cursor intialized')

while 1:
print('In while 1')
connection, address = s.accept() # connection is a new socket
print('Accepted a connection')

# TODO:
# verify connection is from powernet so we don't get random
connections to port 80

# best way to get the data would be:
# read until we get all the headers
# parse Content header to get length of data
# read for that many additional bytes

while 1:
print('In while 1 2')
data = connection.recv (4096) # receive up to 4K bytes
print('First set of data receieved:')
#print(data)
'''
if data:
print data
#connection.sen d(response) #send response; OK 200 HTTP message
else:
break
'''
if data:
# second read should recieve the rest of the packet
# finding how long the content is
length = data.find('Cont ent-Length: ')
print('Index of "Content-Length" = ' + str(length))
length = int(data[length+16:lengt h+18])
# adding the bytes from the "Content-Type section:"
length += 51

data += connection.recv (length)
print('Second set of data received:')
print(data)

if data:
#print('Found data:')
#print data #print the string data
packet = DataPacket.Data Packet(data) #use the string data to
create a DataPacket object
print(packet) #print the contents of the object, I hope :)

#we have a packet, let's write it to the mysql DB
packet.mysql_in sert(cursor)

#connection.sen d(response) #send response; OK 200 HTTP
message
print('Sending response')
response = library.respons e_message() #get the response message
totalsent = 0 #initialize a variable to track how much we've sent
so far
while totalsent < len(response): #while we haven't sent everything
sent = connection.send (response[totalsent:]) #send whatever we
can starting where we stopped
totalsent += sent #increment the count of how much we've sent.

else:
print('No data')
break

else:
print('No data')
break

[Joshua Kugler]

Guy Davidson wrote:

Hi Folks,

I'm having some issues with an small socket based server I'm writing,
and I was hoping I could get some help.

My code (attached below) us supposed to read an HTTP Post message
coming from a power meter, parse it, and return a proper HTTP 200 Ok
message. The problem is that the socket fails to send the entire
message as one message, creating a fragmented message which the power
meter then fails to read and accept.

Is there any way to force the socket to send the entire message at
once? Am I doing anything wrong? Is there an easier way to implement
this functionality?

I don't have a solution to the problem presented, per se, but you might have
an easier time implementing this if you use SimpleHTTPServe r or
CGIHTTPServer modules. This will take care of all the socket listening,
and a lot of other low-level details.

Hope that helps!

j

[samwyse]

On Jul 10, 1:50*pm, Guy Davidson <GDavids...@gma il.comwrote:

Hi Folks,

I'm having some issues with an small socket based server I'm writing,
and I was hoping I could get some help.

My code (attached below) us supposed to read an HTTP Post message
coming from a power meter, parse it, and return a proper HTTP 200 Ok
message. The problem is that the socket fails to send the entire
message as one message, creating a fragmented message which the power
meter then fails to read and accept.

Is there any way to force the socket to send the entire message at
once? Am I doing anything wrong? Is there an easier way to implement
this functionality?

By 'message', do you mean a single IP datagram? In general, the
answer is no. Each call to 'connection.sen d()' will (in general, see
the next paragraph) transmit as much data as will fit into a single IP
datagram, given the current MTU for the transmission circuit. The
fact that you're calling it in a loop indicates that the data being
sent may be larger than will fit into a datagram.

Or, by 'message', do you mean a single TCP segment? Again, the answer
is no. Your network stack will try to make the TCP segments the right
size to fit within a single IP datagram, leading to the same result as
above.

From your description, I get the feeling that your power meter has a
broken network stack, and you're trying to program around it. You
need to repair the meter.

[Guy Davidson]

On Jul 10, 12:38*pm, samwyse <samw...@gmail. comwrote:

On Jul 10, 1:50*pm, Guy Davidson <GDavids...@gma il.comwrote:

Hi Folks,

I'm having some issues with an small socket based server I'm writing,
and I was hoping I could get some help.

My code (attached below) us supposed to read an HTTP Post message
coming from a power meter, parse it, and return a proper HTTP 200 Ok
message. The problem is that the socket fails to send the entire
message as one message, creating a fragmented message which the power
meter then fails to read and accept.

Is there any way to force the socket to send the entire message at
once? Am I doing anything wrong? Is there an easier way to implement
this functionality?

By 'message', do you mean a single IP datagram? *In general, the
answer is no. *Each call to 'connection.sen d()' will (in general, see
the next paragraph) transmit as much data as will fit into a single IP
datagram, given the current MTU for the transmission circuit. *The
fact that you're calling it in a loop indicates that the data being
sent may be larger than will fit into a datagram.

Or, by 'message', do you mean a single TCP segment? *Again, the answer
is no. *Your network stack will try to make the TCP segments the right
size to fit within a single IP datagram, leading to the same result as
above.

From your description, I get the feeling that your power meter has a
broken network stack, and you're trying to program around it. *You
need to repair the meter.

Here's the weird thing. I know it should be able to be done.

Let me try and explain in some more depth what I'm trying to do:

The meter sends HTTP Post messages, and expects, as a reply, an HTTP
200/Ok message.

I try to send the following message, using the socket.send() command:

'HTTP/1.1 200 OK\r\nDate: Thu, 10 July 2008 14:07:50 GMT\r\nServer:
Apache/2.2.8 (Fedora)\r\nX-Powered-By: PHP/5.2.4\r\nConten t-Length: 4\r
\nConnection: close\r\nConten t-Type: text/html; charset=UTF-8\r\n\r
\n[0]\n'

However, when I snoop on the packets in wireshark, here's what I see:

HTTP/1.1 200 Ok:

HTTP/1.1 200 OK
Date: Wed, 09 July 2008 14:55:50 GMT
Server: Apache/2.2.8 (Fedora)
X-Powered-By:

Continuation or non-HTTP traffic:

PHP/5.2.4
Content-Length: 4
Connection: close
Content-Type: text/html; charset=UTF-8

[0]

It splits into two packages, which the meter can't read, and the
communication breaks down there.

Any ideas?

Thanks.

[Gabriel Genellina]

En Thu, 10 Jul 2008 18:10:46 -0300, Guy Davidson <GD********@gma il.com>
escribi�:

On Jul 10, 12:38Â*pm, samwyse <samw...@gmail. comwrote:

>On Jul 10, 1:50Â*pm, Guy Davidson <GDavids...@gma il.comwrote:

My code (attached below) us supposed to read an HTTP Post message
coming from a power meter, parse it, and return a proper HTTP 200 Ok
message. The problem is that the socket fails to send the entire
message as one message, creating a fragmented message which the power
meter then fails to read and accept.

From your description, I get the feeling that your power meter has a
broken network stack, and you're trying to program around it. Â*You
need to repair the meter.

The meter sends HTTP Post messages, and expects, as a reply, an HTTP
200/Ok message.

I try to send the following message, using the socket.send() command:

'HTTP/1.1 200 OK\r\nDate: Thu, 10 July 2008 14:07:50 GMT\r\nServer:
Apache/2.2.8 (Fedora)\r\nX-Powered-By: PHP/5.2.4\r\nConten t-Length: 4\r
\nConnection: close\r\nConten t-Type: text/html; charset=UTF-8\r\n\r
\n[0]\n'

However, when I snoop on the packets in wireshark, here's what I see:

HTTP/1.1 200 Ok:

HTTP/1.1 200 OK
Date: Wed, 09 July 2008 14:55:50 GMT
Server: Apache/2.2.8 (Fedora)
X-Powered-By:

Continuation or non-HTTP traffic:

PHP/5.2.4
Content-Length: 4
Connection: close
Content-Type: text/html; charset=UTF-8

[0]

It splits into two packages, which the meter can't read, and the
communication breaks down there.

Any ideas?

As Guy Davidson has already pointed out, this is a problem in the meter
TCP implementation, and you should ask the vendor to fix it. (Anyway,
looks like "somewhere" there is a buffer size of 100 bytes or so, very
small).

As a workaround, try to shorten your HTTP response; I guess the Server and
X-Powered-By headers are not required; the Date probably isn't either; and
if all your responses are like "[0]" a simple "Content-Type: text/plain"
may suffice (the meter might just ignore it, anyway).

--
Gabriel Genellina

[samwyse]

On Jul 10, 4:10*pm, Guy Davidson <GDavids...@gma il.comwrote:

I try to send the following message, using the socket.send() command:

'HTTP/1.1 200 OK\r\nDate: Thu, 10 July 2008 14:07:50 GMT\r\nServer:
Apache/2.2.8 (Fedora)\r\nX-Powered-By: PHP/5.2.4\r\nConten t-Length: 4\r
\nConnection: close\r\nConten t-Type: text/html; charset=UTF-8\r\n\r
\n[0]\n'

However, when I snoop on the packets in wireshark, here's what I see:

HTTP/1.1 200 Ok:

HTTP/1.1 200 OK
Date: Wed, 09 July 2008 14:55:50 GMT
Server: Apache/2.2.8 (Fedora)
X-Powered-By:

Continuation or non-HTTP traffic:

PHP/5.2.4
Content-Length: 4
Connection: close
Content-Type: text/html; charset=UTF-8

[0]

It splits into two packages, which the meter can't read, and the
communication breaks down there.

OK, it looks like a single TCP segment is being sent by you (which is
consistent with only one socket.send() command being needed), but
something along the way to the meter is using an MTU (Maximum
Transmission Unit) of about 100 bytes, producing two IP datagrams.
How many hops are there between the PC and the meter? Are you
sniffing on the same subnet as the PC, the meter, or somewhere in
between? MTU is normally set to about 1500 (and MSS is generally
MTU-40), but you can generally change these values.

You should be able to set the do-not-fragment flag on your IP packets,
but that may cause them to get dropped instead of sent. You could
also try setting a smaller ICP MSS (Maximum Segment Size), which the
meter might be able to assemble, even if it can't handle fragmented IP
datagrams. Try http://www.cisco.com/en/US/tech/tk87...html#prob_desc
for more help.

You really need to get a networking guru involved if you want to go
down this path. I used to be one, but that was years ago. Or, you
can take Gabriel Genellina's advice and try coding smaller responses.

[samwyse]

On Jul 11, 3:46*am, "Gabriel Genellina" <[e-mail].a r>
wrote:

As Guy Davidson has already pointed out, this is a problem in the meter *
TCP implementation, and you should ask the vendor to fix it.

That would have been me, not Guy.

[Gabriel Genellina]

En Mon, 14 Jul 2008 17:46:12 -0300, samwyse <sa*****@gmail. comescribió:

On Jul 11, 3:46*am, "Gabriel Genellina" <[e-mail].a r>
wrote:

>As Guy Davidson has already pointed out, this is a problem in the meter

That would have been me, not Guy.

Indeed, sorry the misattribution!

--
Gabriel Genellina