473,324 Members | 2,567 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,324 software developers and data experts.

Problems Returning an HTTP 200 Ok Message

Hi Folks,

I'm having some issues with an small socket based server I'm writing,
and I was hoping I could get some help.

My code (attached below) us supposed to read an HTTP Post message
coming from a power meter, parse it, and return a proper HTTP 200 Ok
message. The problem is that the socket fails to send the entire
message as one message, creating a fragmented message which the power
meter then fails to read and accept.

Is there any way to force the socket to send the entire message at
once? Am I doing anything wrong? Is there an easier way to implement
this functionality?

Thanks,

Guy Davidson

Code:

# server application to read data from power meter
# listens on port 80
# receives HTTP POST messages

# author: Maria Kazandjieva <ma******@cs.stanford.edu>, Guy Davidson
<gu********@yahoo.com>
# updated: July 7th, 2008: staring to add database writing
functionality

import time
import datetime #datetime for response message
from socket import * #socket module
import DataPacket #local file with data packet class
import library
import MySQLdb

#formats a proper response message

myHost = ""
myPort = 80

#response = "HTTP/1.1 200 OK\r\nDate: Fri, 20 June 2008 20:40:34 GMT\r
\nServer:SING\r\nX-Powered-By: maria\r\nContent-Length: 7\r
\nConnection:close\r\nContent-Type: text/html\r\n\r\n[0!:20]\n"

#initializing the socket:
s = socket(AF_INET, SOCK_STREAM) # create a TCP socket
s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) # allow port reuse? I hope
s.bind((myHost, myPort)) # bind it to the server port
s.listen(50) # allow 50 simultaneous
pending connections
print('Socket initialized')

#intializing the database
db = MySQLdb.connect(host='localhost', user='powernet',
passwd='jplicks', db='test')
cursor = db.cursor()
print('Database cursor intialized')

while 1:
print('In while 1')
connection, address = s.accept() # connection is a new socket
print('Accepted a connection')

# TODO:
# verify connection is from powernet so we don't get random
connections to port 80

# best way to get the data would be:
# read until we get all the headers
# parse Content header to get length of data
# read for that many additional bytes

while 1:
print('In while 1 2')
data = connection.recv(4096) # receive up to 4K bytes
print('First set of data receieved:')
#print(data)
'''
if data:
print data
#connection.send(response) #send response; OK 200 HTTP message
else:
break
'''
if data:
# second read should recieve the rest of the packet
# finding how long the content is
length = data.find('Content-Length: ')
print('Index of "Content-Length" = ' + str(length))
length = int(data[length+16:length+18])
# adding the bytes from the "Content-Type section:"
length += 51

data += connection.recv(length)
print('Second set of data received:')
print(data)

if data:
#print('Found data:')
#print data #print the string data
packet = DataPacket.DataPacket(data) #use the string data to
create a DataPacket object
print(packet) #print the contents of the object, I hope :)

#we have a packet, let's write it to the mysql DB
packet.mysql_insert(cursor)

#connection.send(response) #send response; OK 200 HTTP
message
print('Sending response')
response = library.response_message() #get the response message
totalsent = 0 #initialize a variable to track how much we've sent
so far
while totalsent < len(response): #while we haven't sent everything
sent = connection.send(response[totalsent:]) #send whatever we
can starting where we stopped
totalsent += sent #increment the count of how much we've sent.

else:
print('No data')
break

else:
print('No data')
break
Jul 10 '08 #1
7 3994
Guy Davidson wrote:
Hi Folks,

I'm having some issues with an small socket based server I'm writing,
and I was hoping I could get some help.

My code (attached below) us supposed to read an HTTP Post message
coming from a power meter, parse it, and return a proper HTTP 200 Ok
message. The problem is that the socket fails to send the entire
message as one message, creating a fragmented message which the power
meter then fails to read and accept.

Is there any way to force the socket to send the entire message at
once? Am I doing anything wrong? Is there an easier way to implement
this functionality?
I don't have a solution to the problem presented, per se, but you might have
an easier time implementing this if you use SimpleHTTPServer or
CGIHTTPServer modules. This will take care of all the socket listening,
and a lot of other low-level details.

Hope that helps!

j

Jul 10 '08 #2
On Jul 10, 1:50*pm, Guy Davidson <GDavids...@gmail.comwrote:
Hi Folks,

I'm having some issues with an small socket based server I'm writing,
and I was hoping I could get some help.

My code (attached below) us supposed to read an HTTP Post message
coming from a power meter, parse it, and return a proper HTTP 200 Ok
message. The problem is that the socket fails to send the entire
message as one message, creating a fragmented message which the power
meter then fails to read and accept.

Is there any way to force the socket to send the entire message at
once? Am I doing anything wrong? Is there an easier way to implement
this functionality?
By 'message', do you mean a single IP datagram? In general, the
answer is no. Each call to 'connection.send()' will (in general, see
the next paragraph) transmit as much data as will fit into a single IP
datagram, given the current MTU for the transmission circuit. The
fact that you're calling it in a loop indicates that the data being
sent may be larger than will fit into a datagram.

Or, by 'message', do you mean a single TCP segment? Again, the answer
is no. Your network stack will try to make the TCP segments the right
size to fit within a single IP datagram, leading to the same result as
above.

From your description, I get the feeling that your power meter has a
broken network stack, and you're trying to program around it. You
need to repair the meter.
Jul 10 '08 #3
On Jul 10, 12:38*pm, samwyse <samw...@gmail.comwrote:
On Jul 10, 1:50*pm, Guy Davidson <GDavids...@gmail.comwrote:
Hi Folks,
I'm having some issues with an small socket based server I'm writing,
and I was hoping I could get some help.
My code (attached below) us supposed to read an HTTP Post message
coming from a power meter, parse it, and return a proper HTTP 200 Ok
message. The problem is that the socket fails to send the entire
message as one message, creating a fragmented message which the power
meter then fails to read and accept.
Is there any way to force the socket to send the entire message at
once? Am I doing anything wrong? Is there an easier way to implement
this functionality?

By 'message', do you mean a single IP datagram? *In general, the
answer is no. *Each call to 'connection.send()' will (in general, see
the next paragraph) transmit as much data as will fit into a single IP
datagram, given the current MTU for the transmission circuit. *The
fact that you're calling it in a loop indicates that the data being
sent may be larger than will fit into a datagram.

Or, by 'message', do you mean a single TCP segment? *Again, the answer
is no. *Your network stack will try to make the TCP segments the right
size to fit within a single IP datagram, leading to the same result as
above.

From your description, I get the feeling that your power meter has a
broken network stack, and you're trying to program around it. *You
need to repair the meter.
Here's the weird thing. I know it should be able to be done.

Let me try and explain in some more depth what I'm trying to do:

The meter sends HTTP Post messages, and expects, as a reply, an HTTP
200/Ok message.

I try to send the following message, using the socket.send() command:

'HTTP/1.1 200 OK\r\nDate: Thu, 10 July 2008 14:07:50 GMT\r\nServer:
Apache/2.2.8 (Fedora)\r\nX-Powered-By: PHP/5.2.4\r\nContent-Length: 4\r
\nConnection: close\r\nContent-Type: text/html; charset=UTF-8\r\n\r
\n[0]\n'

However, when I snoop on the packets in wireshark, here's what I see:

HTTP/1.1 200 Ok:

HTTP/1.1 200 OK
Date: Wed, 09 July 2008 14:55:50 GMT
Server: Apache/2.2.8 (Fedora)
X-Powered-By:

Continuation or non-HTTP traffic:

PHP/5.2.4
Content-Length: 4
Connection: close
Content-Type: text/html; charset=UTF-8

[0]

It splits into two packages, which the meter can't read, and the
communication breaks down there.

Any ideas?

Thanks.
Jul 10 '08 #4
En Thu, 10 Jul 2008 18:10:46 -0300, Guy Davidson <GD********@gmail.com>
escribi�:
On Jul 10, 12:38Â*pm, samwyse <samw...@gmail.comwrote:
>On Jul 10, 1:50Â*pm, Guy Davidson <GDavids...@gmail.comwrote:
My code (attached below) us supposed to read an HTTP Post message
coming from a power meter, parse it, and return a proper HTTP 200 Ok
message. The problem is that the socket fails to send the entire
message as one message, creating a fragmented message which the power
meter then fails to read and accept.

From your description, I get the feeling that your power meter has a
broken network stack, and you're trying to program around it. Â*You
need to repair the meter.

The meter sends HTTP Post messages, and expects, as a reply, an HTTP
200/Ok message.

I try to send the following message, using the socket.send() command:

'HTTP/1.1 200 OK\r\nDate: Thu, 10 July 2008 14:07:50 GMT\r\nServer:
Apache/2.2.8 (Fedora)\r\nX-Powered-By: PHP/5.2.4\r\nContent-Length: 4\r
\nConnection: close\r\nContent-Type: text/html; charset=UTF-8\r\n\r
\n[0]\n'

However, when I snoop on the packets in wireshark, here's what I see:

HTTP/1.1 200 Ok:

HTTP/1.1 200 OK
Date: Wed, 09 July 2008 14:55:50 GMT
Server: Apache/2.2.8 (Fedora)
X-Powered-By:

Continuation or non-HTTP traffic:

PHP/5.2.4
Content-Length: 4
Connection: close
Content-Type: text/html; charset=UTF-8

[0]

It splits into two packages, which the meter can't read, and the
communication breaks down there.

Any ideas?
As Guy Davidson has already pointed out, this is a problem in the meter
TCP implementation, and you should ask the vendor to fix it. (Anyway,
looks like "somewhere" there is a buffer size of 100 bytes or so, very
small).

As a workaround, try to shorten your HTTP response; I guess the Server and
X-Powered-By headers are not required; the Date probably isn't either; and
if all your responses are like "[0]" a simple "Content-Type: text/plain"
may suffice (the meter might just ignore it, anyway).

--
Gabriel Genellina

Jul 11 '08 #5
On Jul 10, 4:10*pm, Guy Davidson <GDavids...@gmail.comwrote:
I try to send the following message, using the socket.send() command:

'HTTP/1.1 200 OK\r\nDate: Thu, 10 July 2008 14:07:50 GMT\r\nServer:
Apache/2.2.8 (Fedora)\r\nX-Powered-By: PHP/5.2.4\r\nContent-Length: 4\r
\nConnection: close\r\nContent-Type: text/html; charset=UTF-8\r\n\r
\n[0]\n'

However, when I snoop on the packets in wireshark, here's what I see:

HTTP/1.1 200 Ok:

HTTP/1.1 200 OK
Date: Wed, 09 July 2008 14:55:50 GMT
Server: Apache/2.2.8 (Fedora)
X-Powered-By:

Continuation or non-HTTP traffic:

PHP/5.2.4
Content-Length: 4
Connection: close
Content-Type: text/html; charset=UTF-8

[0]

It splits into two packages, which the meter can't read, and the
communication breaks down there.
OK, it looks like a single TCP segment is being sent by you (which is
consistent with only one socket.send() command being needed), but
something along the way to the meter is using an MTU (Maximum
Transmission Unit) of about 100 bytes, producing two IP datagrams.
How many hops are there between the PC and the meter? Are you
sniffing on the same subnet as the PC, the meter, or somewhere in
between? MTU is normally set to about 1500 (and MSS is generally
MTU-40), but you can generally change these values.

You should be able to set the do-not-fragment flag on your IP packets,
but that may cause them to get dropped instead of sent. You could
also try setting a smaller ICP MSS (Maximum Segment Size), which the
meter might be able to assemble, even if it can't handle fragmented IP
datagrams. Try http://www.cisco.com/en/US/tech/tk87...html#prob_desc
for more help.

You really need to get a networking guru involved if you want to go
down this path. I used to be one, but that was years ago. Or, you
can take Gabriel Genellina's advice and try coding smaller responses.
Jul 14 '08 #6
On Jul 11, 3:46*am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
wrote:
As Guy Davidson has already pointed out, this is a problem in the meter *
TCP implementation, and you should ask the vendor to fix it.
That would have been me, not Guy.
Jul 14 '08 #7
En Mon, 14 Jul 2008 17:46:12 -0300, samwyse <sa*****@gmail.comescribió:
On Jul 11, 3:46*am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
wrote:
>As Guy Davidson has already pointed out, this is a problem in the meter
That would have been me, not Guy.
Indeed, sorry the misattribution!

--
Gabriel Genellina

Jul 15 '08 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Pete | last post by:
I've run into an interesting memory problem. I think I'm running out of heap space, but I'm not sure.... I'm creating two new arrays like such.... pImage = new UBYTE ; pImageTemp = new...
4
by: Mark Anderson | last post by:
Sorry if this is a rookie mistake... I've been through all the FAQs and the books I have but I can't see the mistake so I guess it's something simple <g> - I'm an occasional JS user. I've got...
14
by: Dave | last post by:
Hello all, Can anybody help with the problem below? I'm trying to define a conversion function that converts objects to function pointers and am getting a compile error on the line indicated...
10
by: BBFrost | last post by:
We just recently moved one of our major c# apps from VS Net 2002 to VS Net 2003. At first things were looking ok, now problems are starting to appear. So far ... (1) ...
7
by: Railinc | last post by:
Hi... I'm trying to write a web service client and am running into a problem with getting a response back from the server. I know that a full xml message is being returned (thanks to a tcp/ip...
21
by: Doug Lerner | last post by:
I'm working on a client/server app that seems to work fine in OS Firefox and Windows IE and Firefox. However, in OS X Safari, although the UI/communications themselves work fine, if the...
11
by: Steve Smith | last post by:
I have written a winforms application that launches approximately 150 threads with Thread.ThreadStart() Each thread uses CDO 1.21 to logon to a different Exchange mailbox and send/receive a...
36
by: Chuck Faranda | last post by:
I'm trying to debug my first C program (firmware for PIC MCU). The problem is getting serial data back from my device. My get commands have to be sent twice for the PIC to respond properly with...
10
by: connyledin | last post by:
Im trying to create a version of the game Wumpus. Mine is called Belzebub. But im STUCK! And its due tuesday 2 maj. Im panicing! Can some one help me?? here is the file:...
9
by: Frank | last post by:
Hi, imagine there's a WEB application reading data from an Oracle database to visualize in using DataGrids in the clients browser. Yes, sounds simple, just create OracleConnection + OracleCommand...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.