performance problem of streaming data over TCP

[Changhao]

Hi, friends,

I am implementing a protocol on top of 'asyncore.dispa tcher' to
send streaming multimedia data over TCP socket. However, I found that
the throughput of my current implementation is surprisingly low.

Below is a snippet of my code with a note that: the packet sent
over the socket is of variable length, of which the first a couple bytes
indicates the length of the packet. I implemented a 'var_str_to_int '
function to get the 'length' and the offset of the payload.

############### ############### ############### ##
def handle_read(sel f):
self.socket.set blocking(1)
while (True) :
data =
self.assemble_m sg()
if (not data) :
self.socket.set blocking(0)
return
(length,index) = var_str_to_int( data)
self._dispatch_ cmd(data[index:])
if (not self.recv_buffe r) :
break
self.socket.set blocking(0)

def assemble_msg(se lf) :
if (self.recv_buff er) :
data = self.recv_buffe r
else :
data = self.socket.rec v(self.BUFFER_S IZE)
if (len(data) == 0) :
return None
length,index = var_str_to_int( data)
while (length +index > len(data)) :
data += self.socket.rec v(self.BUFFER_S IZE)
if (length + index == len(data)) :
self.recv_buffe r = ''
else :
self.recv_buffe r = data[length+index:]
data = data[:length+index]
return data
############### ############### ############### ##

I found it took around 7 seconds to receive a packet of 40KB sent
from localhost. The throughput is even not enough to support any
streaming multimedia file for live playback in a LAN.

I read some threads in the group. It sounds like the problem might
have to do with the TCP option, setting TCP_NODELAY probably could solve
the problem. But I tried the following on both a Linux and a windows XP
machine. Neither of them worked:

############### ############### ############### #
self.socket.set sockopt(socket. IPPROTO_TCP, socket.TCP_NODE LAY, 1)
############### ############### ############### #

Any comments or suggests is highly appreciated.

Thanks,
Changhao

[Changhao]

I figured out the reason. It was because of asyncore.dispat cher's
inefficient implementation of messange sending. Instead of using its
'push' method. I directly call the underlying 'socket.send' and got the
problem solved. Sorry about the spam.

Thanks

Changhao wrote:

Hi, friends,

I am implementing a protocol on top of 'asyncore.dispa tcher' to send
streaming multimedia data over TCP socket. However, I found that the
throughput of my current implementation is surprisingly low.

Below is a snippet of my code with a note that: the packet sent over
the socket is of variable length, of which the first a couple bytes
indicates the length of the packet. I implemented a 'var_str_to_int '
function to get the 'length' and the offset of the payload.

############### ############### ############### ##
def handle_read(sel f):
self.socket.set blocking(1)
while (True) :
data =
self.assemble_m sg()
if (not data) :
self.socket.set blocking(0)
return
(length,index) = var_str_to_int( data)
self._dispatch_ cmd(data[index:])
if (not self.recv_buffe r) :
break
self.socket.set blocking(0)

def assemble_msg(se lf) :
if (self.recv_buff er) :
data = self.recv_buffe r
else :
data = self.socket.rec v(self.BUFFER_S IZE)
if (len(data) == 0) :
return None
length,index = var_str_to_int( data)
while (length +index > len(data)) :
data += self.socket.rec v(self.BUFFER_S IZE)
if (length + index == len(data)) :
self.recv_buffe r = ''
else :
self.recv_buffe r = data[length+index:]
data = data[:length+index]
return data
############### ############### ############### ##

I found it took around 7 seconds to receive a packet of 40KB sent
from localhost. The throughput is even not enough to support any
streaming multimedia file for live playback in a LAN.

I read some threads in the group. It sounds like the problem might
have to do with the TCP option, setting TCP_NODELAY probably could solve
the problem. But I tried the following on both a Linux and a windows XP
machine. Neither of them worked:

############### ############### ############### #
self.socket.set sockopt(socket. IPPROTO_TCP, socket.TCP_NODE LAY, 1)
############### ############### ############### #

Any comments or suggests is highly appreciated.

Thanks,
Changhao

[Fredrik Lundh]

Changhao wrote:

I am implementing a protocol on top of 'asyncore.dispa tcher' to
send streaming multimedia data over TCP socket. However, I found that
the throughput of my current implementation is surprisingly low.

I'm not sure what you think you're doing in your code, but I'm quite
sure that using a home-brewed blocking loop inside an event handler in a
high-performance asynchronous framework isn't exactly the best way to
get good performance. Have you tried using asyncore for asynchronous
communication ?

</F>