473,289 Members | 1,940 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,289 software developers and data experts.

Re: Problem with writing fast UDP server

On Fri, 21 Nov 2008 00:20:49 -0200, Gabriel Genellina <ga*******@yahoo.com.arwrote:
>En Thu, 20 Nov 2008 14:24:20 -0200, Krzysztof Retel
<Kr*************@googlemail.comescribió:
>>On Nov 20, 4:00Â*pm, bieff...@gmail.com wrote:
>>>On 20 Nov, 16:03, Krzysztof Retel <Krzysztof.Re...@googlemail.com>
wrote:

I am struggling writing fast UDP server. It has to handle around 10000
UDP packets per second. I started building that with non blocking
socket and threads. Unfortunately my approach does not work at all.
I wrote a simple case test: client and server. The client sends 2200
packets within 0.137447118759 secs. The tcpdump received 2189 packets,
which is not bad at all.
But the server only handles 700 -- 870 packets, when it is non-
blocking, and only 670 – 700 received with blocking sockets.
The client and the server are working within the same local network
and tcpdump shows pretty correct amount of packets received.

I wonder if there is a kind of setting for socket to allow no delays?

I've used this script to test sending UDP packets. I've not seen any
delays.

<code>
[snip]
</code>

Start the server before the client.
If you want to try this program out on POSIX, make sure you change the
time.clock() calls to time.time() calls instead, otherwise the results
aren't very meaningful.

I gave this a try on an AMD64 3200+ running a 32 bit Linux installation.
Here's the results I got on the server:

Packet count 91426
Total bytes 8228340 bytes
Total time 8.4 secs
Avg size / packet 90 bytes
Max size / packet 90 bytes
Max time / packet 41070.9 us
Min time / packet 79.9 us
Avg time / packet 92.3 us
Max speed 1100.4 Kbytes/sec
Min speed 2.1 Kbytes/sec
Avg speed 952.0 Kbytes/sec

And on the client:

Packet count 91426
Total bytes 8228340 bytes
Total time 8.4 secs
Avg size / packet 90 bytes
Max size / packet 90 bytes
Max time / packet 40936.0 us
Min time / packet 78.9 us
Avg time / packet 92.3 us
Max speed 1113.7 Kbytes/sec
Min speed 2.1 Kbytes/sec
Avg speed 952.1 Kbytes/sec

Both processes ran on the same machine and communicated over localhost.

For comparison, I tried running the client against a Twisted-based UDP
server. Here are the results from that server:

Packet count 91426
Total bytes 8228340 bytes
Total time 11.8 secs
Avg size / packet 90 bytes
Max size / packet 90 bytes
Max time / packet 55393.9 us
Min time / packet 8.8 us
Avg time / packet 128.7 us
Max speed 9963.2 Kbytes/sec
Min speed 1.6 Kbytes/sec
Avg speed 682.7 Kbytes/sec

This seemed a bit low to me though, so I tried writing an alternate client
and re-ran the measurement. Here are the new server results:

Packet count 91426
Total bytes 8228340 bytes
Total time 2.9 secs
Avg size / packet 90 bytes
Max size / packet 90 bytes
Max time / packet 38193.0 us
Min time / packet 8.8 us
Avg time / packet 32.2 us
Max speed 9963.2 Kbytes/sec
Min speed 2.3 Kbytes/sec
Avg speed 2726.7 Kbytes/sec

And then tried the new client against the original server, with these
results:

Packet count 91426
Total bytes 8228340 bytes
Total time 3.8 secs
Avg size / packet 90 bytes
Max size / packet 90 bytes
Max time / packet 23675.0 us
Min time / packet 6.9 us
Avg time / packet 41.7 us
Max speed 12711.7 Kbytes/sec
Min speed 3.7 Kbytes/sec
Avg speed 2109.0 Kbytes/sec

So it does seem that handling 10k datagrams per second should be no
problem, assuming comparable hardware, at least if whatever work you
have to do to process each one doesn't take more than about 24 25ths
of a millisecond (leaving you the remaining 1 part out of 25 of every
millisecond to receive a packet).

For reference, here's the Twisted UDP client code:

from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor
from twisted.internet.task import LoopingCall

msg = 'xyxabc123' * 10

class EchoClientDatagramProtocol(DatagramProtocol):

def startProtocol(self):
self.transport.connect('127.0.0.1', 8000)
LoopingCall(self.sendDatagrams).start(0.00005)

def sendDatagrams(self):
for i in xrange(50):
self.transport.write(msg)

def main():
protocol = EchoClientDatagramProtocol()
t = reactor.listenUDP(0, protocol)
reactor.run()

if __name__ == '__main__':
main()

And here's the Twisted UDP server code:

from time import time as clock

from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor

class EchoUDP(DatagramProtocol):
history = []
t0 = clock()
def datagramReceived(self, datagram, address):
t1 = clock()
self.history.append((len(datagram), t1 - self.t0))
self.t0 = t1
if len(self.history) == 91427:
self.history.pop(0)
show_stats(self.history)
self.history = []

def main():
reactor.listenUDP(8000, EchoUDP())
reactor.run()

def show_stats(history):
npackets = len(history)
bytes_total = sum([item[0] for item in history])
bytes_avg = float(bytes_total) / npackets
bytes_max = max([item[0] for item in history])
time_total = sum([item[1] for item in history])
time_max = max([item[1] for item in history])
time_min = min([item[1] for item in history])
time_avg = float(time_total) / npackets
speed_max = max([item[0]/item[1] for item in history if item[1]>0])
speed_min = min([item[0]/item[1] for item in history if item[1]>0])
speed_avg = float(bytes_total) / time_total
print "Packet count %8d" % npackets
print "Total bytes %8d bytes" % bytes_total
print "Total time %8.1f secs" % time_total
print "Avg size / packet %8d bytes" % bytes_avg
print "Max size / packet %8d bytes" % bytes_max
print "Max time / packet %8.1f us" % (time_max*1e6)
print "Min time / packet %8.1f us" % (time_min*1e6)
print "Avg time / packet %8.1f us" % (time_avg*1e6)
print "Max speed %8.1f Kbytes/sec" % (speed_max/1024)
print "Min speed %8.1f Kbytes/sec" % (speed_min/1024)
print "Avg speed %8.1f Kbytes/sec" % (speed_avg/1024)
print

if __name__ == '__main__':
main()

Nov 21 '08 #1
1 5162
On Nov 21, 3:52*am, Jean-Paul Calderone <exar...@divmod.comwrote:
>
Start the server before the client.

If you want to try this program out on POSIX, make sure you change the
time.clock() calls to time.time() calls instead, otherwise the results
aren't very meaningful.

I gave this a try on an AMD64 3200+ running a 32 bit Linux installation.
Here's the results I got on the server:
Jean-Paul, thanks very much for the code snippets.
I have tried this out with number of tests. Both approaches are
working fine when I run them on the same machine.
When I run the client (twisted version) on another machine and the udp
server (non twisted) on server machine, I got some strange behaviour.
The client send 255 messages and than it cause an error: socket.error:
(11, 'Resource temporarily unavailable')

Any idea what could be wrong?

Nov 21 '08 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

37
by: Ubaidullah Nubar | last post by:
Hi, How well is Python suited for developing database based applications? I am new to Python so please bear with me if some of the questions are too simple. I specifically have the following...
3
by: Ben Taylor | last post by:
Hi I am trying to build, more for a learning project than anything, an ASP.NET web application in C# that will run on my local machine, which has now successully got IIS installed on it (XP pro)....
9
by: Chibi | last post by:
I'm writing an application that's bandwidth intense. My available bandwidth is 1.5Mbps (cable modem) and I know I can reach around 1.4Mbps, if not more. The application I am writing right now...
29
by: John Rivers | last post by:
Hello, What good reason there is for not allowing methods in ASPX pages I can't imagine, but here is how to get around that limitation: (START) <body MS_POSITIONING="FlowLayout"> <form...
3
by: Froggy / Froggy Corp. | last post by:
First thx for your quick answer :) iostat dont return bad load average. I never grow up to 1Mb/s and the IDE drive can go faster. I really can't change the server, its a locative server which...
35
by: rbt | last post by:
I've been doing some file system benchmarking. In the process, I need to create a large file to copy around to various drives. I'm creating the file like this: fd = file('large_file.bin', 'wb')...
4
by: eselk | last post by:
I've never really setup or used MS SQL Server (just a couple hours, one day, several months ago). I think MS SQL Server has the ability to use "linked tables", like MS Access does. Is this...
6
by: Abandoned | last post by:
Hi.. I use the threading module for the fast operation. But i have some problems.. This is my code sample: ================= conn =...
11
by: Krzysztof Retel | last post by:
Hi guys, I am struggling writing fast UDP server. It has to handle around 10000 UDP packets per second. I started building that with non blocking socket and threads. Unfortunately my approach...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.