473,574 Members | 3,249 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Broadcast server

I would like to write a server with the low level API of python (
socket+select and/or socket+thread ) that allow me to register client
and update them every X seconds ( could be the time, the temperature, a
stock quote, a message , ... ).

How to write the server that keep hot connections with clients and
update them when events are trigerred. I don't know how to begin this ,
i look at the python doc but the doc is more related to client updating
the server and i am not sure of the right design that could be use
here.

Thx
Manu

Aug 31 '06 #1
8 3214
On Thu, 31 Aug 2006 10:14:15 +0100, <sw***@netcourr ier.comwrote:
I would like to write a server with the low level API of python (
socket+select and/or socket+thread ) that allow me to register client
and update them every X seconds ( could be the time, the temperature, a
stock quote, a message , ... ).

How to write the server that keep hot connections with clients and
update them when events are trigerred. I don't know how to begin this ,
i look at the python doc but the doc is more related to client updating
the server and i am not sure of the right design that could be use
here.
Kamaelia ( http://kamaelia.sf.net/ ) - the project I'm working on, feels
like it may be well suited to this kind of task. Its based around having
individual components that pass messages to each other. So you can
envisage an event source propogating a message to all subscribed clients
quite easily.

Your source's of events can be components, that publish their events to a
'backplane' - a kind of distribution board that replicates messages sent
to it to all subscribers.

from Axon.Component import component
from Kamaelia.Util.B ackplane import Backplane, subscribeTo, publishTo
from Kamaelia.Util.P ipelineComponen t import pipeline

class EventSource(com ponent):
def main(self):
while 1:
if event_happens:
self.send( "EVENT HAPPENED!\n", "outbox")
yield 1

pipeline( EventSource(),
publishTo("Even ts"),
).activate()

Backplane("Even ts").activate ()

You can have any number of event sources, all 'publishing' to the same
backplane.

Then to handle clients connecting I'd create a server component, with a
factory function that creates a component to handle each client
connection. In this case, a component that 'subscribes' to messages coming
from the backplane:

from Kamaelia.Chassi s.ConnectedServ er import SimpleServer

def clientHandlerFa ctory():
return subscribeTo("Ev ents")

SimpleServer(cl ientHandlerFact ory).activate()

Then I'd start the system running:

from Axon.Scheduler import scheduler

scheduler.run.r unThreads()

Things like the client handling could be extended relatively easily to
marshall complex data types to strings, suitable for network transmission;
or to manage clients choosing what data sources to subscribe to.

If you think this approach might work for you, I'm happy to give you a
hand with getting Kamaelia up and running, and with learning how to write
your own components to make the system more sophisticated.

We hang about on #kamaelia on freenode (irc) - please do drop in for a
chat!

Hope this helps!
Matt
--
| Matt Hammond
| Research Engineer, Tech. Group, BBC, Kingswood Warren, Tadworth, Surrey,
UK
| http://kamaelia.sf.net/
| http://www.bbc.co.uk/rd/
Aug 31 '06 #2
Hi Matt ,
It sounds very interesting and i will definitely take a deeper look
but i was more considering something simpler as a learning exercise
more that finding a package that should already do what i want to
achieve.
I want to find the basic guidelines to write that kind of client/server
architecture and make some tests of different solutions(socke t+select
or socket+thread or mix of them)

Your project seems very great and will try to read in to learn.

Aug 31 '06 #3
sw***@netcourri er.com wrote:
I would like to write a server with the low level API of python (
socket+select and/or socket+thread ) that allow me to register client
and update them every X seconds ( could be the time, the temperature, a
stock quote, a message , ... ).

How to write the server that keep hot connections with clients and
update them when events are trigerred. I don't know how to begin this ,
i look at the python doc but the doc is more related to client updating
the server and i am not sure of the right design that could be use
here.
I'd suggest to find the Richard W. Stevens book "Unix Network Programing".
You'll learn a lot about networking from that book. It's based on C, but
you'll see that the concepts can be easily used with Python too.

Find that book, and don't forget the stdlib docs:
http://docs.python.org/lib/module-socket.html
http://docs.python.org/lib/module-select.html

--
damjan
Sep 1 '06 #4
Yes but before going deeper ( too deep ) i would like to play with a
toy python example. Basically what i want to study first is a socket or
socket+select server and 5 clients that get time updated from the
server.

Don't really know how to keep hot connections without blocking the
server ? If someone can show me a quick and dirty example of the
client/server i'll apprciate it a lot.

Thx
Manu

Sep 1 '06 #5
sw***@netcourri er.com wrote:
It sounds very interesting and i will definitely take a deeper look
but i was more considering something simpler as a learning exercise
more that finding a package that should already do what i want to
achieve.
The bulk of this post dissects (roughly) how Kamaelia handles this
internally since I figure that might also be useful if you're after
this as a learning exercise :-) (It should hopefully be clear enough
for this to be useful) At minimum it might go some way to explain why
people generally tend to suggest using an existing framework :-)
I want to find the basic guidelines to write that kind of client/server
architecture and make some tests of different solutions(socke t+select
or socket+thread or mix of them)
I think if you're doing socket & select it sounds easy/simple until you
start dealing with all the cases you need to to do it properly. You're
probably best looking at the python cookbook for a good initial starting
point.
Your project seems very great and will try to read in to learn.
Kamaelia *might* still be useful here in some respects, since despite being
*effectively* event based, due to the use of generators code tends to look
linear.

The low level structure of the server breaks down as follows:

(I'm spending a bit of time here since it might be of general interest,
sorry if it isn't)

* Kamaelia.Intern et.TCPServer is a simple component that sits waiting for
a user to connect.

It makes the server port, adds any user socket options, makes it non
blocking, binds it to the requested (if any) host and either to a user
defined or random port number. (latter is for testing)

It then registers the socket with the piece of code that's taking care
of select handling, and sits to wait for any notifications of activity
on the socket. (Which would mean for something acting as a server a new
connection).

It then performs a socket accept, and creates an object to handle the
raw (now connected) socket - a connected socket adaptor. It then
registers that socket and object with the piece of code that's taking
care of select handling. (So that the select handling can tell the
piece of code handling the raw socket to do some work on the socket)

code: http://tinyurl.com/pocm8

* Kamaelia.Intern et.ConnectedSoc ketAdaptor is actually more complicated
than you might expect. The reason is because the socket is
non-blocking, and where-ever you would normally block you have
to maintain some sort of buffering in case the action fails.
This is more a symptom of non-blocking behaviour than it is about
Kamaelia.

Anyhow the basics of this is to take any data that is to be sent to the
socket, and add that to a sending queue. (this data will generally
come from the code handling the protocol)

Then if the select handling code has said that we can send/should data,
we take data from the send queue and send as much as we can until our
sending fails (after all, we're non-blocking, so we expect it to fail).

Then if the select handling code has said that there's data ready to
receive the component reads from the socket until an error (again we
expect an error due to non-blocking behaviour). This data is then
passed on to a location where a protocol handler can expect to take the
data and do what it likes with it.

The bulk of the code for the CSA actually revolves around resends, and
the necessary error handling. (It's also one of the oldest bits of code
in Kamaelia so possibly a little horrid to look at!)

code: http://tinyurl.com/nj8lk

* Kamaelia.Intern et.Selector is also slightly more complex than you might
think. This is because whilst you tell select "let me know whenever any
of these sockets is ready for (reading|writin g|exceptional)" , it
actually cares about what you do *after* it's given you an answer.

For example if you don't read from all the things that are ready to
read, or don't write to all the things that are ready to write or
(worse) try asking it to do things with regard to closed or invalid
sockets, then you're just asking for trouble.

There's lots of ways round this and you can do something like:
looping:
(readables, writeables, excepts) = select( ...)
for readable in readables:
read from readable and do something
for writable in writeables:
write to writeables and do something
for exceptional in excepts:
handle exceptional

However then you add new problems - what about if you want to do a
database request as result of one? Render a image for another? Wait for
keyboard press? etc.

As a result that's why we use explicit message passing, though many
people use a reactor (or sometimes a pro-actor) pattern to handle
events. (if you're familiar with simulation it's very similar to an
event wheel approach).

For us we simply have a service (a selector) that sits an waits to be
told "send me a message when this is ready to do X". Since the thing
handling select can't know (in our system because we don't want any
accidental serialisation) whether the event (socket read to ready,
socket ready to write) has been handled, our selector removes the
socket from the read/write/exceptional set when it's been found to be
ready to do something.

This means that when the thing using has read from the socket it tells
the selector "can I have more please", and if it's written to the
socket has to tell the selector "I want to do more, but it bust - can
you tell me when it's working again".

This means the code managing the select loop and the socket sets the
select call operates on has to deal with all these issues. (You'd have
to anyway, but in some respects it's more explicit at the socket-level
in Kamaelia)

As a result you'll see explicit code adding sockets to "readers",
"writers", and "exceptiona ls", and for removing sockets from them when
an event's happened. This puts responsibility for making this work with
the code that cares - the code handling the socket activity directly.
The rationale for this is that it then also means that the same code
that's used for sockets can be used for file handles. (one of the
points of select on unix after all)

code: http://tinyurl.com/mfqyr

For completeness:
* Kamaelia.Chassi s.ConnectedServ er.SimpleServer takes the above
components and packages them up in an easy to use way. (meaning you can
avoid all the lowlevel hassle).

It's called a Chassis because like you add components onto a car
chassis (wheels, seats, doors, engine) in order to make a car, you need
to add someting to the SimpleServer to make a server. Specifically you
need to give it something that can create components to handle
connections.

As a result, the way SimpleServer works is whenever a new connection
comes in, it gets told about it by being passed a connected socket
adaptor by the tcp server. It then creates a protocol handler to talk
to the connected socket adaptor. As a result, any information that
comes in from the socket gets passed to the protocol handler. Any
information the protocol handler generates gets sent out the socket.

As a result the logic for the SimpleServer is:
* Create the TCP Server
* Loop
* When the TCP Server gives us a connected socket adaptor,
create a protocol handler (from the given function), and wire
the two together.
* If the connection dies (due to the protocol handler saying
"shutdown" or due to the socket dying) the everyone interested
finds out and propogates the change. (Meaning you don't get
dead file descriptors in the select call)
code: http://tinyurl.com/pts72

You'll note that the explanation of select itself is by far the longest
here! You may also find "man select_tut" interesting and useful if you're
doing this as a learning exercise.

If you're looking to do this more generally, you should really consider
using SocketServer or asyncore that come with python, or Twisted or Kamelia
if you want to run a production system.

Twisted is more well known and more deployed and as a result more battle
tested (meaning seasoned programmers would sensibly trust it more!), and
follows a more common approach for coding all this. (Boils down to something
similar though) As a result if hiring people to work on code is something
to think about Twisted operates is probably a better choice. Kamaelia is
however designed to be easy to pick up.

Finally nipping back to your example here:
5 clients that get time updated from the server
The code to handle on the server side is trivial.

----(start)----
import time
from Axon.ThreadedCo mponent import threadedcompone nt
from Kamaelia.Chassi s.Pipeline import Pipeline
from Kamaelia.Chassi s.ConnectedServ er import SimpleServer
from Kamaelia.Util.S tringify import Stringify
from Kamaelia.Util.B ackplane import *

class periodictime(th readedcomponent ):
def main(self):
while 1:
time.sleep(0.1)
self.send(time. time(), "outbox")

Backplane("peri odictime").acti vate()
Pipeline(period ictime(),
Stringify(), # Make suitable for network
PublishTo("peri odictime")).act ivate()

def getTimeProtocol ():
return SubscribeTo("pe riodictime")

SimpleServer(pr otocol=getTimeP rotocol, port=1600).run( )
----(finish)----

Telnet to 127.0.0.1 1600 to see the result here.

If you want to have the data source something from a client (eg event info
coming in on port 1599) with clients of the event source coming in from
elsewhere, this would change the server as follows:

----(start)----
from Kamaelia.Chassi s.Pipeline import Pipeline
from Kamaelia.Chassi s.ConnectedServ er import SimpleServer
from Kamaelia.Util.B ackplane import *

Backplane("peri odictime").acti vate()

def publishTime(): return PublishTo("peri odictime")
def getTimeProtocol (): return SubscribeTo("pe riodictime")

SimpleServer(pr otocol=publishT ime, port=1599).run( )
SimpleServer(pr otocol=getTimeP rotocol, port=1600).run( )
----(finish)----

And the time (event) source would look like this:

----(start)----
import time
from Kamaelia.Intern et.TCPClient import TCPClient
from Axon.ThreadedCo mponent import threadedcompone nt
from Kamaelia.Chassi s.Pipeline import Pipeline
from Kamaelia.Util.S tringify import Stringify

class periodictime(th readedcomponent ):
def main(self):
while 1:
time.sleep(0.1)
self.send(time. time(), "outbox")

perdiodictimese rver = "127.0.0.1"
Pipeline(period ictime(),
Stringify(), # Make suitable for network
TCPClient(perdi odictimeserver, 1599)).activate ()
----(finish)----

Anyway, I hope the explanation of what's going on inside the core is useful
since in many respects if you're writing your own select handling loop
(which I would encourage you to do if you're learning about this!), the
basics of what you have to do stay the same. (check activity, clear, when
errors happen ask again, buffer data which needs to get sent, and decouple
everything as best as makes sense whilst trying to avoid accidental
serialisations) .

The reason for the examples in the end is merely for completeness. (I'll
probably add these to our SVN distribution since the question does seem to
crop up fairly often generally speaking!)

If you're looking to do this in a production environment I'm personally an
advocate of learning what's going on in the core and then using an existing
library.

(The reason Kamaelia exists is because I wondered if there was an
alternative, potentially clearer way of writing these things, most
people would quite sensibly just use Twisted - especially given you
can buy a book on it! I personally think Kamaelia is cleaner, but then
I would think that :)

Have fun!
Michael

Sep 3 '06 #6
<sw***@netcourr ier.comwrote:
Yes but before going deeper ( too deep ) i would like to play with a
toy python example. Basically what i want to study first is a socket or
socket+select server and 5 clients that get time updated from the
server.

Don't really know how to keep hot connections without blocking the
server ? If someone can show me a quick and dirty example of the
client/server i'll apprciate it a lot.
Download the zipfile with the examples from Python in a Nutshell (freely
available at
<http://examples.oreill y.com/pythonian/pythonian-examples.zip>) and look
at the examples for chapter 19 -- several sample servers and clients for
a trivial "echo" protocol that progress from a server that's only able
to deal with one client at a time, to multithreaded ones, to ones
implemented with the better scalability of asynchronous socket handling,
via select, asyncore, asynchat, and Twisted (there are also UDP clients
and servers for an otherwise perfectly analogous task).

I strongly recommend asynchronous socket handling, by far the best
scalable and best performing approach to network programming. Doing it
all by yourself with the select module is quite the learning experience,
although in real-life I'd always recommend Twisted instead (it's an
extremely rich framework, normally used at vastly higher semantics
level, but in the mentioned examples I show instead how to use it at
nearly the same semantics level as asyncore &c).
Alex
Sep 3 '06 #7
Thx Alex,
This is exactly what i want to look at . Async will definitely be
something i will look at . Does it make sense to mix async and threaded
server ( let say a threaded server accepting max 10 connections and
dealing with client request asynchronously ). Does it sounds you a good
design.

Of course i will have a look a twisted that sounds me doing all the job
( for sure better that i can do ) .

Thx
Manu

Sep 4 '06 #8
<sw***@netcourr ier.comwrote:
Thx Alex,
This is exactly what i want to look at . Async will definitely be
something i will look at . Does it make sense to mix async and threaded
server ( let say a threaded server accepting max 10 connections and
dealing with client request asynchronously ). Does it sounds you a good
design.
I would spawn a worker thread only for specific jobs that cannot be done
asynchronously, and would dispatch work requests to it from the main
thread, and get results back, via instances of Queue.Queue. One example
might be interfacing with a database which does not directly support any
asynchronous operation: all work on that DB should be done by one
dedicated thread that does nothing else. I have a whole chapter of the
Nutshell dedicated to multiprocessing (processes and threads), and in
that chapter I devote ample space (relative to the Nutshell's always
tight space constraints;-) to recommending the architectures based on
Queue instances that experience teaches me work best.

Of course i will have a look a twisted that sounds me doing all the job
( for sure better that i can do ) .
It's a well-crafted and rich framework -- in particular, it already
wraps on your behalf some cases of the need for an auxiliary thread, as
sketched above. On the other hand, learning in depth the "bare metal"
on which Twisted's elegant architecture rests is a highly recommended
endeavor: remember Spolsky's "Law of Leaky Abstractions".. . ``all
abstractions leak'', and thus, to use abstractions for the best, you'd
better have a solid understanding of the underlying layers (I strongly
believe that having designed microchips makes me a better assembly
language programmer, having programmed in assembly language makes me a
better C programmer, having programmed in C makes me a better Python
programmer...!-).
Alex
Sep 4 '06 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
4156
by: Olivier Hoarau | last post by:
Hello, I have build a client/server application with the socket module. The server mades UDP broadcasting and the client only reads UDP broadcast messages. All work fine. Now I want to use for the same thing the socketserver module, it's ok for the client, but I don't succeed in making work the server :-(( Here is the client (which works)
6
10302
by: pekspro | last post by:
I need some code that gets the address from a server. I read somewhere that you could do this by starting some broadcast server using UDP. The client should send an broadcast message, and when the server answering the client gets the address. But how do I implement this? I did this simple quick-hack: Server: Socket server = new...
2
3013
by: gregory_may | last post by:
First the research links: IPv6 spec (look for 'jumbo payload'): http://www.cs-ipv6.lancs.ac.uk/ipv6/documents/rfcs/archive/rfc1883.txt IPv6 Sample C# Client/Server http://www.codeproject.com/csharp/ipv6.asp I am developing an application where I need to broadcast information to clients. I have hit a 64K limit in UDP broadcasting. I...
2
2416
by: Shawn G. | last post by:
I'm going around in circles here and need some help. Scenario: I am using (have used) either the UDPClient class or Sockets class to broadcast some bytes to 255.255.255.255 (udp protocol). Upon doing so, a device (of which I will know nothing about) will receive my broadcast and send out a broadcast of its own. Using Ethereal (LAN packet...
1
2002
by: Dave | last post by:
I can't figure out why my code does not recieve the message. The only message that it receives is the one going out from my machine. I am suppose to send out a message to a server. Upon receiving that message on the server side the server is suppose to send a message back. I can't get my app to receive that message. It only receives the...
0
1105
by: Dave | last post by:
I can't figure out why my code does not recieve the message. The only message that it receives is the one going out from my machine. I am suppose to send out a message to a server. Upon receiving that message on the server side the server is suppose to send a message back. I can't get my app to receive that message. It only receives the...
7
2299
by: GTi | last post by:
I need a code snippet for sending a message to all computers located on the same IP subnet. This is only a simple message like; "Here I am" I'm developing a server/client application. Instead of configuration of each clients where the server is located I want the server to broadcast a simple message that the server is present. This solution...
0
2255
by: salman | last post by:
Hello I have local network with ip(s) 192.168.1.x, and i am creating VPN through RAS- Remote Access Service (window server 2003) of ip range from 10.0.0.10 - 10.0.0.255. client are connecting to vpn correctly. The problem is that when i want to broadcast the message to vpn network by sending through broadcast ip of 10.0.0.255 it is not...
9
3684
by: Irmen de Jong | last post by:
Hello Sorry this might be a bit offtopic but I don't really know where else to post this question. If you could point me in the right direction that is much appreciated. I'm running into a weird thing on Windows XP. I'm using Python 2.5 with latest pywin32 from Mark Hammond.
1
7608
by: raviskar | last post by:
Hi, I have a problem in receiving UDP broadcast data. We have a simple application which listens for the UDP broadcast data from the external servers. We can receive the UDP broadcast data from one of the servers. For the other external server, our program does not receive any data in the sockets, but we can see the broadcast data in tcpdump...
0
7760
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
8271
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7858
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
8137
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
6511
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
0
5335
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3774
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
1
1369
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
1099
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.