471,854 Members | 1,448 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,854 software developers and data experts.

Broadcast server

I would like to write a server with the low level API of python (
socket+select and/or socket+thread ) that allow me to register client
and update them every X seconds ( could be the time, the temperature, a
stock quote, a message , ... ).

How to write the server that keep hot connections with clients and
update them when events are trigerred. I don't know how to begin this ,
i look at the python doc but the doc is more related to client updating
the server and i am not sure of the right design that could be use
here.

Thx
Manu

Aug 31 '06 #1
8 3096
On Thu, 31 Aug 2006 10:14:15 +0100, <sw***@netcourrier.comwrote:
I would like to write a server with the low level API of python (
socket+select and/or socket+thread ) that allow me to register client
and update them every X seconds ( could be the time, the temperature, a
stock quote, a message , ... ).

How to write the server that keep hot connections with clients and
update them when events are trigerred. I don't know how to begin this ,
i look at the python doc but the doc is more related to client updating
the server and i am not sure of the right design that could be use
here.
Kamaelia ( http://kamaelia.sf.net/ ) - the project I'm working on, feels
like it may be well suited to this kind of task. Its based around having
individual components that pass messages to each other. So you can
envisage an event source propogating a message to all subscribed clients
quite easily.

Your source's of events can be components, that publish their events to a
'backplane' - a kind of distribution board that replicates messages sent
to it to all subscribers.

from Axon.Component import component
from Kamaelia.Util.Backplane import Backplane, subscribeTo, publishTo
from Kamaelia.Util.PipelineComponent import pipeline

class EventSource(component):
def main(self):
while 1:
if event_happens:
self.send( "EVENT HAPPENED!\n", "outbox")
yield 1

pipeline( EventSource(),
publishTo("Events"),
).activate()

Backplane("Events").activate()

You can have any number of event sources, all 'publishing' to the same
backplane.

Then to handle clients connecting I'd create a server component, with a
factory function that creates a component to handle each client
connection. In this case, a component that 'subscribes' to messages coming
from the backplane:

from Kamaelia.Chassis.ConnectedServer import SimpleServer

def clientHandlerFactory():
return subscribeTo("Events")

SimpleServer(clientHandlerFactory).activate()

Then I'd start the system running:

from Axon.Scheduler import scheduler

scheduler.run.runThreads()

Things like the client handling could be extended relatively easily to
marshall complex data types to strings, suitable for network transmission;
or to manage clients choosing what data sources to subscribe to.

If you think this approach might work for you, I'm happy to give you a
hand with getting Kamaelia up and running, and with learning how to write
your own components to make the system more sophisticated.

We hang about on #kamaelia on freenode (irc) - please do drop in for a
chat!

Hope this helps!
Matt
--
| Matt Hammond
| Research Engineer, Tech. Group, BBC, Kingswood Warren, Tadworth, Surrey,
UK
| http://kamaelia.sf.net/
| http://www.bbc.co.uk/rd/
Aug 31 '06 #2
Hi Matt ,
It sounds very interesting and i will definitely take a deeper look
but i was more considering something simpler as a learning exercise
more that finding a package that should already do what i want to
achieve.
I want to find the basic guidelines to write that kind of client/server
architecture and make some tests of different solutions(socket+select
or socket+thread or mix of them)

Your project seems very great and will try to read in to learn.

Aug 31 '06 #3
sw***@netcourrier.com wrote:
I would like to write a server with the low level API of python (
socket+select and/or socket+thread ) that allow me to register client
and update them every X seconds ( could be the time, the temperature, a
stock quote, a message , ... ).

How to write the server that keep hot connections with clients and
update them when events are trigerred. I don't know how to begin this ,
i look at the python doc but the doc is more related to client updating
the server and i am not sure of the right design that could be use
here.
I'd suggest to find the Richard W. Stevens book "Unix Network Programing".
You'll learn a lot about networking from that book. It's based on C, but
you'll see that the concepts can be easily used with Python too.

Find that book, and don't forget the stdlib docs:
http://docs.python.org/lib/module-socket.html
http://docs.python.org/lib/module-select.html

--
damjan
Sep 1 '06 #4
Yes but before going deeper ( too deep ) i would like to play with a
toy python example. Basically what i want to study first is a socket or
socket+select server and 5 clients that get time updated from the
server.

Don't really know how to keep hot connections without blocking the
server ? If someone can show me a quick and dirty example of the
client/server i'll apprciate it a lot.

Thx
Manu

Sep 1 '06 #5
sw***@netcourrier.com wrote:
It sounds very interesting and i will definitely take a deeper look
but i was more considering something simpler as a learning exercise
more that finding a package that should already do what i want to
achieve.
The bulk of this post dissects (roughly) how Kamaelia handles this
internally since I figure that might also be useful if you're after
this as a learning exercise :-) (It should hopefully be clear enough
for this to be useful) At minimum it might go some way to explain why
people generally tend to suggest using an existing framework :-)
I want to find the basic guidelines to write that kind of client/server
architecture and make some tests of different solutions(socket+select
or socket+thread or mix of them)
I think if you're doing socket & select it sounds easy/simple until you
start dealing with all the cases you need to to do it properly. You're
probably best looking at the python cookbook for a good initial starting
point.
Your project seems very great and will try to read in to learn.
Kamaelia *might* still be useful here in some respects, since despite being
*effectively* event based, due to the use of generators code tends to look
linear.

The low level structure of the server breaks down as follows:

(I'm spending a bit of time here since it might be of general interest,
sorry if it isn't)

* Kamaelia.Internet.TCPServer is a simple component that sits waiting for
a user to connect.

It makes the server port, adds any user socket options, makes it non
blocking, binds it to the requested (if any) host and either to a user
defined or random port number. (latter is for testing)

It then registers the socket with the piece of code that's taking care
of select handling, and sits to wait for any notifications of activity
on the socket. (Which would mean for something acting as a server a new
connection).

It then performs a socket accept, and creates an object to handle the
raw (now connected) socket - a connected socket adaptor. It then
registers that socket and object with the piece of code that's taking
care of select handling. (So that the select handling can tell the
piece of code handling the raw socket to do some work on the socket)

code: http://tinyurl.com/pocm8

* Kamaelia.Internet.ConnectedSocketAdaptor is actually more complicated
than you might expect. The reason is because the socket is
non-blocking, and where-ever you would normally block you have
to maintain some sort of buffering in case the action fails.
This is more a symptom of non-blocking behaviour than it is about
Kamaelia.

Anyhow the basics of this is to take any data that is to be sent to the
socket, and add that to a sending queue. (this data will generally
come from the code handling the protocol)

Then if the select handling code has said that we can send/should data,
we take data from the send queue and send as much as we can until our
sending fails (after all, we're non-blocking, so we expect it to fail).

Then if the select handling code has said that there's data ready to
receive the component reads from the socket until an error (again we
expect an error due to non-blocking behaviour). This data is then
passed on to a location where a protocol handler can expect to take the
data and do what it likes with it.

The bulk of the code for the CSA actually revolves around resends, and
the necessary error handling. (It's also one of the oldest bits of code
in Kamaelia so possibly a little horrid to look at!)

code: http://tinyurl.com/nj8lk

* Kamaelia.Internet.Selector is also slightly more complex than you might
think. This is because whilst you tell select "let me know whenever any
of these sockets is ready for (reading|writing|exceptional)", it
actually cares about what you do *after* it's given you an answer.

For example if you don't read from all the things that are ready to
read, or don't write to all the things that are ready to write or
(worse) try asking it to do things with regard to closed or invalid
sockets, then you're just asking for trouble.

There's lots of ways round this and you can do something like:
looping:
(readables, writeables, excepts) = select( ...)
for readable in readables:
read from readable and do something
for writable in writeables:
write to writeables and do something
for exceptional in excepts:
handle exceptional

However then you add new problems - what about if you want to do a
database request as result of one? Render a image for another? Wait for
keyboard press? etc.

As a result that's why we use explicit message passing, though many
people use a reactor (or sometimes a pro-actor) pattern to handle
events. (if you're familiar with simulation it's very similar to an
event wheel approach).

For us we simply have a service (a selector) that sits an waits to be
told "send me a message when this is ready to do X". Since the thing
handling select can't know (in our system because we don't want any
accidental serialisation) whether the event (socket read to ready,
socket ready to write) has been handled, our selector removes the
socket from the read/write/exceptional set when it's been found to be
ready to do something.

This means that when the thing using has read from the socket it tells
the selector "can I have more please", and if it's written to the
socket has to tell the selector "I want to do more, but it bust - can
you tell me when it's working again".

This means the code managing the select loop and the socket sets the
select call operates on has to deal with all these issues. (You'd have
to anyway, but in some respects it's more explicit at the socket-level
in Kamaelia)

As a result you'll see explicit code adding sockets to "readers",
"writers", and "exceptionals", and for removing sockets from them when
an event's happened. This puts responsibility for making this work with
the code that cares - the code handling the socket activity directly.
The rationale for this is that it then also means that the same code
that's used for sockets can be used for file handles. (one of the
points of select on unix after all)

code: http://tinyurl.com/mfqyr

For completeness:
* Kamaelia.Chassis.ConnectedServer.SimpleServer takes the above
components and packages them up in an easy to use way. (meaning you can
avoid all the lowlevel hassle).

It's called a Chassis because like you add components onto a car
chassis (wheels, seats, doors, engine) in order to make a car, you need
to add someting to the SimpleServer to make a server. Specifically you
need to give it something that can create components to handle
connections.

As a result, the way SimpleServer works is whenever a new connection
comes in, it gets told about it by being passed a connected socket
adaptor by the tcp server. It then creates a protocol handler to talk
to the connected socket adaptor. As a result, any information that
comes in from the socket gets passed to the protocol handler. Any
information the protocol handler generates gets sent out the socket.

As a result the logic for the SimpleServer is:
* Create the TCP Server
* Loop
* When the TCP Server gives us a connected socket adaptor,
create a protocol handler (from the given function), and wire
the two together.
* If the connection dies (due to the protocol handler saying
"shutdown" or due to the socket dying) the everyone interested
finds out and propogates the change. (Meaning you don't get
dead file descriptors in the select call)
code: http://tinyurl.com/pts72

You'll note that the explanation of select itself is by far the longest
here! You may also find "man select_tut" interesting and useful if you're
doing this as a learning exercise.

If you're looking to do this more generally, you should really consider
using SocketServer or asyncore that come with python, or Twisted or Kamelia
if you want to run a production system.

Twisted is more well known and more deployed and as a result more battle
tested (meaning seasoned programmers would sensibly trust it more!), and
follows a more common approach for coding all this. (Boils down to something
similar though) As a result if hiring people to work on code is something
to think about Twisted operates is probably a better choice. Kamaelia is
however designed to be easy to pick up.

Finally nipping back to your example here:
5 clients that get time updated from the server
The code to handle on the server side is trivial.

----(start)----
import time
from Axon.ThreadedComponent import threadedcomponent
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Chassis.ConnectedServer import SimpleServer
from Kamaelia.Util.Stringify import Stringify
from Kamaelia.Util.Backplane import *

class periodictime(threadedcomponent):
def main(self):
while 1:
time.sleep(0.1)
self.send(time.time(), "outbox")

Backplane("periodictime").activate()
Pipeline(periodictime(),
Stringify(), # Make suitable for network
PublishTo("periodictime")).activate()

def getTimeProtocol():
return SubscribeTo("periodictime")

SimpleServer(protocol=getTimeProtocol, port=1600).run()
----(finish)----

Telnet to 127.0.0.1 1600 to see the result here.

If you want to have the data source something from a client (eg event info
coming in on port 1599) with clients of the event source coming in from
elsewhere, this would change the server as follows:

----(start)----
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Chassis.ConnectedServer import SimpleServer
from Kamaelia.Util.Backplane import *

Backplane("periodictime").activate()

def publishTime(): return PublishTo("periodictime")
def getTimeProtocol(): return SubscribeTo("periodictime")

SimpleServer(protocol=publishTime, port=1599).run()
SimpleServer(protocol=getTimeProtocol, port=1600).run()
----(finish)----

And the time (event) source would look like this:

----(start)----
import time
from Kamaelia.Internet.TCPClient import TCPClient
from Axon.ThreadedComponent import threadedcomponent
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Util.Stringify import Stringify

class periodictime(threadedcomponent):
def main(self):
while 1:
time.sleep(0.1)
self.send(time.time(), "outbox")

perdiodictimeserver = "127.0.0.1"
Pipeline(periodictime(),
Stringify(), # Make suitable for network
TCPClient(perdiodictimeserver, 1599)).activate()
----(finish)----

Anyway, I hope the explanation of what's going on inside the core is useful
since in many respects if you're writing your own select handling loop
(which I would encourage you to do if you're learning about this!), the
basics of what you have to do stay the same. (check activity, clear, when
errors happen ask again, buffer data which needs to get sent, and decouple
everything as best as makes sense whilst trying to avoid accidental
serialisations).

The reason for the examples in the end is merely for completeness. (I'll
probably add these to our SVN distribution since the question does seem to
crop up fairly often generally speaking!)

If you're looking to do this in a production environment I'm personally an
advocate of learning what's going on in the core and then using an existing
library.

(The reason Kamaelia exists is because I wondered if there was an
alternative, potentially clearer way of writing these things, most
people would quite sensibly just use Twisted - especially given you
can buy a book on it! I personally think Kamaelia is cleaner, but then
I would think that :)

Have fun!
Michael

Sep 3 '06 #6
<sw***@netcourrier.comwrote:
Yes but before going deeper ( too deep ) i would like to play with a
toy python example. Basically what i want to study first is a socket or
socket+select server and 5 clients that get time updated from the
server.

Don't really know how to keep hot connections without blocking the
server ? If someone can show me a quick and dirty example of the
client/server i'll apprciate it a lot.
Download the zipfile with the examples from Python in a Nutshell (freely
available at
<http://examples.oreilly.com/pythonian/pythonian-examples.zip>) and look
at the examples for chapter 19 -- several sample servers and clients for
a trivial "echo" protocol that progress from a server that's only able
to deal with one client at a time, to multithreaded ones, to ones
implemented with the better scalability of asynchronous socket handling,
via select, asyncore, asynchat, and Twisted (there are also UDP clients
and servers for an otherwise perfectly analogous task).

I strongly recommend asynchronous socket handling, by far the best
scalable and best performing approach to network programming. Doing it
all by yourself with the select module is quite the learning experience,
although in real-life I'd always recommend Twisted instead (it's an
extremely rich framework, normally used at vastly higher semantics
level, but in the mentioned examples I show instead how to use it at
nearly the same semantics level as asyncore &c).
Alex
Sep 3 '06 #7
Thx Alex,
This is exactly what i want to look at . Async will definitely be
something i will look at . Does it make sense to mix async and threaded
server ( let say a threaded server accepting max 10 connections and
dealing with client request asynchronously ). Does it sounds you a good
design.

Of course i will have a look a twisted that sounds me doing all the job
( for sure better that i can do ) .

Thx
Manu

Sep 4 '06 #8
<sw***@netcourrier.comwrote:
Thx Alex,
This is exactly what i want to look at . Async will definitely be
something i will look at . Does it make sense to mix async and threaded
server ( let say a threaded server accepting max 10 connections and
dealing with client request asynchronously ). Does it sounds you a good
design.
I would spawn a worker thread only for specific jobs that cannot be done
asynchronously, and would dispatch work requests to it from the main
thread, and get results back, via instances of Queue.Queue. One example
might be interfacing with a database which does not directly support any
asynchronous operation: all work on that DB should be done by one
dedicated thread that does nothing else. I have a whole chapter of the
Nutshell dedicated to multiprocessing (processes and threads), and in
that chapter I devote ample space (relative to the Nutshell's always
tight space constraints;-) to recommending the architectures based on
Queue instances that experience teaches me work best.

Of course i will have a look a twisted that sounds me doing all the job
( for sure better that i can do ) .
It's a well-crafted and rich framework -- in particular, it already
wraps on your behalf some cases of the need for an auxiliary thread, as
sketched above. On the other hand, learning in depth the "bare metal"
on which Twisted's elegant architecture rests is a highly recommended
endeavor: remember Spolsky's "Law of Leaky Abstractions"... ``all
abstractions leak'', and thus, to use abstractions for the best, you'd
better have a solid understanding of the underlying layers (I strongly
believe that having designed microchips makes me a better assembly
language programmer, having programmed in assembly language makes me a
better C programmer, having programmed in C makes me a better Python
programmer...!-).
Alex
Sep 4 '06 #9

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

3 posts views Thread by Olivier Hoarau | last post: by
6 posts views Thread by pekspro | last post: by
2 posts views Thread by Shawn G. | last post: by
1 post views Thread by Dave | last post: by
reply views Thread by Dave | last post: by
7 posts views Thread by GTi | last post: by
reply views Thread by salman | last post: by
9 posts views Thread by Irmen de Jong | last post: by
1 post views Thread by raviskar | last post: by
NeoPa
reply views Thread by NeoPa | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.