Design/Architecture question : redundancy for small client/server network

[Derrick]

Hello all;

This is going to be a bit long...sorry...

I am about to start out on a small scale client/server type system.
The server will collect data from various sources (custom hardware over
ethernet), and make that data available to the clients (either
webservices or remoting, not sure yet). Since the hardware is
networked, the clients could talk to it directly, but I don't want to
put the hardware through that load since reads/writes are half slow,
hence the server. The server will also serve web pages to the external
world for monitoring purposes. In addition to "pulling" data from the
server, the clients will also have to "push" information to configure
the hardware.

This is pretty standard stuff, that I've done a bunch of times. But
now I want to be able to add a level of redundancy/high availability.
I don't care if the web pages stop getting served, but I always want
the clients to have access to the hardware data (both setting the
config and reading back).

The obvious solution would be to have a "spare" server, the same way
you would have a server cluster in a web farm. But given the scale of
the system (~10 clients), that seems overkill. Ideally what I'd like
to be able to do is to "promote" one of the clients to the server role
(with the exception of the webpages) until the real server comes back
online.

Has anyone tackled this sort of concept before? Are there any
"patterns" for this type of behaviour? I'm sure that I'm not the first
who'd want to do something like this, and it seems do-able.

The first thing I thought of would be for the server to gather a list
of all clients, and give that list back to each client. If the main
server were to go offline, each client would start going down the list
searching for the backup server - if the next item is "me", then I
become the backup server, otherwise try to contact that item and use it
as the server (this would all happen behind some proxy class). The
problem that appears to exist is the delay between a client switching
to the server role. If "Client A" is in the midst of assuming the
server role, and "Client B" tries to access "Client A" as the
backup-server, it is possible that the switch is incomplete and "Client
A" won't respond, resulting in "Client B" continuing down the list.
Maybe this could be solved using a "retry" scheme, or cycling through
the list a few times...

Thanks for reading all that!!

Derrick