Hello,
I am in the process of designing the following kind of system --
A MySQL database holds hundreds of thousands of entries. All of these
entries are periodically processed by a set of backends. Whenever an
entry is processed, its related fields are updated.
The approaches I am considering for load-balancing are:
1. Have a backend-id associated with each entry. Each backend server
is given
a range of backend-ids it should process. I don't like this for the
reason that
if someone messes up backend-id range allocation among backends, some
entries may not get processed at all (or, could get processed multiple
times).
2. No backend-ids. Each backend server grabs a bunch of unprocessed
entries by
updating a status field. Once it is done processing all of them, it
again updates the status to signal the completion of the processing.
This approach results in two updates to every record in a processing
cycle. Also, it is tricky
to grab unprocessed records (since they may have just been processed
by other backend).
Do I have any other choices apart from above? I would highly
appreciate any ideas.
Thanks,
KP.
