<be************@lycos.com> wrote in message
news:11*********************@c13g2000cwb.googlegro ups.com...
(This Google article suggestion comes from the CleverCS site):
http://labs.google.com/papers/mapreduce-osdi04.pdf
"MapReduce is a programming model and an associated implementation for
processing and generating large data sets. Users specify a map function
that processes a key/value pair to generate a set of intermediate
key/value pairs, and a reduce function that merges all intermediate
values associated with the same intermediate key."
Summarizing groups (and sometimes simultaneously by subgroups) has been a
standard operation for decades in both statistics packages and database
report generators. Python has various versions of the map and reduce
operations that they use. Its dicts can easily be used to group items with
the same key. What is somewhat specific to Google is the need to generate
*multiple* key-value pairs from each input document. What MapReduce does
that is somewhat innovative is automatically parallelize the computation to
run fault-tolerantly on a cluster of up to 1000s of machines with machine
slowdowns and failures 'common'. And indeed, that is the reason to use the
system with trivial map or reduce functions, as they sometimes do.
What this does show is that Google could be regarded as a cluster
supercomputing company, with Web search as the visible development
application.
Terry J. Reedy
