1 the map-reduce framework compiled by mark silberstein, using slides from dan weld’s class at u....
TRANSCRIPT
![Page 1: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/1.jpg)
1
The Map-Reduce Framework
Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other
![Page 2: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/2.jpg)
2
The MapReduce Paradigm
You are given a phonebook: How many entries have the last name
“Smith”? Create a list of all last names, with the
numbers of entries they appear in. How many entries have the first name
“John”? Create a list of all first names, with the
numbers of entries they appear in.
How can these tasks be distributed to many computers?
![Page 3: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/3.jpg)
Motivation
Large-Scale Data Processing Want to use 1000s of CPUs
• But don’t want hassle of managing things
MapReduce framework provides Automatic parallelization & distribution Fault tolerance I/O scheduling Monitoring & status updates
![Page 4: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/4.jpg)
Map/Reduce Map/Reduce
Programming model from Lisp (and other functional languages)
Many problems can be phrased this way
Easy to distribute across nodes Nice retry/failure semantics
![Page 5: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/5.jpg)
Map in Lisp (Scheme) (map f list [list2 list3 …])
(map square ‘(1 2 3 4)) (1 4 9 16)
(reduce + ‘(1 4 9 16)) (+ 16 (+ 9 (+ 4 1) ) ) 30
(reduce + (map square (map – l1 l2))))
Unary operator
Binary operator
![Page 6: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/6.jpg)
Map/Reduce ala Google map(key, val) is run on each item in set
emits new-key / new-val pairs
reduce(new-key, new-vals) is run for each unique new-key emitted by map() emits final output
![Page 7: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/7.jpg)
count words in docs Input consists of (url, contents) pairs
map(key=url, val=contents):▫ For each word w in contents, emit (w, “1”)
reduce(key=word, values=uniq_counts):▫ Sum all “1”s in values list▫ Emit result “(word, sum)”
![Page 8: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/8.jpg)
Count, Illustrated
map(key=url, val=contents):For each word w in contents, emit (w, “1”)
reduce(key=word, values=uniq_counts):Sum all “1”s in values listEmit result “(word, sum)”
see bob throwsee spot run
see 1bob 1 run 1see 1spot 1throw1
bob 1 run 1see 2spot 1throw1
![Page 9: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/9.jpg)
Grep Input consists of (file, regexp) map(file, regexp):
• If contents matches regexp, emit (regexp, #line)
reduce(regexp, #line):•Don’t do anything; just emit line
![Page 10: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/10.jpg)
Reverse Web-Link Graph
Map For each URL linking to target, … Output <target, source> pairs
Reduce Concatenate list of all source URLs Outputs: <target, list (source)> pairs
![Page 11: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/11.jpg)
Inverted Index
Result: <word : pages where it appears>
Map For each page, Output <word: page>
Reduce Concatenate all pages of the word Output <word: page1, page2,...>
![Page 12: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/12.jpg)
12
Shuffle
Map-Reduce FrameworkInput: acollectionof keysand theirvalues
User-written
Mapfunction
Each input(k,v) mappedto set ofintermediatekey-valuepairs
sort allkey-valuepairs by
key
One list ofintermediatevalues foreach key:(k, [v1,…,vn])
Each listof valuesis reducedto a singlevalue forthat key
User-writtenReducefunction
![Page 13: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/13.jpg)
Parallelization
How is this distributed?1. Partition input key/value pairs into
chunks, run map() tasks in parallel 2. Shuffle: After all map()s are complete,
consolidate all emitted values for each unique emitted key
3. Now partition space of output map keys, and run reduce() in parallel
If map() or reduce() fails, reexecute!
![Page 14: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/14.jpg)
Data store 1 Data store nmap
(key 1, values...)
(key 2, values...)
(key 3, values...)
map
(key 1, values...)
(key 2, values...)
(key 3, values...)
Input key*value pairs
Input key*value pairs
== Barrier == : Aggregates intermediate values by output key
reduce reduce reduce
key 1, intermediate
values
key 2, intermediate
values
key 3, intermediate
values
final key 1 values
final key 2 values
final key 3 values
...
Shuffle
![Page 15: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/15.jpg)
Semantics Assumptions on
Map and Reduce operations• Map has no side effects
• Reduce is – Associative (a#b)#c= a#(b#c)– Commutative (a#b)=(b#a)– Idempotent
![Page 16: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/16.jpg)
Another example
In a credit card company: Given:
“credit card number”->”bargain cost”
Find the average bragain cost per month
Map(card, bargain) Reduce(month, {total cost,
#bargains})
![Page 17: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/17.jpg)
17
Implementation
A program forks a master process and many worker processes.
Input is partitioned into some number of splits.
Worker processes are assigned either to perform Map on a split or Reduce for some set of intermediate keys.
![Page 18: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/18.jpg)
18
Distributed Execution Overview User
Program
Worker
Worker
Master
Worker
Worker
Worker
fork fork fork
assignmap
assignreduce
readlocalwrite
remoteread,sort
OutputFile 0
OutputFile 1
write
Split 0Split 1Split 2
Input Data
Problems with efficiency of parallel implementation???
![Page 19: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/19.jpg)
Distributed Execution 2
![Page 20: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/20.jpg)
Another example: distributed sort
Map ? Reduce ? Something is missing .... Shuffle phase must allow distributed
sort!
![Page 21: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/21.jpg)
21
Responsibility of the Master
1. Assign Map and Reduce tasks to Workers.
2. Check that no Worker has died (because its processor failed).
3. Communicate results of Map to the Reduce tasks.
![Page 22: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/22.jpg)
22
Communication from Map to Reduce
Select a number R of reduce tasks. Divide the intermediate keys into R
groups, e.g. by hashing. Each Map task creates, at its own
processor, R files of intermediate key-value pairs, one for each Reduce task.
![Page 23: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/23.jpg)
Locality
Master program divvies up tasks based on location of data: tries to have map() tasks on same machine as physical file data, or at least same rack
map() task inputs are divided into 64-128 MB blocks: same size as Google File System chunks
![Page 24: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/24.jpg)
Fault Tolerance
Master detects worker failuresRe-executes failed map() tasksRe-executes in-progress reduce() tasks
Master notices particular input key/values cause crashes in map(), and skips those values on re-execution.Effect: Can work around bugs in third-
party libraries
![Page 25: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/25.jpg)
Strugglers
No reduce can start until map is complete:A single slow disk controller can rate-
limit the whole process Master redundantly executes “slow”
map tasks; uses results of first correctly-terminating replica
![Page 26: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/26.jpg)
Task Granularity & Pipelining
Fine granularity tasks: #map tasks >> #machines Minimizes time for fault recovery Can pipeline shuffling with map execution Better dynamic load balancing
Often use 200,000 map & 5000 reduce tasks
Running on 2000 machines
![Page 27: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/27.jpg)
Sorting guarantees within each reduce partition
Compression of intermediate data Combiner
Perform reduce on local map results before shuffle
Useful for saving network bandwidth
Other Refinements
![Page 28: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/28.jpg)
Yet another example
Distributed single source shortest path adjacency list representation (node ->
list of neighbors) Until all nodes are done:
•map(node, distance from source, {neighbors})
• reduce(neighbor, {distance,distance,....})– Output: {neighbor, minimal distance}
“Graph Algorithms with MapReduce”: http://www.umiacs.umd.edu/~jimmylin/cloud-computing/Session5.ppt
![Page 29: 1 The Map-Reduce Framework Compiled by Mark Silberstein, using slides from Dan Weld’s class at U. Washington, Yaniv Carmeli and some other](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e7a5503460f94b7ab09/html5/thumbnails/29.jpg)
29
Hadoop
An open-source implementation of Map-Reduce in Java.
Download from: http://apache.org/hadoop
Main referencehttp://labs.google.com/papers/
mapreduce-osdi04.pdf