map-reduce-merge: simplified relational data processing on large clusters h.yang, a. dasdan...
TRANSCRIPT
Map-Reduce-Merge: Simplified Relational Data Processing on
Large ClustersH.Yang, A. Dasdan (Yahoo!), R. Hsiao, D.S.Parker (UCLA)
Shimin ChenBig Data Reading
Group Presentation
Motivation
Map-Reduce framework Compared to relational DBMS “simplified” for data processing in search engines
Problem: join multiple heterogeneous datasets Not quite fit into map-reduce Ad-hoc solutions: map-reduce on one data set while
reading data from the other dataset on the fly
Contribution
Goal: support relational algebra primitives without sacrificing existing generality and simplicity
Proposal: map-reduce-merge
Outline
Introduction Map-Reduce Map-Reduce-Merge Applications to Relational Data Processing Optimizations and Enhancements Case Studies Conclusion
Let’s Refresh Our Memory
Functional programming model
Comments
Low-cost unreliable commodity hardware Failure often occurs during each map/reduce task Coordinator re-run mapper or reducer
Homogenization: for equi-join Transform each dataset into (join key, payload) Then apply map-reduce to merge entries from different
datasets Problem: only equi-joins may take lots of extra disk space, incur excessive
communications
Outline
Introduction Map-Reduce Map-Reduce-Merge Applications to Relational Data Processing Optimizations and Enhancements Case Studies Conclusion
Map-Reduce-Merge Primitiveskey
join
Focusing on Merge
Two sets of inputs generated by multiple reducers: Which α reducers and β reducers match? How to get the next key-value pair? Customized preprocessing for inputs? Merging algorithm?
All of these are customizable
Focusing on Merge
Two sets of inputs generated by multiple reducers: Partition Selector: Which α reducers and β reducers match?
Iterator: How to get the next key-value pair? Processor: Customized preprocessing for inputs? Merger: Merging algorithm?
All of these are customizable
Example: Emp & Dept
Employee Department
Partition Selector
LHS: reduce key:dept-id, emp-id partition key: dept-id RHS: reduce key:dept-id, partition key: dept-id Assuming #reducer is the same, LHS reducer K matches
RHS reducer K
Processor
Pre-processing for each input E.g. building hash table for hash join
This example is sort-merge Processor is empty
Iterator for sort-merge
Merger
Other Iterators
Nested-loop: For each (k,v) of the first input, get all the second input Then rewind the second input and process the next (k,v) of
the first input
Hash join: Read all of one input, then read all of the other input
Outline
Introduction Map-Reduce Map-Reduce-Merge Applications to Relational Data Processing Optimizations and Enhancements Case Studies Conclusion
Relation
Relation R with an attribute set A A is broken down into a key part K, and a
value part V
Relational Operators Generalized selection: choosing a subset of records
Filtering can be done in mapper/reducer/merger Projection: choosing a subset of attributes
User-defined mapper (k,v)(k’,v’) Aggregation
Group-by is performed before reduce Easy to implement aggregation in reducer
Joins (set union, intersection, difference, cartesian product) Sort-merge, hash join, nested-loop
Rename
Outline
Introduction Map-Reduce Map-Reduce-Merge Applications to Relational Data Processing Optimizations and Enhancements Case Studies Conclusion
Partition Selector
In general: LHS has R1 reducers, RHS has R2 reducers, performing cartesian-product like operator
Suppose R1 R2, use R1 merger, where merger j selects: Input from LHS reducer j Input from RHS all reducers Remote reads: R1*(1+R2) = R1 + R1*R2
Natural equi-join case: Let R1==R2==R, use R merger, where merger j selects: LHS reducer j and RHS reducer j Remote reads: 2*R
Combining Phases
Entire workflow consists of multiple map-reduce-merge To avoid remote copying:
ReduceMap, MergeMap:co-locate next mapper with previous reducer or merger
ReduceMerger:co-locate merger with one of the reducer
ReduceMergeMap
Map-Reduce-Merge Library
Put common merge implementations into a library Joins Common iterators etc.
Configuration API for building a Customized Workflow
Map/reduce
Map/reduce/merge Multiple Map/reduce/merge
Outline
Introduction Map-Reduce Map-Reduce-Merge Applications to Relational Data Processing Optimizations and Enhancements Case Studies Conclusion
Webgraphs
Each row: (URL, in-links, out-links) Potentially large number of links Only a few are needed for many operations Store each column of the table in a separate file
Reconstruct the table by join E.g. compute the intersection of in-links and out-
links
TPC-H Query 2
After Combining Phases
Conclusion
Extend map-reduce Support relational operators However, the merge step seems quite
complicated