Download - Big Data Processing with MapReduce and Spark
Matei Zaharia
UC Berkeley AMPLab
spark-project.orgUC BERKELEY
Big Data Processing with MapReduce and Spark
OutlineThe big data problemMapReduce modelLimitations of MapReduceSpark modelFuture directions
The Big Data ProblemData is growing faster than computation speedsGrowing data sources
»Web, mobile, scientific, …
Cheap storage»Doubling every 18
months
Stalling CPU speeds»Even multicores not
enough
ExamplesFacebook’s daily logs: 60 TB1000 genomes project: 200 TBGoogle web index: 10+ PB
Cost of 1 TB of disk: $50Time to read 1 TB from disk: 6 hours (50 MB/s)
The Big Data ProblemSingle machine can no longer process or even store all the data!Only solution is to distribute over large clusters
Google Datacenter
How do we program this thing?
Traditional Network ProgrammingMessage-passing between nodesReally hard to do at scale:
»How to split problem across nodes?• Must consider network, data locality
»How to deal with failures?• 1 server fails every 3 years => 10K nodes see
10 faults/day»Even worse: stragglers (node is not failed,
but slow)Almost nobody does message passing!
Data-Parallel ModelsRestrict the programming interface so that the system can do more automatically“Here’s an operation, run it on all of the data”
»I don’t care where it runs (you schedule that)
»In fact, feel free to run it twice on different nodes
Biggest example: MapReduce
MapReduceFirst widely popular programming model for data-intensive apps on clustersPublished by Google in 2004
»Processes 20 PB of data / day
Popularized by open-source Hadoop project
»40,000 nodes at Yahoo!, 70 PB at Facebook
MapReduce Programming ModelData type: key-value records
Map function:(Kin, Vin) list(Kinter, Vinter)
Reduce function:(Kinter, list(Vinter)) list(Kout, Vout)
Example: Word Countdef mapper(line): foreach word in line.split(): output(word, 1)
def reducer(key, values): output(key, sum(values))
Word Count Execution
the quickbrown
fox
the fox ate the mouse
how now
brown cow
Map
Map
Map
Reduce
Reduce
brown, 2
fox, 2how, 1now, 1the, 3
ate, 1cow, 1mouse,
1quick, 1
the, 1brown, 1
fox, 1
quick, 1
the, 1fox, 1the, 1
how, 1now, 1
brown, 1ate, 1
mouse, 1
cow, 1
Input Map Shuffle & Sort Reduce Output
MapReduce ExecutionAutomatically split work into many small tasks
Send map tasks to nodes based on data localityLoad-balance dynamically as tasks finish
Fault Recovery1. If a task crashes:
»Retry on another node• OK for a map because it had no dependencies• OK for reduce because map outputs are on
disk»If the same task repeatedly fails, end the
jobRequires user code to be
deterministic
Fault Recovery2. If a node crashes:
»Relaunch its current tasks on other nodes»Relaunch any maps the node previously ran• Necessary because their output files were lost
along with the crashed node
Fault Recovery3. If a task is going slowly (straggler):
»Launch second copy of task on another node
»Take the output of whichever copy finishes first, and kill the other one
Example Applications
1. SearchInput: (lineNumber, line) recordsOutput: lines matching a given patternMap:
if(line matches pattern): output(line)
Reduce: identity function–Alternative: no reducer (map-only job)
2. SortInput: (key, value) recordsOutput: same records, sorted by keyMap: identity functionReduce: identify function
Trick: Pick partitioningfunction p so thatk1 < k2 => p(k1) < p(k2)
pigsheepyakzebra
aardvarkant
beecowelephant
Map
Map
Map
Reduce
Reduce
ant, bee
zebra
aardvark,elephant
cow
pig
sheep, yak
[A-M]
[N-Z]
3. Inverted IndexInput: (filename, text) recordsOutput: list of files containing each wordMap:
foreach word in text.split(): output(word, filename)
Reduce: def reduce(word, filenames): output(word, unique(filenames))
Inverted Index Example
afraid, (12th.txt)be, (12th.txt, hamlet.txt)greatness, (12th.txt)not, (12th.txt, hamlet.txt)of, (12th.txt)or, (hamlet.txt)to, (hamlet.txt)
to be or not to
be
hamlet.txt
be not afraid of greatnes
s
12th.txt
to, hamlet.txtbe, hamlet.txtor, hamlet.txtnot, hamlet.txtbe, 12th.txtnot, 12th.txtafraid, 12th.txtof, 12th.txtgreatness, 12th.txt
4. Most Popular WordsInput: (filename, text) recordsOutput: the 100 words occurring in most filesTwo-stage solution:–Job 1:• Create inverted index, giving (word, list(file)) records
–Job 2:• Map each (word, list(file)) to (count, word)• Sort these records by count as in sort job
SummaryBy providing a data-parallel model, MapReduce greatly simplified cluster programming:
»Automatic division of job into tasks»Locality-aware scheduling»Load balancing»Recovery from failures & stragglers
But… the story doesn’t end here!
OutlineThe big data problemMapReduce modelLimitations of MapReduceSpark modelFuture directions
When an Abstraction is Useful…People want to compose it!Most real applications require multiple MR steps
»Google indexing pipeline: 21 steps»Analytics queries (e.g. sessions, top K): 2-5
steps»Iterative algorithms (e.g. PageRank): 10’s
of steps
Problems: programmability & performance
ProgrammabilityMulti-step jobs create spaghetti code
»21 MR steps -> 21 mapper and reducer classes
Lots of boilerplate wrapper code per stepAPI doesn’t provide type safety
PerformanceMR only provides one pass of computation
»Must write out data to file system in-between
Expensive for apps that need to reuse data
»Multi-step algorithms (e.g. PageRank)»Interactive data mining (many queries on
same data)
Users often hand-optimize by merging steps
SparkAims to address both problemsProgrammability: clean, functional API
»Parallel transformations on collections»5-10x less code than MR»Available in Scala, Java and Python
Performance:»In-memory computing primitives»Automatic optimization across operators
Spark Programmability
#include "mapreduce/mapreduce.h"
// User’s map functionclass SplitWords: public Mapper { public: virtual void Map(const MapInput& input) { const string& text = input.value(); const int n = text.size(); for (int i = 0; i < n; ) { // Skip past leading whitespace while (i < n && isspace(text[i])) i++; // Find word end int start = i; while (i < n && !isspace(text[i])) i++; if (start < i) Emit(text.substr( start,i-start),"1"); } }};
REGISTER_MAPPER(SplitWords);
// User’s reduce functionclass Sum: public Reducer { public: virtual void Reduce(ReduceInput* input) { // Iterate over all entries with the // same key and add the values int64 value = 0; while (!input->done()) { value += StringToInt( input->value()); input->NextValue(); } // Emit sum for input->key() Emit(IntToString(value)); }};
REGISTER_REDUCER(Sum);
int main(int argc, char** argv) { ParseCommandLineFlags(argc, argv); MapReduceSpecification spec; for (int i = 1; i < argc; i++) { MapReduceInput* in= spec.add_input(); in->set_format("text"); in->set_filepattern(argv[i]); in->set_mapper_class("SplitWords"); }
// Specify the output files MapReduceOutput* out = spec.output(); out->set_filebase("/gfs/test/freq"); out->set_num_tasks(100); out->set_format("text"); out->set_reducer_class("Sum");
// Do partial sums within map out->set_combiner_class("Sum");
// Tuning parameters spec.set_machines(2000); spec.set_map_megabytes(100); spec.set_reduce_megabytes(100); // Now run it MapReduceResult result; if (!MapReduce(spec, &result)) abort(); return 0; }
Google MapReduce WordCount:
Spark Programmability
Spark WordCount:
val file = spark.textFile(“hdfs://...”)
val counts = file.flatMap(line => line.split(“ ”)) .map(word => (word, 1)) .reduceByKey(_ + _)
counts.save(“out.txt”)
Spark PerformanceIterative algorithms:
0 20 40 60 80 100 120 1404.1
121K-means Clustering
Hadoop MRSpark
sec
0 10 20 30 40 50 60 70 80 900.96
80Logistic Regression
Hadoop MRSpark
sec
Spark ConceptsResilient distributed datasets (RDDs)
»Immutable, partitioned collections of objects»May be cached in memory for fast reuse
Operations on RDDs»Transformations (build RDDs), actions
(compute results)
Restricted shared variables»Broadcast, accumulators
Example: Log MiningLoad error messages from a log into memory, then interactively search for various patternslines = spark.textFile(“hdfs://...”)
errors = lines.filter(_.startsWith(“ERROR”))messages = errors.map(_.split(‘\t’)(2))messages.cache()
Block 1
Block 2
Block 3
Worker
Worker
Worker
Driver
messages.filter(_.contains(“foo”)).countmessages.filter(_.contains(“bar”)).count. . .
tasksresults
Cache 1
Cache 2
Cache 3
Base RDDTransformed
RDD
Action
Result: full-text search of Wikipedia in <1 sec (vs 20
sec for on-disk data)
Result: search 1 TB data in 5-7 sec
(vs 170 sec for on-disk data)
Fault RecoveryRDDs track lineage information that can be used to efficiently reconstruct lost partitionsEx:
messages = textFile(...).filter(_.startsWith(“ERROR”)) .map(_.split(‘\t’)(2))
HDFS File Filtered RDD
Mapped RDDfilter
(func = _.contains(...))map
(func = _.split(...))
Demo
Example: Logistic RegressionGoal: find best line separating two sets of points
+
–
+ ++
+
+
++ +
– ––
–
–
–– –
+
target
–
random initial line
Example: Logistic Regressionval data = spark.textFile(...).map(readPoint).cache()
var w = Vector.random(D)
for (i <- 1 to ITERATIONS) { val gradient = data.map(p => (1 / (1 + exp(-p.y*(w dot p.x))) - 1) * p.y * p.x ).reduce(_ + _) w -= gradient}
println("Final w: " + w)
w automatically shipped to
cluster
Logistic Regression Performance
110 s / iteration
first iteration 80 s
further iterations 1 s
1 10 20 300
102030405060
HadoopSpark
Number of Iterations
Runn
ing
Tim
e (m
in)
Other RDD OperationsTransformation
s(define a new
RDD)
mapfilter
samplegroupByKeyreduceByKey
cogroup
flatMapunionjoin
crossmapValues
...
Actions(output a result)
collectreduce
takefold
countsaveAsTextFile
saveAsHadoopFile...
Spark in Java and PythonJavaRDD<String> lines = sc.textFile(...);lines.filter(new Function<String, Boolean>() { Boolean call(String s) { return s.contains(“error”); }}).count();
lines = sc.textFile(...)lines.filter(lambda x: “error” in x).count()
Shared VariablesSo far we’ve seen that RDD operations can use variables from outside their scopeBy default, each task gets a read-only copy of each variable (no sharing)Good place to enable other sharing patterns!
Example: Collaborative FilteringGoal: predict users’ movie ratings based on past ratings of other movies
R =1 ? ? 4
5 ? 3? ? 3 5
? ? 35 ? 5 ?
? ? 14 ? ? ?
? 2 ?
Movies
Users
Model and AlgorithmModel R as product of user and movie feature matrices A and B of size U×K and M×K
Alternating Least Squares (ALS)»Start with random A & B»Optimize user vectors (A) based on movies»Optimize movie vectors (B) based on users»Repeat until converged
R A= BT
Serial ALSvar R = readRatingsMatrix(...)
var A = // array of U random vectorsvar B = // array of M random vectors
for (i <- 1 to ITERATIONS) { A = (0 until U).map(i => updateUser(i, B, R)) B = (0 until M).map(i => updateMovie(i, A, R))}
Range objects
Naïve Spark ALSvar R = readRatingsMatrix(...)
var A = // array of U random vectorsvar B = // array of M random vectors
for (i <- 1 to ITERATIONS) { A = spark.parallelize(0 until U, numSlices) .map(i => updateUser(i, B, R)) .collect() B = spark.parallelize(0 until M, numSlices) .map(i => updateMovie(i, A, R)) .collect()}
Problem:
R re-sent to all
nodes in each
iteration
Efficient Spark ALSvar R = spark.broadcast(readRatingsMatrix(...))
var A = // array of U random vectorsvar B = // array of M random vectors
for (i <- 1 to ITERATIONS) { A = spark.parallelize(0 until U, numSlices) .map(i => updateUser(i, B, R.value)) .collect() B = spark.parallelize(0 until M, numSlices) .map(i => updateMovie(i, A, R.value)) .collect()}
Solution: mark R
as broadcas
t variable
Result: 3× performance improvement
AccumulatorsApart from broadcast, another common sharing pattern is aggregation
»Add up multiple statistics about data»Count various events for debugging
Spark’s reduce operation does aggregation, but accumulators are another nice way to express it
Usageval badRecords = sc.accumulator(0)val badBytes = sc.accumulator(0.0)
records.filter(r => { if (isBad(r)) { badRecords += 1 badBytes += r.size false } else { true }}).save(...)
printf(“Total bad records: %d, avg size: %f\n”, badRecords.value, badBytes.value / badRecords.value)
Accumulator RulesCreate with SparkContext.accumulator(initialVal)
“Add” to the value with += inside tasks»Each task’s effect only counted once
Access with .value, but only on master»Exception if you try it on workers
Retains efficiency and fault tolerance!
Job SchedulerCaptures RDD dependency graphPipelines functionsinto “stages”Cache-aware fordata reuse & localityPartitioning-awareto avoid shuffles
join
union
groupBy
map
Stage 3
Stage 1
Stage 2
A: B:
C: D:
E:
F:
G:
= cached partition
User Community3000 people attended online training600 meetup members15 companies contributing
OutlineThe big data problemMapReduce modelLimitations of MapReduceSpark modelFuture directions
Future DirectionsAs “big data” starts to be used for more apps, users’ demands are also growing:
»Latency: instead of training a model every night, can you train in real-time?
»High-level abstractions: matrices, graphs, etc – what is the equivalent of Matlab or R for clusters?
Spark StreamingExtends Spark API to do stream processing
»Run as a series of small, deterministic batches
Intermix with batch and ad-hoc queriessc.twitterStream(...) .filter(_.contains(“spark”) ) .map(t => (t.user, 1)) .runningReduce(_ + _)
t = 1:
t = 2:
tweets pairs counts
map reduce
. . .= RDD = partition
Streaming ResultsBetter performance than other models, while providing fault recovery properties they lack
0 20 40 60 80 1000
10
20
30
Nodes in Cluster
Reco
rds/
s (m
il-lio
ns)
Sliding WordCount + Top K
012345 30s ckpts, 20
nodes30s ckpts, 40 nodes
Proc
essi
ng T
ime
(s)
Scalability Fault Recovery
Higher-Level AbstractionsSparkGraph: graph processing modelMLbase: declarative machine learning libraryShark: SQL queries
Selec-tion
020406080
1001.
1
Ag-grega-
tion
0
200
400
600
32
Join0
300600900
120015001800
SharkHadoop
ConclusionCommodity clusters are needed to handle big data, but pose key challenges (faults, stragglers)Data-parallel models like MapReduce and Spark handle these automaticallyLook for similar models for new problems
www.spark-project.org
Other ResourcesHadoop MapReduce: http://hadoop.apache.org/Spark: http://spark-project.org Hadoop video tutorials: www.cloudera.com/hadoop-trainingAmazon Elastic MapReduce:http://aws.amazon.com/elasticmapreduce/
Behavior with Not Enough RAM
Cache disabled
25% 50% 75% Fully cached
020406080
10068
.8
58.1
40.7
29.7
11.5
% of working set in memory
Iter
atio
n ti
me
(s)