lecture 11 spark - github pagesintro to spark • spark is really a different implementation of the...

31

CS639: Data Management for Data Science Lecture 11: Spark Theodoros Rekatsinas 1

Upload: others

Post on 21-Jun-2020

6 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Page 1: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

CS639:DataManagementfor

DataScienceLecture11:Spark

TheodorosRekatsinas

1

Page 2: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

Logistics/Announcements

2

• QuestionsonPA3?

Page 3: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

Today’sLecture

1. MapReduceImplementation

2. Spark

3

Page 4: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

1. MapReduceImplementation

4

Page 5: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

Recall:TheMapReduceAbstractionforDistributedAlgorithms

DistributedDataStorage

Map

Reduce

(Shuffle)

map map map map map map

reduce reduce reduce reduce

Page 6: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

MapReduce:whathappensinbetween?

Page 7: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

MapReduce:thecompletepicture

Page 8: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

Step1:Splitinputfilesintochunks(shards)

Page 9: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

Step2:Forkprocesses

Page 10: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

Step3:RunMapTasks

Page 11: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

Step4:Createintermediatefiles

Page 12: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

Step4a:Partitioning

Page 13: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

Step5:ReduceTask- sorting

Page 14: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

Step6:ReduceTask- reduce

Page 15: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

Step7:Returntouser

Page 16: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

MapReduce:thecompletepicture

Weneedadistributedfilesystem!

Page 17: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

2.Spark

17

Page 18: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

IntrotoSpark

• SparkisreallyadifferentimplementationoftheMapReduceprogrammingmodel

• WhatmakesSparkdifferentisthatitoperatesonMainMemory• Spark:wewriteprogramsintermsofoperationsonresilient

distributeddatasets(RDDs).• RDD(simpleview):acollectionofelementspartitionedacrossthe

nudesofaclusterthatcanbeoperatedoninparallel.• RDD(complexview):RDDisaninterfacefordatatransformation,

RDDreferstothedatastoredeitherinpersistedstore(HDFS)orincache(memory,memory+disk,diskonly)orinanotherRDD

Page 19: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

RDDsinSpark

Page 20: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

MapReducevsSpark

Page 21: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

RDDs

• Partitionsarerecomputedonfailureorcacheeviction• Metadatastoredforinterface:• Partitions– setofdatasplitsassociatedwiththisRDD• Dependencies– listofparentRDDsinvolvedincomputation• Compute– functiontocomputepartitionoftheRDDgiventheparent

partitionsfromtheDependencies• PreferredLocations– whereisthebestplacetoputcomputationsonthis

partition(datalocality)• Partitioner – howthedataissplitintopartitions

Page 22: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

RDDs

Page 23: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

DAG

• DirectedAcyclicGraph– sequenceofcomputationsperformedondata

• Node– RDDpartition• Edge– transformationontopofthedata• Acyclic– graphcannotreturntotheolderpartition• Directed– transformationisanactionthattransitionsdata

partitionsstate(fromAtoB)

Page 24: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

Example:WordCount

Page 25: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

SparkArchitecture

Page 26: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

SparkComponents

Page 27: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

SparkDriver

• EntrypointoftheSparkShell(Scala,Python,R)• TheplacewhereSparkContext iscreated• TranslatesRDDintotheexecutiongraph• Splitsgraphintostages• Schedulestasksandcontrolstheirexecution• StoresmetadataaboutalltheRDDsandtheirpartitions• BringsupSparkWebUI withjobinformation

Page 28: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

SparkExecutor

• StoresthedataincacheinJVMheaporonHDDs• Readsdatafromexternalsources• Writesdatatoexternalsources• Performsallthedataprocessing

Page 29: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

DagScheduler

Page 30: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

MoreRDDOperations

Page 31: Lecture 11 Spark - GitHub PagesIntro to Spark • Spark is really a different implementation of the MapReduce programming model • What makes Spark different is that it operates on

Spark’ssecretisreallytheRDDabstraction

Big Data Basics Hadoop, MapReduce, Hive, Pig, & Spark

Apache Spark beyond Hadoop MapReduce

Introduction to big data Analytics using Spark - WestGrid | · PDF file · 2016-04-14Introduction to big data Analytics using Spark . ... Mapreduce: why we need spark Spark Internals

Clustering and Parallel Empowering Techniques for Hadoop ... · and Spark with their Hadoop distributions III.APACHE SPARK Spark is another execution system. Like MapReduce, it works

Thrill : High-Performance Algorithmic...Batch Processing Google’s MapReduce, Hadoop MapReduce , Apache Spark , Apache Flink (Stratosphere), Google’s FlumeJava. High Performance

MapReduce and Spark (and MPI) (Lecture 22, cs262a)...MapReduce and Spark (and MPI) (Lecture 22, cs262a) Ali Ghodsi and Ion Stoica, UC Berkeley April 11, 2018 Context (1970s—1990s)

Cloud Computing using MapReduce, Hadoop, Spark - …parlab.eecs.berkeley.edu/sites/all/parlab/files/hindman_bootcamp... · Cloud Computing using MapReduce, Hadoop, Spark Benjamin

Cloud Computing using MapReduce, Hadoop, Sparkparlab.eecs.berkeley.edu/sites/all/parlab/files/andy_konwinski_parlab... · Cloud Computing using MapReduce, Hadoop, Spark Andy Konwinski

Beyond MapReduce and Spark: CAP, HBase, and Hive · Beyond MapReduce and Spark: CAP, HBase, and Hive ... achieves scalability by using a weaker consistency model that does not ensure

MapReduce - Cornell Universitymajor performance degradations. MapReduce vs Spark. Perspectives • MapReduce, databases, Spark, all differ: • In engineering needs for a particular

Scrap Your MapReduce - Apache Spark

Introduction to MapReduce, Hadoop, Spark, Shark … to MapReduce, Hadoop, Spark, Shark and Mahout Christan Grant [email protected]ﬂ.edu

Distributed Computing with Spark and MapReducerezab/dao/notes/Intro_Spark.pdf · 2018-05-17 · Limitations of MapReduce MapReduce is great at one-pass computation, but inefﬁcient

Lecture 09: Cluster Computing with MapReduce & Sparkweb.csie.ndhu.edu.tw/showyang/DistrSys2018f/09MapReduceSpark.pdf · Lecture 09: Cluster Computing with MapReduce & Spark What is

Spark: Beyond mapreduce

Spark vs. MapReduce

MapReduce and Spark: Overview

Big Data, Hadoop, SpArk, Scala · b.Versiones"de"Apache"Spark c.Comparación"WordCount,"MapReduce" vsWordCountSpark d.ResilientDistributedDataSet(RDD) 4.Introduccióna’Scala a.¿QuéesScala?

Clash of the Titans: MapReduce vs. Spark for Large … of the Titans: MapReduce vs. Spark for Large Scale Data Analytics Juwei Shiz, Yunjie Qiuy, Umar Farooq Minhasx, Limei Jiaoy,

Getting Started with Apache Spark - MapRinfo.mapr.com/rs/mapr/images/Getting_Started_With_Apache_Spark.pdfGetting Started with Apache Spark ... 100 times faster than Hadoop MapReduce,

A Assignment Problems of Different-Sized Inputs in MapReduce

TR-Spark: Transient Computing for Big Data Analytics · TR-Spark: Transient Computing for Big Data Analytics ... data processing systems such as MapReduce, Spark[7], or ... on Azure

Bigdata Hadoop and Spark Development - Acadgild · • Custom Writable In MapReduce • Custom WritableComparable In MapReduce • Schedulers In YARN • FIFO Scheduler • Capacity

Big Data Technology - Hadoop, MapReduce, and Spark

MapReduce - Duke University · We will be using Apache Spark, which is also a compute engine that can process large sets of data. Then, write Spark programs (like MapReduce program)

Evolving a data-driven company from MapReduce to Spark - Ferran Galí Reniu @ PAPIs Connect

From MapReduce to Apache Spark

Introduction to Apache Spark - University of Waterloocs451/slides/... · Hadoop MapReduce Architecture Hadoop v1.0. Hadoop v1.0. Hadoop v2.0. Spark Architecture. Title: Developing

Big Data Processing with MapReduce and Spark

Distributed Computing with Spark and MapReduce

Computational Frameworks MapReduce - DEIcapri/BDC/MATERIAL/MapReduce1920.pdf · 2020-03-21 · MapReduce-Hadoop-Spark Several software frameworks have been proposed to support MapReduce

Introduction to Apache Spark - tropars.github.io · I Hadoop MapReduce, Apache Spark, Apache Flink, etc 25. Agenda Computing at large scale Programming distributed systems MapReduce

Overview of MapReduce and Spark

Unified Big Data Processing with Apache Spark · Unified Big Data Processing with Apache Spark ... MapReduce was great for batch processing, ... Spark can leverage most of the latest

Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark