from mapreduce to apache spark

From MapReduce to Apache Spark: An Ecosystem Evolves Doug Cutting (@cutting) Chief Architect / Apache Hadoop Co-founder

Upload: jen-aman

Post on 23-Jan-2018

936 views

Category:

Data & Analytics

2 download

Report

Download

Embed Size (px):

TRANSCRIPT

1©Cloudera,Inc.Allrightsreserved.

FromMapReduce toApacheSpark:AnEcosystemEvolvesDougCutting(@cutting)ChiefArchitect/ApacheHadoopCo-founder

2©Cloudera,Inc.Allrightsreserved.

TheMergingEcosystems

3©Cloudera,Inc.Allrightsreserved.

Genomics(Broad Institute)

Physics(CERN)

Healthcare Delivery(CHOA)

TheNewUseCasesareAmazing

4©Cloudera,Inc.Allrightsreserved.

BringYourQuestions AboutLivytotheClouderaBooth

Livy.io

• ForsubmittingSparkjobsfromanyweb/mobileapp,withnoSparkclientinstall — enablingnewarchitecturesandusecases

• Providesmulti-tenancyandfaulttolerancetosupportmultipleusersreliably

• Works instandalonemode,withYARN,orwithMesos

• Nocodechangesneeded

• ApacheLicense.Initialcontributors includeemployeesofCloudera,Intel, andMicrosoft+morewanted!

NewContribution:Livy,anOpenSourceRESTServiceforSpark(Alpha)

5©Cloudera,Inc.Allrightsreserved.

Thankyou@cutting

cloudera.com/spark

Online Course Syllabus Template · Storing & Retrieving data with Apache Hadoop HDFS, MapReduce, and Hive Storing & Retrieving data with Apache Hadoop HDFS and Spark Data lakes: A

Overview of MapReduce and Spark

Hive Now Sparks - events.static.linuxfound.org · Apache Hive: a popular data processing tool for Hadoop • Apache Spark: a data computing framework to succeed MapReduce • Marrying

MapReduce - Cornell Universitymajor performance degradations. MapReduce vs Spark. Perspectives • MapReduce, databases, Spark, all differ: • In engineering needs for a particular

Introduction to Apache Spark - University of Waterloocs451/slides/... · Hadoop MapReduce Architecture Hadoop v1.0. Hadoop v1.0. Hadoop v2.0. Spark Architecture. Title: Developing

Community Spotlight Apache MapReduce - Intel · Apache Hadoop* Community Spotlight Apache* MapReduce “This ‘data locality’ is a key design criterion of the MapReduce framework

Budapest Spark Meetup - Apache Spark @enbrite.ly

CV-WANG Guanhua (2016intern) - Peopleguanhua/CV-WANG... · Ø Gemini: Boosting Spark Performance with GPU accelerators Apache Spark is a MapReduce-based platform for large-scale data

Why Apache Spark is the Heir to MapReduce in the Hadoop Ecosystem

Apache FlinkCan easily integrate with Apache Hadoop, Apache MapReduce, Apache Spark, HBase and other big data tools. In-memory management can be customized for better computation

Integrating Apache Hive with Kafka, Spark, and BI...Community Connection: Integrating Apache Hive with Apache Spark--Hive Warehouse Connector Apache Spark-Apache Hive connection configuration

Spark: Beyond mapreduce

Apache Spark - Courses€¦ · Apache Spark Introduction to Data Science DATA11001 Nitinder Mohan CollaborativeNetworking (CoNe) [email protected]. What is Apache Spark?

Getting Started with Apache Spark - MapRinfo.mapr.com/rs/mapr/images/Getting_Started_With_Apache_Spark.pdfGetting Started with Apache Spark ... 100 times faster than Hadoop MapReduce,

Spark vs. MapReduce

NEW ARCHITECTURES FOR APACHE SPARK TM …...NEW ARCHITECTURES FOR APACHE SPARK AND BIG DATA The anatomy of a MapReduce job includes multiple phases, with I/O to …

Spark SQL | Apache Spark

Managed Solutions Apache Spark® · Apache Spark® Apache Spark™ is a high performing engine for large-scale analytics and data processing, While Apache Spark™ provides advanced

Online Course Syllabus Template - Amazon S3 · Storing & Retrieving data with Apache Hadoop HDFS, MapReduce, and Hive Storing & Retrieving data with Apache Hadoop HDFS and Spark Data

Big Data Analysis with Apache - Broad Institute · PDF fileOutline The big data problem MapReduce Apache Spark How people are using it

Thrill : High-Performance Algorithmic...Batch Processing Google’s MapReduce, Hadoop MapReduce , Apache Spark , Apache Flink (Stratosphere), Google’s FlumeJava. High Performance

MapReduce - Duke University · We will be using Apache Spark, which is also a compute engine that can process large sets of data. Then, write Spark programs (like MapReduce program)

Crunch Spark with Apache From MapReduce to · Spark with Apache Crunch Micah Whitacre @mkwhit. Invested in learning. Invested in learning Setup production clusters. Invested in learning

Apache Spark beyond Hadoop MapReduce

MapReduce and Spark: Overview

Apache Spark Introduction · Apache Spark Introduction. Università degli Studi di Modena e Reggio Emilia 2 imore. Università degli Studi di Modena e Reggio Emilia 3 imore MapReduce

Using Apache Spark Pat McDonough - Databricks. Apache Spark spark.incubator.apache.org github.com/apache/incubator- spark [email protected]

Comparing Spark with MapReduce - Glowworm Swarm ...cs.ndsu.edu/~siludwig/Publish/papers/IGI2018.pdfapplications that companies such as Facebook work with (Apache Spark @Scale, 2017)

1 Big Data Hadoop€¦ · · 2017-09-01Data Sampling and Debugging ... 2 Apache Spark & Scala 1 Introduction to Spark Limitations of MapReduce in Hadoop Objectives ... Cassandra

Unified Big Data Processing with Apache Spark · Unified Big Data Processing with Apache Spark ... MapReduce was great for batch processing, ... Spark can leverage most of the latest

Developing Apache Spark Applications - Cloudera · Apache Spark Quick Start Apache Spark Overview Apache Spark Programming Guide Using the Spark DataFrame API A DataFrame is a distributed

Exploring Adverse Drug Effect Data with Apache Spark ... · 23/09/2015 · Apache Spark Overview Open source, Java-based alternative to Hadoop MapReduce platform Execution engine

Introduction to MapReduce, Hadoop, Spark, Shark … to MapReduce, Hadoop, Spark, Shark and Mahout Christan Grant [email protected]ﬂ.edu

Big Data, Hadoop, SpArk, Scala · b.Versiones"de"Apache"Spark c.Comparación"WordCount,"MapReduce" vsWordCountSpark d.ResilientDistributedDataSet(RDD) 4.Introduccióna’Scala a.¿QuéesScala?

· B17 IN V l) 7 -f V y 7 OSS — 11 Apache Tomcat GCC Eclipse Linux MapReduce oss Apache Hadoop Apache Spark Apache Kafta Apache Storm