hadoop summit europe talk 2014: apache hadoop yarn: present and future

© Hortonworks Inc. 2014

Apache Hadoop YARNPresent and Future

Vinod Kumar Vavilapallivinodkv [at] apache.org@tshooter


A quick show of hands..

• Hadoop 2

Architecting the Future of Big Data

Real life Hadoop Logo


Who am I?

• 6.75 Hadoop-years old• Last thing at School – a two node Tomcat cluster. Three months later,

first thing at job, brought down a 800 node cluster ;)• Previously @Yahoo!• Now @Hortonworks• Two hats

– Hortonworks: Hadoop MapReduce and YARN Development lead– Apache: Apache Hadoop YARN lead. Apache Hadoop PMC, Apache Member

• Worked/working on– YARN, Hadoop MapReduce, HadoopOnDemand, CapacityScheduler, Hadoop

security– Apache Ambari: Kickstarted the project and its first release– Stinger: High performance data processing with Hadoop/Hive

• Lots of trouble shooting on clusters• 99% + code in Apache, Hadoop



Agenda

• Apache Hadoop 2 : Overview• Past• Present• Future



Apache Hadoop 2Next Generation Architecture



What is YARN?

• Resource Management Platform– MapReduce v2– Beyond MapReduce with Tez, Storm, Spark; in Hadoop!– Did I mention Services like HBase, Accumulo on YARN with HoYA/Slider?

• How is it different from Hadoop 1? ..



Hadoop 1 vs Hadoop 2

HADOOP 1.0

HDFS(redundant, reliable storage)

MapReduce(cluster resource management

& data processing)

HDFS2(redundant, highly-available & reliable storage)

YARN(cluster resource management)

MapReduce(data processing)

Others

HADOOP 2.0

Single Use SystemBatch Apps

Multi Purpose PlatformBatch, Interactive, Online, Streaming, …


Key Benefits of YARN

• Scale

• New Programming Models & Services

• Improved cluster utilization

• Agility

• To infinity and beyond ..


Why Migrate?

• 2.0 >= 2 * 1.0– HDFS: Lots of ground-breaking features– YARN: Next generation architecture

• Return on Investment: 2x throughput on same hardware!• Ready for improvements in hardware• Not convinced? Let’s see what others are saying!



Yahoo!

• Leader/Visionary on all things Hadoop!• On YARN (0.23.x)• Moving fast to 2.x


http://developer.yahoo.com/blogs/ydn/hadoop-yahoo-more-ever-54421.html


Twitter



Ebay

• Has one of the largest Hadoop clusters in the industry with many petabytes of data

• Migrated production clusters to Hadoop-2• Go to Mayank’s talk

– “Hadoop-2 @ ebay”!– Thursday, April 3– Track : Deployment and Operations

• Should be convinced by now .. . No?



YARN: the Data Operating System



Present



Apache Hadoop releases

• 15 October, 2013• The 1st GA release of Apache Hadoop 2.x• YARN

– First stable and supported release of YARN– Binary Compatibility for MapReduce applications built on hadoop-1.x– YARN level APIs solidified for the future– Performance– Scale!

• HDFS– High Availability for HDFS– HDFS Federation– HDFS Snapshots– NFSv3 access to data in HDFS

• Support for running Hadoop on Microsoft Windows• Substantial amount of integration testing with rest of projects in the

ecosystem


Apache Hadoop 2.2


Apache Hadoop releases (contd)

• 24 February, 2014• First post GA release for the year 2014

• Alpha features in YARN– ResourceManager HA– Application History– Will cover in the 2.4 content

• HDFS– Details follow..

• Number of bug-fixes, enhancements


Apache Hadoop 2.3


HDFS: Heterogeneous Storage



HDFS: DataNode caching



Apache Hadoop releases (contd)

• Very soon!

• YARN– Details follow..– ResourceManager restart fail-over for high availability– Preemption– Application History and timeline

• HDFS– FileSystem ACLs– Rolling upgrades


Apache Hadoop 2.4


ResourceManager Restart and fail-over


ZooKeeper


Capacity Scheduler Preemption



Application History and Timeline

• Few MR specific implementations: History and web-UI• Not just MR anymore!• History

– MapReduce specific Job History Server– Beyond ResourceManager Restart

• Timeline– Framework specific event collection and UIs

• Run analytics on historical apps!



Future



Future: Operational enhancements

• Rolling upgrades– No/minimal impact to users– Ideal: Always rolling!

• HDFS in• YARN



Future: Enabling more apps

• Beyond MR• Discussing next

– Long running services– Isolation– Multi-dimensional resource

scheduling



Future: Long running services

• You can run them already!• Few enhancements needed

– Logs– Security– Management/monitoring

• Resource sharing across workload types

• Project Slider



Fine-grain isolation for multi-tenancy

• Custom memory-monitoring• Cgroups• Linux Containers• VMs



Multi-resource scheduling

• Today – memory & cpu– Physical memory / virtual memory– Cpu Cores – Virtual cores

• CPU stuff: More bake in• Disks

– Space– IOPS

• Network



Other features

• Application SLAs• Node labels• Node affinity/anti-affinity• Better online queue-management



YARN EcosystemBeyond the core YARN project: Briefly



Eco-system

Applications Powered by YARNApache Giraph – Graph ProcessingApache Hama – BSP

Apache Hadoop MapReduce – BatchApache Tez – Batch/Interactive

Apache S4 – Stream ProcessingApache Samza – Stream ProcessingApache Storm – Stream ProcessingApache Spark – Iterative applicationsHOYA – HBase on YARN YARN Frameworks

Apache TwillREEF by MicrosoftSpring support for Hadoop 2

There's an app for that...YARN App Marketplace!


Apache TEZ

• Moving beyond MR• A data processing framework that can execute a complex DAG

of tasks.

• “Apache Tez - A New Chapter in Hadoop Data Processing”– By Siddharth Seth: YARN & Tez Committer/PMC Member– Thursday, April 3 (4:20-5:00pm)



Recap



Recap


• Apache Hadoop 2 is, at least, twice as good!

• Exciting journey with Hadoop for this decade…– Hadoop is no longer a one-trick pony, err elephant– Beyond just HDFS & MapReduce

• Architecture for the future– Centralized data– Exciting spectrum of application types, workloads and usecases


Couple more things..



The Book is out!


http://yarn-book.com/

http://yarn-book.com/


Thank you!

Download Sandbox: Experience Apache HadoopBoth 2.x and 1.x Versions Available!http://hortonworks.com/products/hortonworks-sandbox/

Questions Time!

hadoop summit europe talk 2014: apache hadoop yarn: present and future

Engineering

big data hortonworks

data processing

reliable storage

hadoop mapreduce

hortonworks

apache

yarn

hadoop