hadoop open platform-as-a-service (hops)/j.dowling.pdfapache hadoop yarn ha/scaleout limitations nm...

Hadoop Open Platform-as-a-Service (Hops)

Academics: Jim Dowling, Seif Haridi

PostDocs: Gautier Berthou (SICS)

PhDs: Salman Niazi, Mahmoud Ismail, Kamal Hakimzadeh, Ali Gholami

R/Engineers: Stig Viaene (SICS), Steffen Grohschmeidt

MSc Students: Theofilos Kakantousis, Nikolaos Stangios, “Sri” Srijeyanthan, Vangelos Savvidis, Seçkin Savaşçı.

What is Systems Research?*

•Systems research is the scientific study, analysis, modeling and engineering of effective software platforms.

•Its challenge is to provide dependable, powerful, performant, secure and scalable solutions within an increasingly complex IT environment.

*Drushel et al, “Fostering Systems Research in Europe”, A White Paper by EuroSys, 2006

Why is Big Data Important?

•In a wide array of academic fields, the ability to effectively process data is superseding other more classical modes of research.

“More data trumps better algorithms”*

*“The Unreasonable Effectiveness of Data” [Halevey, Norvig et al 09]

Bill Gates’ biggest product regret*

http://www.zdnet.com/article/bill-gates-biggest-microsoft-product-regret-winfs/

Windows Future Storage (WinFS*)

•“WinFS was an attempt to bring the benefits of schema and relational databases to the Windows file system. …The WinFS effort was started around 1999 as the successor to the planned storage layer of Cairo and died in 2006 after consuming many thousands of hours of efforts from really smart engineers.”

- [Brian Welcker]*

*http://blogs.msdn.com/b/bwelcker/archive/2013/02/11/the-vision-thing.aspx

Background: Hadoop Filesystem and MapRed

6

HDFS: Hadoop Filesystem

write “/crawler/bot/jd.io/1”

Name node

2 1 3

5 4 6 5 6 6

3 1 4 2 1 3 4 2 5

Heartbeats Rebalance

5

2

1 3

Under-replicated blocks

Re-replicate

blocks

Data nodes Data nodes

Big Data Processing with No Data Locality

Job(“/genomes/jim.bam”)

Workflow Manager

2 1 3 5 6 5 3 6 2 1 4 4 1 5 2 3 4 6

Compute Grid Node Job

submit

This doesn’t scale. Bandwidth is the bottleneck

1 6 3 2 5 4

MapReduce – Data Locality

Job(“/genomes/jim.bam”)

Job Tracker

2 1 3 5 6 5 3 6 2 1 4 4 1 5 2 3 4 6

Task Tracker

Task Tracker

Task Tracker

Task Tracker

Task Tracker

Task Tracker

submit

Job Job Job Job Job Job

DN DN DN DN DN DN

R R R R = resultFile(s)

MapReduce Programming Model

Batch Sequential Processing

Scan Sort

With Fault Tolerance

filter

join

The NameNode

11

HDFS NameNode

•Stores Mappings: path_component -> inode inode -> {block} block -> {replica1,replica2,replica3}

•External API to HDFS Clients

- Internal API to DataNodes

•Monitors Datanodes for failures, corrupted data

•Manages Leases, Quotas, (re-)replication

•Must do all this in a single JVM

- Spotify have a 90GB Heap storing references to 300m files

12

High Availability for the NameNode HDFS 2.x

DN DN DN DN

NN Active

NN Standby

JN JN JN

Shared NN

log stored in

quorum of

journal nodes

NN

Checkpt NN

ZK ZK ZK

Master-Slave

Replication

of NN State.

Agreement on

the Active Master

Faster Recovery,

Cut Journal Log

DOESN’T SCALEOUT !

The Evolution of the NamNode

•HDFS (2006)

- In-memory metadata

•HDFS 0.07 (2006)

- WAL (EditLog)

- FSImage

•HDFS 0.21 (2009)

- Weaken Global Lock

•HDFS 2.0 (2011)

- Eventually Consistent Replication: HA-NameNode

They reinvented

the Database

for the NameNode!

Databases had these features long ago

•Oracle v6 (1988)

- Redo and Undo Logs

- Rollback Segments

•Oracle V7.1 (1994)

- Symmetric Replication

•Oracle 9i RAC (2001)

- Shared State Replication

and have continued to evolve…..

The end of the One-size-fits-All Database

•Columnar Databases

- Vertica, Hana

•NewSQL Databases

- MySQL Cluster, VoltDB, Memstore, AtlasDB, FoundationDB

•Graph Databases

- Neo4J

•RDBMSes

- MySQL, Postgres, DB2, Oracle, SQLServer

•In-Memory Stores

- Memcached, Redis

•Key-Value Stores

- Dynamo, Cassandra, MongoDB, Riak

•Petabyte Databases

- BigQuery (Google), RedShift (Amazon), Impala (Cloudera)

16 Stonebraker et al, “One Size Fits All: An Idea Whose Time Has Come and Gone”, 2005

17

•Distributed, In-memory

•2-Phase Commit

- Replicate DB, not the Log!

•Real-time

- Low TransactionInactive timeouts

•Commodity Hardware

•Scales out

- Millions of transactions/sec

- TB-sized datasets (48 nodes)

•Split-Brain solved with Arbitrator Pattern

•SQL and Native Blocking/Non-Blocking APIs

MySQL Cluster (NDB) – Shared Nothing DB

SQL API NDB API

30+ million update transactions/second

on a 30-node cluster

HopsFS

18

HopsFS

• Customizable and Scalable Metadata

• High throughput for read and write operations

• NameNode failover time≈5 seconds (vs ~1 minute for HDFS)

Request Handling (Apache HDFS vs HopsFS)

Apache HDFS NameNode Request Handling

HopsFS NameNode Request Handling

Fine-Grained Locking, Transactional Updates

21

• NDB gives us READ_COMMITTED isolation-level, not strong enough.

• We implemented Serializability for FS operations using implicit locking

in the DAG and row-level locking in NDB.

[Hakimzadeh, Peiro, Dowling, ”Scaling HDFS with a Strongly Consistent Relational Model for Metadata”, DAIS 2014]

Preventing Deadlocks and Starvation

22

/user/jdowling/dna.bam mv

read

block_report

• Solution: all request threads for inode operations traverse the FS hierarchy

in the same order, acquiring locks in the same order.

• Block-level operations have to

follow the same order.

Per Transaction Cache

•Experimentation revealed many roundtrips to the database per transaction.

•Cache intermediate transaction results at NameNodes.

•We also use Memcached at each NameNode to cache mappings of: path->{inode/blocks/replicas}

Sometimes, Transactions Just ain’t Enough

24

Subtree Operations: 4-phase Protocol • Sacrifices Atomicity, but keeps Isolation and Consistency.

• Batch operations and multithreading for performance.

• Failed NameNodes handled transparently.

• Leases used to handle failed clients.

• Large Subtree Operations with millions of Inodes can’t be executed in a single

Transaction, due to the low timeouts for Transactions (real-time).

Leader Election using the Database (NDB)

•We need a leader NameNode to coordinate replication and lease management

•Use NDB as shared memory for Leader Election.

•No more Zookeeper, yay!

25

HopsFS Internal Protocol Scalability

•On 100PB+ clusters, internal protocols make up most of the network traffic for HDFS

•Block Reporting and Exiting Safe Mode

- Batching and work stealing.

HopsFS Write Performance

27 1 Gbit Network, Nodes: 12-core Xeon X560 @ 2.8 Ghz. 2-Node NDB Cluster.

HopsFS Read Performance

28 1 Gbit Network, Nodes: 12-core Xeon X560 @ 2.8 Ghz. 2-Node NDB Cluster.

HopsFS Erasure Coding

HDFS 2.x Triple Replication

(300%)

2x Replication + XOR (220%)

Reed-Solomon (140%)

HopsFS Erasure Coding

30

Data durability with Triple Replication Data durability with Reed-Solomon

Comparison with HDFS-RAID

HopsFS Snapshots

•Read-Only Root-Level Single Snapshot

- Support rollback on unsuccessful software upgrades

- Prototype developed, ongoing work on integration

- Snapshot rollback order-of-growth is O(N)

We did the same for YARN…

33

34

Apache Hadoop Yarn HA/Scaleout Limitations

NM NM NM NM NM

Standby

RM Primary

RM

Clients Zookeeper

• The Resource Manager (RM) is a bottleneck.

• Zookeeper throughput not high enough to persist all RM state

• Standby resource manager can only recover partial state

• All running jobs must be restarted.

• RM state not queryable.

35

Hops Yarn.

NM NM NM NM NM

RM RM

Client NDB NDB NDB

• The RM is a State-Machine. Almost no session state to manage.

• Transparent failover working.

Hops Yarn

•FIFO Scheduler

•Capacity Scheduler

•Fair Scheduler

•Distributed Resource Tracker Service (ongoing)

•Make YARN more interactive (ongoing)

- Reduce NodeManager Heartbeat Time

36

Hops-Hadoop

NN NN NN

DN

NM

DN

NM

DN

NM

DN

NM

DN

NM

DN

NM

DN

NM

DN

NM

DN

NM

DN

NM

DN

NM

DN

NM

RM RM RM

NDB NDB NDB NDB

Exabyte-Scale Hadoop

HDFS HDFS YARN YARN

The Hops Stack Continued

38

Bringing Data People Together

•Data Owners

- Metadata, Ingestion

- Non-programmers

•Data Scientists

- Data analysts

- Programmers

Hops-HDFS

Hops-YARN

HopsHub Karam

el/PaaS

39

Spark Flink Adam Cuneiform

Perimeter Security and Multi-Tenancy

•HopsHub

- Project-level RBAC

• Hadoop trusted proxy

- Analytics Plugin Framework

• Adam, Cuneiform, Spark, Flink, MR

- REST APIs

Related Hadoop Security Projects

Knox, Sentry, Rhino

Network Isolation

LIMS

LDAP

Kerberos

40

HopsHub Two-Factor Authentication

41

Projects for Multi-Tenancy; Activity Trails

Global Activity Trail

Project

42

Project Membership

43

File Browser (Iceberg)

HDFS Files

Upload Data

Apache Flume

Overcome 3 GB browser upload limit

45

Automated Ingestion of Data

Run Cuneiform Workflows on YARN

46

VCF file FastQ files Results (~250 GB) (~10 GB) (~5 MB)

Variant

Calling Annotate BAM file Align

Ongoing MSc Projects

•Realizing the meta-data dream of WinFS

- Vangelis

•Optimizing YARN’s Resource Tracker Service (interactive YARN)

- Sri

•Interactive Data Analytics (Zeppelin-EE)

- Seckin

47

PaaS support with Chef/Karamel

Support for EC2, Vagrant, Bare Metal. 48

Conclusions

•Hops will be the first European distribution of Hadoop when released.

- First beta release coming in Q1 2015

•Lots of ideas for future work

- Tighter Spark, Flink integration

- BiobankCloud support

•NGS Hadoop Workshop Feb 19-20, Stockholm

- Signup at www.biobankcloud.com

49