ghislain fourny big data for engineers spring 2019 · ghislain fourny big data for engineers spring...

Ghislain Fourny

Big Data for Engineers Spring 20199. Resource Management

artjazz / 123RF Stock Photo

2

Data Technology Stack

Storage

Encoding

Syntax

Data models

Validation

Processing

Indexing

Data stores

User interfaces

Querying

3

Where we are

Storage

Encoding

Syntax

Data models

Validation

Processing

Indexing

Data stores

User interfaces

Querying

4

Last week: MapReduceInput data

Output data

Intermediate data (shuffled)

Map Map Map Map Map Map Map Map

Reduce Reduce Reduce Reduce Reduce Reduce Reduce Reduce

5

Hadoop infrastructure (version 1)Namenode

+JobTracker

/dir/file

Datanode+

TaskTracker

Datanode+

TaskTracker

Datanode+

TaskTracker

Datanode+

TaskTracker

Datanode+

TaskTracker

Datanode+

TaskTracker

6

Responsibilities of the MapReduce JobTracker

Resource Management

MonitoringJob lifecycle

Fault-tolerance

Scheduling

7

Issue 1: scalability

M M MM M MM M MM M M

< 4,000 nodes < 40,000 tasks

8

Issue 2: bottleneck

TaskTracker

JobTracker

TaskTracker TaskTracker TaskTracker TaskTracker TaskTracker

8

Bottleneck

9

Issue 3: Jack of all trades

9

Scheduling

Monitoring

10

Issue 4: Utilization (task slots)

10

Static(Decide on M/R at configuration time)

Fixed-size

11

Issue 5: Not fungible

11

Map Reduce

Working atmaximum capacity

Idle

12

YARN

kirtchanut / 123RF Stock Photo

13

YARN

YetAnotherResourceNegotiator

14

YARN

Scheduling

Applicationmanagement

Monitoring

Resource Manager Application MasterApplication MasterApplication MasterApplication MasterApplication Master

15

Framework-specific application masters

MapReduce

DAG distributed processing

Message Passing Interface

Graph processing

16

Scales more

M

10,000 nodes 100,000 tasks

M M M M

M M M M M

M M M M M

M M M M M

M M M M M

M M M M M

17

YARN architecture

NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager

ResourceManager

18

YARN architecture

NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager

ResourceManager

ContainerContainerContainer

19

Resources

Memory

CPU

Disk

NetworkWor

k in

pro

gres

s

20

Container

X GB

Y TB

W cores, U GHz

Z MBps

21

Container

22

YARN

ResourceManager

NodeManager NodeManager NodeManager NodeManager NodeManager

Container

ContainerContainer

23

YARN

ResourceManager


Container

ContainerContainer

Client

Job

24

YARN

ResourceManager


Container

ContainerContainer

Client

Job

Schedules

25

YARN

ResourceManager


ContainerContainer

Client

Job

Schedules

Application Master

26

YARN

ResourceManager


ContainerContainer

Client

Job

Application Master

27

YARN

ResourceManager


ContainerContainer

Client

Job

Application Master

28

Application Master communicates with containers

Application Master

Container

Container

Container

ContainerExecuteMonitorRestart

29

Pure scheduler

Does not monitor tasks.

Does not restart upon failure.

ResourceManager

30

Scheduling strategies: pluggable scheduler

31


FIFO scheduler

32


Capacity scheduler

Queue 1

Queue 2

33


Capacity scheduler

Queue 1

Queue 2

34


Capacity scheduler

Queue 1

Queue 2

35


Capacity scheduler

Queue 1

Queue 2

36


Capacity scheduler

Queue 1

Queue 2

37


Capacity scheduler

Queue 1

Queue 2

38

Steady Fair Share

40% 10% 50%Math Physics CS

Total: 500 GB

200 GB 50 GB 250 GB

39


Fair scheduler

40


Fair scheduler

41


Fair scheduler

42


Fair scheduler

43


Fair scheduler

44


Fair scheduler

45


Fair scheduler

46


Fair scheduler

47

Summary

Separation between scheduling and monitoring

Scalability

Availability

Multi-tenancy

ghislain fourny big data for engineers spring 2019 · ghislain fourny big data for engineers spring...

Documents