Big Data Management and Systems Design

Dr. Weikuan Yu

Associate Professor

Department of Computer Science

Florida State University

Oct 15, 2015 – CS5935 - S2

Life as a Graduate Student

· Course work– Degrees vs. skill sets

· Research– Goals vs. means

· Career development– Presentation and teaching skills– Communication and social skills– Internships– Networking

Oct 15, 2015 – CS5935 - S3

Big Data Challenge

Sources: 1: http://www.infobarrel.com/Evolution 2:http://visual.ly/big-data-explosion?utm_source=visually_embed

Where is Data Coming From ?

3.0 BillionInternet Users

1.35 BillionFacebook Users

550 MillionTweets Per Day

72 Hrs VideoPer Minutes

4.7 Billion Google Search Per Day

And so many others

The dawn of civilization

Year 2003

5 EXABYTES of DATA 5 EXABYTES of DATAEvery 2 Days

Year 2015

7910EXABYTES

Oct 15, 2015 – CS5935 - S4

Big Data Ecosystem

HDFS Hbase

Storage

DisksNetworks

Mahout, Giraph, Pregel, R

MR/Hadoop

Hive, Pig, Shark, Flume

Storm RamCloudSpark

Infrastructure

Applications

Run-Time

Hardware

MemCached

VM, Containers, Public and Private CloudsOS

ComplexityData

Insight

Processors(Accelerators)

Services

Oct 15, 2015 – CS5935 - S5

Research Strategies?· Descriptive research

– Examine and collect facts to document patterns

· Discovery oriented research– Inductive reasoning from patterns to general discoveries

· Engineering-based research– Use existing techniques and theories to create a

technology or tool.

· Hypothesis-driven research– Make a hypothesis and then test the hypothesis using

deductive reasoning

Oct 15, 2015 – CS5935 - S6

Overview of Hadoop MapReduce

Split MapTask

…. ….

Split MapTask

Split MapTask

MOF

MOF

MOF

MOF ReduceTask

MEM

map

ReduceTask

ReduceTask

ReduceTask

shuffle/merge reduce

DFSDFS

JobTrackerAssign MapTasks Assign ReduceTasks

Oct 15, 2015 – CS5935 - S7

1 2 3 4 5 6 70

10

20

30

40

50

60

0

500

1000

1500

2000

2500

3000

Standalone Execution Time (sec)

Sta

nd

alo

ne E

xecu

tion

Tim

e (s

ec)

Groups

Normalized Execution Time (slowdown)

No

rma

lize

d E

xecu

tion

Tim

e

1.5×

Small Job Starvation within Hadoop

52×

Oct 15, 2015 – CS5935 - S8

Hadoop Fair Scheduler

· The mostly widely used Hadoop scheduler

· It is designed to provide fairness among concurrently running jobs.

· Tasks occupy slots until completion or failure

Job Arrival

J-1

J-1

J-1

J-2

J-2

J-2

J-2

J-2

J-3

J-3 J-3

J-3

J-3

J-2

J-2

Time

shuffle

reduce

Slot-R1

Slot-R2

Slot-R3

Slot-M1

Slot-M2

Slot-M3

Slot-M4

Slot-M5

5 Map Slots

3 Reduce Slots

Oct 15, 2015 – CS5935 - S9

How to achieve both Efficiency and Fairness?

· How to correct the non-preemptive nature of reduce tasks for flexible and dynamic allocation of reduce slots?

– Existing schedulers are not aware of such behavior. Once a reduce task is

launched, it stays with the reduce slot till the end.

· How to better schedule two different types of tasks?

– Hadoop schedules both map and reduce tasks with a similar max-min fair

sharing policy without paying attention to their relationship in a job.

– Map and reduce slots need to be dynamically, and proportionally shared.

Objective

Oct 15, 2015 – CS5935 - S10

Preemptive ReduceTasks

· Different from Linux command “Kill –STOP $PID”.

· Lightweight work-conserving preemption mechanism.

– Provides any-time preemption with negligible performance impact.

– Allows a reduce task to resume from where it was preempted.

– Preserves previous computation and I/O.

Oct 15, 2015 – CS5935 - S11

TaskTracker

Heap

Seg

men

t

Merge

seg seg seg

Retrieve

R1: Before Preempt

Heapseg

R1: After Resume

Index

Seg

men

t

Preemption During Shuffle Phase

· Only merge the in-memory segs, while maintaining on-disk segs untouched

Oct 15, 2015 – CS5935 - S12

R1: Before Preempt

MPQ

R1: After Resume

MPQ

DFS

Index

Flush

Ret

rieve

offset

Preemption During Reduce Phase

· Preemption is occurred at the boundary of intermediate <key, val> pairs.

· Recording the current offset of each segment and minimum priority queue

TaskTracker

Oct 15, 2015 – CS5935 - S13

Evaluation of Preemptive ReduceTask

10% 30% 50% 70% 90%3000

4000

5000

6000

7000Work-Conserving PreemptionKilling PreemptionNo Preemption (Baseline)

Completion Ratio of ReduceTask

Jo

b E

xe

cu

tio

n T

ime

(s

e)

Negligible overhead

Oct 15, 2015 – CS5935 - S14

Fast Completion Scheduler

· Strategy: – Find a reduce task for Preemption and select another for Launching,

and balance the utilization of reduce slots

– Decisions to make

· Which reduce task to preempt and for how many times?

· Which one to launch and on which slot/node?

· How to avoid starvation and achieve locality?

· New progress metrics of a job: – Remaining shuffle time

– Remaining data (<k,v> pairs) to be reduced

Oct 15, 2015 – CS5935 - S15

FCS Algorithms

· Preemption algorithm– Select a reduce task from a job with longer remaining time + more

remaining data

– Task slackness: the number of times a task has been preempted

– Avoid starvation: do not preempt a reduce task that has a big slackness.

· Launching algorithm– Select another reduce task from a job with the least remaining time /

remaining data

– Delay a reduce task to maximize data locality.

– Avoid aggressive delay: set a threshold based on the cluster size.

Oct 15, 2015 – CS5935 - S16

1 2 3 4 5 6 7 8 9 1010

100

1000

10000

FCS HFS

· FCS reduces average execution time by 31% (171 jobs).

· Significantly speeds up small jobs at a small cost of big jobs.

Ave

rage

Exe

cutio

n T

ime

(sec

)

10 Groups of Jobs

1.9 2.4 1.9 2.3

1.6

1.9

1.12.2 0.79

Results for Map-heavy Workload

Oct 15, 2015 – CS5935 - S17

1 2 3 4 5 6 7 8 9 101

10

100

1000

10000FCS HFS

Av

era

ge

Re

du

ce

Ta

sk

Wa

it

Tim

e (

se

c)

· Small jobs are benefited from significantly shortened reduce wait time.

· Waiting time are reduced by 22× for the jobs in the first 6 groups.

Average ReduceTask Wait Time

19.512.4

2221 32.2

1.224.5

0.5

0.8

10 Groups of Jobs

27.2

Oct 15, 2015 – CS5935 - S18

1 2 3 4 5 6 7 8 9 1002468

101214161820

Fair Comple-tion

Max

imum

Slo

wdo

wn

· Nearly uniform maximum slowdown for all groups of jobs.

· FCS improves the fairness by 66.7% on average.

10 groups of jobs

Fairness Evaluation: Maximum Slowdown

Oct 15, 2015 – CS5935 - S19

Summary for Coordinated Scheduling

· Identified the fairness and efficiency issues because of the lack of scheduling coordination.

· Introduced Preemptive ReduceTasks for efficient preemption of reduce tasks from long-running jobs

· Designed and Implemented Fast Completion Scheduler for fast execution of small jobs and better job fairness.

Oct 15, 2015 – CS5935 - S20

Broad Research Interests

Big Data· System Design and

Management– Fast Data Movement– Efficient job management– Multi-purpose framework

· High Performance Computing– Parallel computing models– Scalable I/O & communication – Computation & I/O optimization

· Data Analytics– K-mer indexing for sequence

fingerprinting and alignment– Scalable Image Processing– Fast community detection

· Security and Reliability– Analytics logging and Recovery– Cloud security– Storage security

Oct 15, 2015 – CS5935 - S21

Resource and Capabilities· Software: Unstructured Data

Accelerator (UDA)– Accelerator for Big Data Analytics– Transferred to Mellanox

· In-house big data platform– 22 nodes; InfiniBand and 10GigE– SSD and GPGPU (Phi and Kepler)– Donations from Mellanox, Solarflare,

and NVIDIA

Oct 15, 2015 – CS5935 - S22

Sponsors, Contractors, and Collaborators· Current Sponsors and Contractors:

– NSF: Two active grants on big data analytics, storage and network systems, – LLNL: burst buffer based storage systems

· Past Sponsors and Contractors– NASA: one grant for for I/O Optimization of climate applications– DOE Labs: many contracts for high-performance computing.– Industry: contracts from Intel, Mellanox, NVIDIA and Scitor; – Alabama: Innovation Award; Auburn IGP for TigerCloud.

· Collaborators:– IBM, Intel, Mellanox, Scitor, Solarflare, AMD– LBNL, ORNL, LLNL, SNL, GSFC, LPS– Illinois Tech, Clemson, College of William Mary, NJIT, Georgia Tech, Auburn

(OIT, Physics, Biology, SFWS)

Oct 15, 2015 – CS5935 - S23

Research Directions

· Main Thrusts

· Big Data Analytics and systems design

· Network and data privacy and security

· Interdisciplinary data-driven computational research

· Key: solve challenging problems with novel strategies…

· Collaborations

· Students (systems/network oriented, interdisciplinary)

· Faculty (on and off campus)

· National laboratories

· Industry: IBM, Intel, and many more

Oct 15, 2015 – CS5935 - S24

Student Cultivation and Team Building Numerous Internships

ORNL, Sandia, LANL, LLNL, IBM. Awards and honors

First Prize of ACM Grand Finals SC11 Fellowship ($5000). Outstanding students: 2011-2014.

Alumni (Ph.D. listed) Yuan Tian – ORNL Xinyu Que – IBM T.J. Watson Yandong Wang – IBM Watson Zhuo Liu – Yahoo! Cong Xu – Intel Bin Wang – Arista Networks

Current Team 4 Ph.D. students