Programming clusters with DryadLINQ

Mihai BudiuMicrosoft Research, Silicon Valley

Association of C and C++ Users (ACCU)Mountain View, CA, April 13, 2011

2

Goal

3

Design Space

Throughput(batch)

Latency(interactive)

Internet

Datacenter

Data-parallel

Sharedmemory

DryadSearch

HPC

Grid

Transaction

Execution

Application

Data-Parallel Computation

4

Storage

Language

ParallelDatabases

Map-Reduce

GFSBigTable

CosmosAzure

SQL Server

Dryad

DryadLINQScope

Sawzall,FlumeJava

Hadoop

HDFSS3

Pig, HiveSQL ≈SQL LINQ, SQLSawzall, Java

5

Software Stack: Talk Outline

Windows Server

Cluster services

Cluster storage

Dryad

DryadLINQ

Windows Server

Windows Server

Windows Server

Applications

6

DRYAD

Windows Server

Cluster services

Cluster storage

Dryad

DryadLINQ

Windows Server

Windows Server

Windows Server

Applications

7

Dryad

• Continuously deployed since 2006• Running on >> 104 machines• Sifting through > 10Pb data daily• Runs on clusters > 3000 machines• Handles jobs with > 105 processes each• Platform for rich software ecosystem• Used by >> 100 developers

• Written at Microsoft Research, Silicon Valley

The Dryad by Evelyn De Morgan.

8

Dryad = Execution Layer

Job (application)

Dryad

Cluster

Pipeline

Shell

Machine≈

9

2-D Piping• Unix Pipes: 1-D

grep | sed | sort | awk | perl

• Dryad: 2-D grep1000 | sed500 | sort1000 | awk500 | perl50

10

Virtualized 2-D Pipelines

11

Virtualized 2-D Pipelines

12

Virtualized 2-D Pipelines

13

Virtualized 2-D Pipelines

14

Virtualized 2-D Pipelines• 2D DAG• multi-machine• virtualized

15

Dryad Job Structure

grep

sed

sortawk

perlgrep

grepsed

sort

sort

awk

Inputfiles

Vertices (processes)

Outputfiles

ChannelsStage

16

Channels

X

M

Items

Finite streams of items

• distributed filesystem files (persistent)• SMB/NTFS files (temporary)• TCP pipes (inter-machine)• memory FIFOs (intra-machine)

17

Dryad System Architecture

Files, TCP, FIFO, Networkjob schedule

data plane

control plane

NS,Sched RE RERE

V V V

Job manager cluster

Fault Tolerance

19

DRYADLINQ

Windows Server

Cluster services

Cluster storage

Dryad

DryadLINQ

Windows Server

Windows Server

Windows Server

Applications

20

LINQ

Dryad

=> DryadLINQ

21

LINQ = .Net+ Queries

Collection<T> collection;bool IsLegal(Key);string Hash(Key);

var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value};

22

Collections and Iterators

class Collection<T> : IEnumerable<T>;

Elements of type TIterator(current element)

23

DryadLINQ Data Model

Partition

Collection

.Net objects

24

Collection<T> collection;bool IsLegal(Key k);string Hash(Key);

var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value};

DryadLINQ = LINQ + Dryad

C#

collection

results

C# C# C#

Vertexcode

Queryplan(Dryad job)Data

25

Demo

Example: counting lines

var table = PartitionedTable.Get<LineRecord>(file);int count = table.Count();

Parse, Count

Sum

Example: counting words

var table = PartitionedTable.Get<LineRecord>(file);int count = table

.SelectMany(l => l.line.Split(‘ ‘))

.Count();

Parse, SelectMany, Count

Sum

Example: counting unique wordsvar table = PartitionedTable.Get<LineRecord>(file);int count = table

.SelectMany(l => l.line.Split(‘ ‘))

.GroupBy(w => w)

.Count();

GroupBy;Count

HashPartition

Example: word histogramvar table = PartitionedTable.Get<LineRecord>(file);var result = table.SelectMany(l => l.line.Split(' ')) .GroupBy(w => w) .Select(g => new { word = g.Key, count = g.Count() });

GroupBy;Count

GroupByCountHashPartition

Example: high-frequency wordsvar table = PartitionedTable.Get<LineRecord>(file);var result = table.SelectMany(l => l.line.Split(' '))

.GroupBy(w => w)

.Select(g => new { word = g.Key, count = g.Count() })

.OrderByDescending(t => t.count)

.Take(100);

Sort; Take Mergesort;

Take

Example: words by frequencyvar table = PartitionedTable.Get<LineRecord>(file);var result = table.SelectMany(l => l.line.Split(' '))

.GroupBy(w => w)

.Select(g => new { word = g.Key, count = g.Count() })

.OrderByDescending(t => t.count);

Sample

Histogram

Broadcast

Range-partition

Sort

Example: Map-Reducepublic static IQueryable<S> MapReduce<T,M,K,S>( IQueryable<T> input,

Func<T, IEnumerable<M>> mapper,Func<M,K> keySelector,Func<IGrouping<K,M>,S> reducer)

{ var map = input.SelectMany(mapper); var group = map.GroupBy(keySelector); var result = group.Select(reducer); return result;}

33

Map-Reduce Plan

M

R

G

M

Q

G1

R

D

MS

G2

R

X

X

M

Q

G1

R

D

MS

G2

R

X

M

Q

G1

R

D

MS

G2

R

X

M

Q

G1

R

D

M

Q

G1

R

D

MS

G2

R

X

M

Q

G1

R

D

MS

G2

R

X

M

Q

G1

R

D

MS

G2

R

MS

G2

R

map

sort

groupby

reduce

distribute

mergesort

groupby

reduce

mergesort

groupby

reduce

consumer

map

parti

al a

ggre

gatio

nre

ducedynamic

Expectation Maximization

34

• 160 lines • 3 iterations shown

35

Probabilistic Index MapsImages

features

36

Language Summary

WhereSelectGroupByOrderByAggregateJoin

37

What Is It Good For?

38

What is Kinect?

39

Input device

40

The Innards

Source: iFixit

41

Projected IR pattern

Source: www.ros.org

42

Depth computation

Source: http://nuit-blanche.blogspot.com/2010/11/unsing-kinect-for-compressive-sensing.html

43

Kinect video output30 HZ frame rate

57deg field-of-view

8-bit VGA RGB640 x 480

11-bit depth320 x 240

44

Depth map

Source: www.insidekinect.com

45

Vision Problem: What is a human

• Recognize players from depth map• At frame rate• Minimal resource usage

46

XBox 360 Hardware

Source: http://www.pcper.com/article.php?aid=940&type=expert

• Triple Core PowerPC 970, 3.2GHz• Hyperthreaded, 2 threads/core

• 500 MHz ATI graphics card• DirectX 9.5

• 512 MB RAM

• 2005 performance envelope

• Must handle real-time vision AND a modern game

Why is it hard?

48

Generic Extensible Architecture

Expert 1

Expert 2

Expert 3

Arbiter

StatelessRawdataSensor Skeleton

estimatesFinal

estimate

probabilistic

fuses the hypotheses

Stateful

Background segmentationPlayer separation

Body Part Classifier

One Expert: Pipeline Stages

49

Depth mapSensor

Body Part Identification

Skeleton

50

Sample test frames

51

The Classifier

InputDepth map

OutputBody parts

Classifier

Runs on GPU @ 320x240

52

• Start from ground-truth data– depth paired with body parts

• Train classifier to work across– pose– scene position– Height, body shape

Getting the Ground Truth

53

Getting the Ground Truth

Use synthetic data (3D avatar model)• Inject noise

Motion capture

[Xsens][Vicon]

suit / sensors expensive

very accurate high frame rate

large space calibration

55

Learn from Data

Classifier

Training examplesMachine learning

56

Cluster-based training

Classifier

Training examples

Dryad

DryadLINQ

Machine learning

• > Millions of input frames• > 1020 objects manipulated• Sparse, multi-dimensional data• Complex datatypes

(images, video, matrices, etc.)

57

Highly efficient parallellization

time

mac

hine

58

CONCLUSIONS

Conclusions

59

Cluster

LINQ

Dryad

59

=

60

I can finally explain to my sonwhat I do for a living…