stream processing under latency constraints

Stream Processing

under Latency

Constraints

Björn Lohrmann

Daniel Warneke

Odej Kao

Technische Universität Berlin

Background

20.11.2012 2Björn Lohrmann - Stream Processing Under Latency Constraints

Nephele and PACTs currently focus on batch-job

workloads

-to-

What about streaming workloads?

Generally possible with Nephele

PACTs support is WIP

May have different goals

Meet pipeline latency and throughput requirements

Max/Min other custom metrics

Motivation


Live processing of streamed data is also worth looking

at. Some examples:

Incremental search index updates (Google Percolator

replaced MapReduce!)

Social Media streams (see Twitter Storm)

Sensor Networks in science and industry

Multimedia-Streams

User-Generated content from mobile phones

CCTV cameras

Agenda


1. Latency constrained stream processing with Nephele

1. Internal framework design

2. Meeting latency requirements

1. Current Nephele Design Implications

2. Latency Constraints and Measurement

3. Strategy 1: Adaptive Output Buffer Sizing

4. Strategy 2: Task Chaining

5. Experimental Results

2. Streaming on the PACTs layer (WIP)

1. Sliding Window Semantics

Nephele IO Layer Design


Task

n

Task

n+1

Task

n

Task

n+1

Task

n

Task

n+1

Task

n

Compute Node X

Compute Node Y

Compute Node Z

Input Buffer

Queue

Thread/ProcessOutput

Buffer

Data

Item

Sample Application: Video

Livestreaming


Node 1 Node 2 Node n-1 Node n

Decoder

Merger

Overlay

Encoder

Partitioner

RTP

Server

Latency w/o Optimizations


Setup:

200 nodes, 800 cores

32 KB output buffer

size

6400 video streams

Results:

Latency oscillates

around 4s

Large buffers cause

bursts

Implications for Streaming

Applications


Effects of output buffers:

Large buffer = high tp, high latency

Small buffer = low tp, low latency

Trade-off needs to be found to meet latency goals

Thread/Process Model

1 Task= 1 Thread model is flexible, but has overhead

Thread scheduling, synchronization, communication

Serialization may be necessary (bad for TP & latency)

N Tasks = 1 Thread model can sometimes provide

better better tp and latency

Latency Constraints


QoS goal:

Meet latency constraint X, keep

throughput as high as possible

We designed two strategies:

1. Adaptive Output Buffer Sizing

2. Dynamic Task Chaining

Both strategies

work autonomously (only latency constraint is required)

are applied on-demand at runtime

300ms

Measuring Latency

To meet latency constraints we need to

measure first!

General aproach:

Determine which tasks & channels need

measuring

Add & evaluate periodic timestamps to

Determine wait time caused by buffers

Determining task latency is a little trickier

w/o knowing semantics (e.g. reduce)

Ship measurement data to a collector-node

20.11.2012 Björn Lohrmann - Stream Processing Under Latency Constraints 10

300ms

Measuring Latency

Problem:

Combinatorial explosion of

paths through execution graph

for which constraints must be

evaluated

Infeasible to do on a central node

for large-scale workflows

Current approach:

Split execution graph into subgraphs (heuristic)

Assign each subgraph to a worker node responsible for

collecting measurements & applying runtime optimizations

when constraint is violated

Successfully scaled to 200 nodes in experiments


300ms

Adaptive Output Buffer

Sizing


Only applied when latency constraint violated

For each channel

Determine output buffer latency (obl)

If obl > threshold, decrease buffer size:

If obl < threshold, increase buffer size again

200,98.0

),max(:

r

rsizesize obl

310500,1.1

),min(:

r

rsizesize obl

Task Chaining

Conditions:

Pipeline of unchained tasks

Sum of CPU utilizations is <

90% of capacity of one core

(reuse available Nephele

profiling data)

Apply to longest chainable

pipeline of tasks

Control-flow manipulation

requires mp-like tasks


Task

n

Task

n+1

Compute Node

Task

n

Task

n+1

Compute Node

Again, only applied when overall latency constraint is

violated

Schematic Overview


JM

300ms

TM

TM

TM

TM

Payload data

+

Measurements

(lat, tp)

+

Buffer Size Updates

Chain Commands

Scheduled Tasks

+

Distributed

Measurement Setup

Latency w/ Adaptive Buffer

Sizing


Final Latency:

improvement)

Latency /w ABS+TC


Final Latency:

improvement)

Moving further up the


Key steps to push constraints up to PACTs:

1. Find a model to express stream semantics for the

blocking PACTS (reduce, match, cogroup, cross)? ( and implement them in the PACT-runtime

2. Define latency constraint annotations for PACTs

jobs

3. Adapt PACT-compiler to produce streamable plans

and push constraints to Nephele layer

Stream Semantics

Literature shows many different ways of defining

stream semantics for blocking relational operators

Key aspect: The sliding data window

Degrees of freedom:

Tuple-based: take the N most recent tuples

Time-based: take all tuples whose timestamp is fresher

than T time units

Partition-based: partition the stream and take union of

N most recent tuples from each partition

Slide length (tuples or time units)

Timestamp Sources


Timestamp Sources

Obviously for all window types we need timestamped

tuples

Affects determinism:

Depending on where timestamps are added, replaying

the same data will or will not yield the same results

Degrees of freedom:

Timestamping at the data source outside Stratosphere

Yields identical results upon replay

Timestamping within Stratosphere

Yields different results upon replaying (caused by

scheduling, network, etc)


Video Workflow Translated

to PACTs


data src map mapreduce map reduce

Decoder Merger Overlay Encoder RTP-Server

Reduce-Key: Group-IDWindow-Type: Partition-BasedWindow-Size: 1 Tuple (per Partition)Partitioning-By: Stream-IDSlide-Length: 1

Reduce-Key: Group-IDWindow-Type: Tuple-BasedWindow-Size: 1 TupleSlide-Length: 1

Timestamp Source

(Stream-ID, Packet)

(Group-ID, Frame) (Group-ID, Frame)

(Group-ID, Frame)

(Group-ID, Packet)

Data sink

Proposed Model

Which semantics are needed is largely application

dependent

Therefore provide PACTs with common, user-

configurable window semantics

Key configuration parameters:

Window type and slide length

Force the user to define timestamp source locations


stream processing under latency constraints

Technology

task latency

pipeline latency

tcfinal latency

latency problem

latency constraints2

latency constraints3

latency constraints4

latency constraints5