Load Balancing in Stream Processing Systems

The Power of Both ChoicesPractical Load Balancing for Distributed Stream Processing Engines

Muhammad Anis Uddin Nasir, Gianmarco De Francisci Morales, David Garcia-Soriano Nicolas Kourtellis, Marco SerafiniInternational Conference on Data Engineering (ICDE 2015)

Hey everyone:

I am Anis. A PhD student from KTH Royal Institute of Technology.

The work I will present today is a result of my internship at Yahoo Labs Barcelona

For this work, I was working with people at the top. There is Gianmarco, David, Nicolas and Marco

In our work, we propose a very simple and practical approach for load balancing for Stream processing Engines1Stream Processing EnginesStreaming ApplicationOnline Machine LearningReal Time Query ProcessingContinuous Computation

Streaming FrameworksStorm, Borealis, S4, Samza, Spark Streaming2The Power of Both ChoicesStream processing systems are specialized for low latency and high throughput processing for real time data

Few common domains of streaming applications are online machine learning, real time query processing, continuous computation

Due to the need of real time processing, various frameworks have been proposed in the last decade.2Stream Processing ModelStreaming Applications are represented by Directed Acyclic Graphs (DAGs)WorkerWorkerWorkerSourceSource3The Power of Both ChoicesData StreamOperatorsData ChannelsStreaming Applications are represented as DAGsIn a DAG a vertex are set of operators that are distributed across cluster, which apply various light weight transformation of the incoming stream.For example, filters, join, union, aggregrate are common stream transformation.Edges are data channels that are use to route the data from one operator to anotherIn stream processing systems, there are various stream grouping strategies for routing the keys from one level of operators to the never level of operatorsIn todays talk, I will be concentrating on load balancing between the group of operators at each level of DAG3Stream GroupingKey or Fields GroupingHash-based assignmentStateful operations, e.g., page rank, degree count

Shuffle GroupingRound-robin assignmentStateless operations, e.g., data logging, OLTP

4The Power of Both ChoicesKey grouping uses hash based assignment. It applies a hash function on the incoming data and assign the worker to the message by taking the mod of the hashKey grouping is a good choice for stateful operators like aggregatesShuffle grouping is a simple round roibin assignment scheme.Shuffle grouping is a good choice for stateless operators4Key Grouping5The Power of Both ChoicesKey GroupingScalable Low Memory Load Imbalance

To understand stream grouping strategies, lets take an example of word countSuppose we want to count the words in the tweets from twitter

To implement twitter wordcount, we need to set of operators, A first level of operator to split the tweets into set of words.And the next level of operator to count the words

As we know that many of the real workloads are highly skewed. 5Shuffle Grouping6The Power of Both ChoicesShuffle GroupingLoad Balance Memory O(W) Aggregation O(W)

Problem FormulationInput is a unbounded sequence of messages from a key distribution

Each message is assigned to a worker for processing

Load balance propertiesMemory Load BalanceNetwork Load BalanceProcessing Load Balance

Metric: Load Imbalance

The Power of Both Choices7

Talk about different load balance schemes.

MemoryNetworkProcessing

In particular, we are interested in balancing the workload for processing. So total number of messages processing7

Power of two choicesBalls-and-bins problem

AlgorithmFor each ball, pick two bins uniformly at randomAssign the ball to least loaded of the two bins

IssuesDistributed Consensus on Keys Skewed distribution Continuous DataLoad Information

8The Power of Both ChoicesImg source: http://s17.postimg.org/qqctbpftr/Galton_prime_box.jpg An elegant solution is using the power of 2 choices:

In the past it has been introduced as a balls and bins problem:

Given a ball (that means a key in our setup) and a set of bins (workers)pick 2 random bins, check which one has the least load of balls and send the new ball to that one.

Surprisingly, this simple strategy leads to an imbalance that is independent of the balls thrown at the bins.

So this looks like a good solution.However, when applied in DSPEs, it has some complications

Better than hashing Power of choices takes two bins and put at random, and it is better load balance

8Partial Key Grouping (PKG)Key SplittingSplit each key into two serverAssign each instance using power of two choices

Local Load Estimationeach source estimates load on using the local routing history

9The Power of Both Choices9Partial Key Grouping (PKG)Key SplittingLocal Load EstimationThe Power of Both Choices10SourceSourceWorkerWorkerWorker201102202Partial Key Grouping (PKG)Key SplittingDistributedStatelessHandle Skew

Local load estimationNo coordination among sourcesNo communication with workers

11The Power of Both Choices

11Partial Key GroupingThe Power of Both Choices12PKGLoad Balance Memory O(1) Aggregation O(1)

It uses twice at most memory than key grouping

12Analysis: Chromatic Balls and BinsProblem FormulationIf messages are drawn from a key distribution where probabilities of keys are p1p2p3.. pnEach key has d choices out of n workers

Minimize the difference between maximum and average workload

The Power of Both Choices13

AnalysisNecessary Condition: If pi represents the probability of occurrence of a key i

Bounds:The Power of Both Choices14

Streaming ApplicationsMost algorithms that use Shuffle Grouping can be expressed using Partial Key Grouping to reduce:Memory footprintAggregation overhead

Algorithms that use Key Grouping can be rewritten to achieve load balanceThe Power of Both Choices15Streaming Examples Nave Bayes Classifier

Streaming Parallel Decision Trees

Heavy Hitters and Space Saving

The Power of Both Choices16Motivate the people to read the paper16Stream Grouping: SummaryGroupingProsConsKey GroupingScalableMemoryLoad ImbalanceShuffle GroupingLoad BalanceMemory O(W)Aggregation O(W)Partial Key GroupingScalableLoad BalanceMemory O(1)Aggregation O(1)

The Power of Both Choices17ExperimentsWhat is the effect of key splitting on POTC?

How does local estimation compare to a global oracle?

How does PKG perform on a real deployment on Apache Storm?

The Power of Both Choices18We performed various experiments to assess the performance of our technique.

We compare with pure PoTC to study the effect of key splitting

We study how well local load estimation reaches a solution which is similar in balance as with when having global information

We study how robust PKG is to shifting skew

We implemented the PKG on Apache Storm to see how well it does on a real DSPE system.

18Experimental SetupMetricthe difference of maximum and the average load of the workers at time t

DatasetsTwitter, 1.2G tweets (crawled July 2012)Wikipedia, 22M access logsTwitter, 690K cashtags (crawled Nov 2013)Social Networks, 69M edgesSynthetic, 10M keys19The Power of Both Choices

We measured imbalance in the system:

The maximum load observed across workers the average load across workers.

Compared global, local, and a version that does probing of load on workers at regular intervals

Used different real and synthetic data: tweets, wikipedia page access, etc.

19Effect of Key SplittingThe Power of Both Choices20

Here, we compare PKG with regular POTC, a greedy online and offline algorithm and hashing.

We show average imbalance across time, given different number of workers, for Wikipedia and Twitter

Online greedy picks the least loaded worker to handle a new key.

Offline greedy first sorts the keys by frequency and then executes online greedy.

Hashing just applies a hash function on the keys (so its the KG version)

-> Hashing performs the worst.

-> PKG performs very well and similar to the Greedy algorithms

-> Adding workers increases imbalance.

Hashing is the single choice paradigm

20Local Load EstimationThe Power of Both Choices21

This is the average imbalance across the system and time, for Twitter, Wikipedia, Cashtags and a synthetic lognormal distribution.

We study how well the local load estimation compares with global information.

It is always very close, regardless of how many sources we allow the system to have.

Some different patterns for the various datasets, but the trend is the same.

Hashing is worst.

Hashing is just for reference

21Real deployment: Apache StormThe Power of Both Choices22

These are some results from a real deployment on Storm.

We tried to simulate different working time per key. Fore example, reading 400KB from memory is 0.1ms and 1/10th of disk seek is 1ms

On the left plot, we don't have aggregation phase but apply different CPU load per key and measure the throughput supported.

On the right plot, we keep constant the CPU load to 0.4msecs and vary how often we do aggregation. 0.4 is close to saturation to the system.The less frequent is the aggregation, the more the memory cost.

We see that PKG can offer similar or better throughput than SG for the same or smaller memory cost.

22ConclusionPartial Key Grouping (PKG) reduces the load imbalance by up to seven orders of magnitude compared to Key Grouping

PKG imposes constant memory and aggregation overhead, i.e., O(1), compared to Shuffle Grouping that is O(W)

Apache Storm 60% improvement in throughput45% improvement in latency

PKG has been integrated in Apache Storm ver 0.10.

23The Power of Both ChoicesFuture WorkLoad Balancing for Stateful Operators using key migration

Adaptive Load Balancing for highly skewed data

Load Balancing for graph processing systemsThe Power of Both Choices24The Power of Both ChoicesPractical Load Balancing for Distributed Stream Processing Engines

Muhammad Anis Uddin Nasir, Gianmarco De Francisci Morales, David Garcia-Soriano Nicolas Kourtellis, Marco SerafiniInternational Conference on Data Engineering (ICDE) 2015

25