Download - Drinking From The Firehose - The Erlang Way

V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved

Drinking from the Firehose

Drinking from the

FirehoseMahesh Paolini-Subramanya (@dieswaytoofast)V.P. R&D, Ubiquiti Networks



Drinking from the

Firehose



Drinking from the

FirehoseAka: “Scale Matters”



ARCHITECTURE

DEVICE



Architecture

DEVICE

STATUS



Scale-Fail

DEVICE

STATUS



Data Stores



Data Stores

• State of the World



Data Stores


• Historical Data



Data Stores


• Historical Data

• Configurations



Data Stores


• Historical Data

• Configurations

• Billing

AGILITY



Fault Tolerance

From

http://www.erlang.org/download/armstrong_thesis_2003.pdf



The Big Six

Concurrency

From




The Big Six

Concurrency

Fault detection

From




The Big Six

Concurrency

Fault detection

Fault Identification

From




The Big Six

Concurrency

Fault detection


Error Encapsulation

From




The Big Six

Concurrency

Fault detection


Error Encapsulation

Code upgrade

From




The Big Six

Concurrency

Fault detection


Error Encapsulation

Code upgrade

Stable Storage

From




** exception error: no match of

right hand side value

{error,unhappy}



The Big Six

Concurrency

Fault detection


Error Encapsulation

Code upgrade

Stable Storage

From




** exception error: no match of

right hand side value

{error,unhappy}



The Big Six

Concurrency

Fault detection


Error Encapsulation

Code upgrade

Stable Storage

From




The Fault Tolerant System



The Big Six

Concurrency

Fault detection


Error Encapsulation

Code upgrade

Stable Storage

From




The Big Six

Concurrency

Fault detection


Error Encapsulation

Code upgrade

Stable Storage

From


LOOSE C

OUPLING



Loose Coupling



Loose Coupling

Breeds Trust



Loose Coupling

Breeds Trust

Devotes Brainpower appropriately



Loose Coupling

Breeds Trust

Devotes Brainpower appropriately

Minimizes bugs



Fault Tolerance

60 - 90% of all SW projects fail

10 – 25% of all SW projects get abandoned



The Big Six - Systems

Concurrency

Fault detection


Error Encapsulation

Code upgrade

Stable Storage




Concurrency

Fault detection


Error Encapsulation

Code upgrade

Stable Storage

MONIT

ORING



Monitoring?

Monitoring?• Dashboards


• Out of band systems


• Out of band systems

• Polyglot safety




Concurrency

Fault detection


Error Encapsulation

Code upgrade

Stable Storage




Concurrency

Fault detection


Error Encapsulation

Code upgrade

Stable Storage

Polyglo

t

Persis

tence



Polyglot Persistence

Scale-Fail

DEVICE

STATUS



Data Stores


• Historical Data

• Configurations

• Billing



Data Stores




Data Stores

• State of the World ElasticSearch



Data Stores


• Historical Data



Data Stores


• Historical Data Cassandra



Data Stores



• Configurations

Source: Wholly unscientific research by me.



Data Stores



• Configurations Riak



Data Stores




• Billing



Data Stores




• Billing Postgres



Data Stores





Polyglo

t

Persis

tence



Data Stores





LOOSE C

OUPLING



Data Stores





MONIT

ORING

1

1

1



Split Brain

• Its ubiquitous (thanks aphyr)

http://aphyr.com/



Split Brain


• Zookeeper doesn’t necessarily help

http://aphyr.com/



Split Brain


• Zookeeper doesn’t necessarily help

• Moving the problem around doesn’t necessarily help

http://aphyr.com/

Cheating classroom)



Cheating

• Stale status messages are OK



Cheating


• Charts can miss data



Cheating



• Billing is sharded



Cheating



• Billing is sharded

• Availability zones (google.co.uk)

Scaling




Mahesh Paolini-Subramanya

(@dieswaytoofast)

V.P. R&D, Ubiquiti Networks

Download - Drinking From The Firehose - The Erlang Way

Top Related