sdn presentation full lifecycle government cloud open ... · provides virtualized resources...

Sardina Systems Proprietary. Copyright (C) 2014, 2015, 2016, 2017. Sardina Systems. All rights reserved.

lessons learnt: large scale government OpenStack private cloud


1. the client highly technical, critical operations


the client

government x

disparate user groups, varying and competing needs, long-lived and short-lived workloads

technically astute organization


background

challenges with energy consumption, system efficiency

need to squeeze more out of the IT investment

understands gross wastage within their traditional solution


background

need to meet organizational mission

don’t throw money at the challenge

money does not grow on trees


2. OpenStack

flexible, widely adopted

IaaS platform


what is OpenStack?

at the most basic level: management layer for virtualized compute, networking, storage resources throughout a datacenter

modern web ui dashboard: gives administrators control while empowering self-service by users


what does OpenStack offer?

flexible compute, storage, networking


lifecycle

deploy

operate

upgrade


how do we turn this into a solution for the client?


3. the solution system for production, not-a-toy


route to solution

problem definition and solution design

system architecture

hardware specification

project management: what’s happening when, by who


route to solution

plan for production from day 0

build for production from day 0

it’s not a toy: high availability is not optional

monkey-capable no-magic scaling up process


route to solution

your client != your guinea pig


4. deploy showtime: get solution design to work


from design to deployment

the ingredients: what’s the configuration needed

ingredients ready: make sure hardware are available (servers, storage, networking)

system configuration: single source of truth at all times


automate, automate, automate

system configuration = input for automation

deployment of entire system: fully automated

reduced project risk = increased operational confidence


deploy: 1, 2, 3 … showtime

1. input: configuration and inventory files

2. run deployer

3. 1 or 2 cups of coffee

… done


deployer

pre-tested and proven off-site

ci/cd process

no knowledge of OpenStack necessary: enable factory deployment, integration and testing


5. upgrade reliable, interruption-free upgrade


run at all times

n to n+1 upgrade every 6 months (releases eol in 12 months)

2n upgrades in n years: not an option

it’s not a toy: zero-downtime is not optional

system has to continue operation at all times


zero-downtime

get solution architecture right (from deploy phase) … else, find out problem 6 months later

monkey-capable fully automated upgrade process


6. reliability dead is not an option: averting

service death


new expectations

on demand

it should just work


new roles

operator — vs — consumer


operator

operation of underlying infrastructure

provides virtualized resources (compute, networking, storage) … and stops at this!


consumer

operation within the virtualized environment

not concerned about underlying infrastructure


consumer

don’t care about what it takes to provide the cloud environment

it should just work


operator

responsibility to keep cloud environment running


dead

in event the service is dead: expectation no longer met


dead is not an option

if it is dead, it is too late!


when could there be risk of death?


operate

system is deployed, now … keep services running

high-availability is not a luxury


risk of death

services encountering problems + impacting consumers


death during operate phase

intrinsic — vs — extrinsic

death risks


extrinsic death risk?

dealt with via highly available solution architecture


what is OpenStack?

a series of intercommunicating services

HTTP, MQ


types of services

1. with data + configuration

2. configuration only


safeguards: configuration

infrastructure-as-code operating model

must be able to re-deploy components by re-running deployer


safeguards: data

replication, replication, replication

2-node model or quorum/odd-node model


safeguards: data

traditional high availability operations model


anything else?


what else?

http: it’s a web server! let’s treat it as such!

high availability for web servers


what else?

mq: put stuff in queue, take stuff from queue

replicate reader/writer


extrinsic death risk

keep extra copy available at all times


is that all?

yes


smarts?

it’s in the solution architecture

get it right, else it can come back to bite you!


intrinsic death risk

it’s intrinsic, not externally triggered


intrinsic death risk

traditional operation model: monitor, alert, action


monitor

look for dead or alive

if it looks like it is dead, check again x 3, to be sure


monitor

… but it’s too late if it is dead!


intrinsic death risk?

it’s intrinsic!

highly available solution architecture will not suffice


smarts?

need to detect sick states

sick: alive, not healthy


smarts

correlation between x and y

not just causality (that’s easy!)


OpenStack architecture


causality? correlation?


challenge

non-trivial computational task

plus, “sick” in one environment may not be “sick” in another


data

2 types of data

metrics

logs


it needs to …

example: response time behavior of Web service + disk SMART errors

detect sick states that are otherwise below the radar


how?

logs ingest from OpenStack infrastructure

time series store for machine generated data

watch log data for anomalies


analog

take continuous x-ray pictures of patient

feed through decision engine

if detect certain dark spots, the patient may be “sick”


analog

if decision engine got it wrong, need to re-train

next time, dark spots will be classed as “sick”


health

picture covers a certain configurable/variable time window

constituents of the picture: configurable

operator to trigger re-training on wrong decisions


health

shortcuts

pre-condition: has to be alive to check for “sick”

causality relationships


not a toy

highly available

no-polling, operate on pushed data streams


not a toy

back pressure sensitive: not to overload data pipeline

horizontal scalable data store

load balancing across multiple backends


not a toy

high availability for an evaluation system?


massive volumes of data

how much: x00s metrics per data source

volume: potentially saturating x00 GbE network


smarts?

smart data ingest reduction

imbalanced I/O pattern: large volume of small writes, small number of large reads


deal with risk of death and …

know imminent fault

deal with latent threat, before threat becomes patent

meet consumer expectations


avert death


7. efficiency reduce, optimize away wastage, raise

efficiency, increase utilization


efficiency

it’s about what you do with what you have


reservation

what you asked for


utilization

what is actually used


… what’s your utilization level?


traditional hpc sites: “90+% utilization according to scheduler” … but is that reservation or utilization?

the client: understands the wastage in their traditional environment


a large public cloud operator: “90% memory reservation whilst average cpu was 6% and average max cpu was 15%; this is not unusual”


leave lights on?

do you leave lights on at home for the whole week if you are home for a day?

… that’s 14% utilization!


what to do?

need data, detailed data

what’s happening in each part of cpu operating unit, memory subsystem, i/o subsystem

form full picture of system utilization


have data, then?

large scale n-dimensional jigsaw puzzle problem

each parameter = 1 dimension


2 classes of problems

placement: where to place new VMs

rebalancing: how to optimally lay out VMs, optimal number of hosts


2 classes of problems

placement: where to place new piece

rebalancing: dimensions have changed, re-do jigsaw puzzle

solve jigsaw puzzle with x00s of dimensions


why do we need all these parameters?

answer: try packing parcels while having only height information, not having width nor depth?


the impact

the client: raised utilization from 20% to 60% (leaving 40% head room)

3x utilization increase

66% energy reduction


efficient resource management

operating system-based model, throughout entire facility

dynamic, efficient: lower OpEx + CapEx


8. conditional

flexible, widely adopted

IaaS platform


conditional engine

fast, scalable, highly available

handle large volumes of conditions to evaluate


… topics for another day


9. sardina systems


this is Sardina Systems

full-lifecycle automated OpenStack: deploy, operate, upgrade

AI-driven smart, efficient, super-scalable automation technology


enable organizations to rapidly experience the value of OpenStack cloud and maximize utility of their resources

sectors: finance, government, aerospace, research, academia

in 2015, Sardina FishOS won the IDC HPC Innovation Award


top-10 trader on New York market

U of Edinburgh: top-5 UK academic and research site

150k VMs at classified government site


lessons learnt: large scale government OpenStack private cloud

dr kenneth tan [email protected]

+44 798 941 7838

mailto:[email protected]

sdn presentation full lifecycle government cloud open ... · provides virtualized resources...

Documents