continuous delivery & machine learning · confidential software deployment success is…....

Continuous Delivery &

Machine Learning

Oliver Reesharness.ioharness

Who am I?

Senior Sales Engineer

Lead Systems Engineer

Operations

Developer

Copyright © Harness 2018 Confidential 4

100,000 Components & 100s Engineers

99.9% assembled correctly =100 POF’s

Brand Recognition & Revenue

Win Races

Fast & Reliable Car (3 engines per season)

Ideas & Innovation

Continuous Delivery & Time-To-Market


What is Winning in Formula 1?


Chief Engineer

Change is Constant

50+ New Parts Per Race


Formula 1 Deployment Pipeline

Idea Driver

Verify

Automated

Test

Automated

Build

Semi-Automated

Rollback

Automated

Deploy

Automated

NewPart Mins MinsDays Mins

Wind Tunnel

Simulator

Data Science

CAD

3D Printing

Telemetry

Data Science


Software Deployment Pipeline

CodeCustomer

Verify

Manual

Test

Kinda-Automated

Build

Automated

Rollback

Manual

#WTF

Deploy

Manual

Custom

Scripts

Custom

Scripts

Custom

Scripts

HoursMins Hours/DaysChange

F1 Teams Know the Exact Impact of Every Part/Change

“Increased downforce by 2 points”

“That new front wing is worth 2 tenths”

Verification & Impact is How Teams Win Races

Cars will be ~2 seconds faster by end of season


One BIG Difference (vs. Software Delivery)


Testing & Verification is Critical


Measuring Impact is Also Critical

100,000+ Components, 100s Sensors

TBs of Metrics & Logs over a race weekend

Automation, Context & Feedback is King

Data Science, ML & Modeling

Insight in Minutes vs. Hours or Days


Telemetry From a Formula 1 Car


Automated Verification

New

Baseline

Confidential

Software Deployment Success is….

“Application is up so everything is

fine”

“HomepageWorks”

“Not seeing any alerts”

“No-one has complained yet”


Deploying & Verifying Apps…


Continuous Delivery = Automate Everything

Dev

Stage 1

Static Code Analysis

Unit Tests

Integration Tests

Depend/license Tests

Deployment Pipeline

CustomerCode

QA

Stage 2

Smoke Tests

Regression Tests

Security Tests

Browser Tests

Load/Perf Tests

OnSuccess

Production

Stage 3

Availability Tests

Performance Tests

Quality Tests

Security Tests

Canary VerificationOn

Success

OnCommit

How?


Enter Machine Learning


Disclaimer: Machine Learning Is Not A Silver Bullet

(But its better at doing some things better than humans)


Pop Quiz: Who Identifies Anomalies Faster?

Humans or Machines?


Machine Learning 101

❖ Unsupervised Machine Learning❖ Algorithms identify meaning and context of data on its own❖ E.g. Natural Language Processing (NLP), Neural Nets

❖ Supervised Machine Learning❖ Algorithms require human assistance❖ Neural Feedback (Ignore, Dismiss, Ratings, Yes, No)

❖ NOT….Rules❖ Regex, Thresholds, Configuration or Standard Deviation!


Harness - Applying Machine Learning to CD

Performance

Quality

Security

Availability

Deploy

Rollback

ContinuousVerification

Performance

Quality

Business

Availability

UnsupervisedMachine Learning

Automate Deployment Test/Verify/Rollback

$ Revenue / Throughput


Control and Test Data

❖ Control Data❖ Previous Tests (QA)

❖ Previous Deployments (Prod)

❖ Previous Canary Releases (Prod)

❖ Test Data❖ New Tests

❖ New Deployments

❖ New Canary Releases

Copyright © Harness 2019

Detecting Performance Anomalies

Custom Business Metrics


Automate and Verify Application Deployments

Performance Good


Performance Regression


Revenue Regression

Metric = Timestamp + Name + Value

E.g. “1/1/19 10:01:34, CPU, 99.9%”

This is what Dynatrace, AppDynamics, New Relic, Datadog and Prometheus Provides


Analyze Time-Series Metrics from APM/Monitoring

Symbolic Aggregate approXimation (SAX)


Machine Learning Algorithms Used (time-series)

https://jmotif.github.io/sax-vsm_site/morea/algorithm/SAX.html

https://jmotif.github.io/sax-vsm_site/morea/algorithm/SAX.html


Machine Learning Algorithms Used (time-series)

Hidden Markov Models

https://en.wikipedia.org/wiki/Hidden_Markov_model

Predict when data is incomplete

https://en.wikipedia.org/wiki/Hidden_Markov_model


Detecting Quality Anomalies


Log Providers

Custom Log

Provider


Stack Trace

Error Regressions

Quality Regression


Automate Verification (Unstructured Events)

Usual crap AnomaliesUsual crapAnomalies

Event = Timestamp + Message (N attributes)

[timestamp],[severity],[message],[attribute 1], …

Analyze Splunk, ELK, Sumo Logic Data

Apply Natural Language Processing (NLP)


Tokenize Attributes from Events


Entropy (Frequency of Attributes)

0 1

High EntropyLow Entropy


KMeans Clustering

Group EventsBy

Natural Language Similarity


Jaccard & Cosine Distance

Application Log File has 100 Events

10 of those are NullPointerExceptions (Cluster)

NullPointerExceptions are thrown in 10 different places in the application (different positions relative to each other)

Com.app.package.classA

Com.app.package.classB

Com.app.package.classC


Simple Example


Simple Example

NullPointerException

ArraryOutOfBoundsException

BusinessException

ClassNotFoundException


KMeans Clustering isn’t Perfect

Event Data can contain noise(guids, urls, ip addresses, cookies, …)


Use Trained Neural Nets to Reduce False Positives

Non-Neural Net

This is why EngineersDon’t Trust Machine Learning

Trained Neural Net


Customer Example

$500m+ Revenue eCommerce Retailer


Real Example: Build.com


7 Stage Deployment Pipeline (prod failed)


Deploy & Auto-Verify with New Relic & Sumo Logic


4 New NullPointer Exceptions in Online System


Rollback in 32 Seconds


Traditional Continuous Delivery

Overall Visibility, Dashboards & ReportingCustom

Scripts

Verification

Custom

Scripts

Environment Variables & Pipeline Management

Custom

Scripts

Secrets, Auditing& Compliance

Custom

Scripts

Release Strategy

Custom

Scripts

BLUE/GREEN

Rollback

Custom

Scripts

Continuous Delivery

Continuous Integration

Code

Build & Test Artifacts

Artifact

Cloud Stacks

Custom

Scripts

On-Premises DC

InfrastructureProvisioning

Custom

Scripts

ChangeManagement

Custom

Emails


Continuous Delivery With Harness

Verification

Environment Variables & Pipeline Management

Secrets, Auditing& Compliance

Release Strategy

BLUE/GREEN

Rollback

Continuous Integration

Code

Build & Test Artifacts

Artifact

Cloud Stacks

On-Premises DC

InfrastructureProvisioning

ChangeManagement

Continuous VerificationSmart Automation

Continuous Security

Overall Visibility, Dashboards & Reporting

Continuous Delivery As-A-Service


Thank You

Free 15-Day Trial and Community Editionat www.harness.io

harness

continuous delivery & machine learning · confidential software deployment success is…....

Documents