automation for the humans

Automation for the HumansDeveloping automation with the humans in mind

Michael DucyRakuten Technology Conference - 22/10/2016

Michael DucyDirector of Product Marketing

Chef Software, Inc.@mfdii

What do you have to think about immediately after coming up with the idea for an application?

What do you have to think about immediately after coming up with the idea for an application?

Infrastructure

$COMPANY

$COMPANY $APP

$COMPANY $APP $DB

$COMPANY $APP $DB $WEB

$COMPANY $APP $DB $WEB $INFRASTRUCTU

RE

$COMPANY $APP $DB $WEB $INFRASTRUCTU

RE$TECHNICAL_DE

BT

$IAAS_CLOUD

$IAAS_CLOUD

$VM_SERVICE

$IAAS_CLOUD

$VM_SERVICE

$DB_SERVICE

$IAAS_CLOUD

$VM_SERVICE

$DB_SERVICE

$WEB_LAYER

$CONTAINERS

$CONTAINERS

$WEB

$CONTAINERS

$WEB $APP

$CONTAINERS

$WEB $DB$APP

$WEB $DB$APP

$WEB $DB$APP

$FUNCTION

$WEB $DB$APP

$FUNCTION

$BUILDFILE

$WEB $DB$APP

$FUNCTION

$BUILDFILE

$FIXED_APP

$WEB $DB$APP

$FUNCTION

$BUILDFILE

$FIXED_APP

HMM...

HMM...

PRODUCTION?

PRODUCTION?

$WEB $DB$APP

PRODUCTION?

$APP $DB

$APP $DB$APP

$APP

$APP $DB$APP

$WEB

$WEB $WEB

$WEB

$WEB$WEB

PRODUCTION?

$APP $DB$APP

$APP $DB$APP

$APP $DB$APP

$WEB $WEB

$WEB $WEB

$WEB $WEB

$SUPERVISION

PRODUCTION?

$APP $DB$APP

$APP $DB$APP

$APP $DB$APP

$WEB $WEB

$WEB $WEB

$WEB $WEB

$SUPERVISION

$ORCHESTRATION

PRODUCTION?

$APP $DB$APP

$APP $DB$APP

$APP $DB$APP

$WEB $WEB

$WEB $WEB

$WEB $WEB

$SUPERVISION

$ORCHESTRATION

$SERVICE_DISCOVERY

PRODUCTION?

$APP $DB$APP

$APP $DB$APP

$APP $DB$APP

$WEB $WEB

$WEB $WEB

$WEB $WEB

$SUPERVISION

$ORCHESTRATION

$SERVICE_DISCOVERY

$MONITORING

PRODUCTION?

$APP $DB$APP

$APP $DB$APP

$APP $DB$APP

$WEB $WEB

$WEB $WEB

$WEB $WEB

$SUPERVISION

$ORCHESTRATION

$SERVICE_DISCOVERY

$MONITORING

$SUPERVISION

$SUPERVISION

$ORCHESTRATION

$SERVICE_DISCOVERY

$ORCHESTRATION

$SERVICE_DISCOVERY

$MONITORING

$MONITORING

$APP $DB$APP

$APP $DB$APP

$APP $DB$APP

$WEB $WEB

$WEB $WEB

$WEB $WEB

$SUPERVISION

$ORCHESTRATION

$SERVICE_DISCOVERY

$MONITORING

$SUPERVISION

$SUPERVISION

$ORCHESTRATION

$SERVICE_DISCOVERY

$ORCHESTRATION

$SERVICE_DISCOVERY

$MONITORING

$MONITORING

NO-OPS!

$PAAS

$APP

$PAAS

$APP

$FUNCTION

$PAAS

$FIXED_APP

$PAAS

$FIXED_APP

$ADD-ON

$APP2

$PAAS

$FIXED_APP

$APP3

$APP2

$ADD-ON

$ADD-ON

$PAAS

$FIXED_APP

$ADD-ON

$APP2

$ADD-ON

$APP3

$PAAS

$FIXED_APP

$FIXED_APP

$FIXED_APP

$FIXED_APP

$ADD-ON

$APP2

$ADD-ON

$APP3

$PAAS

$FIXED_APP

$ADD-ON

$APP2

$ADD-ON

$APP3

$PAAS

$FIXED_APP

$COMPANY

$ADD-ON

$APP2

$ADD-ON

$APP3

$PAAS

$FIXED_APP

$LEGACY

$COMPANY

$ADD-ON

$APP2

$ADD-ON

$APP3

$PAAS

$FIXED_APP

$LEGACY $DATACENTER

$COMPANY

$ADD-ON

$APP2

$ADD-ON

$APP3

$PAAS

$FIXED_APP

$LEGACY $DATACENTER

$REWRITE

$COMPANY

$ADD-ON

$APP2

$ADD-ON

$APP3

$PAAS

$FIXED_APP

$LEGACY $DATACENTER

$REWRITE

$SECRETS

$COMPANY

$ADD-ON

$APP2

$ADD-ON

$APP3

$PAAS

$FIXED_APP

$LEGACY $DATACENTER

$REWRITE

$SECRETS

$SECURITY

$COMPANY

$ADD-ON

$APP2

$ADD-ON

$APP3

$PAAS

$FIXED_APP

$LEGACY $DATACENTER

$REWRITE

$SECRETS

$SECURITY

$COMPLIANCE$COMPAN

Y

$ADD-ON

$APP2

$ADD-ON

$APP3

$PAAS

$FIXED_APP

$ADD-ON

$APP3

$LEGACY $DATACENTER

$REWRITE

$SECRETS

$SECURITY

$COMPLIANCE$COMPAN

Y

$REWRITE

$ADD-ON

$APP2

Jevon’s Paradox

91

The Looming Complexity Crisis

92


• Growth of connected devices due to cheap compute

• Growth of data generated by these devices

• Growth of compute required to store and process this information

Current automation cannot scaleto the next level of compute.

95


• Commit to tightly bound solutions, before you write any business logic

• Complexity must be dealt with first or it can derail you later

• Infrastructure choices dictate the design of the app

• Proliferation of app stacks and heterogenous environments

• Infrastructure nightmares

Developing “infrastructure-first”

• Business value is in the application

• We start at the bottom and work our way up

• Backwards approach with unintended side-effects (increased complexity, high error rates)

• Instead, start with the business value the work our way down

Developing Human First

• Automation that Reduce Complexity (or perceived complexity)

• Automation that Informs Humans (rather than mechanizes them)

• Automation that Reduces Error

• Automation that Liberates

98

Developing “application-first”

To be successful, the application would need to

• Know how to configure itself for its eventual infrastructure

• Have all of its behaviors and dependencies defined up front

• Know how to adapt to different topologies/environments

• Respond to dynamic changes automatically

• Act with isolation and immutability

Isolation and immutability

• For isolation, our apps are distributed as artifact

• We could expose an interface to change behavior, without changing bits

• That interface could allow dynamic tuning at runtime

• Isolate from downstream dependencies

• Infrastructure is another downstream dependency

Build on past experience

• Failure to isolate infrastructure choices from the app lead to complexity

• Runtimes (e.g. cloud, paas, containers) are rapidly evolving

• Many runtimes only focus on workload placement

• No restrictions on language or execution model

• Cross platform portability is essential

• Support for legacy apps matters

Develop in an application-first model

• Start with the behavior and business value

• Give your apps isolation and immutability

• Expose a mechanism to dynamically reconfigure

• Defer decisions about infrastructure until runtime

• Apps that automatically respond to change

• Do that anywhere

• Do that with new and legacy apps

Habitat Components

Habitat Components

• Habitat Studio - an application packager

Habitat Components


• Habitat Plans - instructions to download, compile, and install

Habitat Components



• Habitat Depot - a place to upload and download your app packages

Habitat Components



• Habitat Depot - a place to upload and download your app packages

• Habitat Supervisor - an intelligent runtime with deployment coordination and service discovery built in

Package Format (.hart)

Plan Artifact Depot

Bare Metal

Containers

AMI

VM

1. Packaging an application starts with a plan. A Plan is as simple to create as a Dockerfile

2. Start by launching the isolated Build environment called a “Studio”

a. Enter the studio (hab-studio enter)

b. Define your Plan (vim plan.sh) to define the software build from source using standard BASH

c. Define the Configuration template for the software, using Mustache (logic-less scripting language)

d. Define the Configuration defaults, using TOML (configuration language)

e. Define Hooks for supervisor behavior: initializing, running, defining a health check, etc

Habitat Workflow

Build Service

Plan Artifact DepotBuild

Service

A Build Service with a workflow to describe the software and behavior for Habitat applications

Explicit about dependenciesIncludes what is configurable about the application

Packages are developed and built in an isolated build environment (hab studio)

Built Artifacts are post-processed in one step to multiple formats including Docker Images

Source Code Repo

Habitat Technology

Depot Artifact

Supervisor: An Intelligent Run-Time Supervisor w/ REST API (hab-sup)

Supervisor Supervisor


Ring

Leader

Initializer

Stand Alone

Part of an Artifact, the Supervisor manages Application node behavior at runtime, providing automation and intelligence:Utilizes predefined Plan to manage application behaviorBuilt in service discoverySelf-organizes with other Supervisor nodes on a Ring into topologiesRings have no single point of failure

Supervisor executes a variety of hooks for defined behavior:

Init, run, health_check, reconfigure, file_update

Supervisor is accessible via REST API:

Habitat Technology

Security & Management

Secret Payloads

Service Service

Service Service

Pub Key

Symmetric Encryption

Service Service

Service Service

Load Balancer

Enterprise-grade security allows for secure runtime management of configurations and payloadsEncrypted, authenticated run-time configurationAutomatic, safe, atomic software updates based upon policy (example: Blue/green deployments)Dynamic topology updatesHandles inter-service discovery through binding

113

Habitat Technology

Habitat Technology

Plan Artifact DepotBuild

Service



Ring

Bare Metal

Containers

AMI

VM

Depot

Build Service & Workflow

Package Format Artifact Distribution Server

Artifact Distribution Server

Intelligent Run-Time Supervisor w/REST API

Source Code Repo

Applications run as distributed, fully automated, and capable autonomous actors

Immutable applications, but flexible and easy to manage because automation travels with the application

Bundles what your apps need to run and nothing else

Embedded automation choreographs application cluster topology/behavior

A network with no reliance on external services and no single-point-of-failure

Provides continuous deployment without traditional Application Release Automation (ARA) tooling

The solution is the same:

Habitat’s approach

HabitatApplication automation that enables modern

application teams to build, deploy, and manage any application in any environment - from

traditional data-centers to containerized microservices.

> Operating System centric; application automation becomes complex

> Focused on integrated SDKs; “Cloud Native” rewrites

> Building tools around applications; pushing in the old direction

> Focuses on the grid as the unit of automation; apps must be specialized

> Focuses on the container as the unit of automation; apps are opaque.

Wait, doesn’t X do that?

We believe that all applications can be autonomous across any platform

PlanCollaborate

Inspect Build

119

Live Demo

Try Habitat for yourself

• https://www.habitat.sh/try

• Tutorials

• Getting started guide

• Extensive documentation

• Open Source!

– https://github.com/habitat-sh/habitatThank you & t-shirts!