stackengine problem space demo

Managing Docker Applications in Production

Boyd E. Hemphill - @behemphi Mike Raab - @stackengine

History - Boyd

Oracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged), Evangelist

History - Boyd

Oracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged)

Started Austin DevOps In 2012

History - Boyd

Oracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged)

Started Austin DevOps In 2012

Co-Organizer of DevOps Days Austin

History - Boyd

Oracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged)

Started Austin DevOps In 2012

Co-Organizer of DevOps Days Austin

Became involved in Docker Austin in August.

History - BoydOracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged)

Started Austin DevOps In 2012

Co-Organizer of DevOps Days Austin

Became involved in Docker Austin in August.

Working with Docker on the first Docker Days Austin

History - Boyd

Oracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged)

Started Austin DevOps In 2012

Co-Organizer of DevOps Days Austin

Became involved in Docker Austin in August.

Working with Docker for the first Docker Days Austin

Keep an eye on @stackengine to see if we can make it happen!

History -StackEngine

CopperEgg

History -StackEngine

CopperEgg

Started in May 2014 to address challenges facing Docker adopters

History -StackEngine

CopperEgg

Started in May 2014 to address challenges facing Docker adopters

Funding announcement October

History -StackEngine

CopperEgg

Started in May 2014 to address challenges facing Docker adopters

Funding announcement October

Currently in private beta

History -StackEngine

CopperEgg

Started in May 2014 to address challenges facing Docker adopters

Funding announcement October

Currently in private beta

mike@stackengine.com

Common Ground

What is The Goal of Your

Company?

What is The Goal of Your

Company?Make Money!

So … What is DevOps?

Is DevOps a Process?

Is it an intersection of overlapping concerns?

Is DevOps a Culture?

DevOps:

DevOps is the way in which a technology organization embeds itself in a business to the benefit of that business.

The Ideal Docker Application

Netflix (unicorn)Service Oriented Architecture

Benefits

Feature velocity - Increase developer productivity using containers to model application topology

Benefits

Feature velocity - Increase developer productivity using containers to model application topology

Innovation velocity - Decreased build and test times make more time for trying that next great idea!

BenefitsFeature velocity - Increase developer productivity using containers to model application topology

Innovation velocity - Decreased build and test times make more time for trying that next great idea!

Density - Reduce Costs

Benefits

Feature velocity - Increase developer productivity using containers to model application topology

Innovation velocity - Decreased build and test times make more time for trying that next great idea!

Density - Reduce Costs

Separation of Concerns - “Code to an interface” moves to the infrastructure level

Benefits

Taken together a company that can get to the end state has a significant and lasting competitive advantage

Today’s Docker Reality

Concerns

This is still very early days for Docker

Concerns

This is still very early days for Docker

Tools to support Dockerized applications are immature or do not exist

Concerns

This is still very early days for Docker

Tools to support Dockerized applications immature or do not exist

Service oriented architecture is hard

Concerns

This is still very early days for Docker

Tools to support Dockerized applications immature or do not exist

Service oriented architecture is hard

Increasing developer throughput is only a local optima, thus a false economy

Concerns

Docker adoption is the first major DevOps initiative not driven by Ops.

Dev Driven DevOps

DevOps is hallmarked by Ops adopting the best practices of developers, servers as code, check in all the things, test driven infrastructure

Dev Driven DevOps

DevOps is hallmarked by Ops adopting the best practices of developers, servers as code, check in all the things, test driven infrastructure

The Docker Revolution is the first mass DevOps initiative that harkens from Developers applying ops thinking

Dev Driven DevOps

There is a huge social engineering effort along side the tech

Dev Driven DevOps

There is a huge social engineering effort along side the tech

DEFN: The application of the findings of social science to the solution of actual social problems.

SDLC

Developers are moving forward

SDLC

Developers are moving forward

QA sees production modeling, large test grids, more effective integration testing

SDLC

Developers are moving forward

QA sees production modeling, large test grids, more effective integration testing

Build engineers see parallelized build grids for faster feedback and better build quality

xkcd … er … SDLCOperations panics - no tools or instrumentation

The Final Mile

The Final Mile

Developer uses containers

The Final Mile

Developer uses containers

System is built and images are produced

The Final Mile

Developer uses containers

System is built and images are produced

Images stored in a private registry

The Final Mile

Developer uses containers

System is built and images are produced

Images stored in a private registry

Now we need to deploy to a QA environment

The Final Mile

So we build a manual solution depending on human behavior

The Final Mile

So we build a manual solution depending on human behavior

Someone builds a tool

Docker DeploymentCreate a component to represent your registry

Docker DeploymentCreate a component to represent the image you would like to run

Docker DeploymentDistribute that image and run it on the hosts in your Mesh.

Docker DeploymentMagic happens.

Docker DeploymentLife is good

Deployment

StackEngine is developing the tooling to make this policy based

Docker DeploymentFluffy bunnies and rainbows ensue

Container Sprawl

Container Sprawl

First there was VMware and suddenly a large corps departments could easily spin up environments

Container Sprawl

First there was VMware and suddenly a large corps departments could easily spin up environments

Then there was the cloud and developers could complete circumvent Ops

Container SprawlFirst there was VMware and suddenly a large corps departments could easily spin up environments

Then there was the cloud and developers could complete circumvent Ops

Now there is Docker and Developers are creating numerous abstractions that are not managed like traditional VMs

Container Sprawl

Containers on VMs on phyical hardware located in availability zones in data centers in geographic regions.

My Head HurtsContainer Sprawl

Container Sprawl

StackEngine is working to make this policy based

Inventory Management

Where’s My StuffHosts

Where’s My StuffImages

Where’s My StuffContainers

Where’s My StuffGeography, Data Center, Zone

Telemetry

Telemetry

What does monitoring a docker app look like?

Telemetry

What does monitoring a docker app look like?

http://goo.gl/VkAQo4

Telemetry

What does monitoring a docker app look like?

http://goo.gl/VkAQo4

What does log aggregation look like?

Telemetry

What does monitoring a docker app look like?

http://goo.gl/VkAQo4

What does log aggregation look like?

http://goo.gl/uNPe9P

What’s my stuff (not) doing

I have this much stuff.

What’s my stuff (not) doing

How are the containers on a host doing

What’s my stuff (not) doing

What is the container state?

What’s my stuff (not) doingMore Location, Host and Container metrics coming!

Components

Components

Components are infrastructure abstractions (cloud provider, data center, docker registry, docker image)

Components

Components are infrastructure abstractions (cloud provider, data center, docker registry, docker image)

Components can be arranged in logic ways to represent physical infrastructure.

Composing an AppWhere are your components coming from

Composing and AppWhat makes the application

Composing an AppStart the db

Composing and AppStart the web server

Composing and AppRun the application server

Composing and AppRun the Cache

Service Discovery

Service Discovery - Consistency Models

Zookeeper

Etcd / Consul

Chef Server / Heira

Service Discovery

Allows multiple deployments of the same service on the same host.

Service Discovery

Allows multiple deployments of the same service on the same host.

Containers register themselves upon startup and are found by tags rather than URI’s, fixed ports, etc.

Service Discovery

Allows multiple deployments of the same service on the same host.

Containers register themselves upon startup and are found by tags rather than URI’s, fixed ports, etc.

We favor something like consul with a search interface like the chef server.

Service DiscoveryAllows multiple deployments of the same service on the same host.

Containers register themselves upon startup and are found by tags rather than URI’s, fixed ports, etc.

We favor something like consul with a search interface like the chef server.

A step towards cattle containers instead of pet containers

Roadmap

Roadmap

Service Discovery

Roadmap

Service Discovery

User Management

Roadmap

Service Discovery

User Management

Addition of new Components

Roadmap

Service Discovery

User Management

Addition of new Components

Better telemetry

Roadmap

Service Discovery

User Management

Addition of new Components

Better telemetry

Scheduling

Roadmap

Service Discovery

User Management

Addition of new Components

Better telemetry

Scheduling

A/B Deployment support

Live Product DemoAl Gore and his gremlins willing

Boyd E. Hemphill - @behemphi Mike Raab - @stackengine

Thank You for Your Time and Comments.