incrementalism: an industrial strategy for adopting modern automation
TRANSCRIPT
1. Development Practices 2. Secrets Management 3. Packaging 4. Developer-centric, Self-Healing Applications 5. Data Center Aware Services 6. Infrastructure Manipulation
DATA CENTERS ARE YOUR FACTORIES.
NETWORKS ARE YOUR ROADS.
YOU APP PRODUCES WIDGETS.
MICROSERVICES ARE YOUR ROBOTS ON THE FACTORY LINE.
• New program! • Use a piece of user data • Talks to a database • Returns something useful
• Highly-Available • Self-Healing • Feedback
• New program! • Use a piece of user data • Talks to a database • Returns something useful
• Highly-Available • Self-Healing • Feedback
BUSINESS VALUE
• New program! • Use a piece of user data • Talks to a database • Returns something useful
• Highly-Available • Self-Healing • Feedback
INPUT
• New program! • Use a piece of user data • Talks to a database • Returns something useful
• Highly-Available • Self-Healing • Feedback
DEPENDENCY
• New program! • Use a piece of user data • Talks to a database • Returns something useful
• Highly-Available • Self-Healing • Feedback
PROPERTIES OFTHE APPLICATION
• New program! • Use a piece of user data • Talks to a database • Returns something useful
• Highly-Available • Self-Healing • Feedback
VALUE
• New program! • Use a piece of user data • Talks to a database • Returns something useful
• Highly-Available • Self-Healing • Feedback
DISTRACTING, REQUIRED, NECESSARY COMPLEXITY
• New program! • Use a piece of user data • Talks to a database • Returns something useful
• Highly-Available • Self-Healing • Feedback
WIDGET
• New program! • Use a piece of user data • Talks to a database • Returns something useful
• Highly-Available • Self-Healing • Feedback
INDUSTRIAL COMPUTING
RAW MATERIAL
REDUNDANT ROBOTSREPLACEABLE PARTSQUALITY-MANAGEMENT
WIDGETSTORAGE
Codified Development Environment
Reproducible Dev Environments
Disposable R&D Workspace
Shared Developer Workspace
Developer-driven Infrastructure
Terminal
$ $EDITOR Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.box_url = "https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-vagrant.box"
config.vm.network "private_network", ip: "192.168.33.10"
config.vm.provision "shell", path: "setup.sh"
end
$ cat setup.sh
apt-get install postgresql-server tmux
Terminal
my-laptop$ vagrant up --destroy-on-error
my-laptop$ vagrant ssh
vm$ uname -a | tee /vagrant/uname.out
Linux compton 2.6.24-19-server #1 SMP Sat Jul 12 00:40:01 UTC 2008 i686 GNU/Linux
vm$ logout
Shared connection to 192.168.39.130 closed.
my-laptop$ cat uname.out
Linux compton 2.6.24-19-server #1 SMP Sat Jul 12 00:40:01 UTC 2008 i686 GNU/Linux
Terminal
$ $EDITOR Vagrantfile
Vagrant.configure("2") do |config|
config.ssh.shell = "sh"
config.vm.synced_folder ".", "/vagrant", nfs: true, id: "vagrant-root"
end
Terminal
my-laptop$ $EDITOR myapp.go
my-laptop$ GOOS=linux GOARCH=amd64 go build -o myapp-linux
my-laptop$ GOOS=freebsd GOARCH=amd64 go build -o myapp-freebsd
my-laptop$ vagrant ssh
vm$ /vagrant/myapp-linux
vm$ logout
my-laptop$ vagrant up freebsd-vm1
my-laptop$ vagrant ssh freebsd-vm1
freebsd-vm1$ /vagrant/myapp-freebsd
freebsd-vm1$ logout
"I want to deploy this app to prod. Password-less logins are disabled on the databases! Now what?"
Private GitHub repo Commit passwords inline in SCM Switch creds based on $HOSTNAME?
Establish a protocol for acquiring credentials at runtime
Terminal
$ vault read postgresql/creds/readonlyKey Value--- -----lease_id postgresql/creds/readonly/5fec46f2-ab40-d9b8-61a2-887c7946eeb6lease_duration 1h0m0slease_renewable truepassword f8a93086-b11d-10cd-8795-f537a10de712username token-9e57c18f-ac99-8e29-48f2-3fb09066d2b4
Terminal
$ VAULT_ADDR=http://vault.service.consul vault read postgresql/creds/readonlyKey Value--- -----lease_id postgresql/creds/readonly/5fec46f2-ab40-d9b8-61a2-887c7946eeb6lease_duration 1h0m0slease_renewable truepassword f8a93086-b11d-10cd-8795-f537a10de712username token-9e57c18f-ac99-8e29-48f2-3fb09066d2b4
Terminal
$ psql -U postgrespsql (9.6.1)Type "help" for help.
postgres=# \du
Role name | Attributes | Member of-------------------------------------------+--------------------------+------------postgres | Superuser, Create ... | {}token-9e57c18f-ac99-8e29-48f2-3fb09066d2b4 | Password valid until ... | {}
Terminal
$ vault renew postgresql/creds/readonly/5fec46f2-ab40-d9b8-61a2-887c7946eeb6Key Value--- -----lease_id postgresql/creds/readonly/5fec46f2-ab40-d9b8-61a2-887c7946eeb6lease_duration 1h0m0slease_renewable true
Terminal
$ cat myapp-db-config.yml.ctmpl---{{- with secret "postgresql/creds/readonly" }}username: "{{ .Data.username }}"password: "{{ .Data.password }}"database: "myapp"{{- end }}$ consul-template -template="myapp-db-config.yml.ctmpl:myapp-db-config.yml" ./myapp
<CTRL+C>Received interrupt, cleaning up...
Terminal
$ vault write postgresql/roles/readonly \ sql="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"Success! Data written to: postgresql/roles/readonly
Terminal
$ tee my-policy.vault | vault write postgresql/roles/readonly sql=-CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}';GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";Success! Data written to: postgresql/roles/readonly
Codified Build Environment
Reproducible Packaging
Shared Packaging Instructions
Developer-driven Build and Packaging Steps
compile.json 1/3
{ "builders": [{ "name": "myapp", "type": "docker", "image": "centos:6", "commit": true, "privileged": true }], "provisioners": [ { "type": "file", "source": "myapp-linux", "destination": "/usr/local/bin/myapp" }, { "type": "file", "source": "local-config-file.repo", "destination": "/usr/local/etc/myapp.conf" }, { "type": "file", "source": "start_myapp.sh", "destination": "/sbin/start_myapp" },
compile.json 2/3
{ "type": "shell", "inline": [ "/usr/bin/yum -y update", "/usr/bin/yum -y install util-linux-ng patch", "/bin/chmod 0600 /usr/local/etc/myapp.conf", "/bin/chmod 0744 /sbin/start_myapp /usr/local/bin/myapp-linux", "/usr/bin/curl -o /usr/local/etc/some-ca.crt https://host.example.com/pki/ca.crt", ] } ], "post-processors": [ [ { "type": "docker-tag", "repository": "myorg/myapp" }, { "type": "docker-save", "path": "myapp.tar" },
compile.json 3/3
{ "type": "artifice", "files": ["myapp.tar"] }, { "type": "compress", "output": "myapp.tar.gz", "compression_level": 9 }, { "type": "atlas", "artifact": "myorg/myapp", "artifact_type": "archive", "metadata": { "created_at": "{{ timestamp }}" } } ] ]}
compile.json 1/3
{ "builders": [{ "name": "myapp", "type": "docker", "image": "centos:6", "commit": true, "privileged": true }], "provisioners": [ { "type": "shell", "scripts": [ "setup.sh", "prod-app-script.sh" ] }
redis.job
job "redis" { datacenters = ["asia-east1", "asia-northeast1"]
task "redis" { driver = "docker"
config { image = "redis:latest" }
resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 port "redis" {} } } }}
redis-service.job
job "redis" { datacenters = ["asia-east1", "asia-northeast1"] task "redis" { service { name = "redis" # redis.service.consul port = "redis" check { type = "tcp" interval = "30s" timeout = "2s" } }... resources { network { mbits = 10 port "redis" { static = "6379" } } }
Developer-centric Release Management
Reproducible Runtime Environments
Provider Agnostic Runtime
Native Hybrid-Cloud Consumption Model
Self-Healing Infrastructure
* Service Discovery and Secure Introduction Support
myapp.job
job "myapp" { region = "apac" datacenters = ["asia-east1", "asia-northeast1"] type = "service"
group "myapp" { count = 1 task "api" { driver = "docker"
artifact { source = "https://s3.amazonaws.com/myorg/myapp.tar.gz" options { archive = "tar.gz" } }
...
myapp.job
config { load = ["myapp.tar"] image = "myorg/myapp" command = "/sbin/start_myapp" args = [ "-mode=api" ] network_mode = "host" pid_mode = "host" }
service { name = "${TASKGROUP}" # myapp.service.consul tags = [ "api" ] # api.myapp.service.consul port = "api" check { type = "http" path = "/health.txt" interval = "5s" timeout = "2s" } } }
myapp.job
task "web" { driver = "docker" config { load = ["myapp.tar"] image = "myorg/myapp" command = "/sbin/start_myapp" args = [ "-mode=web" ] network_mode = "host" pid_mode = "host" } service { name = "${TASKGROUP}" # myapp.service.consul tags = [ "web" ] # web.myapp.service.consul port = "web" check { type = "http" path = "/health.txt" interval = "5s" timeout = "2s" } }
myapp-green.job
job "myapp-green" { region = "apac" datacenters = ["asia-east1", "asia-northeast1"] type = "service"
group "myapp" { count = 19 task "api" { driver = "docker"
artifact { source = "https://s3.amazonaws.com/myorg/myapp-v123.tar.gz" options { archive = "tar.gz" } }
...
myapp-blue.job
job "myapp-blue" { region = "apac" datacenters = ["asia-east1", "asia-northeast1"] type = "service"
group "myapp" { count = 1 task "api" { driver = "docker"
artifact { source = "https://s3.amazonaws.com/myorg/myapp-v124.tar.gz" options { archive = "tar.gz" } }
...
myapp-blue.job
$ nomad plan myapp-blue.job+ Job: "myapp-blue"+ Task Group: "myapp" (1 create) + Task: "api" (forces create)
Scheduler dry-run:- All tasks successfully allocated.
myapp-green.job
job "myapp-green" { region = "apac" datacenters = ["asia-east1", "asia-northeast1"] type = "service"
group "myapp" { count = 0 task "api" { driver = "docker"
artifact { source = "https://s3.amazonaws.com/myorg/myapp-v123.tar.gz" options { archive = "tar.gz" } }
...
myapp-blue.job
job "myapp-blue" { region = "apac" datacenters = ["asia-east1", "asia-northeast1"] type = "service"
group "myapp" { count = 20 task "api" { driver = "docker"
artifact { source = "https://s3.amazonaws.com/myorg/myapp-v124.tar.gz" options { archive = "tar.gz" } }
...
myapp-blue.job
job "myapp-blue" { region = "apac" datacenters = ["asia-east1", "asia-northeast1"] type = "service"
update { # Stagger updates every 120 seconds stagger = "120s"
# Update a single task at a time max_parallel = 1 }
...
myapp-blue.job
$ nomad status myapp-blueID = myapp-blueName = myapp-blueType = servicePriority = 50Datacenters = asia-east1Status = runningPeriodic = false
SummaryTask Group Queued Starting Running Failed Complete Lostmyapp 0 0 1 0 0 0
AllocationsID Eval ID Node ID Task Group Desired Status Created At24cfd201 81efc2fa 8d0331e9 myapp run running 11/11/16 21:03:19 AEDT
myapp-blue.job
$ nomad alloc-status --verbose a7365fe4ID = a7365fe4-cb28-a6e9-f3d4-f99e49c89776Eval ID = c3c9a1db-dbeb-8afa-0a83-4f1b8b5a03f5Name = myapp-blue.myapp[0]Node ID = 1f029d38-8d4b-a552-261f-e457b60f9b4bJob ID = myapp-blueClient Status = runningCreated At = 11/11/16 22:04:53 AEDTEvaluated Nodes = 1Filtered Nodes = 0Exhausted Nodes = 0Allocation Time = 1.085001msFailures = 0
==> Task ResourcesTask: "api"CPU Memory MB Disk MB IOPS Addresses500 256 300 0 db: 127.0.0.1:38537
Task: "web"CPU Memory MB Disk MB IOPS Addresses
Terminal
$ vault read postgresql/creds/readonlyKey Value--- -----lease_id postgresql/creds/readonly/5fec46f2-ab40-d9b8-61a2-887c7946eeb6lease_duration 1h0m0slease_renewable truepassword f8a93086-b11d-10cd-8795-f537a10de712username token-9e57c18f-ac99-8e29-48f2-3fb09066d2b4
Terminal
$ env VAULT_TOKEN=.... vault read postgresql/creds/readonlyKey Value--- -----lease_id postgresql/creds/readonly/5fec46f2-ab40-d9b8-61a2-887c7946eeb6lease_duration 1h0m0slease_renewable truepassword f8a93086-b11d-10cd-8795-f537a10de712username token-9e57c18f-ac99-8e29-48f2-3fb09066d2b4
myapp.job
job "myapp" { region = "apac" datacenters = ["asia-east1", "asia-northeast1"] type = "service"
group "myapp" { count = 1 task "api" { driver = "docker"
env { VAULT_TOKEN = "7ea47d76-a653-4d43-9507-dbeed3b3747f" }
artifact { source = "https://s3.amazonaws.com/myorg/myapp.tar.gz" options { archive = "tar.gz" } }
myapp.job
job "myapp" { region = "apac" datacenters = ["asia-east1", "asia-northeast1"] type = "service"
group "myapp" { count = 1 task "api" { driver = "docker"
vault { policies = ["myapp", "api"] change_mode = "signal" change_signal = "SIGUSR1" }
artifact { source = "https://s3.amazonaws.com/myorg/myapp.tar.gz" options { archive = "tar.gz" }
Containerized
Virtualized
Standalone
Docker
Windows Server Containers
Qemu / KVM
Hyper-V
Xen
Java Jar
Static Binaries
C#
Rkt
Key Value Store
HTTP API
Host & Service Level Health
Checks
Datacenter Aware
Service
Discovery
HTTP + DNS
CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT
SERVER SERVER SERVER
REPLICATION REPLICATION
RPC
RPC LAN GOSSIP
CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT
SERVER SERVER SERVER
REPLICATION REPLICATION
RPC
RPC LAN GOSSIP
SERVERSERVER SERVER
REPLICATION REPLICATION
WAN GOSSIP
Terminal
$ terraform plan -var-file=yow2016.tfvars -out=yow2016.tfplanRefreshing Terraform state in-memory prior to plan...The refreshed state will be used to calculate this plan, butwill not be persisted to local or remote state storage.
Your plan was also saved to the path below. Call the "apply" subcommandwith this plan file and Terraform will exactly execute this executionplan.
Path: yow2016.tfplan
+ consul_key_prefix.myservice_config datacenter: "<computed>" path_prefix: "myservice/mycomponent/" subkeys.%: "3" subkeys.appParam1: "val1" subkeys.appParam2: "var2" subkeys.dbHostname: "my-db.service.consul"
Plan: 1 to add, 0 to change, 0 to destroy.
Terminal
$ cat myservice-consul-kv-config.tfvariable "path_prefix" { default = "myservice/mycomponent"}
resource "consul_key_prefix" "myservice_config" { path_prefix = "${var.path_prefix}/"
subkeys = { "appParam1" = "val1" "appParam2" = "var2" "dbHostname" = "my-db.service.consul" }}
Terminal
$ git diff myservice-consul-kv-config.tfdiff --git a/myservice-consul-kv-config.tf b/myservice-consul-kv-config.tfindex 76533d8..990d595 100644--- a/myservice-consul-kv-config.tf+++ b/myservice-consul-kv-config.tf@@ -5,7 +5,5 @@ resource "consul_key_prefix" "myservice_config" { "appParam1" = "val1" "appParam2" = "var2" "dbHostname" = "my-db.service.consul"+ "dbHostnameFollower" = "slave.my-db.query.consul"+ "dbHostnameLeader" = "master.my-db.query.consul" } }
Terminal
$ terraform plan -var-file=yow2016.tfvars -out=yow2016.tfplanRefreshing Terraform state in-memory prior to plan...The refreshed state will be used to calculate this plan, butwill not be persisted to local or remote state storage.
consul_key_prefix.myservice_config: Refreshing state... (ID: myservice/mycomponent/)
Your plan was also saved to the path below. Call the "apply" subcommandwith this plan file and Terraform will exactly execute this executionplan.
Path: yow2016.tfplan
~ consul_key_prefix.myservice_config subkeys.%: "3" => "5" subkeys.dbHostnameFollower: "" => "slave.my-db.query.consul" subkeys.dbHostnameLeader: "" => "master.my-db.query.consul"
Plan: 0 to add, 1 to change, 0 to destroy.
Terminal
$ terraform apply yow2016.tfplanconsul_key_prefix.myservice_config: Modifying... subkeys.%: "3" => "5" subkeys.dbHostnameFollower: "" => "slave.my-db.query.consul" subkeys.dbHostnameLeader: "" => "master.my-db.query.consul"consul_key_prefix.myservice_config: Modifications complete
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
The state of your infrastructure has been saved to the pathbelow. This state is required to modify and destroy yourinfrastructure, so keep it safe. To inspect the complete stateuse the `terraform show` command.
State path: terraform.tfstate
Terminal
$ terraform fmt cat myservice-consul-kv-config.tfcat myservice-consul-kv-config.tf$ cat cat myservice-consul-kv-config.tfresource "consul_key_prefix" "myservice_config" { path_prefix = "${var.path_prefix}/"
subkeys = { "appParam1" = "val1" "appParam2" = "var2" "dbHostname" = "my-db.service.consul" "dbHostnameFollower" = "slave.my-db.query.consul" "dbHostnameLeader" = "master.my-db.query.consul" }
Terminal
$ cat myservice-consul-kv-config.tfvariable "path_prefix" { default = "myservice/mycomponent"}
variable "service_db_name" {}
resource "consul_key_prefix" "myservice_config" { path_prefix = "${var.path_prefix}/"
subkeys = { "appParam1" = "val1" "appParam2" = "var2" "dbHostname" = "${var.service_db_name}.service.consul" "dbHostnameFollower" = "slave.${var.service_db_name}.query.consul" "dbHostnameLeader" = "master.${var.service_db_name}.query.consul" }}
Terminal
$ cat yow2016.tfvars"address" = "127.0.0.1:8500"
"datacenter" = "yow2016"
"token" = ""
"service_db_name" = "my-db"
Terminal
$ terraform plan -var-file=yow2016.tfvars -out=yow2016.tfplanRefreshing Terraform state in-memory prior to plan...The refreshed state will be used to calculate this plan, butwill not be persisted to local or remote state storage.
consul_key_prefix.myservice_config: Refreshing state... (ID: myservice/mycomponent/)
No changes. Infrastructure is up-to-date. This means that Terraformcould not detect any differences between your configuration andthe real physical resources that exist. As a result, Terraformdoesn't need to do anything.
Terminal
$ cat myservice-consul-kv-config.tfvariable "db_leader_tag" { default = "leader" }variable "db_follower_tag" { default = "follower" }# snipresource "consul_key_prefix" "myservice_config" { path_prefix = "${var.path_prefix}/"
subkeys = { "appParam1" = "val1" "appParam2" = "var2" "dbHostname" = "${var.service_db_name}.service.consul" "dbHostnameFollower" = "${var.db_follower_tag}.${var.service_db_name}.query.consul" "dbHostnameLeader" = "${var.db_leader_tag}.${var.service_db_name}.query.consul" }}
Terminal
% git diffdiff --git a/myservice-consul-kv-config.tf b/myservice-consul-kv-config.tfindex 76590ef..e22eff0 100644--- a/myservice-consul-kv-config.tf+++ b/myservice-consul-kv-config.tf@@ -1,9 +1,9 @@ variable "db_leader_tag" {- default = "leader"+ default = "rw" }
variable "db_follower_tag" {- default = "follower"+ default = "ro" }
variable "path_prefix" {
Terminal
$ terraform plan -var-file=yow2016.tfvars -out=yow2016.tfplanRefreshing Terraform state in-memory prior to plan...The refreshed state will be used to calculate this plan, butwill not be persisted to local or remote state storage.
consul_key_prefix.myservice_config: Refreshing state... (ID: myservice/mycomponent/)
Your plan was also saved to the path below. Call the "apply" subcommandwith this plan file and Terraform will exactly execute this executionplan.
Path: yow2016.tfplan
~ consul_key_prefix.myservice_config subkeys.dbHostnameFollower: "follower.my-db.query.consul" => "ro.my-db.query.consul" subkeys.dbHostnameLeader: "leader.my-db.query.consul" => "rw.my-db.query.consul"
Plan: 0 to add, 1 to change, 0 to destroy.$ git reset --hardHEAD is now at 2ab88f9 Revise terminology from master/slave to leader/follower
1. Codify Everything 2. Pre-Plan outcomes at build-time 3. Create reproducible artifacts 4. Idempotent APIs and Tooling 5. Developer-Centric Operations 6. Make small, well understood changes changes 7. Start where it makes sense for your organization
FULL STACK VALUE
INCREMENTALISM LIFE CYCLE CODIFY EXAMPLE 1TENETS EXAMPLE 2
BUSINESS OBJECTIVE
MYAPP
NOMAD
SECRETS
VAULT
RISK MITIGATED
INCREMENTALISM LIFE CYCLE CODIFY EXAMPLE 1TENETS EXAMPLE 2
BUSINESS OBJECTIVE
MYAPP
NOMAD
SECRETS
VAULT
SECRETS CONSUL CLUSTERAPPLICATION CONSUL CLUSTER
SELF-HEALING SYSTEMS
INCREMENTALISM LIFE CYCLE CODIFY EXAMPLE 1TENETS EXAMPLE 2
TRIAGE DIAGNOSE
TREATPREVENT
SELF-ASSEMBLING SYSTEMS
INCREMENTALISM LIFE CYCLE CODIFY EXAMPLE 1TENETS EXAMPLE 2
FOUNDATION
PLATFORM
SELF-ASSEMBLING SYSTEMS
INCREMENTALISM LIFE CYCLE CODIFY EXAMPLE 1TENETS EXAMPLE 2
FOUNDATION
PLATFORM
APP
SELF-ASSEMBLING SYSTEMS
INCREMENTALISM LIFE CYCLE CODIFY EXAMPLE 1TENETS EXAMPLE 2
FOUNDATION
PLATFORM
APP"Easy"
SELF-ASSEMBLING SYSTEMS
INCREMENTALISM LIFE CYCLE CODIFY EXAMPLE 1TENETS EXAMPLE 2
FOUNDATION
PLATFORM
APP"Easy"
Tough
SELF-ASSEMBLING SYSTEMS
INCREMENTALISM LIFE CYCLE CODIFY EXAMPLE 1TENETS EXAMPLE 2
FOUNDATION
PLATFORM
APP"Easy"
Hard
Tough
SELF-ASSEMBLING SYSTEMS
INCREMENTALISM LIFE CYCLE CODIFY EXAMPLE 1TENETS EXAMPLE 2
FOUNDATION
PLATFORM
APP
ERROR BUDGET
INCREMENTALISM LIFE CYCLE CODIFY EXAMPLE 1TENETS EXAMPLE 2
Big BudgetLess Important
Smaller BudgetMore Important
RISK MANAGEMENT
INCREMENTALISM LIFE CYCLE CODIFY EXAMPLE 1TENETS EXAMPLE 2
Insiders
OpenSSL
ApplicationVulnerabilities
EMBRACE AUTOMATION
INCREMENTALISM LIFE CYCLE CODIFY EXAMPLE 1TENETS EXAMPLE 2
Creative
Industrious
Lazy
Mental Drift