london hashicorp meetup #8 - testing programmable infrastructure by matt long
Post on 22-Jan-2018
1.327 Views
Preview:
TRANSCRIPT
MATT LONG
TESTING PROGRAMMABLE INFRASTRUCTURE
PROGRAMMABLE INFRASTRUCTURE IS GREAT, BUT WE'RE MISSING SOMETHING.
TESTING.
I'M A TESTERHELLO, I'M MATT
I WORK HERE ↑
I AM NOT A SYSADMIN
WHAT IS PROGRAMMABLE INFRASTRUCTURE?
TESTING PROGRAMMABLE INFRASTRUCTURE
THE APPLICATION OF METHODS AND TOOLING FROM SOFTWARE DEVELOPMENT TO MANAGEMENT OF IT INFRASTRUCTURE
PROGRAMMABLE INFRASTRUCTURE IS..
THE INTERNET
TESTING PROGRAMMABLE INFRASTRUCTURE
EXAMPLES OF PROGRAMMABLE INFRASTRUCTURE
▸ Automated provisioning & configuration
▸ Configuration as code
▸ Version / source controlled
TESTING PROGRAMMABLE INFRASTRUCTURE
TOOLING EXAMPLES
PROGRAMMABLE INFRASTRUCTURE IS AWESOME!
Credit: Vault Boy, Bethesda Softworks
IT'S FAST!
IT'S AUTOMATIC!
IT'S ALL CODE!
BUT IT GETS COMPLEX
TESTING IS USED TO MITIGATE COMPLEXITY & RISK
BUT TESTING IS RARE
Credit: Gunshow, KC Green
TESTING PROGRAMMABLE INFRASTRUCTURE
WHAT I'M GOING TO TALK ABOUT
▸PART 1: Testing a cloud broker
▸PART 2: Building a Kubernetes cluster
▸CONCLUSIONS
TESTING A CLOUD BROKER
AN INFRASTRUCTURE HEAVY PRODUCT
THE PROBLEM
TESTING PROGRAMMABLE INFRASTRUCTURE
WE WANT TO MOVE TO THE CLOUD...
BUT WE'RE WARY OF LOCK IN
Large organisation
TESTING PROGRAMMABLE INFRASTRUCTURE
USERS
USE MULTIPLE CLOUD PROVIDERS
TESTING PROGRAMMABLE INFRASTRUCTURE
PROBLEMS
▸ Different interfaces, feature sets & lingo
▸ Can't switch easily
▸ Spending difficult to track
▸ Temptation to fall back on most popular
TESTING PROGRAMMABLE INFRASTRUCTURE
USERS
CLOUD BROKER
TESTING PROGRAMMABLE INFRASTRUCTURE
BENEFITS
▸Quick, easy provisioning
▸ one team previously took 3 months
▸ Common interface to cloud features
▸ Templates for common dev environments
▸ Built in best practice: monitoring, security
▸ Track spending
THIS IS A REALLY COMPLICATED APPLICATION
TESTING PROGRAMMABLE INFRASTRUCTURE
TESTING PROGRAMMABLE INFRASTRUCTURE
WORKFLOW
▸ Log into Web UI
▸ Fill in information about environment
▸ Broker creates and bootstraps resources
▸ SSH into resources
TESTING PROGRAMMABLE INFRASTRUCTURE
WEB TESTING
▸ Log into Web UI
▸ Fill in information about environment
TESTING PROGRAMMABLE INFRASTRUCTURE
???
▸ Broker creates and bootstraps resources
▸ SSH into resources
HOW DO YOU TEST INFRASTRUCTURE?
TESTING PROGRAMMABLE INFRASTRUCTURE
WHAT TO TEST?
Do our deployment scripts work?
Does the VPN server work?
Can instances access one another?
Are services running?
Can I SSH into a server?
THIS SEEMS FAMILIAR..
TESTING PROGRAMMABLE INFRASTRUCTURE
Does the VPN box work?Can I SSH into a server?
Do our deployment scripts work?
Are services running?
ANOTHER TESTING PYRAMID?
credit: Ubuntu dev quality guidehttps://developer.ubuntu.com/en/phone/platform/quality/
Can instances access one another?
TOOLING
TESTING PROGRAMMABLE INFRASTRUCTURE
TOOLS AVAILABLE
▸ Bats
▸ ShUnit2
▸ Goss
▸ ServerSpec / Inspec / TestInfra
▸ Test Kitchen
UNIT TESTING
TESTING PROGRAMMABLE INFRASTRUCTURE
BATS
▸ "Bash Automated Testing System"
▸ Unit testing for bash
▸ Like JUnit
TESTING PROGRAMMABLE INFRASTRUCTURE
SH UNIT 2
▸ Shell unit testing framework
▸ Runs on all Bourne shells
▸ sh, BASH, DASH, ksh, zsh
▸ No activity or support?
INTEGRATION TESTINGOR: SERVER VALIDATION
TESTING PROGRAMMABLE INFRASTRUCTURE
GOSS
▸ Go based
▸ Specs in YAML
▸ Minimal, fast, and simple
▸ Some neat features
▸ .. have to run on the server
▸ .. no Windows support
TESTING PROGRAMMABLE INFRASTRUCTURE
SERVERSPEC
▸ Server based assertions
▸ Ruby/RSpec based
▸ Probably the most famous
▸ Can SSH into instances
TESTING PROGRAMMABLE INFRASTRUCTURE
INSPEC
▸ Written & maintained by Chef
▸ Very similar to ServerSpec
▸ Different feature set
▸ More focused on compliance
TESTING PROGRAMMABLE INFRASTRUCTURE
TESTINFRA
▸ ServerSpec, but in Python
TEST HARNESS
TESTING PROGRAMMABLE INFRASTRUCTURE
TEST KITCHEN
▸ Orchestrates setup, test, teardown
▸ Runs BATS, shUnit2, RSpec, Serverspec
▸ Popular in the Chef community
▸ Not suitable for our cloud broker
OUR SOLUTION
TESTING PROGRAMMABLE INFRASTRUCTURE
USERS
CLOUD BROKER
TESTING PROGRAMMABLE INFRASTRUCTURE
USERS
WEB TEST FRAMEWORK
TESTING PROGRAMMABLE INFRASTRUCTURE
USERS
INFRASTRUCTURE TEST FRAMEWORK
TESTING PROGRAMMABLE INFRASTRUCTURE
USERS
WEB TESTS
https://github.com/opencredo/test-automation-quickstart
TESTING PROGRAMMABLE INFRASTRUCTURE
INFRASTRUCTURE TESTS
Serverspec
TESTING PROGRAMMABLE INFRASTRUCTURE
INFRASTRUCTURE TESTING STACK
/ Serverspec
???
TESTING PROGRAMMABLE INFRASTRUCTURE
WHY RUBY?
▸ Fantastic testing community
▸ More suitable for SSHing into boxes
▸ "Win RM" gem
▸ Ops already familiar with it
▸ Reduces tech stack
TESTING PROGRAMMABLE INFRASTRUCTURE
SERVERSPEC SMOKE TESTS
▸ Run before everything else
▸ Really quick
▸ Catches obvious errors
▸ Not complex tasks
TESTING PROGRAMMABLE INFRASTRUCTURE
SERVERSPEC EXAMPLE
describe package('jenkins') do it { should be_installed } end
describe service('jenkins') do it { should be_enabled } it { should be_running } end
describe port(8080) do it { should be_listening } end
TESTING PROGRAMMABLE INFRASTRUCTURE
Background: Given environment has been created And the following user details: | user_alias | username | public_key | | userA | envoy | test |
Scenario: IPA - Login via SSH Key authentication succeeds Given user "userA" is authorised to access environment vms When user "userA" starts ssh session in host "env"
Then I should be able to echo "hello world"
CUCUMBER FOR ACCEPTANCE TESTING
TESTING PROGRAMMABLE INFRASTRUCTURE
Background: Given environment has been created And the following user details: | user_alias | username | public_key | | userA | envoy | test |
Scenario: IPA - Login via SSH Key authentication succeeds Given user "userA" is authorised to access environment vms When user "userA" starts ssh session in host "env"
Then I should be able to echo "hello world"
CUCUMBER FOR ACCEPTANCE TESTING
Cloud broker APIs
TESTING PROGRAMMABLE INFRASTRUCTURE
Background: Given environment has been created And the following user details: | user_alias | username | public_key | | userA | envoy | test |
Scenario: IPA - Login via SSH Key authentication succeeds Given user "userA" is authorised to access environment vms When user "userA" starts ssh session in host "env"
Then I should be able to echo "hello world"
CUCUMBER FOR ACCEPTANCE TESTING
Standard Ruby
TESTING PROGRAMMABLE INFRASTRUCTURE
Background: Given environment has been created And the following user details: | user_alias | username | public_key | | userA | envoy | test |
Scenario: IPA - Login via SSH Key authentication succeeds Given user "userA" is authorised to access environment vms When user "userA" starts ssh session in host "env"
Then I should be able to echo "hello world"
CUCUMBER FOR ACCEPTANCE TESTING
RSpec assertions
TESTING PROGRAMMABLE INFRASTRUCTURE
UNDER THE CUCUMBER, PLAIN RUBY
Then(/^I should be able to echo "([^"]*)"$/) do |text| cmd = "echo #{text}" output = @session.exec!(cmd) close_ssh(@session) expect(output.to_s.strip).to eql(text) end
THOUGHTS
TESTING PROGRAMMABLE INFRASTRUCTURE
THE GOOD
▸ Specialised tests for each layer
▸ Really quick, expressive ServerSpec tests
▸ Power of a full programming language for user tests
TESTING PROGRAMMABLE INFRASTRUCTURE
THE BAD
▸ Over reliance on acceptance tests
▸ Awkward switching between two suites
▸ Out of my comfort zone
TESTING PROGRAMMABLE INFRASTRUCTURE
THE UGLY
▸ Starting infrastructure is SLOW.
▸ It's expensive...
IT WAS WORTH ITDESPITE ALL THAT
BUILDING A KUBERNETES CLUSTER
APPLYING TDD TO INFRASTRUCTURE
INTERNAL DEVOPS TRAINING COURSE
I LEARNED A LOT!
Credit: The Simpsons, Fox
TESTING PROGRAMMABLE INFRASTRUCTURE
BUILD THIS:
WITH THESE:
TESTING PROGRAMMABLE INFRASTRUCTURE
NOT A STRAIGHTFORWARD TASK
TESTING PROGRAMMABLE INFRASTRUCTURE
BUT HOW TO TEST IT?
▸ This is a dev activity
▸ Want fast feedback
▸ Complexity is mitigated by testing!
TESTING PROGRAMMABLE INFRASTRUCTURE
▸ Provisions cloud infrastructure
▸ Declarative files
▸ Some support for variables
TERRAFORM
TESTING PROGRAMMABLE INFRASTRUCTURE
TERRAFORM COMMANDS
▸ terraform plan
▸ Tells you what will change
▸ terraform apply
▸ Applies changes
▸ terraform validate
▸ Lints terraform syntax
TESTING PROGRAMMABLE INFRASTRUCTURE
TERRAFORM FILE EXAMPLE
resource "aws_instance" "etcd-node" { count = 3
ami = "ami-7abd0209" # centos availability_zone = "eu-west-1a" # ireland instance_type = "t2.micro" subnet_id = .... private_ip = ....
key_name = "${aws_key_pair.my-key.key_name}" }
TESTING PROGRAMMABLE INFRASTRUCTURE
LINT WITH 'TERRAFORM VALIDATE' COMMAND
Omitting a variable:
TESTING PROGRAMMABLE INFRASTRUCTURE
BUT IT DOESN'T CATCH ALL PROBLEMS
Duplicate subnet CIDRS:
TESTING PROGRAMMABLE INFRASTRUCTURE
LINTING ISN'T ENOUGH
▸ Devs don’t just rely on compilers
▸ We need something more powerful
Credit: Nick Cave, "Soundsuit"
UNIT TESTING
TESTING PROGRAMMABLE INFRASTRUCTURE
TERRAFORM_VALIDATE
▸ Python based unit testing
▸ NOT to be confused with 'validate' command
▸ Builds map of resources & properties
▸ Totally offline
▸ New and incomplete
https://github.com/elmundio87/terraform_validate
TESTING PROGRAMMABLE INFRASTRUCTURE
TERRAFORM_VALIDATE FORK
OC has forked the terraform validate repohttps://github.com/opencredo/terraform_validate
INTEGRATION TESTING
TESTING PROGRAMMABLE INFRASTRUCTURE
GOSS
▸ Easy to get up and running
▸ Doesn’t support remote
# example usage: ./goss-test.sh 34.248.91.167
TARGET='centos@'$1 SSH_KEY_PATH=~/.ssh/aws
ssh -t -t -i $SSH_KEY_PATH $TARGET 'curl -fsSL https://goss.rocks/install | sudo sh' scp ./goss.json $TARGET:~/goss.yaml
ssh -t -t -i $SSH_KEY_PATH $TARGET 'goss validate'
https://gist.github.com/burythehammer/081d6ee11cc33c2f4c4729ae67622f5b
TESTING PROGRAMMABLE INFRASTRUCTURE
▸ Terraform compatibility
▸ Already a talk about this
▸ “Untangling Infrastructure Code” by Nell Shamrell-Harrington
TEST KITCHEN + INSPEC
TESTING PROGRAMMABLE INFRASTRUCTURE
TestCreate Config Destroy ]TEST KITCHEN MANAGES YOUR TEST LIFECYCLE
TESTING PROGRAMMABLE INFRASTRUCTURE
TestCreate Config Destroy
TEST KITCHEN DOESN'T SUPPORT MULTIPLE PROVISIONERS
TESTING PROGRAMMABLE INFRASTRUCTURE
TEST KITCHEN DOESN'T SUPPORT MULTIPLE PROVISIONERS
https://github.com/test-kitchen/test-kitchen/issues/329
TESTING PROGRAMMABLE INFRASTRUCTURE
TERRAFORM 'NULL RESOURCE'
resource "null_resource" "ansible" { triggers { instance_ids = "${join(",", aws_instance.etcd-node.*.id)}"
}
provisioner "local-exec" { command = "sleep 20 && cd ../ansible/ && ansible-playbook etcd.yaml" }
}
TESTING PROGRAMMABLE INFRASTRUCTURE
TestCreate
Config
Destroy
THOUGHTS
TESTING PROGRAMMABLE INFRASTRUCTURE
THE GOOD
▸ Tooling exists!
▸ You can totally get a test suite working
Credit: Overwatch, Blizzard Entertainment
TESTING PROGRAMMABLE INFRASTRUCTURE
THE BAD
▸ Unit testing extremely immature
▸ Tools immature in general
Credit: Overwatch, Blizzard Entertainment
TESTING PROGRAMMABLE INFRASTRUCTURE
THE HACKY
▸ Be prepared to hack
▸ It might not even be possible
Credit: Overwatch, Blizzard Entertainment
THIS IS BRAND NEW GROUND
REMEMBER:
TESTING TOOLS DEPEND ON YOUR STACK
CONCLUSIONS
TESTING IS IMPORTANT
BUT OFTEN IGNORED
TESTERS AND OPS SHOULD WORK TOGETHER
WE NEED TO GET OUT OF OUR COMFORT ZONES
TOOLS EXISTBUT BE PREPARED
TO HACK
FINALLY...
TESTING PROGRAMMABLE INFRASTRUCTURE
THE APPLICATION OF METHODS AND TOOLING FROM SOFTWARE DEVELOPMENT TO MANAGEMENT OF IT INFRASTRUCTURE
PROGRAMMABLE INFRASTRUCTURE IS..
TESTING IS A SOFTWARE DEVELOPMENT METHOD
WE SHOULD APPLY IT TO INFRASTRUCTURE
top related