Chef Cookbook Testing and Continuous IntegrationJulian C. DunnSenior Consultant, Opscode, Inc.<[email protected]>
Friday, June 14, 13
Why Write Cookbook Tests?
• Catch development errors• Catch regressions• Code to requirements (TDD, BDD)• Smaller batch sizes == lower blast radius• Continuous delivery of cookbooks!• Build confidence that your infrastructure is
working correctly
Friday, June 14, 13
Software Development Lifecycle for Sysadmins
Friday, June 14, 13
Devops Is A Two-Way Street• It’s great when
developers care about• uptime!• scaling!• deployment!
• Put them on call! etc. etc. etc.
Friday, June 14, 13
Devops Is A Two-Way Street
• Systems Administrators also have as much or more to learn from developers as well!
Friday, June 14, 13
Source Code
Compiler
Artifact
Test
Typical Development Workflow
• Typical Software Development Workflow:
• Write Source
• Compile Source
• Deploy Artifacts
• Write Tests
• Run Tests
Friday, June 14, 13
Source Code
Compiler
Artifact
Test
Typical Development Workflow
• Source Code
• The recipe for a computer program
• Edited directly
• Managed via Source Control Software
Friday, June 14, 13
Source Code
Compiler
Artifact
Test
Typical Development Workflow
• Compiler
• Takes source code and converts it into executable programs
Friday, June 14, 13
Source Code
Compiler
Artifact
Test
Typical Development Workflow
• Artifact
• Artifacts are executable programs created by compilers.
• Compiled artifacts cannot be edited directly. Source code must be changed and re-compiled to produce a new build artifact.
Friday, June 14, 13
Source Code
Compiler
Artifact
Test
Typical Development Workflow
• Testing
• Write tests to verify that code works as intended
• Run tests at different stages of the code lifecycle to ensure correctness
Friday, June 14, 13
Source Code
Compiler
Artifact
Test
Typical Development Workflow
• When developing software, most time isn’t actually spent coding
• 10-second changes to source code can take minutes to vet
• Compiling code
• Deploying code
• Writing & Running tests
Friday, June 14, 13
Friday, June 14, 13
Source Code
Compiler
Artifact
Test
Typical Development WorkflowFun!
Boring!
• Things that are fun:
• Designing programs!
• Writing source code!
• Things that are boring:
• Compiling code
• Deploying artifacts
• Running tests
Friday, June 14, 13
Look familiar to any Chefs?Friday, June 14, 13
Worksta(on:+Knife+cookbook+
create+Worksta(on:+Edit+cookbook+
Worksta(on:+Knife+cookbook+
upload+
Provision+target+Bootstrap+target+
Worksta(on:+Edit+target+run+
list+
ssh+target+ Target:+Run+chef>client+
Traditional Cookbook Development: First Chef Run
Friday, June 14, 13
Worksta(on:+edit+cookbook+
Worksta(on:+knife+cookbook+
upload+
Target:+Run+chef9client+
Traditional Cookbook Development: Subsequent Runs
Friday, June 14, 13
Source'Code'
Compiler'
Ar/fact'
Test'
Boooooooring• Too much time doing
“paperwork”• vi recipes/something.rb
• knife cookbook upload
• sudo pkill -USR1 chef-client
• #%$#%$ something broke, let me do that all again
• Not enough time doing fun stuff!
• Writing recipes
Less Fun!
More Boring!
X
Friday, June 14, 13
Source'Code'
Compiler'
Ar/fact'
Test'
Boooooooring• Too much time doing
“paperwork”• vi recipes/something.rb
• knife cookbook upload
• sudo pkill -USR1 chef-client
• #%$#%$ something broke, let me do that all again
• Not enough time doing fun stuff!
• Writing recipes
Less Fun!
More Boring!
XTHIS
SUCKS!
Friday, June 14, 13
Chefs on a Plane
What if...
Friday, June 14, 13
Chefs on a Plane
What if...Worksta(on:+Knife+cookbook+
create+Worksta(on:+Edit+cookbook+
Worksta(on:+Knife+cookbook+
upload+
Provision+target+Bootstrap+target+
Worksta(on:+Edit+target+run+
list+
ssh+target+ Target:+Run+chef>client+
Target:(Run(chef/client(
Worksta6on:(edit(cookbook(
Worksta6on:(knife(cookbook(
upload(
...we could automate all
of this...
Friday, June 14, 13
Chefs on a Plane
What if...Worksta(on:+Knife+cookbook+
create+Worksta(on:+Edit+cookbook+
Worksta(on:+Knife+cookbook+
upload+
Provision+target+Bootstrap+target+
Worksta(on:+Edit+target+run+
list+
ssh+target+ Target:+Run+chef>client+
Target:(Run(chef/client(
Worksta6on:(edit(cookbook(
Worksta6on:(knife(cookbook(
upload(
...we could automate all
of this...
...to run entirely on this...
Friday, June 14, 13
Chefs on a Plane
What if...
...even while aboard this?
Worksta(on:+Knife+cookbook+
create+Worksta(on:+Edit+cookbook+
Worksta(on:+Knife+cookbook+
upload+
Provision+target+Bootstrap+target+
Worksta(on:+Edit+target+run+
list+
ssh+target+ Target:+Run+chef>client+
Target:(Run(chef/client(
Worksta6on:(edit(cookbook(
Worksta6on:(knife(cookbook(
upload(
...we could automate all
of this...
...to run entirely on this...
Friday, June 14, 13
Source'Code'
Compiler'
Ar/fact'
Test'
Developing for Chef: Rapid Iteration
• Less time waiting around for cookbook deploys and Chef runs
• More frequent testing
• Better code
• Business needs met more quickly
More Fun!
Less Boring!
X
Friday, June 14, 13
Code Can (Help)Friday, June 14, 13
The Toolchain
Friday, June 14, 13
The Toolchain
• You’re becoming a developer!• Reasonably powerful computer• Good editor
Friday, June 14, 13
Have a Good Computer• Virtualization is used to run
acceptance tests• Running on a real virtualized
“node”• Lots of memory (4GB+, 8GB
recommended)• Fast disk (SSD)• Good processor (Intel i5+)• Modern OS
XFriday, June 14, 13
Tools• Chef (obviously)• Virtualization provider• Virtualization automation
(Vagrant)• Cookbook dependency
manager (Berkshelf, Librarian)
• Unit and acceptance testing frameworks
Friday, June 14, 13
Software Testing Terminology and Chef
Friday, June 14, 13
Different Types of Testing
• There are formal Software Engineering definitions for testing
Friday, June 14, 13
Different Types of Testing
• There are formal Software Engineering definitions for testing• Unit test
Friday, June 14, 13
Different Types of Testing
• There are formal Software Engineering definitions for testing• Unit test• Integration test
Friday, June 14, 13
Different Types of Testing
• There are formal Software Engineering definitions for testing• Unit test• Integration test• Acceptance test
Friday, June 14, 13
Different Types of Testing
• There are formal Software Engineering definitions for testing• Unit test• Integration test• Acceptance test
• An easy explanation for SAs is ...
Friday, June 14, 13
• Unit Test: Signal Input
What and When To Test
Flickr user: Rain Rabbit
Friday, June 14, 13
• Unit Test: Signal Input
• Did I send Chef the correct command?
What and When To Test
Flickr user: Rain Rabbit
Friday, June 14, 13
• Unit Test: Signal Input
• Did I send Chef the correct command?
• Signal Processing
What and When To Test
Flickr user: Rain Rabbit
Friday, June 14, 13
• Unit Test: Signal Input
• Did I send Chef the correct command?
• Signal Processing
• Did Chef interpret my command correctly?
What and When To Test
Flickr user: Rain Rabbit
Friday, June 14, 13
• Unit Test: Signal Input
• Did I send Chef the correct command?
• Signal Processing
• Did Chef interpret my command correctly?
• Acceptance Test: Signal Output
What and When To Test
Flickr user: Rain Rabbit
Friday, June 14, 13
• Unit Test: Signal Input
• Did I send Chef the correct command?
• Signal Processing
• Did Chef interpret my command correctly?
• Acceptance Test: Signal Output
• Did my expressed intent, executed by Chef, achieve the desired result?
What and When To Test
Flickr user: Rain Rabbit
Friday, June 14, 13
Chef Testing Tools
Friday, June 14, 13
Chef Testing Tools
• Signal Input• ChefSpec
Friday, June 14, 13
Chef Testing Tools
• Signal Input• ChefSpec
• Signal out• Chef Minitests & Minitest Handler
Friday, June 14, 13
Chef Testing Tools
• Signal Input• ChefSpec
• Signal out• Chef Minitests & Minitest Handler
• Signal processing• Who cares; that’s what Opscode’s own tests for
Chef are for!
Friday, June 14, 13
General Philosophy for Chef Testing
Friday, June 14, 13
General Philosophy for Chef Testing
• Unit tests• ChefSpec assertion for every resource of interest• Each recipe should get its own test file
Friday, June 14, 13
General Philosophy for Chef Testing
• Unit tests• ChefSpec assertion for every resource of interest• Each recipe should get its own test file
• Acceptance tests• Anything that you couldn’t test in a unit test
Friday, June 14, 13
General Philosophy for Chef Testing
• Unit tests• ChefSpec assertion for every resource of interest• Each recipe should get its own test file
• Acceptance tests• Anything that you couldn’t test in a unit test
• Don’t repeat yourself!
Friday, June 14, 13
Unit Testing: ChefSpec
Friday, June 14, 13
ChefSpec Overview
• Built on top of RSpec• Converge a Chef run in memory• Overrides all providers to take no action• Make assertions about the in-memory Chef run• Mock Ohai data (automatic attributes) with Fauxhai
Friday, June 14, 13
Example Recipe Part Oneinclude_recipe “java”
user node['sauceproxy']['server']['user'] do comment "SauceLabs Proxy User" system true action :createend
directory node['sauceproxy']['server']['install_dir'] do owner node['sauceproxy']['server']['user'] mode 00755 action :createend
# Can't assume we have unzippackage "unzip" do action :installend
execute "unzip-saucelabs-proxy" do cwd node['sauceproxy']['server']['install_dir'] command "unzip -o #{Chef::Config[:file_cache_path]}/#{node['sauceproxy']['server']['zipfile']}" action :nothing notifies :restart, "service[sauceproxy]"end
https://github.com/juliandunn/sauceproxy/blob/master/recipes/server.rb
Friday, June 14, 13
Example Recipe Part Tworemote_file "#{Chef::Config[:file_cache_path]}/#{node['sauceproxy']['server']['zipfile']}" do
source "#{node['sauceproxy']['server']['download_url']}/#{node['sauceproxy']['server']['zipfile']}" action :create_if_missing notifies :run, "execute[unzip-saucelabs-proxy]", :immediatelyend
template "/etc/init.d/sauceproxy" do source "sauceproxy.init.erb" mode 00755 owner "root" group "root" action :createend
template "/etc/sysconfig/sauceproxy" do source "sauceproxy.sysconfig.erb" mode 00644 owner "root" group "root" action :createend
service "sauceproxy" do supports :restart => true action [:enable, :start]end
Friday, June 14, 13
Example ChefSpec Test: Setupdescribe 'sauceproxy::server' do
let (:chef_run) do runner = ChefSpec::ChefRunner.new( platform: 'centos', version: '6.3' ) runner.node.set['sauceproxy']['server']['user'] = 'fake' runner.node.set['sauceproxy']['server']['install_dir'] = '/tmp/fake' runner.node.set['sauceproxy']['server']['version'] = '3.14159' runner.converge('sauceproxy::server') end
Friday, June 14, 13
Example ChefSpec Test: Expectations it 'should create a sauceproxy directory with the right ownership' do expect(chef_run).to create_directory('/tmp/fake') expect(chef_run.directory('/tmp/fake')).to be_owned_by('fake') end
it 'should start a service called sauceproxy' do expect(chef_run).to start_service('sauceproxy') expect(chef_run).to set_service_to_start_on_boot('sauceproxy') endend
Friday, June 14, 13
ChefSpec In Action
Friday, June 14, 13
ChefSpec: Expectations
• Many expectations (assertions) possible• https://github.com/acrmp/chefspec#making-
assertions• Don’t (just) restate static resources
• Mock out data• Static resource assertions: good for regressions
• Whoops, I deleted the service resource
Friday, June 14, 13
Acceptance Testing: MiniTest
Friday, June 14, 13
Chef Client Run
Friday, June 14, 13
Chef Client Run
Friday, June 14, 13
Acceptance Tests as Report Handler
• Acceptance testing as a Chef Report Handler• Many tools (serverspec, bats, minitest)• I’ll demo Chef-MiniTest-Handler; most widely used
Friday, June 14, 13
MiniTest Handler Cookbook
• http://community.opscode.com/cookbooks/minitest-handler• Installs Minitest Gems• Installs Chef Minitest Gems• Installs the Chef-Minitest-Handler Notification
Handler for Chef-Client• Places test files from cookbooks on the target node
as part of a Chef-client run
Friday, June 14, 13
MiniTest Files
• Each recipe gets an individual test file.• recipes/server.rb• files/default/test/server_test.rb
• Tests for each recipe are automatically loaded by the handler
Friday, June 14, 13
Example MiniTestrequire 'minitest/spec'
describe_recipe 'sauceproxy::server' do
it 'runs as a daemon' do service('sauceproxy').must_be_running end
end
http://tinyurl.com/minitest-examplesfor many more example tests
Friday, June 14, 13
Make this go on a
with virtualization!Friday, June 14, 13
Using Vagrant & Berkshelf to Iterate
• Vagrant is virtualization middleware• Driven from Vagrantfile• Defines VM images, customization parameters,
provisioners (Chef in our case)• Berkshelf is a cookbook dependency manager
• Get dependent cookbooks from community API• Feed them to Vagrant Chef provisioner
Friday, June 14, 13
Example VagrantfileVagrant.configure("2") do |config| config.vm.hostname = "sauceproxy-berkshelf" config.vm.box = "opscode-centos-6.4" config.vm.box_url = "https://opscode-vm.s3.amazonaws.com/vagrant/opscode_centos-6.4_chef-11.4.4.box" config.ssh.max_tries = 40 config.ssh.timeout = 120 config.vm.provision :chef_solo do |chef| chef.json = {} chef.run_list = [ "recipe[minitest-handler]", "recipe[sauceproxy::server]" ] endend
Friday, June 14, 13
% vagrant up
Bringing Up Vagrant
borkbork:~/Dropbox/devel/github/juliandunn/sauceproxy (master)$ vagrant upBringing machine 'default' up with 'virtualbox' provider...[default] Importing base box 'centos-6.4'...[default] Matching MAC address for NAT networking...[default] Setting the name of the VM...[default] Clearing any previously set forwarded ports...[Berkshelf] Updating Vagrant's berkshelf: '/Users/juliandunn/.berkshelf/vagrant/berkshelf-20130611-20810-rt7k01'[Berkshelf] Using sauceproxy (0.1.8) at path: '/Users/juliandunn/Dropbox/devel/github/juliandunn/sauceproxy'[Berkshelf] Using minitest-handler (0.2.1)[Berkshelf] Using java (1.11.4)[Berkshelf] Using windows (1.8.10)[Berkshelf] Using chef_handler (1.1.4)[default] Creating shared folders metadata...[default] Clearing any previously set network interfaces...[default] Preparing network interfaces based on configuration...[default] Forwarding ports...[default] -- 22 => 2222 (adapter 1)[default] Booting VM...[default] Waiting for VM to boot. This can take a few minutes.
Friday, June 14, 13
% vagrant up
Bringing Up Vagrant
borkbork:~/Dropbox/devel/github/juliandunn/sauceproxy (master)$ vagrant upBringing machine 'default' up with 'virtualbox' provider...[default] Importing base box 'centos-6.4'...[default] Matching MAC address for NAT networking...[default] Setting the name of the VM...[default] Clearing any previously set forwarded ports...[Berkshelf] Updating Vagrant's berkshelf: '/Users/juliandunn/.berkshelf/vagrant/berkshelf-20130611-20810-rt7k01'[Berkshelf] Using sauceproxy (0.1.8) at path: '/Users/juliandunn/Dropbox/devel/github/juliandunn/sauceproxy'[Berkshelf] Using minitest-handler (0.2.1)[Berkshelf] Using java (1.11.4)[Berkshelf] Using windows (1.8.10)[Berkshelf] Using chef_handler (1.1.4)[default] Creating shared folders metadata...[default] Clearing any previously set network interfaces...[default] Preparing network interfaces based on configuration...[default] Forwarding ports...[default] -- 22 => 2222 (adapter 1)[default] Booting VM...[default] Waiting for VM to boot. This can take a few minutes.
Friday, June 14, 13
Vagrant Run Continued[default] Waiting for VM to boot. This can take a few minutes.[default] VM booted and ready for use!GuestAdditions 4.2.12 running --- OK.[default] Setting hostname...[default] Configuring and enabling network interfaces...[default] Mounting shared folders...[default] -- /vagrant[default] -- /tmp/vagrant-chef-1/chef-solo-1/cookbooks[default] Running provisioner: chef_solo...Generating chef JSON and uploading...Running chef-solo...[2013-06-12T02:52:26+00:00] INFO: *** Chef 11.4.4 ***[2013-06-12T02:52:26+00:00] INFO: Setting the run_list to ["recipe[minitest-handler]", "recipe[sauceproxy::server]"] from JSON[2013-06-12T02:52:26+00:00] INFO: Run List is [recipe[minitest-handler], recipe[sauceproxy::server]][2013-06-12T02:52:26+00:00] INFO: Run List expands to [minitest-handler, sauceproxy::server][2013-06-12T02:52:26+00:00] INFO: Starting Chef Run for sauceproxy-berkshelf
Friday, June 14, 13
Vagrant Run Continued[2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds[2013-06-12T02:55:22+00:00] INFO: Running report handlersRun options: -v --seed 31883
# Running tests:
recipe::java::openjdk#test_0001_installs the correct version of the jdk =0.11 s = .recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =0.04 s = .recipe::sauceproxy::server#test_0001_runs as a daemon =0.09 s = .
Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.
3 tests, 3 assertions, 0 failures, 0 errors, 0 skips[2013-06-12T02:55:22+00:00] INFO: Report handlers complete
Friday, June 14, 13
Vagrant Run Continued[2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds[2013-06-12T02:55:22+00:00] INFO: Running report handlersRun options: -v --seed 31883
# Running tests:
recipe::java::openjdk#test_0001_installs the correct version of the jdk =0.11 s = .recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =0.04 s = .recipe::sauceproxy::server#test_0001_runs as a daemon =0.09 s = .
Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.
3 tests, 3 assertions, 0 failures, 0 errors, 0 skips[2013-06-12T02:55:22+00:00] INFO: Report handlers complete
Friday, June 14, 13
Minitests from Java cookbook ran too![2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds[2013-06-12T02:55:22+00:00] INFO: Running report handlersRun options: -v --seed 31883
# Running tests:
recipe::java::openjdk#test_0001_installs the correct version of the jdk =0.11 s = .recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =0.04 s = .recipe::sauceproxy::server#test_0001_runs as a daemon =0.09 s = .
Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.
3 tests, 3 assertions, 0 failures, 0 errors, 0 skips[2013-06-12T02:55:22+00:00] INFO: Report handlers complete
Friday, June 14, 13
Minitests from Java cookbook ran too![2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds[2013-06-12T02:55:22+00:00] INFO: Running report handlersRun options: -v --seed 31883
# Running tests:
recipe::java::openjdk#test_0001_installs the correct version of the jdk =0.11 s = .recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =0.04 s = .recipe::sauceproxy::server#test_0001_runs as a daemon =0.09 s = .
Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.
3 tests, 3 assertions, 0 failures, 0 errors, 0 skips[2013-06-12T02:55:22+00:00] INFO: Report handlers complete
Friday, June 14, 13
New Test/Dev CycleWrite Unit Tests
Worked?
Run ChefSpec
No
Write Recipe Code
Write Acceptance Tests
Yes
Worked?
Run Vagrant + Minitest Handler
No
vagrant destroy
Yes Commit/Tag Code, etc.
Friday, June 14, 13
CI Pipeline Integration
Friday, June 14, 13
Continuous Integration Pipeline
• Run unit tests in your pipeline • Run acceptance tests in your
pipeline• Drive cookbook uploads as the
output!
Friday, June 14, 13
Travis-CI Example for SauceProxylanguage: rubygemfile: - gemfiles/travis.gemfilervm: - "1.9.2" - "1.9.3"script: bundle exec rake test:syntax test:lint test:specnotifications: email: - [email protected]
Friday, June 14, 13
Rakefile Example
• Too long to post here• Rake tasks for Foodcritic
and ChefSpec• https://github.com/
juliandunn/sauceproxy/blob/master/Rakefile
Friday, June 14, 13
Travis-CI Example Dashboard
Friday, June 14, 13
^#$%(#$ I broke the build
Friday, June 14, 13
Test Kitchen (alpha)
• Run acceptance test suite on multiple OSes
• Different fixtures for each, if desired• Different drivers (vagrant, ec2, lxc,
etc.)• Hook up to Jenkins or other CI
system if desired
Friday, June 14, 13
Test Kitchen Config File---driver_plugin: vagrantplatforms:- name: centos-6.4 driver_config: box: opscode-centos-6.4 box_url: https://opscode-vm.s3.amazonaws.com/vagrant/opscode_centos-6.4_provisionerless.box require_chef_omnibus: true- name: centos-5.9 driver_config: box: opscode-centos-5.9 box_url: https://opscode-vm.s3.amazonaws.com/vagrant/opscode_centos-5.9_provisionerless.box require_chef_omnibus: true
suites:- name: default run_list: ["recipe[minitest-handler]", "recipe[sauceproxy::server]"] attributes: {}
Friday, June 14, 13
Test Kitchen Demo
Friday, June 14, 13
Resources
Friday, June 14, 13
Resources• ChefSpec: https://github.com/acrmp/chefspec• Chef Minitest Handler: https://github.com/calavera/
minitest-chef-handler • Berkshelf: http://berkshelf.com/• Vagrant: http://vagrantup.com/• Test Kitchen: https://github.com/opscode/test-kitchen• Bento: https://github.com/opscode/bento• SauceProxy Sample Cookbook: https://github.com/
juliandunn/sauceproxyFriday, June 14, 13
More on Testing
• Chef-NYC: The Hows and Whys of Cookbook Testing
• Seth Vargo (Author of ChefSpec, Fauxhai, many others)
• http://www.meetup.com/Chef-NYC/events/122219772/
• June 25th in Manhattan
Friday, June 14, 13
Chef Fundamentals Training
• Boston, July 11-12 (CompuWorks, 263 Summer St.)
• eventbrite.com/event/6652057483
• Use code BOSTON-MEETUP to save 25%!
Friday, June 14, 13
Questions and PrizesAlso, we’re hiring! Automate all the things for fun and profit!http://www.opscode.com/blog/careers/(and/or see me after)
Friday, June 14, 13