cooking with chef, knife, vagrant and co

© Zühlke 2013

Torben Knerr

Cooking with Chef, Knife, Vagrant & Co 4 Ingredients for tasty “Infrastructure-As-Code”

27. September 2013 Slide 1

© Zühlke 2013

With typical components:

• Application Servers

• Database Servers

• Caching Servers

• Load Balancer / Reverse Proxy

• Clustering & Failover

A Typical Web App Project…

Cooking with Chef, Knife, Vagrant & Co | Torben Knerr

nginx

Tomcat Tomcat

DB Master

DB Slave

App App

memcached


© Zühlke 2013

With a typical Deployment Pipeline:


Local DEV TEST PROD



© Zühlke 2013

With typical (?) problems:

• All environments are somehow different

• Nobody knows the differences in detail

• Setting up a new environment takes weeks

• And again it looks different from the others

• Totally no Dev/Prod Parity

• And last but not least…




© Zühlke 2013

First Approach: Hey, let’s build VMs!

This already helps:

• Use VMs for a consistent work environment

• Manually set up a local developer VM with Tomcat, PostgreSQL, nginx, etc…

• Carefully hand-crafted with love!

• Distribute it to the team

Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 7

© Zühlke 2013

But…

• Manual management of VMs is cumbersome

• How do you version a 40GB VM?

• How do you take care of updates?

• Can you diff the changes?

• Once it works locally, how can you apply this to the rest of the deployment pipeline?

• How do you parameterize such VMs?

Better than nothing, but still far from perfect!


First Approach: Hey, let’s build VMs!

© Zühlke 2013

Why Scripting / Automation?

• Lightweight scripts vs. fat VMs – Allows for proper versioning and diffing! – Easy to share

• Infrastructure can be reliably recreated – Minimize manual (= boring & error prone) tasks – Basis for “clean” infrastructure deployments – Anywhere in the deployment pipeline

• Explicit design of infrastructure – Described as an executable specification – Not formed “by accident”

Treat your infrastructure like code!

Better Solution: Automate your VMs!


© Zühlke 2013

Cooking with Chef, Knife, Vagrant & Co

Torben Knerr

Welcome to the DevOps Kitchen! Infrastructure á la Carte


© Zühlke 2013

Goal: set up a simple server infrastructure, fully automated

• For example on a developer machine

• Separate VMs for app server and database server

• Anywhere reproducible („clean setup“)

Our Example Scenario


App Server VM

Tomcat

Database Server VM

PostgreSQL

.war


© Zühlke 2013


Torben Knerr

Ingredient #1 We need VMs


© Zühlke 2013

Ingredient #1: we need VMs Enter Vagrant What is Vagrant?

• Vagrant is “automation for VMs”

• Different VM providers: VirtualBox, aws, rackspace, etc..

• The configuration of VMs is described in a Vagrantfile

• Simple commands for interacting with VMs: – vagrant up – vagrant ssh – vagrant halt – …

• http://vagrantup.com


http://vagrantup.com/

Ingredient #1: we need VMs – A simple Vagrantfile


Basebox


App Server VM


Database Server VM

Ingredient #1: we need VMs – Interacting with Vagrant VMs

vagrant up

Ingredient #1: we need VMs – Interacting with Vagrant VMs

vagrant ssh

© Zühlke 2013

Ingredient #1: we need VMs A quick look at the VirtualBox GUI


Is this the App Server VM? Or the Database Server VM?


Ingredient #1: we need VMs – More VM configuration


Name, CPUs, RAM, etc…


vagrant reload

© Zühlke 2013

Ingredient #1: we need VMs Looking at the VirtualBox GUI again


and the VM Parameters are applied too

Better names


© Zühlke 2013


Torben Knerr

Ingredient #2: Provisioning


© Zühlke 2013

Ingredient #2: Provisioning Back to our Example Scenario

“Naked” VMs are not enough

• On the database server VM we need PostgreSQL

• On the app server VM we need Tomcat

We need a tool (“provisioner”) for installing and configuring software components on top of naked VMs


© Zühlke 2013

Ingredient #2: Provisioning Enter Chef

What is Chef?

• Chef is a “Configuration Management” system

• Installs & configures software on nodes

• Provides a DSL for describing the configuration of a node in a readable and platform-independent way

More information:

• http://www.opscode.com/chef/

Vagrant supports provisioning of VMs via Chef out-of-the-box!


http://www.opscode.com/chef/

Ingredient #2: Provisioning – Installing PostgreSQL


run the PostgreSQL Server recipe


configuration parameters

vagrant up db

What’s inside the PostgreSQL Recipe?

install the “postgresql“ package

create the config file

Erzeuge die Konfig-Datei

create another config file

start the „postgresql“ service

we also need the postgresql client…

Ingredient #2: Provisioning – Installing Tomcat


run the Tomcat default recipe (short for “tomcat::default“)


tomcat specific configuration

vagrant up web

© Zühlke 2013

Chef Philosophy

Specification of the target state

• Idempotence

• Convergence (“self-healing“)

Chef DSL abstracts the OS-specific details via

• Resources

• Providers

Naming: kitchen metaphor

• Chef, Cookbook, Recipe, Knife, …


© Zühlke 2013


Torben Knerr

Ingredient #3: Managing Cookbooks


© Zühlke 2013

Ingredient #3: Managing Cookbooks Back to our Example Scenario


where is the PostgreSQL cookbook / recipe coming from???


© Zühlke 2013

Ingredient #3: Managing Cookbooks Typical Sources for “Community Cookbooks”


http://community.opscode.com/cookbooks/

https://github.com/search?q=cookbook


http://community.opscode.com/cookbooks/

https://github.com/search?q=ntp+cookbook

https://github.com/search?q=ntp+cookbook

© Zühlke 2013

Ingredient #3: Managing Cookbooks Enter Librarian

What is Librarian?

• Librarian is a Dependency Management tool for Cookbooks

• Downloads and extracts cookbooks for you

Dependencies are specified in a Cheffile

• Transitive dependencies are resolved as well

• Different cookbook sources and version constraints are respected

More information:

• https://github.com/applicationsonline/librarian-chef


https://github.com/applicationsonline/librarian-chef



© Zühlke 2013

Ingredient #3: Managing Cookbooks A simple Cheffile

Cheffile


Ingredient #3: Managing Cookbooks – A simple Cheffile

Librarian „installs“ the cookbooks

© Zühlke 2013


Torben Knerr

Ingredient #4: Application Cookbooks


© Zühlke 2013

Ingredient #4: Application Cookbooks Back to our Example Scenario

Re-using Community Cookbooks is not enough

• We need a specific database and database user for our application

• We need to deploy our application’s .war file in Tomcat

• Existing Community Cookbooks don’t “know” our specific requirements!

We need to write our own “Application Cookbook” for this purpose!


© Zühlke 2013

Did you know? You can create cookbooks with a knife!


Ingredient #4: Application Cookbooks Enter Knife


© Zühlke 2013

Ingredient #4: Application Cookbooks Structure of our “myapp” cookbook

Single cookbook with separate recipes for setting up the database and webapp

Goal: hide implementation details from the user


myapp

attributes

recipes

db.rb

web.rb

db.rb

default.rb

web.rb


© Zühlke 2013


Attributes files expose the “configuration parameters” of a cookbook and supply sensible defaults


myapp

attributes

recipes

db.rb

web.rb

db.rb

default.rb

web.rb


© Zühlke 2013


• myapp/attributes/db.rb

• myapp/attributes/web.rb


© Zühlke 2013


The “db” recipe first installs PostgreSQL, then creates the database and database user


myapp

attributes

recipes

db.rb

web.rb

db.rb

default.rb

web.rb


first install the PostgreSQL server

installs prerequisites for interacting with PostgreSQL from within this recipe

creates the database user

creates the database

© Zühlke 2013

Ingredient #4: Application Cookbooks Use “myapp::db” in the Vagrantfile

Vagrantfile


instead of “postgresql::server”


vagrant provision db

© Zühlke 2013


The “web” recipe first installs tomcat, then configures and deploys the our web app


myapp

attributes

recipes

db.rb

web.rb

db.rb

default.rb

web.rb


we need to install Tomcat first

there is our .war file located

database configuration (via context.xml)

deploy the webapp to Tomcat

vagrant provision web

Is it running now?

Or whatever is in your context… (we have borrowed the “yabe“ example from Play! here: http://www.playframework.org/documentation/1.0/samples)

http://www.playframework.org/documentation/1.0/samples

http://www.playframework.org/documentation/1.0/samples

© Zühlke 2013

Recap: Ingredients for cooking in the DevOps Kitchen


Ingr. #1 We need VMs

Ingr. #2 Provisioning

Ingr. #3 Cookbook Management

Ingr. #4 Application Cookbooks

+

+

+


© Zühlke 2013

Fully Automated Infrastructure via Textual Specification

• No shell scripts, but specification of the target state via Chef DSL

• Crisp and concise, but not cryptic

• SCM-friendly: Kilobytes instead of Gigabytes, diffable

• Complex infrastructure available at the push-of-a-button

• Reproducible anywhere in the deployment pipeline

• Good software engineering principles applied to infrastructure

What have we achieved?


© Zühlke 2013

We have barely touched the surface…

There’s much more of Chef, Vagrant & Co we have not seen today:

• More Vagrant features – Alternative Providers (e.g. vmware, aws, rackspace, managed-servers, etc…) – Alternative Provisioners (e.g. puppet, shell, ansible, salt, etc…) – Vagrant Plugins (e.g. vagrant-omnibus, vagrant-berkshelf, vagrant-cachier, …) – …

• More Chef concepts – Writing Custom Resources und Providers (LWRPs) – Data bags – Environments – Search – Chef Solo vs. Chef Server – …

• And most importantly…


© Zühlke 2013 Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 85

…if we treat infrastructure as code, then we have to test!

Linting

• foodcritic

Unit-Level Testing

• chefspec

• fauxhai

Integration Testing (from the inside)

• chef-minitest-handler

• test-kitchen

Acceptance Testing (outside-in)

• cucumber-nagios

© Zühlke 2013

Resources Some useful stuff to check out… • Code Examples from this talk

https://github.com/tknerr/oop-chef-demo

• Newer Example based on Vagrant / Chef / Berkshelf https://github.com/tknerr/sample-application-cookbook

• Bills Kitchen – all you need for cooking with Chef & Vagrant on Windows https://github.com/tknerr/bills-kitchen https://ikm.zuehlke.com/topics/Pages/Bill%27s%20Kitchen.aspx

• “Chef, Vagrant & Co” Yammer Group https://www.yammer.com/zuehlke.com/#/threads/inGroup?type=in_group&feedId=2051695

• “Test-Driven Infrastructure with Chef (2nd ed.)” – highly recommended! http://shop.oreilly.com/product/0636920030973.do







https://github.com/tknerr/sample-application-cookbook





https://github.com/tknerr/bills-kitchen



https://ikm.zuehlke.com/topics/Pages/Bill's Kitchen.aspx

https://www.yammer.com/zuehlke.com/#/threads/inGroup?type=in_group&feedId=2051695

https://www.yammer.com/zuehlke.com/#/threads/inGroup?type=in_group&feedId=2051695

http://shop.oreilly.com/product/0636920030973.do

© Zühlke 2013

Resources More of the Vagrant Ecosystem… • Bindler – Manage your Vagrant Plugins via `plugins.json`

https://github.com/fgrehm/bindler

• Selection of useful Vagrant plugins https://github.com/schisamo/vagrant-omnibus – installs Chef on a “provisionerless” basebox https://github.com/riotgames/vagrant-berkshelf – resolves cookbook dependencies on `vagrant up` https://github.com/fgrehm/vagrant-cachier – transparently caches downloaded packages https://github.com/tmatilai/vagrant-proxyconf – sets the proxy inside the VM …

• Selection of Vagrant Providers https://github.com/mitchellh/vagrant-aws – provision EC2 instances in the AWS cloud with Vagrant https://github.com/tknerr/vagrant-managed-servers – provision “non-VMs” with Vagrant https://github.com/fgrehm/vagrant-lxc - provision LXC containers …




https://github.com/schisamo/vagrant-omnibus



https://github.com/opscode/bento#current-baseboxes




https://github.com/riotgames/vagrant-berkshelf



https://github.com/fgrehm/vagrant-cachier/



https://github.com/tmatilai/vagrant-proxyconf



https://github.com/mitchellh/vagrant-aws



https://github.com/tknerr/vagrant-managed-servers





https://github.com/fgrehm/vagrant-lxc



© Zühlke 2013


Torben Knerr

Discussion Time!

20. Juli 2012

Puppet, Berkshelf, Foo, Bar, Baz

DevOps, NoOps, Blablabla

Vagrant, Chef, Deployment Stuff

Holy Cloud! Slide 88

Thank You! And let’s continue the discussion here:

http://linkedin.com/in/tknerr

http://xing.to/tknerr

https://twitter.com/tknerr_de

mailto:[email protected]

cooking with chef, knife, vagrant and co

Technology