cooking with chef, knife, vagrant and co

Torben Knerr

Cooking with Chef, Knife, Vagrant & Co 4 Ingredients for tasty “Infrastructure-As-Code”

27. September 2013 Slide 1

With typical components:

• Application Servers

• Database Servers

• Caching Servers

• Load Balancer / Reverse Proxy

• Clustering & Failover

A Typical Web App Project…

Cooking with Chef, Knife, Vagrant & Co | Torben Knerr

Tomcat Tomcat

DB Master

DB Slave

App App

memcached

With a typical Deployment Pipeline:

Local DEV TEST PROD

With typical (?) problems:

• All environments are somehow different

• Nobody knows the differences in detail

• Setting up a new environment takes weeks

• And again it looks different from the others

• Totally no Dev/Prod Parity

• And last but not least…

First Approach: Hey, let’s build VMs!

This already helps:

• Use VMs for a consistent work environment

• Manually set up a local developer VM with Tomcat, PostgreSQL, nginx, etc…

• Carefully hand-crafted with love!

• Distribute it to the team

Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 7

But…

• Manual management of VMs is cumbersome

• How do you version a 40GB VM?

• How do you take care of updates?

• Can you diff the changes?

• Once it works locally, how can you apply this to the rest of the deployment pipeline?

• How do you parameterize such VMs?

Better than nothing, but still far from perfect!

First Approach: Hey, let’s build VMs!

Why Scripting / Automation?

• Lightweight scripts vs. fat VMs – Allows for proper versioning and diffing! – Easy to share

• Infrastructure can be reliably recreated – Minimize manual (= boring & error prone) tasks – Basis for “clean” infrastructure deployments – Anywhere in the deployment pipeline

• Explicit design of infrastructure – Described as an executable specification – Not formed “by accident”

Treat your infrastructure like code!

Better Solution: Automate your VMs!

Cooking with Chef, Knife, Vagrant & Co

Torben Knerr

Welcome to the DevOps Kitchen! Infrastructure á la Carte

Goal: set up a simple server infrastructure, fully automated

• For example on a developer machine

• Separate VMs for app server and database server

• Anywhere reproducible („clean setup“)

Our Example Scenario

App Server VM

Tomcat

Database Server VM

PostgreSQL

Torben Knerr

Ingredient #1 We need VMs

Ingredient #1: we need VMs Enter Vagrant What is Vagrant?

• Vagrant is “automation for VMs”

• Different VM providers: VirtualBox, aws, rackspace, etc..

• The configuration of VMs is described in a Vagrantfile

• Simple commands for interacting with VMs: – vagrant up – vagrant ssh – vagrant halt – …

• http://vagrantup.com

Ingredient #1: we need VMs – A simple Vagrantfile

Basebox

App Server VM

Database Server VM

Ingredient #1: we need VMs – Interacting with Vagrant VMs

vagrant up

Ingredient #1: we need VMs – Interacting with Vagrant VMs

vagrant ssh

Ingredient #1: we need VMs A quick look at the VirtualBox GUI

Is this the App Server VM? Or the Database Server VM?

Ingredient #1: we need VMs – More VM configuration

Name, CPUs, RAM, etc…

Ingredient #1: we need VMs – More VM configuration

vagrant reload

Ingredient #1: we need VMs Looking at the VirtualBox GUI again

and the VM Parameters are applied too

Better names

Torben Knerr

Ingredient #2: Provisioning

Ingredient #2: Provisioning Back to our Example Scenario

“Naked” VMs are not enough

• On the database server VM we need PostgreSQL

• On the app server VM we need Tomcat

We need a tool (“provisioner”) for installing and configuring software components on top of naked VMs

Ingredient #2: Provisioning Enter Chef

What is Chef?

• Chef is a “Configuration Management” system

• Installs & configures software on nodes

• Provides a DSL for describing the configuration of a node in a readable and platform-independent way

More information:

• http://www.opscode.com/chef/

Vagrant supports provisioning of VMs via Chef out-of-the-box!

Ingredient #2: Provisioning – Installing PostgreSQL

run the PostgreSQL Server recipe

Ingredient #2: Provisioning – Installing PostgreSQL

configuration parameters

vagrant up db

What’s inside the PostgreSQL Recipe?

install the “postgresql“ package

create the config file

Erzeuge die Konfig-Datei

create another config file

start the „postgresql“ service

we also need the postgresql client…

Ingredient #2: Provisioning – Installing Tomcat

run the Tomcat default recipe (short for “tomcat::default“)

Ingredient #2: Provisioning – Installing Tomcat

tomcat specific configuration

vagrant up web

Chef Philosophy

Specification of the target state

• Idempotence

• Convergence (“self-healing“)

Chef DSL abstracts the OS-specific details via

• Resources

• Providers

Naming: kitchen metaphor

• Chef, Cookbook, Recipe, Knife, …

Torben Knerr

Ingredient #3: Managing Cookbooks

Ingredient #3: Managing Cookbooks Back to our Example Scenario

where is the PostgreSQL cookbook / recipe coming from???

Ingredient #3: Managing Cookbooks Typical Sources for “Community Cookbooks”

http://community.opscode.com/cookbooks/

https://github.com/search?q=cookbook

Ingredient #3: Managing Cookbooks Enter Librarian

What is Librarian?

• Librarian is a Dependency Management tool for Cookbooks

• Downloads and extracts cookbooks for you

Dependencies are specified in a Cheffile

• Transitive dependencies are resolved as well

• Different cookbook sources and version constraints are respected

More information:

• https://github.com/applicationsonline/librarian-chef

Ingredient #3: Managing Cookbooks A simple Cheffile

Cheffile

Ingredient #3: Managing Cookbooks – A simple Cheffile

Librarian „installs“ the cookbooks

Torben Knerr

Ingredient #4: Application Cookbooks

Ingredient #4: Application Cookbooks Back to our Example Scenario

Re-using Community Cookbooks is not enough

• We need a specific database and database user for our application

• We need to deploy our application’s .war file in Tomcat

• Existing Community Cookbooks don’t “know” our specific requirements!

We need to write our own “Application Cookbook” for this purpose!

Did you know? You can create cookbooks with a knife!

Ingredient #4: Application Cookbooks Enter Knife

Ingredient #4: Application Cookbooks Structure of our “myapp” cookbook

Single cookbook with separate recipes for setting up the database and webapp

Goal: hide implementation details from the user

attributes

recipes

web.rb

default.rb

web.rb

Attributes files expose the “configuration parameters” of a cookbook and supply sensible defaults

attributes

recipes

web.rb

default.rb

web.rb

• myapp/attributes/db.rb

• myapp/attributes/web.rb

The “db” recipe first installs PostgreSQL, then creates the database and database user

attributes

recipes

web.rb

default.rb

web.rb

first install the PostgreSQL server

installs prerequisites for interacting with PostgreSQL from within this recipe

creates the database user

creates the database

Ingredient #4: Application Cookbooks Use “myapp::db” in the Vagrantfile

Vagrantfile

Ingredient #4: Application Cookbooks Use “myapp::db” in the Vagrantfile

Vagrantfile

instead of “postgresql::server”

vagrant provision db

Ingredient #4: Application Cookbooks Extended Cheffile

The “web” recipe first installs tomcat, then configures and deploys the our web app

attributes

recipes

web.rb

default.rb

web.rb

we need to install Tomcat first

there is our .war file located

database configuration (via context.xml)

deploy the webapp to Tomcat

Ingredient #4: Application Cookbooks Use “myapp::web” in the Vagrantfile

Vagrantfile

instead of the “tomcat::default“ recipe

Vagrantfile

C connect to this database host (here: ip address of the db vm)

vagrant provision web

Is it running now?

Or whatever is in your context… (we have borrowed the “yabe“ example from Play! here: http://www.playframework.org/documentation/1.0/samples)

Torben Knerr

The DevOps Kitchen – Recap

Recap: Ingredients for cooking in the DevOps Kitchen

Ingr. #1 We need VMs

Ingr. #2 Provisioning

Ingr. #3 Cookbook Management

Ingr. #4 Application Cookbooks

Fully Automated Infrastructure via Textual Specification

• No shell scripts, but specification of the target state via Chef DSL

• Crisp and concise, but not cryptic

• SCM-friendly: Kilobytes instead of Gigabytes, diffable

• Complex infrastructure available at the push-of-a-button

• Reproducible anywhere in the deployment pipeline

• Good software engineering principles applied to infrastructure

What have we achieved?

We have barely touched the surface…

There’s much more of Chef, Vagrant & Co we have not seen today:

• More Vagrant features – Alternative Providers (e.g. vmware, aws, rackspace, managed-servers, etc…) – Alternative Provisioners (e.g. puppet, shell, ansible, salt, etc…) – Vagrant Plugins (e.g. vagrant-omnibus, vagrant-berkshelf, vagrant-cachier, …) – …

• More Chef concepts – Writing Custom Resources und Providers (LWRPs) – Data bags – Environments – Search – Chef Solo vs. Chef Server – …

• And most importantly…

…if we treat infrastructure as code, then we have to test!

Linting

• foodcritic

Unit-Level Testing

• chefspec

• fauxhai

Integration Testing (from the inside)

• chef-minitest-handler

• test-kitchen

Acceptance Testing (outside-in)

• cucumber-nagios

Resources Some useful stuff to check out… • Code Examples from this talk

https://github.com/tknerr/oop-chef-demo

• Newer Example based on Vagrant / Chef / Berkshelf https://github.com/tknerr/sample-application-cookbook

• Bills Kitchen – all you need for cooking with Chef & Vagrant on Windows https://github.com/tknerr/bills-kitchen https://ikm.zuehlke.com/topics/Pages/Bill%27s%20Kitchen.aspx

• “Chef, Vagrant & Co” Yammer Group https://www.yammer.com/zuehlke.com/#/threads/inGroup?type=in_group&feedId=2051695

• “Test-Driven Infrastructure with Chef (2nd ed.)” – highly recommended! http://shop.oreilly.com/product/0636920030973.do

Resources More of the Vagrant Ecosystem… • Bindler – Manage your Vagrant Plugins via `plugins.json`

https://github.com/fgrehm/bindler

• Selection of useful Vagrant plugins https://github.com/schisamo/vagrant-omnibus – installs Chef on a “provisionerless” basebox https://github.com/riotgames/vagrant-berkshelf – resolves cookbook dependencies on `vagrant up` https://github.com/fgrehm/vagrant-cachier – transparently caches downloaded packages https://github.com/tmatilai/vagrant-proxyconf – sets the proxy inside the VM …

• Selection of Vagrant Providers https://github.com/mitchellh/vagrant-aws – provision EC2 instances in the AWS cloud with Vagrant https://github.com/tknerr/vagrant-managed-servers – provision “non-VMs” with Vagrant https://github.com/fgrehm/vagrant-lxc - provision LXC containers …

Torben Knerr

Discussion Time!

20. Juli 2012

Puppet, Berkshelf, Foo, Bar, Baz

DevOps, NoOps, Blablabla

Vagrant, Chef, Deployment Stuff

Holy Cloud! Slide 88

Thank You! And let’s continue the discussion here:

cooking with chef, knife, vagrant and co

Technology

ser1906bu vmware and chef - leveraging the vsphere … ·...

vagrant workshop

itb2015 - winning with vagrant, puppet and chef

knife unit tips from chef ross. knife sharpening when...

new batches info - quality thought › wp-content ›...

new batches info - quality thought · chef 3. terraform /...

giacomo marketing – malaysian favourite household nameonly...

the chef knife use and care knife cuts. knife safety always...

creating encrypted data bags using knife solo plugin for...

using chef and vagrant at gengo

topics covered : summary · topics covered : summary ......

vagrant story

vagrant presentation

vagrant + chef + mongodb · vagrant+chef’+ mongodb’...

consistent development environment with vagrant and chef

consistent development environment using vagrant and chef

infrastructure as code using chef - cmwg.org · chef...

the chef knife

vagrant intro

cooking with chef, knife, vagrant and co