Chef Cookbook Design Patterns

Eric KrupnikSoftware EngineerCopyright Clearance Center

05/02/20232

Who Am I?

• Software Engineer

• Platform Engineering at CCC

• Chef user since May, 2013

ekrupnik@copyright.com

ekrupnik@gmail.com

github.com/ekrupnik

05/02/20233

Who Are You?

• Who has heard of Chef?

• Who uses Chef?– Other configuration management tools?

05/02/20234

Agenda

• Introduction

• Chef overview

• Cookbook types

• Data bags vs attributes

• Run lists and cookbook management

• Q and A

05/02/20235

INTRODUCTION

05/02/20236

Issues To Address

• Large volume of applications– Number of apps

– Many environments

– Too much to handle manually!

• Error prone configuration during deployments

05/02/20237

Sheer Volume

• Many environments (columns)

• Many applications (rows)

• This leads to error prone deployments and inevitable configuration issues

• As of May, 2015 more than 180 deployable components over 6 major environments!

05/02/20238

CHEF OVERVIEW

05/02/20239

What is Chef? Infrastructure as Code?

• Configuration management tool

• Infrastructure as code– Executable text

• Don’t write MS Word documents anymore

– Version control• Track how infrastructure changes over time• Shareable

– Idempotent• Write code to define how a machine should look• Run Chef often to prevent configuration drift

– Testable

05/02/202310

Chef Terms

• A cookbook is the versioned unit of distribution– We will cover cookbook types later

– Contains 1+ recipe(s)

• A recipe is where a user defines a set of resources Chef manages

• Recipe should manage a group of related resources

• A resource is “something” Chef manages and configures– Package, directory, file, service, user

– Can be configured with attributes

– Can create custom resources

05/02/202311

Chef Resource Example

directory "/tmp/folder" do

owner 'root'

group 'root'

.

.

mode node['folder_priv']

action :create

end

Declaration of resource

Hard coded attribute

Hard coded attribute

.

.

Node attribute

Action to perform (create, delete)

end

05/02/202312

COOKBOOK TYPES

05/02/202313

Cookbook Types

• Application cookbook

• Library cookbook

• Wrapper cookbook– Run differently configured instances of application

– Won’t cover this today because it is rare

• Community cookbook– The community maintains many cookbooks you can use

– Won’t cover this today

05/02/202314

Application Cookbook

• One recipe per deployable component

• Define attributes and configurations needed for application– Attributes like application name

– Property file(s)

• Should be fairly lightweight

• Should depend on one or more library cookbooks

• Suggested naming conventions:– Cookbook – for the application it manages

– Recipes – for the individual deployable component

05/02/202315

Application Cookbook

cookbooks/myapp /attributes ui.rb rest_svc.rb /files/test ui_test.rb rest_svc_test.rb /recipes ui.rb rest_svc.rb /templates ui_properties.erb rest_svc_properties.erb

metadata.rb Berksfile .kitchen.yml

05/02/202316

Library Cookbook

• Defines a standard way to deploy an artifact

• Consumed by application cookbooks

• Can link multiple library cookbooks together

• Keeps application cookbooks DRY

• Contains any custom resources– Act on attributes defined by application cookbook

– Has defaults, can be overwritten by application cookbook

05/02/202317

Custom Resources

• This is called an LWRP (Lightweight Resource Provider)– Heavy weight resource providers also exist

• These should be the biggest worker in your cookbooks. It’s reusable, configurable, and defines how an application should be deployed.

• Two primary files– Resource: defines accepted attributes/configuration

– Provider: defines the “how” and does the work

– Each named after the resource you are defining

05/02/202318

Custom Resources (Resource)

lib_cookbook/resources/webapp.rb

actions :install, :delete

attribute :app_name, :kind_of => String, :required => trueattribute :port, :kind_of => String, :required => true

.

.

attribute :property_file_tempalte, :kind_of => String, :required => false

default_action :install

05/02/202319

Custom Resources (Provider)lib_cookbook/provider/webapp.rb

action :install do

app = new_resource.app_name java_version = Helpers.get_java_version(app)

java_install java_version do java_version java_version install_dir “/ccc/java” end

remote_file “#{app}” do source Helpers.get_artifactory_url(app) owner “root” end

template “/ccc/apps/#{app}/config/#{app}.properties” do source property_file_template mode 0644 variables( :data_bag_values => Helpers.get_data_bag_values(app) ) end

end

05/02/202320

DATA BAGS VS ATTRIBUTES

05/02/202321

Defining Configurations

• So far, we have covered setting attributes in application cookbooks– Pros:

• Forced standard in all environments• Versioned as part of a cookbook

– Cons:• Can’t change attribute values between environments• Hard to change “quickly and easily”

• Data bags are the answer!– Data bag contains 1 data bag item (JSON file) per environment

– 1:1 data bag to cookbook (same name even!)

– Live configuration data including application version

05/02/202322

Data Bag Example

data_bags/myapp/DEV.json

{ “id”: “DEV”, “myapp.welcomePage.title”: “Welcome to our app!”, “myapp.error.message”: “This is DEV – what did you expect! Of course it crashed!”, “myapp.error.email.recipient.list”: “ekrupnik@copyright.com, jdoe@copyright.com” }

data_bags/myapp/QA.json

data_bags/myapp/UAT.json

data_bags/myapp/PRD.json

{ “id”: “PRD”, “myapp.welcomePage.title”: “Welcome to our app!”, “myapp.error.message”: “We’re sorry something went wrong.”, “myapp.error.email.recipient.list”: “errors@copyright.com” }

05/02/202323

Data Bags in Action

• Remember this from the custom resource? template “/ccc/apps/#{app}/config/#{app}.properties” do source property_file_template variables( :data_bag_values => Helpers.get_data_bag_values(app) ) end

• Since the templates are erb files which allow deploy time variable replacement, we can use data bag values to generate a properties file template for the appropriate environment as we deploy.

05/02/202324

Data Bags in Action

• Properties file template: cookbooks/myapp/templates/myapp.properties.erb

myapp.welcomePage.title=<%= @data_bag_values[‘myapp.welcomPage.title’] %> myapp.error.message=<%= @data_bag_values[‘myapp.error.message’] %> myapp.error.email.recipient.list=<%= @data_bag_values[‘myapp.error.email.recipient.list’] %>

• Generated at deploy time by our custom resource we get: DEV: myapp.welcomePage.title=Welcome to our app! myapp.error.message=This is DEV – what did you expect! Of course it crashed! myapp.error.email.recipient.list=ekrupnik@copyright.com, jdoe@copyright.com

PRD: myapp.welcomePage.title=Welcome to our app! myapp.error.message=We’re sorry something went wrong. myapp.error.email.recipient.list=errors@copyright.com

05/02/202325

RUN LISTS AND COOKBOOK MANAGEMENT

05/02/202326

Run Lists

• A run list defines which recipes are needed to run during a Chef run.

• Run lists are stored on the Chef Server, and are specific to a node.

• Since library cookbooks only get called by application cookbooks, no library cookbook recipes should go into the run list

• Run lists are most commonly defined in either:– Chef role

– Chef environment file

05/02/202327

Roles vs Environment Files

• Both are unversioned, but roles tend to be more dangerous due to more global effect

• Both CAN define global or environment-wide attributes

• Environment files contain cookbook version to use

• Simple to keep cookbook versions and run lists in same place

• We do not use roles for these reasons!

05/02/202328

Environment Files Example

environments/DEV.rb name “DEV” cookbook_versions({ “myapp” => “= 3.1.0”, “newapp” => “= 1.4.0”, . . )} run_lists({ “ui-server” => “recipe[my_app::ui]”, “svc-server” => “recipe[my_app::rest_svc], recipe[newapp::rest_svc]” )}

environments/QA.rbenvironments/UAT.rb

environments/PRD.rb name “PRD” cookbook_versions({ “myapp” => “= 3.1.0”, . . )} run_lists({ “ui-server” => “recipe[my_app::ui]”, “svc-server” => “recipe[my_app::rest_svc]” )}

05/02/202329

Cookbook Dependencies

• Cookbooks can depend on other cookbooks– Application cookbook depends on library cookbook(s)

– Library cookbook depends on other library cookbook(s)

• Each cookbook has a metadata.rb file– Defines info on cookbook

– Defines cookbook dependencies

– Doesn’t matter if library or application cookbook

• There are tools like Berkshelf and Librarian– Berkshelf is now included in Chef-DK!

05/02/202330

Cookbook metadata.rb Example

name “myapp”

maintainer “Copyright Clearance Center”

maintainer_email “cookbooks@copyright.com”

description “App cookbook to install myapp”

version “3.1.0”

depends “library_cookbook_1”, “=5.0.0”

depends “library_cookbook_2”

depends “library_cookbook_3”, “~> 1.0”

depends “community_cookbook”

05/02/202331

Summary

• Recipes should be light weight and very focused

• Use library cookbooks with custom resources

• Library cookbooks can use other library cookbooks

• Use a mix between attributes and data bags– Standard configurations in attributes

– Environment specific configurations in data bags

• Use environment files for cookbook versions and run lists

05/02/202332

QUESTIONS?

Thank you!Eric Krupnik & CCC Platform Engineering team