introduction to chef
DESCRIPTION
A presentation I gave at the 2013 Salishan High Speed Computing conference. With 2 hours' notice :)TRANSCRIPT
![Page 1: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/1.jpg)
An Extemporaneous IntroductionTo Chef
Kevin A. SmithDirector of Server Engineering
![Page 2: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/2.jpg)
Who am I?
• Director of Server Engineering @ Opscode
• Software developer for 17 years
• 7 years with Erlang
• Alumni of DCRI, SAS, Red Hat, Basho
• Erlang In Practice @ PragProg
![Page 3: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/3.jpg)
Agenda
• Infrastructure as Code
• Configuration Management
• Chef 101
• Chef in Large Environments
http://www.flickr.com/photos/koalazymonkey/3590953001/
![Page 4: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/4.jpg)
Infrastructure as Code
![Page 5: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/5.jpg)
Building and managing infrastructure programmatically
Infrastructure as Code
![Page 6: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/6.jpg)
Enable the reconstruction of the business from
nothing but a source code repository, an application
data backup, and bare metal resources.
Infrastructure as Code
![Page 7: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/7.jpg)
Configuration Management
![Page 8: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/8.jpg)
The Old Way
![Page 9: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/9.jpg)
Manual Configuration
• Labor intensive
• Error prone
• Hard to reproduce
![Page 10: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/10.jpg)
Scripting
• Very brittle
• Throw away, one off scripts
• grep sed awk perl
• curl | bash
![Page 11: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/11.jpg)
File Distribution
• NFS mounts
• rdist
• scp-on-a-for-loop
• rsync on cron
![Page 12: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/12.jpg)
This does not scale!
for i in `cat servers.txt` ; do scp ntp.conf root@$i:/etc/ntpd.conf ; donefor i in `cat servers.txt` ; do ssh root@$i /etc/init.d/ntpd restart ; donefor i in `cat servers.txt` ; do ssh root@$i chkconfig ntpd on ; done
![Page 13: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/13.jpg)
See nodes grow.
Load Balancer
ApplicationServer
Database
ApplicationServer
![Page 14: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/14.jpg)
Load Balancer
ApplicationServer
Database
ApplicationServer
Load Balancer
Database
Grow, nodes. Grow!
![Page 15: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/15.jpg)
Datacenter #1
Load Balancer
AppServer
Database
AppServer
Load Balancer
Database
Datacenter #2
Load Balancer
AppServer
Database
AppServer
Load Balancer
Database
Internet
There are a lot of nodes!
![Page 16: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/16.jpg)
A New Way
![Page 17: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/17.jpg)
Declarative Configuration
• Define policy
• Say what, not how
• Abstract interface to resources
![Page 18: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/18.jpg)
Idempotence
• Property of a declarative interface
• f(x) = x
• Eliminates brittleness
• Safe to run over and over
package "ntp" do action :installend
template "/etc/ntp.conf" do source "ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, "service[ntpd]"end
service "ntpd" do action [:enable,:start]end
![Page 19: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/19.jpg)
Convergence
• Running an agent “converges” a system onto desired state
• Fights entropy and unauthorized changes
• Update function inputs to deal with changing requirements
$ echo “boom” > /etc/ntp.conf$ chef-client
$ grep server /etc/ntp.conf | head -n 1us.pool.ntp.org
$ ps -e | grep ntp 1799 ? 00:00:00 ntpd
$ /etc/init.d/ntpd stop$ chef-client
ps -e | grep ntp 1822 ? 00:00:00 ntpd
![Page 20: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/20.jpg)
Chef 101
http://www.flickr.com/photos/lapstrake/2711240606/in/photostream/
![Page 21: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/21.jpg)
The chef-client runs on your systems.
![Page 22: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/22.jpg)
Clients talk to a Chef server.
![Page 23: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/23.jpg)
Client server conversations are protected with SSL and
RSA signatures.
![Page 24: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/24.jpg)
Each system running Chef is called a Managed Node.
![Page 25: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/25.jpg)
Chef API Server
RDBMS
Search Engine
Asset Store
Managed Node
ChefClient
System Architecture
![Page 26: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/26.jpg)
Nodes have attributes
{ "kernel": { "machine": "x86_64", "name": "Darwin", "os": "Darwin", "version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386", "release": "10.4.0" }, "platform_version": "10.6.4", "platform": "mac_os_x", "platform_build": "10F569", "domain": "local", "os": "darwin", "current_user": "mray", "ohai_time": 1278602661.60043, "os_version": "10.4.0", "uptime": "18 days 17 hours 49 minutes 18 seconds", "ipaddress": "10.13.37.116", "hostname": "morbo", "fqdn": "morbomorbo.local", "uptime_seconds": 1619358 }
Platform
Kernel
Hostname, etc.
![Page 27: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/27.jpg)
Node attributes are searchable.
$ knife search node ‘platform:mac_os_x’
search(:node, ‘platform:mac_os_x’)
![Page 28: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/28.jpg)
Nodes have “to do” lists.
![Page 29: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/29.jpg)
Nodes have a Run List
% knife node show hadoop-prod.example.com -r{ "run_list": [ "role[base]", "role[hadoop-worker]" ]}
![Page 30: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/30.jpg)
Nodes can have Roles.
![Page 31: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/31.jpg)
Aspirational Roles
• webserver
• database_master
• monitoring
• hadoop-worker
![Page 32: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/32.jpg)
Roles have Attributes and a run list.
![Page 33: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/33.jpg)
Roles
name "hadoop-worker"description "Hadoop cluster member”
run_list( "role[base]", "recipe[java]", "recipe[hadoop]", “recipe[hadoop-config]")
default_attributes( "hadoop-config" => { "config_path" => “/etc/hadoop” })
![Page 34: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/34.jpg)
chef-client configures resources on managed nodes.
cookbook_file
template service
package
deploy
git
http_request
link
ruby_block
logbash
execute
remote_file
user
![Page 35: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/35.jpg)
Chef Resources
• Have a type.
• Have a name.
• Have parameters.
• Take action to put the resource in the declared state.
• Can send notifications to other resources.
package "apache2" do action :installend
template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 notifies :restart, "service[apache2]"end
service "apache2" do supports :restart => true action [:enable, :start]end
![Page 36: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/36.jpg)
package “hadoop” { yum install hadoopapt-get install hadooppacman sync hadooppkg_add -r hadoop
Chef Providers
![Page 37: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/37.jpg)
Recipes are collections of resources.
![Page 38: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/38.jpg)
Chef Recipes
• Resources are evaluated in the order they appear.
package "haproxy" do action :installend
template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"end
service "haproxy" do supports :restart => true action [:enable, :start]end
![Page 39: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/39.jpg)
Chef Recipes
• Recipes can include other recipes.
• Included recipes are also evaluated in order.
include_recipe "apache2"include_recipe "apache2::mod_rewrite"include_recipe "apache2::mod_deflate"include_recipe "apache2::mod_headers"include_recipe "apache2::mod_php5"
![Page 40: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/40.jpg)
Chef Recipes
• Extend recipes with Ruby.
%w{ php5 php5-dev php5-cgi }.each do |pkg|
package pkg do action :install end
end
![Page 41: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/41.jpg)
• Dynamic configuration through search.
pool_members = search("node", "role:app_server")
template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members notifies :restart, "service[haproxy]"end
Chef Recipes
![Page 42: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/42.jpg)
Cookbooks are packages for recipes and related files.
![Page 43: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/43.jpg)
Cookbook Metadata
maintainer "Opscode, Inc."maintainer_email "[email protected]"license "Apache 2.0"description "Installs/Configures tomcat"long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))version "0.10.3"
%w{ java jpackage }.each do |cb| depends cbend
%w{ debian ubuntu centos redhat fedora }.each do |os| supports osend
recipe "tomcat::default", "Installs and configures Tomcat"
![Page 44: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/44.jpg)
Cookbooks are Source Code
% git logcommit d640a8c6b370134d7043991894107d806595cc35Author: jtimberman <[email protected]>
Import nagios version 1.0.0
commit c40c818498710e78cf73c7f71e722e971fa574e7Author: jtimberman <[email protected]>
installation and usage instruction docs
commit 99d0efb024314de17888f6b359c14414fda7bb91Author: jtimberman <[email protected]>
Import haproxy version 1.0.1
commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4Author: jtimberman <[email protected]>
add mediawiki cookbook
commit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700Author: jtimberman <[email protected]>
multiple environments in data bag for mediawiki
![Page 45: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/45.jpg)
OSS & Community Oriented
• Apache 2.0 License
• Wiki, mailing lists, shared cookbook repos
• http://community.opscode.com
• Healthy ecosystem
• 20k+ users
• Hundreds of contributors
• Community tooling: Food Critic, Test Kitchen, Berkshelf
![Page 46: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/46.jpg)
Chef In“Large” Environments
![Page 47: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/47.jpg)
New Server
• Ground up rewrite Ruby/C Erlang
• Order of magnitude more scalable
• 2k nodes 20k+ nodes per server*
*Depending on specific work load
![Page 48: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/48.jpg)
High Scalability Users
• Cycle Computing
• edmunds.com
![Page 49: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/49.jpg)
Push Execution
• Converge infrastructure on demand
• Real-timey view of managed infrastructure
• Reduces change latency
• 4k nodes now, 10k soon
![Page 50: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/50.jpg)
Network Automation
• Network provisioning and configuration
• VLANs, QoS, etc.
• Partnered w/Arista on PoC (Fall 2012)
• More coming soon!
![Page 51: Introduction to Chef](https://reader036.vdocument.in/reader036/viewer/2022062513/5554c5e2b4c90559398b51c0/html5/thumbnails/51.jpg)
Thank You