Transcript
Page 1: Squashing the Heisenbugs

SQUASHING THE HEISENBUGSTrotter Cashion / Chariot CI Event 2010

http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/photostream/

Page 2: Squashing the Heisenbugs

SQUASHING THE HEISENBUGSTrotter Cashion / Chariot CI Event 2010

http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/photostream/

BOHRBUGS

Page 3: Squashing the Heisenbugs

SQUASHING THE BOHRBUGSTrotter Cashion / Chariot CI Event 2010

http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/photostream/

Page 4: Squashing the Heisenbugs

THIS WILL BE ON THE INTERNET!

Page 5: Squashing the Heisenbugs

CO-FOUNDER

http://mashion.net

Page 7: Squashing the Heisenbugs

http://gems.github.com/octocat.png

github.com/trotter

Page 8: Squashing the Heisenbugs

AGENDA

Page 9: Squashing the Heisenbugs

HEISENBUGS AND BOHRBUGS

Page 10: Squashing the Heisenbugs

THE MANUAL PROBLEM

Page 11: Squashing the Heisenbugs

CHEF

Page 12: Squashing the Heisenbugs

TYING CHEF INTO CI

Page 13: Squashing the Heisenbugs

QUESTIONS (AQAFT)

Page 14: Squashing the Heisenbugs

HEISENBUGS

http://upload.wikimedia.org/wikipedia/commons/8/84/Hindenburg_burning.jpg

Page 15: Squashing the Heisenbugs

OBSERVER EFFECT

EventYou

Page 16: Squashing the Heisenbugs

http://en.wikipedia.org/wiki/File:Bohr-atom-PAR.svg

BOHRBUGS

Page 17: Squashing the Heisenbugs

MANDELBUGS

Page 18: Squashing the Heisenbugs

SCHRÖDINBUGS

Page 19: Squashing the Heisenbugs

NOOBBUGS

Page 20: Squashing the Heisenbugs

THE MANUAL PROBLEM

Page 21: Squashing the Heisenbugs

http://www.dolphinsc.com/images/Dolphin.jpg

MYSQL

Page 22: Squashing the Heisenbugs

http://images.buycostumes.com/mgen/merchandiser/32494.jpg

POSTGRES

Page 25: Squashing the Heisenbugs
Page 26: Squashing the Heisenbugs

Memcached

Page 27: Squashing the Heisenbugs

MemcachedRedis

Page 28: Squashing the Heisenbugs

MemcachedRedis

Mongodb

Page 29: Squashing the Heisenbugs

MemcachedRedis

Mongodb Riak

Page 30: Squashing the Heisenbugs

MemcachedRedis

Mongodb Riak

Cassandra

Page 31: Squashing the Heisenbugs

MemcachedRedis

Mongodb Riak

Cassandra

Project Voldemort

Page 32: Squashing the Heisenbugs

MemcachedRedis

Mongodb Riak

Cassandra

Project Voldemort

Ubuntu

Page 33: Squashing the Heisenbugs

MemcachedRedis

Mongodb Riak

Cassandra

Project Voldemort

Ubuntu

Redhat

Page 34: Squashing the Heisenbugs

MemcachedRedis

Mongodb Riak

Cassandra

Project Voldemort

Ubuntu

RedhatSolaris

Page 35: Squashing the Heisenbugs

MemcachedRedis

Mongodb Riak

Cassandra

Project Voldemort

Ubuntu

RedhatSolaris

iptables

Page 36: Squashing the Heisenbugs

MemcachedRedis

Mongodb Riak

Cassandra

Project Voldemort

Ubuntu

RedhatSolaris

iptables

sshd

Page 37: Squashing the Heisenbugs

MemcachedRedis

Mongodb Riak

Cassandra

Project Voldemort

Ubuntu

RedhatSolaris

iptables

sshd

DNS Servers

Page 38: Squashing the Heisenbugs

MemcachedRedis

Mongodb Riak

Cassandra

Project Voldemort

Ubuntu

RedhatSolaris

iptables

sshd

DNS Servers

TCP/IP internals

Page 39: Squashing the Heisenbugs

We Idolize the Artisan Sysadmin

Page 40: Squashing the Heisenbugs

... but what do we end up with?

Page 41: Squashing the Heisenbugs

http://www.enviro-bc.ca/wp-content/uploads/2009/04/bowlake-aval_clair-israelson.jpg

SNOWFLAKES... that become avalanches

Page 43: Squashing the Heisenbugs

http://assets.troubledteenswizard.com/uploaded/WoW/nomoney1.jpg

NO MORE MONEY!

Page 44: Squashing the Heisenbugs

We’re really left with two choices

Page 45: Squashing the Heisenbugs

Automate... or die.

Page 46: Squashing the Heisenbugs

Automation is not hard!

Page 47: Squashing the Heisenbugs

BASH

Page 51: Squashing the Heisenbugs

http://www.opscode.com/

WRITTEN BY

Page 52: Squashing the Heisenbugs

http://www.prestonlee.com/wp-content/uploads/2008/09/ruby.png

WRITTEN IN

Page 53: Squashing the Heisenbugs

Two Delicious Flavors

Page 55: Squashing the Heisenbugs

HOW IT WORKS

Page 56: Squashing the Heisenbugs

HOW IT WORKS

Your Computer

Page 57: Squashing the Heisenbugs

HOW IT WORKS

Your Computer

Box to Provision

Page 58: Squashing the Heisenbugs

HOW IT WORKS

Your Computer

Chef ServerBox to Provision

Page 59: Squashing the Heisenbugs

HOW IT WORKS

Your Computer

Chef ServerBox to Provision

Prepare

Page 60: Squashing the Heisenbugs

HOW IT WORKS

Your Computer

Chef ServerBox to Provision

Prepare

What now?

Page 61: Squashing the Heisenbugs

HOW IT WORKS

Your Computer

Chef ServerBox to Provision

Prepare

What now?

Cookbooks

Page 62: Squashing the Heisenbugs

HOW IT WORKS

Your Computer

Chef ServerBox to Provision

Prepare

What now?

Cookbooks

Run!

Page 64: Squashing the Heisenbugs

HOW IT WORKS

Page 65: Squashing the Heisenbugs

HOW IT WORKS

Your Computer

Page 66: Squashing the Heisenbugs

HOW IT WORKS

Your Computer

Box to Provision

Page 67: Squashing the Heisenbugs

HOW IT WORKS

Your Computer

Box to Provision

Prepare

Page 68: Squashing the Heisenbugs

HOW IT WORKS

Your Computer

Box to ProvisionCookbooks

Prepare

Page 69: Squashing the Heisenbugs

HOW IT WORKS

Your Computer

Box to ProvisionCookbooks

Prepare

Go!

Page 70: Squashing the Heisenbugs

Chef Structure

Page 71: Squashing the Heisenbugs

CHEF HEIRARCHY

Node

Roles

Cookbooks

Cookbooks

Recipes Attributes Templates Definitions

Page 72: Squashing the Heisenbugs

ROLES ARE REUSABLE!

config/dev.json config/db.json

roles/db.rb

Page 73: Squashing the Heisenbugs

http://www.flickr.com/photos/seandreilinger/959864706/sizes/o/

GETTINGGOING

Page 74: Squashing the Heisenbugs

WHAT YOU NEED

Page 75: Squashing the Heisenbugs

WHAT YOU NEED

Ruby & Rubygems

Page 76: Squashing the Heisenbugs

WHAT YOU NEED

Ruby & Rubygems

`gem install chef` Chef Gem

Page 77: Squashing the Heisenbugs

WHAT YOU NEED

Ruby & Rubygems

`gem install chef` Chef Gem

`git clone http://github.com/opscode/chef-repo.git`

Chef Repo

Page 78: Squashing the Heisenbugs

http://cookbooks.opscode.com/

COOKBOOKS.OPSCODE.COM

Page 79: Squashing the Heisenbugs

TERMINAL JUNKIES

Page 80: Squashing the Heisenbugs

TERMINAL JUNKIES

•spatula search mysql

Page 81: Squashing the Heisenbugs

TERMINAL JUNKIES

•spatula search mysql

•spatula install mysql

Page 82: Squashing the Heisenbugs

TERMINAL JUNKIES

•spatula search mysql

•spatula install mysql

•spatula show mysql

Page 83: Squashing the Heisenbugs

TERMINAL JUNKIES

•spatula search mysql

•spatula install mysql

•spatula show mysql

•... knife has similar functionality

Page 84: Squashing the Heisenbugs

GET THE BOX READY

spatula prepare db-one.host

ssh db-one.host “install ruby”

ssh db-one.host “install chef”

=

Page 85: Squashing the Heisenbugs

GO GO GO!

spatula cook db-one.host db ssh db-one.host “sudo \ chef-solo -c config/solo.rb \ -j config/db.json”

rsync repo to db-one.host

=

Page 86: Squashing the Heisenbugs

WITH CHEF SERVER

install dependencies

turn on client

send cookbooks to client

Page 87: Squashing the Heisenbugs

http://media.photobucket.com/image/minecraft/bluekabal/roma/minesofroma.jpg

DIGGING DEEPER

Page 88: Squashing the Heisenbugs

CHEF HEIRARCHY

Node

Roles

Cookbooks

Cookbooks

Recipes Attributes Templates Definitions

Page 89: Squashing the Heisenbugs

// ./config/db_one.json{ "run_list": ["role[db]"] }

A NODE

Recipes and Roles go here

Page 90: Squashing the Heisenbugs

CHEF HEIRARCHY

Node

Roles

Cookbooks

Cookbooks

Recipes Attributes Templates Definitions

Page 91: Squashing the Heisenbugs

# ./roles/db.rbname "database"description "Be a DB"run_list [ "apache2", "mysql::server" ]

A ROLE

Page 92: Squashing the Heisenbugs

# ./roles/db.rbname "database"description "Be a DB"run_list [ "apache2", "mysql::server" ]

A ROLE

Recipes to run

Page 93: Squashing the Heisenbugs

# ./roles/db.rbname "database"description "Be a DB"run_list [ "apache2", "mysql::server" ]

A ROLE

./cookbooks/apache2/recipes/default.rb

Page 94: Squashing the Heisenbugs

# ./roles/db.rbname "database"description "Be a DB"run_list [ "apache2", "mysql::server" ]

A ROLE

./cookbooks/mysql/recipes/server.rb

Page 95: Squashing the Heisenbugs

CHEF HEIRARCHY

Node

Roles

Cookbooks

Cookbooks

Recipes Attributes Templates Definitions

Page 96: Squashing the Heisenbugs

# ./cookbooks/mysql/attributes/server.rbdefault[:mysql][:server_debian_password] = secure_passworddefault[:mysql][:server_root_password] = secure_passworddefault[:mysql][:server_repl_password] = secure_passworddefault[:mysql][:bind_address] = ipaddressdefault[:mysql][:datadir] = "/var/lib/mysql"

# Tunablesdefault[:mysql][:tunable][:key_buffer] = "250M"default[:mysql][:tunable][:max_connections] = "800"default[:mysql][:tunable][:wait_timeout] = "180"

ATTRIBUTES

Page 97: Squashing the Heisenbugs

// ./config/db.json{ "mysql": { "server_root_password": "beefcake" },  "recipes": [ "mysql" ] }

OVERRIDING ATTRIBUTES

# ./roles/db.rbname "database"description "Be a DB"run_list [ "mysql::server" ]default_attributes :mysql => { :server_root_password =>

"beefcake" }

Node

Role

Page 98: Squashing the Heisenbugs

CHEF HEIRARCHY

Node

Roles

Cookbooks

Cookbooks

Recipes Attributes Templates Definitions

Page 99: Squashing the Heisenbugs

# ./cookbooks/mysql/recipes/server.rbinclude_recipe "mysql::client"

THE RECIPE... PART 1

Page 100: Squashing the Heisenbugs

package "mysql-server" do action :installend

THE RECIPE... PART 2

Page 101: Squashing the Heisenbugs

template "/etc/mysql/my.cnf" do source "my.cnf.erb" owner "root" group "root" mode "0644" notifies :restart, resources(:service => "mysql"), :immediatelyend

THE RECIPE... PART 3

Page 102: Squashing the Heisenbugs

CHEF HEIRARCHY

Node

Roles

Cookbooks

Cookbooks

Recipes Attributes Templates Definitions

Page 103: Squashing the Heisenbugs

user = mysqlpid-file = /var/run/mysqld/mysqld.pidsocket = /var/run/mysqld/mysqld.sockdatadir = <%= @datadir %>bind-address = <%= @node[:mysql][:bind_address] %>

A TEMPLATE SAMPLE

Page 104: Squashing the Heisenbugs

user = mysqlpid-file = /var/run/mysqld/mysqld.pidsocket = /var/run/mysqld/mysqld.sockdatadir = <%= @datadir %>bind-address = <%= @node[:mysql][:bind_address] %>

A TEMPLATE SAMPLE

OMG, ERB!!

Page 105: Squashing the Heisenbugs

user = mysqlpid-file = /var/run/mysqld/mysqld.pidsocket = /var/run/mysqld/mysqld.sockdatadir = <%= @datadir %>bind-address = <%= @node[:mysql][:bind_address] %>

A TEMPLATE SAMPLE

From Attributes

Page 106: Squashing the Heisenbugs

user = mysqlpid-file = /var/run/mysqld/mysqld.pidsocket = /var/run/mysqld/mysqld.sockdatadir = <%= @datadir %>bind-address = <%= @node[:mysql][:bind_address] %>

A TEMPLATE SAMPLE

From Template

Page 107: Squashing the Heisenbugs

template "/etc/mysql/my.cnf" do source "my.cnf.erb" owner "root" group "root" mode "0644" variables :datadir => “/var/lib/mysql”end

ASSIGNING VARIABLES

Page 108: Squashing the Heisenbugs

CHEF HEIRARCHY

Node

Roles

Cookbooks

Cookbooks

Recipes Attributes Templates Definitions

Page 109: Squashing the Heisenbugs

define :apache_conf do dir = node[:apache][:dir] template "#{dir}/mods-available/#{params[:name]}.conf" do source "mods/#{params[:name]}.conf.erb" notifies :restart, resources(:service => "apache2") end end

# In your recipeapache_conf "mod_ssl"

DEFINITIONS

Page 110: Squashing the Heisenbugs

TYING IT INTO CI

Page 111: Squashing the Heisenbugs

USE CHEF TO PROVISION CI

http://images.nationalgeographic.com/wpf/media-live/photos/000/148/custom/heart-ngk0207_14824_470x300.jpg

Page 112: Squashing the Heisenbugs

attribution

CHEF AS PART OF TEST SCRIPT

Page 113: Squashing the Heisenbugs

CONTINUOUS DEPLOYMENT

Page 114: Squashing the Heisenbugs

http://www.flickr.com/photos/andresthor/3946773501/sizes/l/in/photostream/

LET’SWRAPIT UP

Page 115: Squashing the Heisenbugs

http://www.flickr.com/photos/kenfagerdotcom/3409813881/

AUTOMATE EVERYTHING!!

Page 116: Squashing the Heisenbugs

Deploying == Provisioning

Page 117: Squashing the Heisenbugs

http://www.flickr.com/photos/martin_borjesson/4083726805/sizes/o/

USE CHEF!!!!!

Page 118: Squashing the Heisenbugs

ON THE INTERNETS

Page 119: Squashing the Heisenbugs

ON THE INTERNETS

•twitter - @cashion

Page 120: Squashing the Heisenbugs

ON THE INTERNETS

•twitter - @cashion

•github - http://github.com/trotter

Page 121: Squashing the Heisenbugs

ON THE INTERNETS

•twitter - @cashion

•github - http://github.com/trotter

•email - cashion on the gmailz

Page 122: Squashing the Heisenbugs

http://www.flickr.com/photos/dancoulter/21042744/sizes/o/

THANK YOU!

Page 123: Squashing the Heisenbugs

RESOURCES

Page 124: Squashing the Heisenbugs

RESOURCES

•http://wiki.opscode.com/display/chef/Resources

Page 125: Squashing the Heisenbugs

RESOURCES

•http://wiki.opscode.com/display/chef/Resources

•#chef on freenode


Top Related