michelin starred cooking with chef presentation

Michelin Starred Cooking with Chef

Jon Cowie, [email protected]

@jonlives

Wednesday, June 27, 12

mailto:[email protected]

mailto:[email protected]

What?


What?

• Chef at Etsy


What?

• Chef at Etsy

• Familiarity and Understanding


What?

• Chef at Etsy


• Critical Approach and Experimentation


What?

• Chef at Etsy



• Use The Source


What?

• Chef at Etsy



• Use The Source

• A liberal sprinkling of screwups


What?

• Chef at Etsy



• Use The Source


• Open Sourced Goodness - We’re all here!


What?

• Chef at Etsy



• Use The Source


• Open Sourced Goodness - We’re all here!

• [x] = http://tiny.cc/velocity2012


http://tiny.cc/velocity2012

http://tiny.cc/velocity2012

Opscode is Orange,Velocity is Blue.In Soviet Russia,

Cookbook writes you.


Chef at Etsy


Our Setup


Our Setup• Open Source chef server 0.10.4



• Backup to Opscode Platform




• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)





• KVM & lxc virts, self hosted too.






• Never test in production!






• Never test in production!

• Many chefs don’t spoil our soup


Familiarity and Understanding


not

http://www.flickr.com/photos/photo_secessionist/5555167113/ (C) Alexander McQueen


http://www.flickr.com/photos/photo_secessionist/5555167113/




• Insight


• Insight

• Simplicity


• Insight

• Simplicity

• Standards


Insight


Insight

• You should never have to say “I don’t know.”


Insight


• What, where, when, why, how long?


Insight


• What, where, when, why, how long?

• Easy, and I’ll show you how!


Chef Dashboard


Chef Dashboard• Chef handler sends metrics to graphite [4]


Chef Dashboard• Chef handler sends metrics to graphite [4]• git clone git://github.com/etsy/chef-handlers.git



• Set your graphite server’s URL in graphite.rb




• Add the following to client.rb




• Add the following to client.rb• require "<clonedir>/graphite.rb"

graphite_handler = GraphiteReporting.newreport_handlers << graphite_handlerexception_handlers << graphite_handler




• Add the following to client.rb• require "<clonedir>/graphite.rb"

graphite_handler = GraphiteReporting.newreport_handlers << graphite_handlerexception_handlers << graphite_handler

• Etsy dashboards framework [5]


[12:54:01] <irccat> Chef run failed on gfernandez.vm.ny4dev.etsy.com[12:54:02] <irccat> https://github.etsycorp.com/gist/384228[12:54:02] <irccat>

[12:54:07] <irccat> Chef run failed on buildtest11.ny4dev.etsy.com[12:54:07] <irccat> https://github.etsycorp.com/gist/384227[12:54:07] <irccat>


https://github.etsycorp.com/gist/384228




Chef irccat Alerts


Chef irccat Alerts

• Chef handler send fails to irc[4] via irccat [6]


Chef irccat Alerts

• Chef handler send fails to irc[4] via irccat [6]• git clone git://github.com/etsy/chef-handlers.git


Chef irccat Alerts


• Set your irccat[6] server’s URL in logtoirc.rb


Chef irccat Alerts





Chef irccat Alerts



• Add the following to client.rb• require "<clonedir>/logtoirc.rb"

exception_handlers << Etsy::LogToIRC.new


~ > knife node lastrun buildtest11.ny4dev.etsy.comStatus failed Elapsed Time 4.628171438 Start Time 2012-06-18 10:06:28 +0000End Time 2012-06-18 10:06:32 +0000

Recipe Action Resource Type Resource

Backtrace<snip>

ExceptionChef::Exceptions::Package: package[php] (php::buildtest line 20) had an error: Version 5.3.10-1.el5 of php not found. Did you specify both version and release? (version-release, e.g. 1.84-10.fc6)


~ > knife search node 'lastrun_debug_formatted_exception:Chef\:\:Exceptions\:\:Package*' -a lastrun.debug.formatted_exception

5 items found

id: masterrestore.ny4.etsy.comlastrun.debug.formatted_exception: Chef::Exceptions::Package: package[postgresql-server] (postgresql::server-8.3 line 1) had an error: Installed package postgresql-server-8.3.16-1PGDG_id is newer than candidate package postgresql-server-8.3.11-1PGDG_id.rhel5

id: buildtest11.ny4dev.etsy.comlastrun.debug.formatted_exception: Chef::Exceptions::Package: package[php] (php::buildtest line 20) had an error: Version 5.3.10-1.el5 of php not found. Did you specify both version and release? (version-release, e.g. 1.84-10.fc6)

<snip>


Chef lastrun Info


Chef lastrun Info

• Chef handler and knife plugin [7]


Chef lastrun Info


• gem install knife-lastrun


Chef lastrun Info





Chef lastrun Info



• Add the following to client.rb• require "lastrun_update"

handler = LastRunUpdateHandler.newreport_handlers << handlerexception_handlers << handler


Chef lastrun Info



• Add the following to client.rb• require "lastrun_update"

handler = LastRunUpdateHandler.newreport_handlers << handlerexception_handlers << handler

• knife node lastrun <nodename>


Simplicity


Simplicity

• Think of yourself at 3AM!


Simplicity


• Please, won’t you think of the new guy?


Simplicity



• Minimize the logics!


Simplicity



• Minimize the logics!

• As few logical steps from start to finish as possible.


Simplicity - Not!Date: Mon Dec 05 2011 23:07:18 GMT+0000 (GMT)

Subject: so close to death

# Don't install v2 on search or Cent 5.6 nodes-if node[:fqdn] !~ /\b(^(preprod-)?search[0-9]{2}|ny4dev\.etsy\.com|^(preprod-)?giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|îmgcache[0-9]{2}|imgconvert[0-9]{2}|îmgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})\b/ and node.role?("Web56") == false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false+if node[:fqdn] !~ /\b(^(preprod-)?search[0-9]{2}|ny4dev\.etsy\.com|^(preprod-)?giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|îmgcache[0-9]{2}|imgconvert[0-9]{2}|îmgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})\b/ and node.role?("Web56") == false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false and node.role?("Auth") == false


Simplicity - Better!if node.chef_environment == "libmemcached_upgrade"

package "libmemcached" do

version "1.0.4-1"

action :install

end

<snip>

else

package "libmemcached" do

version "0.53-1.1"

action :install

end

<snip>

end


Simplicity - Complexity



• Sometimes you need complex behaviour




• Don’t fight it, try to abstract it.




• Don’t fight it, try to abstract it.

• Case in point: Syslog-ng refactor


Case Study: Syslog-ng



• 36 recipes



• 36 recipes

• 30 versions of syslog-ng.conf



• 36 recipes


• 27 manually configured files in /etc/syslog-ng.d on central server



• 36 recipes


• 27 manually configured files in /etc/syslog-ng.d on central server

• Edge cases and exceptions galore



• Down to:



• Down to:

• 2 recipes (one client, one server)



• Down to:


• 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d)



• Down to:



• Attributes in roles



• Down to:



• Attributes in roles

• Not open sourced yet, sorry :(


Case Study: Syslog-ng"syslog": {

"group": "preprod_web",

"items": {

"web_apache_access_log": {

"source": "/var/log/httpd/access_log",

"source_program_override": "APACHEACCESS: ",

"destination": "<snip>/access.log",

"destination_filters": [

"host('^preprod-web')",

"match('APACHEACCESS')"

],

"destination_options": [

"template_escape(no)"

]

},

}

}


Remember, No Panacea!



• A new package hits the repo.




• Are you in control of when it goes out?





• Memcached Outage






• Do you know what services are going to restart and when?






• Do you know what services are going to restart and when?

• Image Service Outage


Standards


Standards

• Not going to talk about testing [8]!


Standards

• Not going to talk about testing [8]!

• But I don’t have time for standards!


Standards - No Time!



• I won’t say “Make Time”, but you should...




• For a quick win, try Foodcritic





• Good out of the box rules






• Jenkins integration in seconds







• Supports custom rules







• Supports custom rules

• Plays well with others


Foodcritic


Foodcritic•gem install foodcritic



•foodcritic <cookbook_repo>




•Simple Jenkins job:




•Simple Jenkins job:

#!/usr/bin/env rvm-shell 1.9.3foodcritic -f correctness .


Standards at Etsy


Standards at Etsy• “style” not “correctness”[9]


Standards at Etsy• “style” not “correctness”[9]• ETSY001 - Package or yum_package resource used with :upgrade action



• ETSY002 - Execute resource used to run git commands




• ETSY003 - Execute resource used to run curl or wget commands





• ETSY004 - Execute resource defined without conditional or action :nothing






• ETSY005 - Action :restart sent to a core service







• ETSY006 - Execute resource used to run chef-provided command







• ETSY006 - Execute resource used to run chef-provided command

• ETSY007 - Package or yum_package resource used to install core package without specific version number


Standards at Etsy


Standards at Etsy

• ETSY001 - Written after Memcached Outage


Standards at Etsy


• Package or yum_package resource used with :upgrade action


Standards at Etsy


• Package or yum_package resource used with :upgrade action

• package "memcached" do action :upgradeend


Standards at Etsy


Standards at Etsy

• ETSY005 - Written after a total Image service outage


Standards at Etsy


• Action :restart sent to a core service


Standards at Etsy


• Action :restart sent to a core service• cookbook_file "/etc/httpd/conf.d/myvhost.conf" do

source "myvhost.conf" notifies :restart, resources(:service => "httpd")end


Critical Approach and Experimentation


CA&E


CA&E

• Chef is by necessity generic


CA&E


• ...so don’t take Opscode’s word for it.


CA&E



• If it doesn’t work well for you, change it!


CA&E



• If it doesn’t work well for you, change it!

• Case Study - Etsy Environments rollout


Environments Rollout



• We knew we needed them




• Simple enough, right?




• Simple enough, right?

• Let’s look at the workflow...


Env: Standard Workflow



• knife cookbook show php




• Change version number in metadata.rb





• Change version constraint in foo.json






• Commit and push changes to git







• knife cookbook upload php --freeze







• knife cookbook upload php --freeze

• knife environment from file foo.json


Env: Our Issues


Env: Our Issues

• 41 people with Chef repo access


Env: Our Issues


• Most with knife keys


Env: Our Issues


• Most with knife keys

• All editing the same 2 files with every change...


Env: Solutions?


Env: Solutions?

• Go with it and hope for the best?


Env: Solutions?


• Don’t use environments?


Env: Solutions?



• Write a totally new workflow?


Env: Solutions?



• Write a totally new workflow?

• Tweak the existing one with some tooling?


Env: Solution!


Env: Solution!

• Tweak with some tooling!


Env: Solution!

• Tweak with some tooling!

• Presenting knife-spork [9]...


Spork: Workflow


Spork: Workflow

• Wrapper around standard environments workflow


Spork: Workflow


• check - cookbook versioning


Spork: Workflow



• bump - increment version component


Spork: Workflow




• upload - upload and freeze


Spork: Workflow




• upload - upload and freeze

• promote - set env constraints


Spork: Check$> knife spork check apache2

Checking versions for cookbook apache2...

Current local version: 1.0.6

Remote versions (Max. 5 most recent only):*1.0.6, frozen1.0.5, frozen<snip>

DANGER: Your local cookbook has same version number as the starred version above!

Please bump your local version or you won't be able to upload.


Spork: Bump

$> knife spork bump apache2 <major|minor|patch|manual>

Bumping patch level of the apache2 cookbook from 1.0.6 to 1.0.7


Spork: Upload

$> knife spork upload apache2

Uploading and freezing apache2 [1.0.7]upload complete


Spork: Promote$> knife spork promote foo php

Adding version constraint php = 1.0.6

Saving changes into foo.json

Promotion complete! Please remember to upload your changed Environment file to the Chef Server.

---

$> knife spork promote foo php --remoteAdding version constraint php = 0.1.0

Saving changes into foo.json

Uploading foo to server


Spork


Spork

• Worked well, avoided the issues it was designed for


Spork


• Subsequently evolved


Spork



• A lot of input & work came from Devs


Spork




• Organic evolution


Spork




• Organic evolution

• Open sourced, of course


Spork: Evolution


Spork: Evolution

• Safety Checks


Spork: Evolution

• Safety Checks

• Extra Features


Spork: Safety Checks



• Before promoting, check version is uploaded...



$> knife spork promote php --remote

<snip>

WARNING: It looks like you have multiple cookbook paths defined so I can't tell if you're running inside a git repo.

Checking that php version 0.1.93 exists on the server before promoting (any error means it hasn't been uploaded yet)...

ERROR: The object you are looking for could not be found

Response: Cannot find a cookbook named php with version 0.1.93



• Before promoting, check version is uploaded...

• Check if you’re promoting changes to more than you thought....


Spork: Safety ChecksWARNING: You're about to promote changes to several cookbooks:

WARNING: ganglia: = 0.1.26 changed to = 0.1.25installerz: = 0.1.66 changed to = 0.1.65php: = 0.1.92 changed to = 0.1.93

Are you sure you want to continue? (Y/N) N

You said no, so I'm done here.

Would you like to reset your local development.json to match the server?? (Y/N) Y

<snip>

development.json reset.


Spork: Features


Spork: Features• Default Environments



• Git support



• Git support

• Chat notifications + Gist (upload and promote)



• Git support


• IRCCat



• Git support


• IRCCat

• Hipchat (courtesy of Secondmarket)


Features: Chat Notifications

[19:43:46] <irccat> CHEF: pmcdonnell uploaded and froze cookbook immount version 0.0.24

[19:44:00] <irccat> CHEF: pmcdonnell uploaded environment production https://github.etsycorp.com/gist/385043

[19:44:01] <irccat> CHEF: pmcdonnell uploaded environment development https://github.etsycorp.com/gist/385044







• Git support


• IRCCat


• Graphite (on promote)


Features: Graphite



• Git support


• IRCCat


• Graphite (on promote)

• Foodcritic (on upload)Wednesday, June 27, 12

Features: Foodcritic

$> knife spork upload system

<snip>

Lint checking system...

ERROR: Lint check failed. Halting upload.

ERROR: Lint check output:

ERROR: ETSY003: Execute resource used to run curl or wget commands: /Users/jcowie/dev/etsy/chef/cookbooks/system/recipes/dev-ssl.rb:41


Use the Source!

tiny.cc/velocity2012


We’re hiring!

BoF on Tuesdayor just come and say Hi :)


michelin starred cooking with chef presentation

Technology

chef dashboard chef

chef dashboardwednesday

setup open source chef

chef jon cowie

git clone git

etsy familiarity

selfhosted nodes

graphite servers url