michelin starred cooking with chef presentation

Michelin Starred Cooking with Chef

Jon Cowie, Etsy.comjcowie@etsy.com

@jonlives

Wednesday, June 27, 12

• Chef at Etsy

• Familiarity and Understanding

• Chef at Etsy

• Critical Approach and Experimentation

• Chef at Etsy

• Use The Source

• Chef at Etsy

• Use The Source

• A liberal sprinkling of screwups

• Chef at Etsy

• Use The Source

• Open Sourced Goodness - We’re all here!

• Chef at Etsy

• Use The Source

• Open Sourced Goodness - We’re all here!

• [x] = http://tiny.cc/velocity2012

Opscode is Orange,Velocity is Blue.In Soviet Russia,

Cookbook writes you.

Chef at Etsy

Our Setup

Our Setup• Open Source chef server 0.10.4

• Backup to Opscode Platform

• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)

• KVM & lxc virts, self hosted too.

• Never test in production!

• Many chefs don’t spoil our soup

Familiarity and Understanding

http://www.flickr.com/photos/photo_secessionist/5555167113/ (C) Alexander McQueen

• Insight

• Simplicity

• Insight

• Simplicity

• Standards

Insight

• You should never have to say “I don’t know.”

Insight

• What, where, when, why, how long?

Insight

• What, where, when, why, how long?

• Easy, and I’ll show you how!

Chef Dashboard

Chef Dashboard• Chef handler sends metrics to graphite [4]

Chef Dashboard• Chef handler sends metrics to graphite [4]• git clone git://github.com/etsy/chef-handlers.git

• Set your graphite server’s URL in graphite.rb

• Add the following to client.rb

• Add the following to client.rb• require "<clonedir>/graphite.rb"

graphite_handler = GraphiteReporting.newreport_handlers << graphite_handlerexception_handlers << graphite_handler

• Add the following to client.rb• require "<clonedir>/graphite.rb"

graphite_handler = GraphiteReporting.newreport_handlers << graphite_handlerexception_handlers << graphite_handler

• Etsy dashboards framework [5]

[12:54:01] <irccat> Chef run failed on gfernandez.vm.ny4dev.etsy.com[12:54:02] <irccat> https://github.etsycorp.com/gist/384228[12:54:02] <irccat>

[12:54:07] <irccat> Chef run failed on buildtest11.ny4dev.etsy.com[12:54:07] <irccat> https://github.etsycorp.com/gist/384227[12:54:07] <irccat>

Chef irccat Alerts

• Chef handler send fails to irc[4] via irccat [6]

Chef irccat Alerts

• Chef handler send fails to irc[4] via irccat [6]• git clone git://github.com/etsy/chef-handlers.git

Chef irccat Alerts

• Set your irccat[6] server’s URL in logtoirc.rb

Chef irccat Alerts

• Add the following to client.rb• require "<clonedir>/logtoirc.rb"

exception_handlers << Etsy::LogToIRC.new

~ > knife node lastrun buildtest11.ny4dev.etsy.comStatus failed Elapsed Time 4.628171438 Start Time 2012-06-18 10:06:28 +0000End Time 2012-06-18 10:06:32 +0000

Recipe Action Resource Type Resource

Backtrace<snip>

ExceptionChef::Exceptions::Package: package[php] (php::buildtest line 20) had an error: Version 5.3.10-1.el5 of php not found. Did you specify both version and release? (version-release, e.g. 1.84-10.fc6)

~ > knife search node 'lastrun_debug_formatted_exception:Chef\:\:Exceptions\:\:Package*' -a lastrun.debug.formatted_exception

5 items found

id: masterrestore.ny4.etsy.comlastrun.debug.formatted_exception: Chef::Exceptions::Package: package[postgresql-server] (postgresql::server-8.3 line 1) had an error: Installed package postgresql-server-8.3.16-1PGDG_id is newer than candidate package postgresql-server-8.3.11-1PGDG_id.rhel5

id: buildtest11.ny4dev.etsy.comlastrun.debug.formatted_exception: Chef::Exceptions::Package: package[php] (php::buildtest line 20) had an error: Version 5.3.10-1.el5 of php not found. Did you specify both version and release? (version-release, e.g. 1.84-10.fc6)

<snip>

Chef lastrun Info

• Chef handler and knife plugin [7]

Chef lastrun Info

• gem install knife-lastrun

Chef lastrun Info

• Add the following to client.rb• require "lastrun_update"

handler = LastRunUpdateHandler.newreport_handlers << handlerexception_handlers << handler

Chef lastrun Info

• Add the following to client.rb• require "lastrun_update"

handler = LastRunUpdateHandler.newreport_handlers << handlerexception_handlers << handler

• knife node lastrun <nodename>

Simplicity

• Think of yourself at 3AM!

Simplicity

• Please, won’t you think of the new guy?

Simplicity

• Minimize the logics!

Simplicity

• Minimize the logics!

• As few logical steps from start to finish as possible.

Simplicity - Not!Date: Mon Dec 05 2011 23:07:18 GMT+0000 (GMT)

Subject: so close to death

# Don't install v2 on search or Cent 5.6 nodes-if node[:fqdn] !~ /\b(^(preprod-)?search[0-9]{2}|ny4dev\.etsy\.com|^(preprod-)?giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|îmgcache[0-9]{2}|imgconvert[0-9]{2}|îmgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})\b/ and node.role?("Web56") == false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false+if node[:fqdn] !~ /\b(^(preprod-)?search[0-9]{2}|ny4dev\.etsy\.com|^(preprod-)?giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|îmgcache[0-9]{2}|imgconvert[0-9]{2}|îmgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})\b/ and node.role?("Web56") == false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false and node.role?("Auth") == false

Simplicity - Better!if node.chef_environment == "libmemcached_upgrade"

package "libmemcached" do

version "1.0.4-1"

action :install

<snip>

package "libmemcached" do

version "0.53-1.1"

action :install

<snip>

Simplicity - Complexity

• Sometimes you need complex behaviour

• Don’t fight it, try to abstract it.

• Case in point: Syslog-ng refactor

Case Study: Syslog-ng

• 36 recipes

• 30 versions of syslog-ng.conf

• 36 recipes

• 27 manually configured files in /etc/syslog-ng.d on central server

• 36 recipes

• 27 manually configured files in /etc/syslog-ng.d on central server

• Edge cases and exceptions galore

• Down to:

• 2 recipes (one client, one server)

• Down to:

• 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d)

• Down to:

• Attributes in roles

• Down to:

• Attributes in roles

• Not open sourced yet, sorry :(

Case Study: Syslog-ng"syslog": {

"group": "preprod_web",

"items": {

"web_apache_access_log": {

"source": "/var/log/httpd/access_log",

"source_program_override": "APACHEACCESS: ",

"destination": "<snip>/access.log",

"destination_filters": [

"host('^preprod-web')",

"match('APACHEACCESS')"

"destination_options": [

"template_escape(no)"

Remember, No Panacea!

• A new package hits the repo.

• Are you in control of when it goes out?

• Memcached Outage

• Do you know what services are going to restart and when?

• Image Service Outage

Standards

• Not going to talk about testing [8]!

Standards

• Not going to talk about testing [8]!

• But I don’t have time for standards!

Standards - No Time!

• I won’t say “Make Time”, but you should...

• For a quick win, try Foodcritic

• Good out of the box rules

• Jenkins integration in seconds

• Supports custom rules

• Plays well with others

Foodcritic

Foodcritic•gem install foodcritic

•foodcritic <cookbook_repo>

•Simple Jenkins job:

#!/usr/bin/env rvm-shell 1.9.3foodcritic -f correctness .

Standards at Etsy

Standards at Etsy• “style” not “correctness”[9]

Standards at Etsy• “style” not “correctness”[9]• ETSY001 - Package or yum_package resource used with :upgrade action

• ETSY002 - Execute resource used to run git commands

• ETSY003 - Execute resource used to run curl or wget commands

• ETSY004 - Execute resource defined without conditional or action :nothing

• ETSY005 - Action :restart sent to a core service

• ETSY006 - Execute resource used to run chef-provided command

• ETSY007 - Package or yum_package resource used to install core package without specific version number

Standards at Etsy

• ETSY001 - Written after Memcached Outage

Standards at Etsy

• Package or yum_package resource used with :upgrade action

Standards at Etsy

• Package or yum_package resource used with :upgrade action

• package "memcached" do action :upgradeend

Standards at Etsy

• ETSY005 - Written after a total Image service outage

Standards at Etsy

• Action :restart sent to a core service

Standards at Etsy

• Action :restart sent to a core service• cookbook_file "/etc/httpd/conf.d/myvhost.conf" do

source "myvhost.conf" notifies :restart, resources(:service => "httpd")end

Critical Approach and Experimentation

• Chef is by necessity generic

• ...so don’t take Opscode’s word for it.

• If it doesn’t work well for you, change it!

• Case Study - Etsy Environments rollout

Environments Rollout

• We knew we needed them

• Simple enough, right?

• Let’s look at the workflow...

Env: Standard Workflow

• knife cookbook show php

• Change version number in metadata.rb

• Change version constraint in foo.json

• Commit and push changes to git

• knife cookbook upload php --freeze

• knife environment from file foo.json

Env: Our Issues

• 41 people with Chef repo access

Env: Our Issues

• Most with knife keys

Env: Our Issues

• Most with knife keys

• All editing the same 2 files with every change...

Env: Solutions?

• Go with it and hope for the best?

Env: Solutions?

• Don’t use environments?

Env: Solutions?

• Write a totally new workflow?

Env: Solutions?

• Write a totally new workflow?

• Tweak the existing one with some tooling?

Env: Solution!

• Tweak with some tooling!

Env: Solution!

• Tweak with some tooling!

• Presenting knife-spork [9]...

Spork: Workflow

• Wrapper around standard environments workflow

Spork: Workflow

• check - cookbook versioning

Spork: Workflow

• bump - increment version component

Spork: Workflow

• upload - upload and freeze

Spork: Workflow

• upload - upload and freeze

• promote - set env constraints

Spork: Check$> knife spork check apache2

Checking versions for cookbook apache2...

Current local version: 1.0.6

Remote versions (Max. 5 most recent only):*1.0.6, frozen1.0.5, frozen<snip>

DANGER: Your local cookbook has same version number as the starred version above!

Please bump your local version or you won't be able to upload.

Spork: Bump

$> knife spork bump apache2 <major|minor|patch|manual>

Bumping patch level of the apache2 cookbook from 1.0.6 to 1.0.7

Spork: Upload

$> knife spork upload apache2

Uploading and freezing apache2 [1.0.7]upload complete

Spork: Promote$> knife spork promote foo php

Adding version constraint php = 1.0.6

Saving changes into foo.json

Promotion complete! Please remember to upload your changed Environment file to the Chef Server.

$> knife spork promote foo php --remoteAdding version constraint php = 0.1.0

Saving changes into foo.json

Uploading foo to server

• Worked well, avoided the issues it was designed for

• Subsequently evolved

• A lot of input & work came from Devs

• Organic evolution

• Open sourced, of course

Spork: Evolution

• Safety Checks

Spork: Evolution

• Safety Checks

• Extra Features

Spork: Safety Checks

• Before promoting, check version is uploaded...

$> knife spork promote php --remote

<snip>

WARNING: It looks like you have multiple cookbook paths defined so I can't tell if you're running inside a git repo.

Checking that php version 0.1.93 exists on the server before promoting (any error means it hasn't been uploaded yet)...

ERROR: The object you are looking for could not be found

Response: Cannot find a cookbook named php with version 0.1.93

• Before promoting, check version is uploaded...

• Check if you’re promoting changes to more than you thought....

Spork: Safety ChecksWARNING: You're about to promote changes to several cookbooks:

WARNING: ganglia: = 0.1.26 changed to = 0.1.25installerz: = 0.1.66 changed to = 0.1.65php: = 0.1.92 changed to = 0.1.93

Are you sure you want to continue? (Y/N) N

You said no, so I'm done here.

Would you like to reset your local development.json to match the server?? (Y/N) Y

<snip>

development.json reset.

Spork: Features

Spork: Features• Default Environments

• Git support

• Chat notifications + Gist (upload and promote)

• Git support

• IRCCat

• Git support

• IRCCat

• Hipchat (courtesy of Secondmarket)

Features: Chat Notifications

[19:43:46] <irccat> CHEF: pmcdonnell uploaded and froze cookbook immount version 0.0.24

[19:44:00] <irccat> CHEF: pmcdonnell uploaded environment production https://github.etsycorp.com/gist/385043

[19:44:01] <irccat> CHEF: pmcdonnell uploaded environment development https://github.etsycorp.com/gist/385044

• Git support

• IRCCat

• Graphite (on promote)

Features: Graphite

• Git support

• IRCCat

• Graphite (on promote)

• Foodcritic (on upload)Wednesday, June 27, 12

Features: Foodcritic

$> knife spork upload system

<snip>

Lint checking system...

ERROR: Lint check failed. Halting upload.

ERROR: Lint check output:

ERROR: ETSY003: Execute resource used to run curl or wget commands: /Users/jcowie/dev/etsy/chef/cookbooks/system/recipes/dev-ssl.rb:41

Use the Source!

tiny.cc/velocity2012

We’re hiring!

BoF on Tuesdayor just come and say Hi :)

michelin starred cooking with chef presentation

chef dashboard chef

chef dashboardwednesday

setup open source chef

chef jon cowie

git clone git

etsy familiarity

selfhosted nodes

graphite servers url

Technology

guest chef dinner with michelin-starred chef christophe...

michelin starred cooking with chef

gourmet food travel - andronis exclusive spa... · 2020. 7....

palmerhouse - palady homesfor a special occasion, you could...

white - edition hotelssoaring penthouse; a restaurant by...

celebrity constellation · • menus by a michelin-starred...

173054 the loftings amenities leaflet 4pp 210 x...

beautifully prepared by michelin-starred chef ettore

the food nomad - luxury cruises: amazon, mekong river...

celebrity edge · eat. drink. meet. • globally inspired...

press kit 2018 - monte carlo sbm corporate · paolo sari at...

benu - phaidon · benu the debut book from the celebrated,...

alcohol and other drug use in michelin-starred kitchen

the michelin guide i shanghai michelin€¦ · michelin...

great britain & ireland michelin guide 2015 summary...

the cuisine · 2016. 12. 7. · 1 michelin star, chef...

andlight mana restaurant dubai design report · designers...

global compact report 2012 - composites | gelcoats...7 /60...

ahmed baladi lawyer partner, allen &...

presentazione di powerpoint - starhotels · 2018. 9....