austin agile conf 2012 infrastructure automation-gmiranda

Agility Through Infrastructure Automation

George [email protected]

Austin Agile Conference 2012November 16, 2012

Friday, November 16, 12

mailto:[email protected]


Introductions# finger $(whoami)Login: gmiranda! ! ! ! ! ! ! Name: George MirandaDirectory: /home/gmiranda! ! ! Shell: /bin/bashOn since Mon 14 Apr 1997 18:01 (GMT) on tty1 from :0No mail on [email protected]:

twitter:! gmiranda23github:!! gmiranda23irc:!! ! gmiranda23! (irc.freenode.net - #chef)community:!gmiranda23! (community.opscode.com)role:! ! consultant, evangelist, trainer, *:*




Scope


ScopeAutomation + Culture = Agility



• Infrastructure Automation Approaches




• Infrastructure & Automation Best Practices





• Cultural Pitfalls






• Making more awesome






• Making more awesome

What this talk is not

• Chef vs. Puppet

• Cloud All The Things!!!

• How to structure your Organization

• Which Development Model to adopt


System Build Approaches

http://www.flickr.com/photos/dancedaoc/3083836988/sizes/z/in/photostream/




Complications

• “That one host” you know you can’t rebuild

• Untracked configuration change

• Collections of Bash, PERL, Python, ???

• Rebuild from: wiki, cheatsheets, folklore

http://www.flickr.com/photos/humblog/4996661110/sizes/l/in/photostream/




Unprecedented Growth

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989

Virtual Nodes

Physical Hardware

1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015



1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989

Virtual Nodes

Physical Hardware

1980Mainframe

1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015



1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989

Virtual Nodes

Physical Hardware

1980Mainframe

1990Client/Server

1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015



1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989

Virtual Nodes

Physical Hardware

1980Mainframe

1990Client/Server

2000Datacenter

1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015



1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989

Virtual Nodes

Physical Hardware

1980Mainframe

1990Client/Server

2000Datacenter

2010+Cloud

1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015



1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989

Virtual Nodes

Physical Hardware

1980Mainframe

1990Client/Server

2000Datacenter

2010+Cloud

The things that got us here…

…must change to get us here!

1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015


The Rise of Configuration Management

http://www.flickr.com/photos/24375810@N06/6611017007




We have a problem at scale


Here’s a hint


Cabling?


Close...


http://www.flickr.com/photos/michaelheiss/3090102907/

Complexity




Infrastructure


Items we manipulate• Routes

• Users

• Groups

• Tasks

• Packages

• Software

• Services

• Nodes

• Networking

• Files

• Directories

• Symlinks

• Mounts

• Ruby Gems

• Python Modules

• Java Artifacts

• Disks

• Volumes

• Filesystems

• Firewall Rules


See Node

Application


See Nodes

Application

Application Database


See Nodes Grow

Application

Application Databases


Grow Nodes

App Servers



...Grow

Load Balancer


App Servers


Grow Nodes, Grow

Load Balancers


App Servers


Grow Nodes, Grow

Load Balancers

App DB Cache

App Servers



Infrastructure has a Topology

Load Balancers

App DB Cache

App Servers



Infrastructure IS a Snowflake

Load Balancers

App DB Cache

App Servers


Floating IP?


Complexity Increases Quickly

App LBs

App Servers

NoSQL

DB slaves

Cache

DB Cache

DBs


... Increases Very Quickly

DC1 DC3

DC2


Configuration Management

http://www.flickr.com/photos/philliecasablanca/3354734116/




Sysadmins


The Past


• Labor intensive

• Error prone

• Hard to reproduce

• Unsustainable

http://www.flickr.com/photos/pureimaginations/4805330106/

Manual Configuration




• Typically very brittle

• Throw away, one off scripts

• grep sed awk perl

• curl | bash

http://www.flickr.com/photos/40389360@N00/2428706650/

Scripting




• NFS mounts

• rdist

• scp-on-a-for-loop

• rsync on cron

http://www.flickr.com/photos/walkadog/4317655660

File Distribution


http://www.flickr.com/photos/walkadog/4317655660/sizes/o/in/photostream/

http://www.flickr.com/photos/walkadog/4317655660/sizes/o/in/photostream/

for i in `cat servers.txt` ; do scp ntp.conf root@$i:/etc/ntpd.conf ; done

for i in `cat servers.txt` ; do ssh root@$i /etc/init.d/ntpd restart ; done

for i in `cat servers.txt` ; do ssh root@$i chkconfig ntpd on ; done

• ^ does not scale

http://www.flickr.com/photos/alexerde/3479006495

This used to be awesome.




• Cluster SSH

• ISConf

• Golden Images

Execution Management


Typical Boring Infrastructure

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite



Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

• Move SSH off port 22

• Lets put it on 2022



Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite



• edit /etc/ssh/sshd_config



Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite



• edit /etc/ssh/sshd_config

1 2

3

4

5

6


Maintenance Window

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite


Maintenance Window

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite 1 2

3

8

5 64 7

9

10 11

12

• Launch, Delete

• Repeat

• Typically manually


Maintenance Window

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite 1 2

3

8

5 64 7

9

10 11

12

• Launch, Delete

• Repeat

• Typically manually

• Don’t break anything!

• Bob just got fired =(


Different IP Addresses?

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

• Invalid configs!


Systems Integration

• Keep a list of current resources

• Collect vast amounts of data on those resources

• Quickly search through stacks of current resource data

• Generate your Infrastructure Topology from a current source of truth

http://www.flickr.com/photos/fotos_medem/3399096196/




So when this...

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite


... becomes this...

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite


That can happen automatically

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite


Copyright © 2010 Opscode, Inc - All Rights Reserved 41Friday, November 16, 12

http://www.opscode.com/













































































































Managing Complexity Today


Managing Complexity TodayHow Do we Manage This at Cloud Scale?

• Thousands of infrastructure dependencies and configurations needed for each change.

• Huge Amounts of Time

• Increased Cost of Correction of Manual Errors

• Huge Need for Talent

• Risk of Critical Skills Shortage


Google, Amazon, Microsoft, Yahoobuilt their own tools


but it was “secret sauce”


everyone else was here

... inexperienced & poorly equipped for the world they must now operate in.


Infrastructure

"It is common to think in terms of individual machines rather than view an entire infrastructure as a combined whole"

“A good infrastructure, whether departmental, divisional, or enterprise-wide, is a single loosely-

coupled virtual machine, with hundreds or thousands of hard drives and CPU's.”

-- Bootstrapping an Infrastructure USENIX LISA ’98


Infrastructure as Code

• Programmatically provision and configure

• Treat like any other code base

• Gives you tools to manage complexity while being flexible enough to evolve with your Infrastructure

• Reconstruct the business from code repository, data backup, and baremetal resources


Declarative Syntax

• Define policy

• Say what, not how

• Abstraction between platforms

• Many positive side effects


Idempotence

• You’ll hear this a lot

• Property of declarative interface

• Eliminates brittleness of scripting

• Identity function: f(x)=x

• Safe to repeat


Chef is a Tool

http://www.flickr.com/photos/wessexandy/7690486884/sizes/c/in/pool-96164123@N00/




Wax Philosophical

• We are artists & masters of our craft

• Everyone needs great tools

• Nobody remember’s Picasso’s paintbrush

http://www.flickr.com/photos/vgm8383/2686128924/sizes/l/




The core ideas in this talk:

Automation + Culture =AGILITY!


Pitfalls

http://www.flickr.com/photos/nesposit/2787559303/sizes/o/in/photostream/




This should sound familiar...


Traditional thinking

Dev’s job is to add new featuresOps’ job is to keep the site stable and fast

http://www.flickr.com/photos/stewart/461099066/

Slide Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr




http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

Ops’ job is NOT to keep the site stable and fast

Dev’s job is NOT to add new features


OUR jobis to ENABLE our business


Our business REQUIRES change


BUT CHANGE IS THE CAUSE OF MOST OUTAGES!


Choose:Discourage change in the interests of

stabilityOR

Allow change to happen as often as it needs to


http://www.flickr.com/photos/gsfc/6795048198/sizes/o/in/photostream/

The Great Abyss




The right culture is a requirement for survival & success at web

scale.


Lessons Learned:Every Post-mortem


Lessons Learned:Every Post-mortem

Ever...


Root Cause:“Bad Luck... it was a perfect storm of impossible events”


Lesson #1“We have a bunch of manual processes which we need to

automate”


Lesson #2“We introduced too many changes

at once”


Slide Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change


http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change


Image Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change

RAAAWR!!! I’m SCARY!




Slide Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change




Images Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change

I can haz cuddle?




MAKEMORE

AWESOME!!!


Continuous Delivery

Faster Time to ValueHigher Availability

Happier TeamsMore Cool Stuff


Stuff Suits Care About

• Visibility & Accountability

• Reduce Risk

• Business Agility


Stuff Engineers care about

• Change when we need it

• Innovate Faster

• Constant Improvements

• Application & Site Resiliency


Recap


Recap

•Step 1) Automate your Infrastructure


Recap


•Step 2) Bridge the Cultural Divide


Recap



•Step 3) Profit!


Recap



•Step 3) Profit!

•Automation + Culture = Agility


Try it out!

• Hosted Chef is a SaaS product hosted by Opscode

• http://manage.opscode.com

• Our wiki: http://wiki.opscode.com

• Fast start guide:

• http://wiki.opscode.com/display/chef/Fast+Start+Guide

• Our Community site: http://community.opscode.com

• Cookbooks in our Github account: http://github.com/opscode/cookbooks

• The materials for our 3-day Chef Fundamentals class are online:

• https://github.com/opscode/chef-fundamentals


http://manage.opscode.com

http://manage.opscode.com

http://wiki.opscode.com

http://wiki.opscode.com

http://wiki.opscode.com/display/chef/Fast+Start+Guide

http://wiki.opscode.com/display/chef/Fast+Start+Guide

http://community.opscode.com

http://community.opscode.com

http://github.com/opscode/cookbooks

http://github.com/opscode/cookbooks

https://github.com/opscode/chef-fundamentals

https://github.com/opscode/chef-fundamentals

Supported Platforms

• Ubuntu (10.04, 10.10, 11.04, 11.10, 12.04)

• Debian (5.x, 6.x)

• RHEL & CentOS (5.x, 6.x)

• Fedora 10+

• SUSE Enterprise (11.2)

• openSUSE (12.1)

• Solaris (5.9, 5.10, 5.11 -- x86 and SPARC)

• Mac OS X (10.4, 10.5, 10.6, 10.7)

• Windows 7

• Windows Server 2003 R2, 2008, 2008 R2


Additional Resources• Opscode Youtube Channel:

• http://www.youtube.com/opscode

• Jesse Robbins, Changing Culture & Being a force for Awesome

• http://www.youtube.com/watch?v=OU8ihx3nT6I

• Matt Ray on Automating Continuous Deployment

• http://www.opscode.com/blog/2012/11/13/automating-continuous-deployment-wchef/

• Continuous Delivery by Jez Humble & David Farley

• http://continuousdelivery.com/


http://www.youtube.com/opscode

http://www.youtube.com/opscode

http://www.youtube.com/watch?v=OU8ihx3nT6I

http://www.youtube.com/watch?v=OU8ihx3nT6I

http://www.opscode.com/blog/2012/11/13/automating-continuous-deployment-wchef/




http://continuousdelivery.com/

http://continuousdelivery.com/

Thanks!

• George Miranda

• [email protected]

• @gmiranda23




Questions?

• On freenode: #chef and #chef-hacking

• http://lists.opscode.com

• http://tickets.opscode.com

• http://help.opscode.com

• @opscode and @opscode_status on Twitter


http://lists.opscode.com

http://lists.opscode.com

http://tickets.opscode.com

http://tickets.opscode.com

http://help.opscode.com

http://help.opscode.com

austin agile conf 2012 infrastructure automation-gmiranda

Technology

scope automation

unprecedented growth

system build approaches

folklore http

gmiranda23 github

gmiranda23 irc

puppet cloud

george miranda directory