managing distributed systems with chef

Managing Distributed Systems with Chef

Mandi Wallsmandi@opscode.com

RiCon 2012October 10, 2012

whoami

• Senior Technical Evangelist (Consultant) at Opscode

• @lnxchk

• mandi@opscode.com

Chef

• Configuration management system built with the cloud in mind

• http://www.opscode.com

Chef is a Tool

http://www.flickr.com/photos/wessexandy/7690486884/sizes/c/in/pool-96164123@N00/

Quick Chef Bits

• Resources: things you manage (files, directories, services)

• Nodes: hosts you manage, run chef-client

• Recipes: collections of resources

• Templates: dynamically generate configuration

• Cookbooks: packages for recipes, usually a functional piece of software

• Chef Server: stores info, cookbooks, runs an API and a search engine

What is a Distributed System

“A distributed system is a collection of independent computers that appears to its users as a single coherent system”

~ Tanenbaum and van Steen, Distributed Systems, 2002

What Do You Distribute?

• Hits: with a load balancer

• Reads: with some slaves

• Workload: with some compute nodes

• Storage: with some storage nodes

Infrastructure Management

• Complex distributed systems require the proper tools to configure them to meet their goals

• Distributed systems are built by connecting bricks together in pleasing landscapes

http://www.flickr.com/photos/53486765@N02/

How Do You Distribute?

• Client-Server N:1

• Master-Slave-Client: 1:N:M

• Mesh or Cluster: N!

• omg science

Goals

• Create system topologies that are as complex as needed to meet the requirements of my architecture

• Allow configurations to dynamically update when nodes join or disappear

N:1 Client Server

• Basic examples

• Load balancer finding web nodes

• Application servers finding a datastore

example: Load Balancer

• Using Chef roles

• Roles are essentially used to create types of nodes

• I want a webserver, I create a webserver role that includes stuff like Apache, or nginx, or php, or whatever I need

• The webserver I build today using the role will be the same as the one I build next week or next month

• Roles are searchable!

Chef Roles

example: Load Balancer

example: App Servers and Data

• Maybe I don’t want my servers to dynamically go looking for an element

• I want to be able to tell them where to find it

• Use Chef attributes

Chef Attributes

example: App Servers and Data

example: App Servers and Data

Master:Slaves:Clients

• Services in complex topologies have more than one access pattern

• Combinations of Chef Roles and Attributes create more interesting relationships

Master Role

Slave Role

Client Role

What Was All That?

Clusters

• Complex topologies in which every component should know about all or most of the other components

Clusters Using Environments

• Chef Environments allow you to logically partition your infrastructure

• Canonical example is a Dev/Test/Stage/Prod model

• But!

• Create a logical group dedicated to your cluster

Chef Environments

Environments in Recipes

• Nodes belonging to the mob environment will have tony as their boss

• Nodes belonging to the herd environment will have cowboy as their boss

• The same software can be used to manage these two clusters, and their environments keep them together

Chef Databags

• “Bags of Holding”

• Whatever random kind of stuff you need to share, in JSON

• Not limited to cookbooks, roles, recipes: global data

• Usually you want them to be saved in your source repository

• Set of directories and files like data_bags/clusters/herd.json and data_bags/clusters/mob.json

Writing to Databags from Nodes

• Little bit dangerous

• Little bit racy

http://www.gcpvd.org/2011/09/27/riverzedge-industrial-ball-and-sander-drag-races-october-14/

Clusters Using Databags• Update the cluster’s databag in real time from the nodes

• Let’s say the herd cluster elected sheepdog as its boss instead of cowboy

• We could write a piece of node data and search, or we could abuse a databag

Write a Databag Item

• Write out a new value in the clusters databag, herd item

Read Databag Items• The rest of the herd will get the new value by reading it out of the

databag

• The convergence interval is as long as you wait between chef-client executions on the nodes

Things to Consider

• Your nodes converge by running the chef-client agent

• chef-client can be run on an interval, or on demand

Other Chefy Things

• Start, Report, and Error Handlers

• Lightweight Resources and Providers

• http://wiki.opscode.com

Thanks!

• http://www.opscode.com

• http://community.opscode.com

• @lnxchk

• mandi@opscode.com