managing distributed systems with chef
TRANSCRIPT
whoami
• Senior Technical Evangelist (Consultant) at Opscode
• @lnxchk
Chef
• Configuration management system built with the cloud in mind
• http://www.opscode.com
Chef is a Tool
http://www.flickr.com/photos/wessexandy/7690486884/sizes/c/in/pool-96164123@N00/
Quick Chef Bits
• Resources: things you manage (files, directories, services)
• Nodes: hosts you manage, run chef-client
• Recipes: collections of resources
• Templates: dynamically generate configuration
• Cookbooks: packages for recipes, usually a functional piece of software
• Chef Server: stores info, cookbooks, runs an API and a search engine
What is a Distributed System
“A distributed system is a collection of independent computers that appears to its users as a single coherent system”
~ Tanenbaum and van Steen, Distributed Systems, 2002
What Do You Distribute?
• Hits: with a load balancer
• Reads: with some slaves
• Workload: with some compute nodes
• Storage: with some storage nodes
Infrastructure Management
• Complex distributed systems require the proper tools to configure them to meet their goals
• Distributed systems are built by connecting bricks together in pleasing landscapes
http://www.flickr.com/photos/53486765@N02/
How Do You Distribute?
• Client-Server N:1
• Master-Slave-Client: 1:N:M
• Mesh or Cluster: N!
• omg science
Goals
• Create system topologies that are as complex as needed to meet the requirements of my architecture
• Allow configurations to dynamically update when nodes join or disappear
N:1 Client Server
• Basic examples
• Load balancer finding web nodes
• Application servers finding a datastore
example: Load Balancer
• Using Chef roles
• Roles are essentially used to create types of nodes
• I want a webserver, I create a webserver role that includes stuff like Apache, or nginx, or php, or whatever I need
• The webserver I build today using the role will be the same as the one I build next week or next month
• Roles are searchable!
Chef Roles
example: Load Balancer
example: App Servers and Data
• Maybe I don’t want my servers to dynamically go looking for an element
• I want to be able to tell them where to find it
• Use Chef attributes
Chef Attributes
example: App Servers and Data
example: App Servers and Data
Master:Slaves:Clients
• Services in complex topologies have more than one access pattern
• Combinations of Chef Roles and Attributes create more interesting relationships
Master Role
Slave Role
Client Role
What Was All That?
Clusters
• Complex topologies in which every component should know about all or most of the other components
Clusters Using Environments
• Chef Environments allow you to logically partition your infrastructure
• Canonical example is a Dev/Test/Stage/Prod model
• But!
• Create a logical group dedicated to your cluster
Chef Environments
Environments in Recipes
• Nodes belonging to the mob environment will have tony as their boss
• Nodes belonging to the herd environment will have cowboy as their boss
• The same software can be used to manage these two clusters, and their environments keep them together
Chef Databags
• “Bags of Holding”
• Whatever random kind of stuff you need to share, in JSON
• Not limited to cookbooks, roles, recipes: global data
• Usually you want them to be saved in your source repository
• Set of directories and files like data_bags/clusters/herd.json and data_bags/clusters/mob.json
Writing to Databags from Nodes
• Little bit dangerous
• Little bit racy
http://www.gcpvd.org/2011/09/27/riverzedge-industrial-ball-and-sander-drag-races-october-14/
Clusters Using Databags• Update the cluster’s databag in real time from the nodes
• Let’s say the herd cluster elected sheepdog as its boss instead of cowboy
• We could write a piece of node data and search, or we could abuse a databag
Write a Databag Item
• Write out a new value in the clusters databag, herd item
Read Databag Items• The rest of the herd will get the new value by reading it out of the
databag
• The convergence interval is as long as you wait between chef-client executions on the nodes
Things to Consider
• Your nodes converge by running the chef-client agent
• chef-client can be run on an interval, or on demand
Other Chefy Things
• Start, Report, and Error Handlers
• Lightweight Resources and Providers
• http://wiki.opscode.com
Thanks!
• http://www.opscode.com
• http://community.opscode.com
• @lnxchk