configuration management 101 - a tale of disaster recovery using cfengine 3
TRANSCRIPT
![Page 1: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/1.jpg)
A tale of disaster recoveryCFEngine everyday, practices and tools
Nicolas Charles <[email protected]>Jonathan Clarke <[email protected]>
RMLL 2011 @Strasbourg, France
![Page 2: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/2.jpg)
About the speakers
Nicolas Charles
CFEngine contributor
CFEngine ”Community Champion” (C3)
Jonathan Clarke
CFEngine contributor
Contributor to various LDAP FLOSS projects
But we get on pretty well!(mostly...)
Scala Developer Sysadmin
![Page 3: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/3.jpg)
1) Configuration Management 101
2) A tale of disaster recovery
3) Our choice of tool
4) About CFEngine 3
Agenda
![Page 4: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/4.jpg)
A bit aboutConfiguration Management...
![Page 5: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/5.jpg)
Configuration management What is it?
Configuration Management is a field of management that focuses on establishing and maintaining consistency of a system (..) throughout its life
Software configuration management is the task of tracking and controlling changes in the software
Sources:http://en.wikipedia.org/wiki/Configuration_managementhttp://en.wikipedia.org/wiki/Software_configuration_management
![Page 6: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/6.jpg)
A server crashed.
Install a new one, peoplecan't work without it!
OK, it'll be done inabout two days...
There's a new critical security patchwe must deploy on all our servers!
Get it out quickly!
Right, I'll put the wholeteam on it.
Why configuration management?
![Page 7: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/7.jpg)
Why configuration management?
Automation
IndustrializationReproducibility
![Page 8: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/8.jpg)
Why configuration management?
How do we setupservice X?
Ask Jim, he'sthe expert on that.
But he left the company...
Huh, this server has been loggingerrors for a few weeks.
Oh? I think Michael changedsomething on it recently...He'll tell you what it was.
Damn, he's on vacation!
![Page 9: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/9.jpg)
Why configuration management?
Building-upknowledge
HistoryDocumentation
![Page 10: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/10.jpg)
Why configuration management?
An intruder just stole our datausing a vulnerability in amodule we don't need...
I thought the project specificationensured that we disabled that?
Er, it did, but we enabled it tosolve a problem and forgot todisable it afterwards... sorry...
![Page 11: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/11.jpg)
Why configuration management?
Vigilance
AlertsAutomatic repairs
![Page 12: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/12.jpg)
Why configuration management?
I don't understand how thisserver is setup. It doesn't matchour best-practices.
Oh, that's a legacy server...
Well, it's a collection of littlethings, here and there...
Give me details on ourcurrent security policy.
Ah... Well, OK.Tell me: is it fully appliedon all our critical servers?
Er...
![Page 13: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/13.jpg)
Why configuration management?
Rationalization
ControlNormalization
![Page 14: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/14.jpg)
Configuration management
Rationalization
ControlNormalization
Vigilance
AlertsAutomatic repairs
Building-upknowledge
HistoryDocumentation
Automation
IndustrializationReproducibility
![Page 15: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/15.jpg)
An ill-fated talefrom the recent past
Disaster Recovery
(CASE STUDY)
![Page 16: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/16.jpg)
Before the disaster... Our company's IT infrastructure
Small company: small requirements Web site, email Git repository, Redmine...
Small company: small budget All on one hosted server
![Page 17: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/17.jpg)
Asking for trouble? Just one hosted server! Critical services!
No, a ”safe” configuration: Redundant hardware, 3 disk RAID-5 array All services automatically installed and setup
using Configuration Management Backups: daily (several off-site locations) Several VMs to separate services
![Page 18: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/18.jpg)
A critical failure 2 hard drives fail simultaneously
→ RAID-5 array is down
→ Almost all services fail immediately
→ ”The end of the world as we know it”
→ Need to rebuild everything NOW
![Page 19: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/19.jpg)
Recovering Step 1: Panic! Step 2: Get a new server Step 3: Reinstall base OS + virtualization Step 4: Restore VM configuration Step 4: Re-create the VMs manually Step 5: Reinstall each OS in each VM...
whoops
![Page 20: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/20.jpg)
Recovering Step 6: Installation Configuration Management Step 7: Sit back and watch all the services
coming back online as if by magic! Step 8: Huh, where's my data? Step 9: Manually restore backups Step 10: Make a list of missing data...
![Page 21: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/21.jpg)
Lessons learned
1) Hard disks fail reliably
2) Restoring virtualization setups:● Backing up the config files would have helped● Need CM tools to describe the desired state!
(Cfengine Nova does this)
3) Configuration Management should tie in to our backup system
4) Backups were lacking some files: always test!
![Page 22: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/22.jpg)
Wishlist and discussion Integrating Configuration Management tools
and backup systems is a crucial step for CM to be efficient for disaster recovery
What do others do?
Provisioning VMs and their resources (disks, network) should be automated too
Cloud providers are one solution What about ”plain” virtualization?
![Page 23: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/23.jpg)
What we chose, and why
Configuration ManagementTools
![Page 24: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/24.jpg)
Our choice Back in mid 2009 Needed a configuration management tool Criteria:
Open source Multi-platform agent (including Windows) Resilient Non-disruptive
![Page 25: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/25.jpg)
Our choice: candidates
CFEngine 3 Puppet Chef
![Page 26: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/26.jpg)
Our choice: candidates
CFEngine 3
More on thischoice later...
![Page 27: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/27.jpg)
A bit about CFEngine 3...
Sources: across the Internet
![Page 28: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/28.jpg)
CFEngine: History
Sou
rce:
http
://ve
rtic
alsy
sadm
in.c
om/b
log/
unca
tego
rized
/rel
ativ
e-or
igin
s-o
f-cf
engi
ne-c
hef-
and-
pupp
et
![Page 29: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/29.jpg)
CFEngine 3: Intro Configuration management software Written in C Two versions :
Community (GPL v3) Nova (closed source)
Community + extra features Some features released in Community
Backed by CFEngine AS – Norway based company founded in 2009
![Page 30: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/30.jpg)
CFEngine 3: Features
Large user base and community
Multi-agent technologyLightweight, non-intrusive
AutonomousFault-tolerant
Multi platform
Adapted toheterogeneousenvironments
Highly scalableProgressive
roll-out
![Page 31: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/31.jpg)
CFEngine 3: Components Cf-agent
Runs on all managed hosts Applies configuration – this is the heart Can connect to cf-serverd to get policies / files
Cf-serverd Distributes policies and files Must be run on policy server(s) Usually run on all hosts to enable remote runs
Cf-monitord Collects statistics on all nodes
![Page 32: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/32.jpg)
Memory usage Daemon consumption on managed hosts
![Page 33: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/33.jpg)
CFEngine 3: Usage examples Large companies Critical systems: Joint Australia Tsunami
Warning Centre Personal computers Mobile devices: Nokia N900 Underwater devices: army submarines Small and medium companies... Community
![Page 34: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/34.jpg)
Feature: Multi-platform Define a configuration for all operating systems
Windows, Linux Make it ”transparent” (forget about the
complexity) Existing standard library handling the
differences between each OS and distribution
![Page 35: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/35.jpg)
CFEngine 3: Promises Configuration rules are called promises
”Promise” to be in the desired state Cfengine agent handles the steps to get there:
convergence
Promise theory is based on research done in the University of Oslo
![Page 36: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/36.jpg)
Feature: File editing Only change what you need to
You like your distribution's defaults? You have various different systems already
setup and just need to change something?
Search for lines and replace/delete/add them Only change one field in a file
/etc/passwd for example
![Page 37: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/37.jpg)
Feature: Complex tasks Powerful class system to trigger promises
Based on nodes itself Based on time Based on whatever you might imagine
Complex workflow can be created
![Page 38: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/38.jpg)
Configuration example Install the LAMP stack
bundle agent caller { vars:
"pkg_list" slist => { "httpd", "php5", "mysql" };
packages: "${pkg_list}" package_method => generic, package_method => "addupdate";}
![Page 39: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/39.jpg)
RMLL 2011
Thank you !
![Page 40: Configuration management 101 - A tale of disaster recovery using CFEngine 3](https://reader033.vdocument.in/reader033/viewer/2022060109/55582395d8b42a5e468b50dc/html5/thumbnails/40.jpg)
CFEngine 3: Features
According to Kuleven comparative study of configuration management systems:
Very mature Cross platform (*BSD, AIX, HP-UX, Linux, Mac
OS X, Solaris, Windows) Strongly distributed Based on state description and convergence Very high scalabily ( > 10000 nodes ) Very small footprint
Source: http://distrinet.cs.kuleuven.be/software/sysconfigtools/overview