living system or build factory - chris maxwell
TRANSCRIPT
Living System or
Build Factory?!
Chris Maxwell !
@WrathOfChris
Cloud guy @ Ubiquiti Networks
Who is this talk for?
• We build systems
• We turn design into implementation
• We operate systems forever**not forever
We keep systems from failing
Holding up the system?
We do it wrong
Not helping
What is a Living System?
The business has a need
Instructions are unclear
What about THIS space?
Instructions are not followed
...except a VCR, CD’s, and a remote?
Production loads change unexpectedly
Additional capacity added
Wrong tools at hand
12ga crimp on 24ga wire doesn’t work
Patches are put into production
Patches become production
Production is unexpected
Unexpected Scotsman
Parts not available
No ventilation ducting available
Inconsistent Origins
One of these things
is not like the other
http://www.flickr.com/photos/msittig/4801606785/
• Has a long life
• Mutates over time
• Each system slightly different
• The system is the state
Living System
http://www.flickr.com/photos/anijdam/2362660336/
Problems!
Can you rebuild it … fast?
… accurately?http://www.flickr.com/photos/adactio/2856955322/
Problems!
Do you keep: • HA failover? • Standby host?
Problems!
Have you tested? !
When both fail?
What is a Build Factory?
For business…:
Run the linehttp://www.flickr.com/photos/pasukaru76/6893926948/
For devs…:
make clean && make
For ops…:
Kill it with firehttp://www.flickr.com/photos/comedynose/3549464622/
Looking back on Build Factory
Copying systems
Cloning systems
Early build factories took time
...hours on 10baseT
...tumbleweeds...
It was a manual process
How to restore a VM image...
...in 7 easy pages of screenshots
Sorry, couldn’t release these pics :( It was just lots of screenshots
+ do this, then that instructions
It was a manual process
...and still manual work afterhttp://spalinux.com/wp-content/uploads/2008/10/
windows_join_domain.png
Scripting is a start
I will replace you with a short script
Makefiles track dependencies
or a Makefile
it builds your kernel, and probably your OS
Today’s tools
Not a complete list
Describe
Write down all of the steps
http://www.flickr.com/photos/spree2010/4960430323/
Automate
“too complex for any one man’s memory”http://www.flickr.com/photos/sepblog/3679949333/
Replicate
Build 3 the same, tie them together
Repeat
Training makes elements replaceable
Build Factory
• Has a short life
• Replaced for each change
• Each system the same
• The procedure is the state
http://upload.wikimedia.org/wikipedia/commons/5/50/Krispy_Kreme_Doughnuts.jpg
When something jams !
deep in the system
a scrap of paper caught in the drive gear
Replace It!•fix it later •send it for
repair •get production
working!
When production patches take over
Production maintenance changes the system
Rebuild it!
Sometimes you get to rebuild them
In the cloud?
!
live replace might cost up to 1 hour
of billing
In the datacentre?
!
live replace can be done with
spare capacity
Takeaway
• Describe it
• Automate it
• Produce it
• Repeat it
Questions?!
@WrathOfChris !
Like the Loch Ness Monster,Answers are over there...
• Forgotten attributions: • ghost-1:http://www.all-internet-security.com/images/pc-copy.jpg
• ghost-2:http://www2.expta.com/uploaded_images/Ghost-741732.png
• ghost-3:http://internship.wesmilliron.com/wp-content/uploads/2012/08/vmware-norton-ghost.jpg
• laplink: http://thumbs1.ebaystatic.com/d/l225/m/mkhsfLBMjkjjMw9skfQUNag.jpg
• lap link:http://www.old-games.com/screenshot/5736-1-lap-link-3-for-dos.jpg