![Page 1: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/1.jpg)
Living System or
Build Factory?!
Chris Maxwell !
@WrathOfChris
Cloud guy @ Ubiquiti Networks
![Page 2: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/2.jpg)
Who is this talk for?
• We build systems
• We turn design into implementation
• We operate systems forever**not forever
![Page 3: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/3.jpg)
We keep systems from failing
![Page 4: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/4.jpg)
Holding up the system?
![Page 5: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/5.jpg)
We do it wrong
![Page 6: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/6.jpg)
Not helping
![Page 7: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/7.jpg)
What is a Living System?
![Page 8: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/8.jpg)
The business has a need
![Page 9: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/9.jpg)
Instructions are unclear
![Page 10: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/10.jpg)
What about THIS space?
![Page 11: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/11.jpg)
Instructions are not followed
![Page 12: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/12.jpg)
...except a VCR, CD’s, and a remote?
![Page 13: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/13.jpg)
Production loads change unexpectedly
![Page 14: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/14.jpg)
Additional capacity added
![Page 15: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/15.jpg)
Wrong tools at hand
![Page 16: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/16.jpg)
12ga crimp on 24ga wire doesn’t work
![Page 17: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/17.jpg)
Patches are put into production
![Page 18: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/18.jpg)
Patches become production
![Page 19: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/19.jpg)
Production is unexpected
![Page 20: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/20.jpg)
Unexpected Scotsman
![Page 21: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/21.jpg)
Parts not available
![Page 22: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/22.jpg)
No ventilation ducting available
![Page 23: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/23.jpg)
Inconsistent Origins
![Page 24: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/24.jpg)
One of these things
is not like the other
http://www.flickr.com/photos/msittig/4801606785/
![Page 25: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/25.jpg)
• Has a long life
• Mutates over time
• Each system slightly different
• The system is the state
Living System
http://www.flickr.com/photos/anijdam/2362660336/
![Page 26: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/26.jpg)
Problems!
Can you rebuild it … fast?
… accurately?http://www.flickr.com/photos/adactio/2856955322/
![Page 27: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/27.jpg)
Problems!
Do you keep: • HA failover? • Standby host?
![Page 28: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/28.jpg)
Problems!
Have you tested? !
When both fail?
![Page 29: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/29.jpg)
What is a Build Factory?
![Page 30: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/30.jpg)
For business…:
![Page 31: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/31.jpg)
Run the linehttp://www.flickr.com/photos/pasukaru76/6893926948/
![Page 32: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/32.jpg)
For devs…:
![Page 33: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/33.jpg)
make clean && make
![Page 34: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/34.jpg)
For ops…:
![Page 35: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/35.jpg)
Kill it with firehttp://www.flickr.com/photos/comedynose/3549464622/
![Page 36: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/36.jpg)
Looking back on Build Factory
![Page 37: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/37.jpg)
Copying systems
![Page 38: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/38.jpg)
![Page 39: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/39.jpg)
Cloning systems
![Page 40: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/40.jpg)
Early build factories took time
![Page 41: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/41.jpg)
...hours on 10baseT
![Page 42: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/42.jpg)
...tumbleweeds...
![Page 43: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/43.jpg)
It was a manual process
![Page 44: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/44.jpg)
How to restore a VM image...
...in 7 easy pages of screenshots
Sorry, couldn’t release these pics :( It was just lots of screenshots
+ do this, then that instructions
![Page 45: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/45.jpg)
It was a manual process
![Page 46: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/46.jpg)
...and still manual work afterhttp://spalinux.com/wp-content/uploads/2008/10/
windows_join_domain.png
![Page 47: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/47.jpg)
Scripting is a start
![Page 48: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/48.jpg)
I will replace you with a short script
![Page 49: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/49.jpg)
Makefiles track dependencies
![Page 50: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/50.jpg)
or a Makefile
it builds your kernel, and probably your OS
![Page 51: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/51.jpg)
Today’s tools
![Page 52: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/52.jpg)
Not a complete list
![Page 53: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/53.jpg)
Describe
![Page 54: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/54.jpg)
Write down all of the steps
http://www.flickr.com/photos/spree2010/4960430323/
![Page 55: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/55.jpg)
Automate
![Page 56: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/56.jpg)
“too complex for any one man’s memory”http://www.flickr.com/photos/sepblog/3679949333/
![Page 57: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/57.jpg)
Replicate
![Page 58: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/58.jpg)
Build 3 the same, tie them together
![Page 59: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/59.jpg)
Repeat
![Page 60: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/60.jpg)
Training makes elements replaceable
![Page 61: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/61.jpg)
Build Factory
• Has a short life
• Replaced for each change
• Each system the same
• The procedure is the state
http://upload.wikimedia.org/wikipedia/commons/5/50/Krispy_Kreme_Doughnuts.jpg
![Page 62: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/62.jpg)
When something jams !
deep in the system
![Page 63: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/63.jpg)
a scrap of paper caught in the drive gear
![Page 64: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/64.jpg)
Replace It!•fix it later •send it for
repair •get production
working!
![Page 65: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/65.jpg)
When production patches take over
![Page 66: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/66.jpg)
Production maintenance changes the system
![Page 67: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/67.jpg)
Rebuild it!
![Page 68: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/68.jpg)
Sometimes you get to rebuild them
![Page 69: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/69.jpg)
In the cloud?
!
live replace might cost up to 1 hour
of billing
![Page 70: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/70.jpg)
In the datacentre?
!
live replace can be done with
spare capacity
![Page 71: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/71.jpg)
Takeaway
• Describe it
• Automate it
• Produce it
• Repeat it
![Page 72: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/72.jpg)
Questions?!
@WrathOfChris !
Like the Loch Ness Monster,Answers are over there...
![Page 73: Living system or build factory - Chris Maxwell](https://reader035.vdocument.in/reader035/viewer/2022062404/554bcbedb4c905ac708b45c5/html5/thumbnails/73.jpg)
• Forgotten attributions: • ghost-1:http://www.all-internet-security.com/images/pc-copy.jpg
• ghost-2:http://www2.expta.com/uploaded_images/Ghost-741732.png
• ghost-3:http://internship.wesmilliron.com/wp-content/uploads/2012/08/vmware-norton-ghost.jpg
• laplink: http://thumbs1.ebaystatic.com/d/l225/m/mkhsfLBMjkjjMw9skfQUNag.jpg
• lap link:http://www.old-games.com/screenshot/5736-1-lap-link-3-for-dos.jpg