mistakes were made
DESCRIPTION
TRANSCRIPT
![Page 1: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/1.jpg)
Some Conference
OSCO
N 2011
Mistakes were madeSelena Deckelmann
@selenamarie
![Page 2: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/2.jpg)
Some Conference
OSCO
N 2011
This goes out to all the sysadmins.
![Page 3: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/3.jpg)
Some Conference
OSCO
N 2011
You can never think about failure too much.
![Page 4: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/4.jpg)
Some Conference
OSCO
N 2011
![Page 5: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/5.jpg)
Some Conference
OSCO
N 2011
Some goals around pessimism.
![Page 6: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/6.jpg)
Some Conference
OSCO
N 2011
Plan for the worst.
![Page 7: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/7.jpg)
Some Conference
OSCO
N 2011
Minimize risk.
![Page 8: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/8.jpg)
Some Conference
OSCO
N 2011
Recover, gracefully.
![Page 9: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/9.jpg)
Some Conference
OSCO
N 2011
Plan for the worst.Minimize risk.
Recover, gracefully.
![Page 10: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/10.jpg)
Some Conference
OSCO
N 2011
![Page 11: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/11.jpg)
Some Conference
OSCO
N 2011
![Page 12: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/12.jpg)
Some Conference
OSCO
N 2011
Tales of failure to...
Document
Test
Verify
Imagine
Implement
![Page 13: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/13.jpg)
Some Conference
OSCO
N 2011
Failure to document.
![Page 14: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/14.jpg)
Some Conference
OSCO
N 2011
Moving Day
Thanks, David Prior!
![Page 15: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/15.jpg)
Some Conference
OSCO
N 2011
Prevent documentation failures.
• Write documentation.
• Update documentation.
• Make documenting a step in your written process.
• Assign a fixed amount of time to that step.
![Page 16: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/16.jpg)
Some Conference
OSCO
N 2011
Documentation tools
• Graphic designers. (Pretty wikis. Pretty docs. (Sphinx?) Diagrams.)
• Timelines.
• Bug tracking.
• Ordered todo lists.
![Page 17: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/17.jpg)
Some Conference
OSCO
N 2011
Failure to test.
![Page 18: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/18.jpg)
Some Conference
OSCO
N 2011
“My first day posing as a sysadmin (~1990, no previous training....) I deleted all zero length files on a Sun workstation.”
![Page 19: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/19.jpg)
Some Conference
OSCO
N 2011
Prevent testing failures.
• Verify success criteria.
• Write tests.
• Test with a buddy.
• Have a plan.
![Page 20: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/20.jpg)
Some Conference
OSCO
N 2011
Testing tools
• Your favorite test framework
• Repeatable shell scripts
• Staging environments
![Page 21: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/21.jpg)
Some Conference
OSCO
N 2011
Failure to verify.
![Page 22: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/22.jpg)
Some Conference
OSCO
N 2011
“What does ‘-d’ actually do?”
![Page 23: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/23.jpg)
Some Conference
OSCO
N 2011
Prevent verification failures.
• Have a plan for things going wrong.
• Have a staging environment.
• Test your rollback plan, not just your implementation plan.
![Page 24: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/24.jpg)
Some Conference
OSCO
N 2011
Verification tools
• Staging environments
• Your buddy
![Page 25: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/25.jpg)
Some Conference
OSCO
N 2011
Failure to imagine.
![Page 26: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/26.jpg)
Some Conference
OSCO
N 2011
For my group the bottom line was
"don't trust anyone".
Thanks, Maggie!
![Page 27: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/27.jpg)
Some Conference
OSCO
N 2011
Recover from failures to imagine.
• Share your stories of failure.
• Talk with people who are different from you.
• Act out implementation scenarios.
![Page 28: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/28.jpg)
Some Conference
OSCO
N 2011
Failure to implement.
![Page 29: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/29.jpg)
Some Conference
OSCO
N 2011
Re-implement.
• Learn from mistakes.
![Page 30: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/30.jpg)
Some Conference
OSCO
N 2011
Reflection.(or, the Post-Mortem)
![Page 31: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/31.jpg)
Some Conference
OSCO
N 2011
Before
• Document the plan with numbered steps and a timeline.
• Test the plan and the rollback plan.
• Identify a “point of no return”.
![Page 32: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/32.jpg)
Some Conference
OSCO
N 2011
During
• Screen sharing: UNIX screen, VNC, etc.
• Chatroom: AIM, Campfire (scrollback!)
• Voice: Campfire, Skype, VOIP, POTS call line
• Headsets!
• Designated time-keeper.
![Page 33: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/33.jpg)
Some Conference
OSCO
N 2011
After
• Documentation updates
• Post-mortems to identify areas of success and areas for improvement.
• Limit improvements to 1-2 things.
![Page 34: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/34.jpg)
Some Conference
OSCO
N 2011
Plan for the worst.Minimize risk.
Recover, gracefully.
![Page 35: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/35.jpg)
Some Conference
OSCO
N 2011
Thanks!
![Page 36: Mistakes were made](https://reader034.vdocument.in/reader034/viewer/2022051109/54843dfbb4af9f690d8b4b22/html5/thumbnails/36.jpg)
Some Conference
OSCO
N 2011
Photo credits
• Flickr: sheepguardingllama