the journey of chaos engineering begins with a single step
TRANSCRIPT
#PDSummit16#PDSummit16
The Journey of Chaos EngineeringBegins with a Single Step
#PDSummit16#PDSummit16
Bruce WongSenior Engineering Manager
Twilio@bruce_m_wong
https://www.linkedin.com/in/brucemwong
#PDSummit16#PDSummit16
#PDSummit16#PDSummit16
2009
2012
2014http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html https://github.com/Netflix/SimianArmy http://techblog.netflix.com/2015/09/chaos-engineering-upgraded.html
#PDSummit16http://readwrite.com/2014/09/17/netflix-chaos-engineering-for-everyone/ http://techblog.netflix.com/2014/09/introducing-chaos-engineering.html
#PDSummit16#PDSummit16
The journey of a thousand miles begins with a single
step.
-Lao Tzu
#PDSummit16#PDSummit16
James BurnsTech Lead
Twilio@1mentat
#PDSummit16 https://www.linkedin.com/in/james-burns-7816a82
#PDSummit16#PDSummit16
PreparationPre-Launch Log Aggregation System -Stage env -Synthetic Traffic
#PDSummit16
The Master of Disaster•Network Issues
•Partitions
•Thundering Herds
•Cascading Failures
•Resource Starvation
•CPU
•Memory
•Disk IO
•Network IO
•Application Load
> sudo halt
#PDSummit16
Incident Start
#PDSummit16
Impact?
#PDSummit16
Post-Mortem
#PDSummit16
#PDSummit16#PDSummit16
#PDSummit16
Round 2•Network Issues
•Partitions
•Thundering Herds
•Cascading Failures
•Resource Starvation
•CPU
•Memory
•Disk IO
•Network IO
•Application Load
> sudo halt
#PDSummit16
Third-Party API Failure
#PDSummit16
#PDSummit16
Well, that’s not what I expected to see
#PDSummit16
Outcomes
Instrument
Instrument
Instrument
API SLAs
Architectural Change!
#PDSummit16
Recap
• Start Simple • Instrumentation
Gaps • Understand your
dashboards • Prevent outages
#PDSummit16http://www.crisistextline.org/ http://polarisproject.org/befree-textline http://trekmedics.org/ https://www.twilio.org/
#PDSummit16
When you wish upon a blue moon…
#PDSummit16#PDSummit16
Please provide feedback for this
session by filling out the feedback survey