what the world needs more resilient systems · what is an example of sev 0? sev name: sev 0 runaway...
TRANSCRIPT
![Page 1: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/1.jpg)
Chaos Engineering: Why the world needs more resilient systems
@tammybutow
![Page 2: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/2.jpg)
Oh hai, nice to meet you! @tammybutow
@tammybutow
tammybutow
Principal SRE @ Gremlin
Tech Advisory Board @ Greenpeace
Enjoys Skateboarding, Snowboarding, Metal, Punk & Breaking Things On Purpose.
![Page 3: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/3.jpg)
Dropbox
DigitalOcean
National Australia Bank
Queensland University of Technology
Netflix
Amazon
Salesforce
Our Gremlin Team Were Previously @
PagerDuty Datadog
![Page 4: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/4.jpg)
More Resilient Systems!Why the world needs:
![Page 5: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/5.jpg)
A resilient system is a highly available and durable system. A resilient system can maintain an acceptable level of service
in the face of failure.
A resilient system can weather the storm (a misconfiguration, a large scale natural disaster or controlled chaos engineering).
What is a resilient system?
![Page 6: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/6.jpg)
Resilient Systems
Let’s review industry examples to understand why we need:
![Page 7: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/7.jpg)
Cardiac monitoring is now done via a bluetooth device implanted in the body and a mobile app.
The patient takes no action. Resilience of the device is the only thing the patient cares about.
Med Tech Industry:
![Page 8: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/8.jpg)
![Page 9: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/9.jpg)
People are changing jobs, moving homes, traveling and more. Systems need to not only
keep up but also provide value anytime/anywhere.
Fin Tech Industry:
![Page 10: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/10.jpg)
A “technical issue related to some routine maintenance”. Impacted the purchase of over 2000 homes.
![Page 11: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/11.jpg)
People are traveling so frequently for work and leisure. They need to be able to get where they
need to go with no hassles.
Transport Tech Industry:
![Page 12: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/12.jpg)
![Page 13: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/13.jpg)
More remote learning than ever before. Many students learn remotely. They need reliable access
to teachers, students and learning materials.
Edu Tech Industry:
![Page 14: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/14.jpg)
![Page 15: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/15.jpg)
People need protection from bushfires, tsunamis, earthquakes and storms. Many of the warning systems for these disasters are legacy
unreliable systems.
Enviro Tech Industry:
![Page 16: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/16.jpg)
Insert photo of tsunami
Saturday, 7 February 2009 - Australia’s all-time worst bushfire disaster
![Page 17: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/17.jpg)
Saturday, 7 February 2009 - Australia’s all-time worst bushfire disasters
![Page 18: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/18.jpg)
Saturday, 7 February 2009 - Australia’s all-time worst bushfire disasters
![Page 19: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/19.jpg)
What do these systems have in common?The primary concern of the user is resilience of the system,
in particular high availability.
![Page 20: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/20.jpg)
A great future for everyoneLet’s figure out how to create:
![Page 21: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/21.jpg)
What does a great future look like?
![Page 22: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/22.jpg)
More Resilient Systems?How do we create:
![Page 23: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/23.jpg)
Introducing: Chaos Engineering
![Page 24: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/24.jpg)
Chaos Engineering?What is
![Page 25: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/25.jpg)
Thoughtful, planned experiments designed to reveal the weakness in our systems.
Chaos Engineering:
![Page 26: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/26.jpg)
Inject something harmful, in order to build an immunity
![Page 27: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/27.jpg)
![Page 28: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/28.jpg)
We can inject harm in hosts, containers, pods, applications and
more.
![Page 29: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/29.jpg)
Chaos Engineer?What is a
![Page 30: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/30.jpg)
A vaccine research computer scientist.
Chaos Engineer:
SREs / Production Engineers commonly practice Chaos Engineering.
![Page 31: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/31.jpg)
A vaccine research computer scientist. Chaos Engineer:
![Page 32: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/32.jpg)
A vaccine research computer scientist. Chaos Engineer:
http://www.cancerresearchuk.org/about-cancer/cancer-in-general/treatment/immunotherapy/types/vaccines-to-treat-cancer
![Page 33: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/33.jpg)
The Bad Database Vaccine
Bad DB Vaccine
What happens when the database is unreachable?
Does the database have reliable and trustworthy monitoring?
Does the database fail gracefully?
![Page 34: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/34.jpg)
Injecting Harm in DynamoDB
https://www.gremlin.com/community/tutorials/gremlin-gameday-breaking-dynamodb/
![Page 35: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/35.jpg)
Chaos EngineeringWhat do you need before you can start doing:
![Page 36: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/36.jpg)
Prerequisites for Chaos Engineering
![Page 37: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/37.jpg)
1. High Severity Incident Management2. Monitoring 3. Measure the Impact of Downtime
Prerequisites for Chaos Engineering
![Page 38: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/38.jpg)
High Severity Incident ManagementChaos Engineering Prerequisite #1:
![Page 39: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/39.jpg)
The practice of recording, triaging, tracking, and assigning business value to problems that
impact critical systems.
High Severity Incident Management:
![Page 41: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/41.jpg)
SEVs?What are
![Page 42: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/42.jpg)
What are SEVs?The term SEV is derived from “High Severity Incident”
![Page 43: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/43.jpg)
What are SEVs?
![Page 44: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/44.jpg)
How Do You Determine SEV levels?
![Page 45: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/45.jpg)
What is an example of SEV 0?
SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!)
SEV Description: Nintendo Switch eShop is down and not working
SEV Start Time: 08:40am Dec 25 2017 (Christmas Day)
What is the availability impact? 100%
What is the outage duration? 5 hours and 40 minutes
![Page 46: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/46.jpg)
What is an example of SEV 0?
![Page 47: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/47.jpg)
The SEV Lifecycle?What is the
![Page 48: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/48.jpg)
![Page 50: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/50.jpg)
How do you identify your critical systems?
![Page 51: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/51.jpg)
What are your critical tier 0 systems?
TrafficDatabaseStorage
![Page 52: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/52.jpg)
MonitoringChaos Engineering Prerequisite #2:
![Page 53: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/53.jpg)
MonitoringWhy Do You Need:
![Page 54: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/54.jpg)
Why Monitor - The Google SRE Book
https://landing.google.com/sre/book/chapters/monitoring-distributed-systems.html
![Page 55: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/55.jpg)
How Should You Use Monitoring
![Page 57: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/57.jpg)
The Four Golden Signals - The Google SRE Book
https://landing.google.com/sre/book/chapters/monitoring-distributed-systems.html
![Page 58: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/58.jpg)
The Four Golden Signals - The Google SRE Book
https://landing.google.com/sre/book/chapters/monitoring-distributed-systems.html
Monitoring Signal
Description Example
Latency The time it takes to service a request. HTTP 500 error triggered due to loss of connection to a database
Traffic A measure of how much demand is being placed on your system
For a web service, this measurement is usually HTTP requests per second
Errors The rate of requests that fail, either explicitly, implicitly or by policy.
Catching HTTP 500s at your load balancer can do a decent job of catching all completely failed requests.
Saturation How "full" your service is. Should also signal impending saturation.
It looks like your database will fill its hard drive in 4 hours.
![Page 59: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/59.jpg)
What Happens If You Do Chaos Engineering Without Monitoring?
![Page 60: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/60.jpg)
You won’t know what’s happening
![Page 61: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/61.jpg)
Measure The Impact Of DowntimeChaos Engineering Prerequisite #3:
![Page 62: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/62.jpg)
Measure The Impact Of DowntimeWe need to understand how SEV 0s impact our customers and business.
![Page 63: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/63.jpg)
Measure The Impact Of Downtime
System Impact: • Availability• Durability
Customer/Business Impact: • Outcome• Cost• Time
![Page 64: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/64.jpg)
What is the impact of the Nintendo Switch eShop SEV 0?
SEV Description: Nintendo Switch eShop is down and not working
What is the availability impact? 100%
Time? 5 hours and 40 minutes
Cost? ______
Outcome? Switch users all over the world can’t buy games
![Page 65: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/65.jpg)
Chaos EngineeringNow we’re ready to get started with:
![Page 66: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/66.jpg)
Chaos Engineering Use Case: Twilio
![Page 67: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/67.jpg)
Chaos Engineering Case Study: Twilio
Ratequeue Chaos has 3 goals: 1. Pick a shard 2. Kill primary 3. Monitor recovery.
![Page 68: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/68.jpg)
Share TheChaos Engineering
Journey Widely
![Page 69: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/69.jpg)
• Do a Chaos Engineering Kick Off @ All Hands• Send email updates & progress reports • Run Monthly Metrics Reviews• Deliver Presentations
Share The Chaos Engineering Journey Widely
![Page 70: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/70.jpg)
Don’t Surprise Everyone!
![Page 71: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/71.jpg)
Gremlin?What is
![Page 72: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/72.jpg)
What is Gremlin?
![Page 73: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/73.jpg)
Gremlin Chaos Engineering AttacksThere are a range of attacks built-in and ready to run on Linux.
Type of Attack Attack Gremlin Support (March 2018)
Resource CPU ✅
Resource Disk ✅
Resource IO ✅
Resource Memory ✅
State Process Killer ✅
State Shutdown ✅
State Time Travel ✅
Network Blackhole ✅
Network DNS ✅
Network Latency ✅
Network Packet Loss ✅
![Page 74: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/74.jpg)
Live Chaos EngineeringDemo
![Page 76: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/76.jpg)
Create a Kubernetes Cluster
Master
Node 1 Node 2 Node 3
159.65.85.204
159.65.85.158 159.65.85.169 159.65.85.202
![Page 80: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/80.jpg)
View Your Kubernetes Pods
![Page 82: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/82.jpg)
Monitor Your Chaos Engineering Attack
![Page 83: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/83.jpg)
Monitor Your Chaos Engineering Attack
![Page 85: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/85.jpg)
The Path To Chaos EngineeringLet’s Review:
![Page 86: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/86.jpg)
The Path To Chaos Engineering
High Severity Incident Management
Monitoring
Make & Measure Improvements
Chaos Engineering
Measure the impact of downtime
![Page 87: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/87.jpg)
Blast Radius and Advanced Chaos
High Severity Incident Management
Monitoring
Make & Measure Improvements
Chaos Engineering
Measure the impact of downtime
![Page 88: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/88.jpg)
Make Improvements?How do you
![Page 89: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/89.jpg)
1. Build - Build a new system / improve existing 2. Borrow - Use open source / contribute to OS3. Buy - Use 3rd party systems4. Brush up - GameDays / Team training5. Break - Chaos Engineering / Failure injection6. Begone - Decommission systems / delete code
How do you make improvements?
![Page 90: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/90.jpg)
Always Measure ImprovementsTell a story of before and after with metrics
![Page 91: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/91.jpg)
More Resilient SystemsThe world needs:
![Page 92: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/92.jpg)
More Resilient Systems!You can create:
![Page 93: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/93.jpg)
Join us on this journey!gremlin.com/community
gremlin.com/slack
![Page 94: What the world needs more resilient systems · What is an example of SEV 0? SEV Name: SEV 0 Runaway Cow (auto generated code names help your team remember and refer to SEVs!) SEV](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec61f29cf1ca866e415653b/html5/thumbnails/94.jpg)
Thanks!@tammybutow gremlin.com