Download - Protect your Users with Circuit breakers
![Page 1: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/1.jpg)
Protect your users with Circuit Breakers
@scott_triglia
Jim’s
![Page 2: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/2.jpg)
Yelp’s Mission:Connecting people with great
local businesses.
![Page 3: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/3.jpg)
@scott_triglia
I work with 💵💶
![Page 4: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/4.jpg)
Let’s talk Circuit Breakers
![Page 5: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/5.jpg)
![Page 6: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/6.jpg)
![Page 8: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/8.jpg)
![Page 9: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/9.jpg)
![Page 10: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/10.jpg)
Our goals today: introduce a basic circuit
breaker
![Page 11: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/11.jpg)
Our goals today: a modular circuit breaker
![Page 12: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/12.jpg)
Our goals today: test it out on several
scenarios
![Page 13: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/13.jpg)
![Page 14: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/14.jpg)
![Page 15: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/15.jpg)
15
12
34
![Page 16: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/16.jpg)
the fundamental rule: your systems will fail
what’s your response?
![Page 17: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/17.jpg)
12
![Page 18: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/18.jpg)
12
3
![Page 19: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/19.jpg)
12
34
![Page 20: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/20.jpg)
Nygard’s circuit breaker
![Page 21: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/21.jpg)
![Page 22: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/22.jpg)
![Page 23: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/23.jpg)
![Page 24: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/24.jpg)
![Page 25: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/25.jpg)
Circuit Breaker States: * Healthy (or “closed”) * Recovering (or “half-open”) * Unhealthy (or “open”)
![Page 26: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/26.jpg)
![Page 27: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/27.jpg)
Recovery:
* Wait for recovery_timeout seconds* Send a trial request, trust its results
![Page 28: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/28.jpg)
Before a circuit breaker
![Page 29: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/29.jpg)
Assume the kitchen gets slow
![Page 30: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/30.jpg)
Kitchen’s backlog grows
![Page 31: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/31.jpg)
Diners wait much longer to get food
![Page 32: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/32.jpg)
New diners make issues worse
![Page 33: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/33.jpg)
And your entire system collapses
![Page 34: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/34.jpg)
And your entire system collapses
![Page 35: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/35.jpg)
With a circuit breaker
![Page 36: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/36.jpg)
CB sees backlog, stops orders
![Page 37: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/37.jpg)
Fewer frustrated diners
![Page 38: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/38.jpg)
Reduced load on the kitchen
![Page 39: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/39.jpg)
A well defined failure mode
![Page 40: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/40.jpg)
How can we do better?
![Page 41: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/41.jpg)
Improvements
![Page 42: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/42.jpg)
1) Detecting Unhealthiness
![Page 43: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/43.jpg)
2) Mitigating downtime
![Page 44: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/44.jpg)
3) Recovering Effectively
![Page 45: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/45.jpg)
Detecting Unhealthiness
Component 1:
![Page 46: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/46.jpg)
![Page 47: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/47.jpg)
def signal_overload(cb): if len(jobs) > THRESH: cb.mark_unhealthy()
![Page 48: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/48.jpg)
New Behavior:
* CB gets signals from anywhere * Signal combining logic
![Page 49: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/49.jpg)
* Allows many (many) new signals
* Must combine signals * Adds complexity to system
![Page 50: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/50.jpg)
Mitigating Downtime
Component 2:
![Page 51: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/51.jpg)
![Page 52: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/52.jpg)
![Page 53: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/53.jpg)
![Page 54: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/54.jpg)
![Page 55: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/55.jpg)
New Behavior: * Code can check in advance about healthiness of system * Automatic monitoring!
![Page 56: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/56.jpg)
* Build features on top of system health status
* Requires a single source of truth?
![Page 57: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/57.jpg)
Recovering Effectively
Component 3:
![Page 58: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/58.jpg)
![Page 59: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/59.jpg)
Dark launch:
* Reject but process normally * Dangerous with side effects
Block User Request Try to process anyway!
![Page 60: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/60.jpg)
Synthetic:
* Dark launching with fake requests * Not necessarily representative
Block User Request Process fake requests
![Page 61: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/61.jpg)
New Behavior:
* Traffic determines health * Removal of recovery timeouts
![Page 62: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/62.jpg)
* Representative recovery * No timeout tuning required * Dark launching not always possible * Synthetic can be unrepresentative
![Page 63: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/63.jpg)
in summary
![Page 64: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/64.jpg)
Your system will fail, have a plan!
![Page 65: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/65.jpg)
The basic CB is better than nothing
![Page 66: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/66.jpg)
Questions to ask:
* What is “unhealthy” for my system? * How should I react to unhealthiness? * How do we recover?
![Page 67: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/67.jpg)
Questions to ask:
* What is “unhealthy” for my system? * How should I react to unhealthiness? * How do we recover?
![Page 68: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/68.jpg)
Questions to ask:
* What is “unhealthy” for my system? * How should I react to unhealthiness? * How do we recover?
![Page 69: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/69.jpg)
Questions to ask:
* What is “unhealthy” for my system? * How should I react to unhealthiness? * How do we recover?
![Page 70: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/70.jpg)
…and much more!
![Page 71: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/71.jpg)
Much comes down to your use case
![Page 72: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/72.jpg)
@YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp
[email protected] @scott_triglia
![Page 73: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/73.jpg)
![Page 74: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/74.jpg)
Downtime options besides rejecting
requests?
![Page 75: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/75.jpg)
http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html
![Page 76: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/76.jpg)
How do I safely test out a new circuit breaker?
![Page 77: Protect your Users with Circuit breakers](https://reader031.vdocument.in/reader031/viewer/2022021814/58ce74e81a28abdc578b6441/html5/thumbnails/77.jpg)
https://engineering.heroku.com/blogs/2015-06-30-improved-production-stability-with-circuit-breakers/