circuit breakers @ api world 2016
TRANSCRIPT
Handle API Failure gracefully with Circuit Breakers
Scott Triglia
1
Jim’s
2
Yelp’s Mission:Connecting people with great
local businesses.
Yelp StatsAs of Q2 2016
92M monthly
users
32 countries
72% of searches from
mobile
108M reviews
Scott Triglia
4
@scott_triglia
Work with the $$$
Let’s talk Circuit Breakers
5
6
7
9
10
Our goals today: introduce a basic circuit breaker
11
Our goals today: a modular circuit breaker
12
Our goals today: test it out on several scenarios
13
14
15
1616
1
2
34
the fundamental rule: your systems will fail
what’s your response?
17
1818
1
2
1919
1
2
3
2020
1
2
34
21
Nygard’s circuit breaker
22
23
24
25
26
Circuit Breaker States: * Healthy (or “closed”) * Recovering (or “half-open”) * Unhealthy (or “open”)
27
28
Recovery:
* Wait for recovery_timeout seconds* Send a trial request, trust its results
29
Before a circuit breaker
30
Assume the kitchen gets slow
31
Kitchen’s backlog grows
32
Diners wait much longer to get food
33
New diners make issues worse
34
And your entire system collapses
35
And your entire system collapses
36
With a circuit breaker
37
CB sees backlog, stops orders
38
Fewer frustrated diners
39
Reduced load on the kitchen
40
A well defined failure mode
41
How can we do better?
42
Improvements
43
1) Detecting Unhealthiness
44
2) Mitigating downtime
45
3) Recovering Effectively
46
Detecting Unhealthiness
Component 1:
47
48
def signal_overload(cb): if len(jobs) > THRESH: cb.mark_unhealthy()
49
New Behavior:
* CB gets signals from anywhere * Signal combining logic
50
* Allows many (many) new signals
* Must combine signals * Adds complexity to system
51
Mitigating Downtime
Component 2:
52
53
54
55
56
New Behavior: * Code can check in advance about healthiness of system * Automatic monitoring!
57
* Build features on top of system health status
* Requires a single source of truth?
58
Recovering Effectively
Component 3:
59
60
Dark launch:
* Reject but process normally * Dangerous with side effects
Block User Request Try to process anyway!
61
Synthetic:
* Dark launching with fake requests * Not necessarily representative
Block User Request Process fake requests
62
New Behavior:
* Traffic determines health * Removal of recovery timeouts
63
* Faster(?) recovery * No timeout tuning required * Dark launching not always possible * Synthetic can be unrepresentative
64
in summary
65
Your system will fail, have a plan!
66
The basic CB is better than nothing
67
Questions to ask:
* What is “unhealthy” for my system? * How should I react to unhealthiness? * How do we recover?
68
Questions to ask:
* What is “unhealthy” for my system? * How should I react to unhealthiness? * How do we recover?
69
Questions to ask:
* What is “unhealthy” for my system? * How should I react to unhealthiness? * How do we recover?
70
Questions to ask:
* What is “unhealthy” for my system? * How should I react to unhealthiness? * How do we recover?
71
…and much more!
Much comes down to your use case
72
@YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp
[email protected] @scott_triglia
75
Can’t we do better than rejecting requests?
http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html
77
How do I safely test out a new circuit breaker?
https://engineering.heroku.com/blogs/2015-06-30-improved-production-stability-with-circuit-breakers/