monitoring complex systems - chicago erlang, 2014
DESCRIPTION
Imagine being responsible for monitoring 100 servers. Now imagine 1000. Each server has 100 different things to keep track of. What do you pay attention to and what do you ignore? What is important? In this talk Brian will show how Erlang can be used to capture more information without compromising clarity — i.e. to keep track of the forest without loosing site of the trees!TRANSCRIPT
![Page 1: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/1.jpg)
Monitoring Complex Systems
![Page 2: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/2.jpg)
I do things to/with computers.
![Page 3: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/3.jpg)
I build real-time systems.
![Page 4: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/4.jpg)
I build fault-tolerant systems.
![Page 5: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/5.jpg)
I build critical systems.
![Page 6: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/6.jpg)
AdRoll
![Page 7: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/7.jpg)
Less this.
![Page 8: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/8.jpg)
More this.
![Page 9: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/9.jpg)
Engineering + Mathematics = ads
![Page 10: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/10.jpg)
Engineering + Mathematics = ads
(you’re welcome)
![Page 11: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/11.jpg)
R E A L - T I M E B I D D I N G
![Page 12: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/12.jpg)
The Problem Domain• Low latency ( < 100 ms per transaction)
![Page 13: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/13.jpg)
The Problem Domain• Low latency ( < 100 ms per transaction)
• Firm real-time system
![Page 14: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/14.jpg)
The Problem Domain• Low latency ( < 100 ms per transaction)
• Firm real-time system
• Highly concurrent (90 billion transactions per day)
![Page 15: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/15.jpg)
The Problem Domain• Low latency ( < 100 ms per transaction)
• Firm real-time system
• Highly concurrent (90 billion transactions per day)
• Global, 24/7 operation
![Page 16: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/16.jpg)
I build Complex Systems
![Page 17: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/17.jpg)
Complex Systems
• Non-linear feedback
• Tightly coupled to external systems
• Difficult to model, understand
![Page 18: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/18.jpg)
Bad things happen when Complex Systems fail.
![Page 19: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/19.jpg)
Humans are bad at predicting the performance of complex systems(…). Our ability to create large and complex systems fools us into believing that we’re also entitled to understand them.
C A R L O S B U E N O “ M AT U R E O P T I M I Z AT I O N H A N D B O O K ”
![Page 20: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/20.jpg)
Complex Systems often create worse problems than those they solve.
![Page 21: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/21.jpg)
The key challenge to sustaining a complex
system is maintaining our understanding of it.
![Page 22: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/22.jpg)
What can be done?
![Page 23: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/23.jpg)
Ahead of time verification is not sufficient.
!
(don’t scrimp on it, though)
![Page 24: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/24.jpg)
Compile-time guarantees are not sufficient.
!
(don’t scrimp on them, either)
![Page 25: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/25.jpg)
We need insight into the running system.
![Page 26: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/26.jpg)
• VM killers
What are we looking for?
![Page 27: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/27.jpg)
• VM killers
• Application performance regressions
What are we looking for?
![Page 28: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/28.jpg)
• VM killers
• Application performance regressions
• Abnormal application behavior
What are we looking for?
![Page 29: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/29.jpg)
• VM killers
• Application performance regressions
• Abnormal application behavior
• Surprises
What are we looking for?
![Page 30: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/30.jpg)
INSTRUMENTATION
![Page 31: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/31.jpg)
The BEAM is ready to ride.
![Page 32: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/32.jpg)
erlang:memory/1
![Page 33: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/33.jpg)
erlang:statistics/1
![Page 34: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/34.jpg)
erlang:system_info/1
![Page 35: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/35.jpg)
What about our own work?
![Page 36: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/36.jpg)
Exometer
![Page 37: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/37.jpg)
Important Termsmetric a measurement
entry a receiver and aggregator of metrics
reporter that which samples entries periodically and ships them to another system
subscription the definition of the regular interval on which reporters sample entries
![Page 38: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/38.jpg)
These are all loosely coupled at runtime.
![Page 39: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/39.jpg)
Configuration is static, but you can adapt it on the fly.
![Page 40: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/40.jpg)
Why exometer over the alternatives?
![Page 41: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/41.jpg)
It is extensively documented.
![Page 42: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/42.jpg)
It’s vigorously maintained.
![Page 43: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/43.jpg)
It’s vigorously maintained.
(Ulf Wiger fan club day.)
![Page 44: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/44.jpg)
It is silly fast.
![Page 45: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/45.jpg)
Okay, great. We have instrumentation.
![Page 46: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/46.jpg)
Now what?
![Page 47: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/47.jpg)
MONITORING
![Page 48: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/48.jpg)
This is the hard part.
![Page 49: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/49.jpg)
Visualization
![Page 50: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/50.jpg)
Alerting
![Page 51: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/51.jpg)
Analysis
![Page 52: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/52.jpg)
Visualization tells you how things look but not why.
![Page 53: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/53.jpg)
A good day.
![Page 54: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/54.jpg)
Uh oh.
![Page 55: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/55.jpg)
Growth is good, but steady is better.
![Page 56: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/56.jpg)
Bids are
stable.
![Page 57: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/57.jpg)
Budgets are
stable.
![Page 58: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/58.jpg)
What happened?
![Page 59: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/59.jpg)
We forgot about Labor Day.
![Page 60: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/60.jpg)
![Page 61: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/61.jpg)
Alerting tells you that something happened, but not why.
![Page 62: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/62.jpg)
A normal day.
![Page 63: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/63.jpg)
Wat?
![Page 64: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/64.jpg)
That’s some cliff.
![Page 65: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/65.jpg)
Timeouts look good.
![Page 66: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/66.jpg)
Errorsprior are okay.
![Page 67: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/67.jpg)
What happened?
![Page 68: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/68.jpg)
“Uh, hey guys, you know Facebook is down, right?”
![Page 69: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/69.jpg)
Analysis gives you why but only if you know how to ask for what.
![Page 70: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/70.jpg)
The memory use of a bidder.
![Page 71: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/71.jpg)
ಠ_ಠ
![Page 72: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/72.jpg)
It’s all binaries.
![Page 73: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/73.jpg)
Not in processes.
![Page 74: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/74.jpg)
Not in ETS.
![Page 75: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/75.jpg)
Come on now.
![Page 76: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/76.jpg)
What happened?
![Page 77: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/77.jpg)
A jiffy bug.
![Page 78: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/78.jpg)
A jiffy bug.
(we think)
![Page 79: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/79.jpg)
Shout out to Miriam Pena for spending two weeks
tracking this down.
![Page 80: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/80.jpg)
Okay, great. We have monitoring and instrumentation.
![Page 81: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/81.jpg)
Now all our problems are solved, right?
![Page 82: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/82.jpg)
Not quite.
![Page 83: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/83.jpg)
Instruments make up for our lack of insight.
![Page 84: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/84.jpg)
Monitoring makes up for our frailty.
![Page 85: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/85.jpg)
Every solution brings its own problems.
![Page 86: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/86.jpg)
Instruments may be misleading.
![Page 87: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/87.jpg)
Instruments may be overwhelming.
![Page 88: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/88.jpg)
Instruments may be
inaccurate.
![Page 89: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/89.jpg)
Instruments may be ignored.
![Page 90: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/90.jpg)
What can be done?
![Page 91: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/91.jpg)
A little paranoia never hurt
anyone.
![Page 92: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/92.jpg)
Use glass displays.
![Page 93: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/93.jpg)
Train.
![Page 94: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/94.jpg)
Keep sight of the main goal.
![Page 95: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/95.jpg)
Have resources you’re willing to sacrifice.
![Page 96: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/96.jpg)
AdRoll is Hiring! :D
![Page 97: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.vdocument.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/97.jpg)
Thanks, folks!<3
@bltroutwine