![Page 1: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/1.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Monitorama
June 4-6, 2018
Allan Espinosa
Site Reliability Engineer
Autoscaling Containers…
with Math
![Page 2: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/2.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
@me
![Page 3: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/3.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Your Distributed System
![Page 4: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/4.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Your Air Conditioner
![Page 5: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/5.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Autoscaling
https://docs.run.pivotal.io/appsman-services/autoscaler/using-autoscaler.html
![Page 6: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/6.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Control Theory
• Influencing dynamic systems
• Corrections based on feedback loops
• Math describes effectiveness
![Page 7: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/7.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Centrifugal Governor
R. Routledge, Discoveries & Inventions of the Nineteenth
Century, 13th Edition, 1901
![Page 8: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/8.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Theory
![Page 9: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/9.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Control Theory
• Target Output
• Input
• Actual Output
• Disturbances
![Page 10: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/10.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Controller Loop Code
![Page 11: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/11.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Sample Output
![Page 12: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/12.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Linear-Time Invariant Systems
![Page 13: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/13.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Desired Properties
• Stability
• Accuracy
• Settling time
• Overshoot
![Page 14: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/14.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Stability – Unstable Systems
![Page 15: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/15.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Unstable Systems
![Page 16: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/16.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Unstable Systems
![Page 17: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/17.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Stable Systems – Damping
![Page 18: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/18.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Stable Systems
![Page 19: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/19.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Accuracy
![Page 20: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/20.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Settling time
![Page 21: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/21.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Overshoots
![Page 22: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/22.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Controllers
![Page 23: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/23.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Proportional Control
![Page 24: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/24.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Proportional Control
• Inherently inaccurate
• Proportional constant KP increases overshoot and settling time
![Page 25: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/25.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Integral Control
• Reduces steady-state errors
• Increases settling times
![Page 26: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/26.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Derivative Control
• Decreases settling time
• Sensitive to noise
![Page 27: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/27.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Non-linearity
![Page 28: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/28.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Other applications
• Response time vs. keep-alive time-outs
• Memory vs. worker connections
• Multiple input-multiple output
Auto-tuning, Auto-configuration
![Page 29: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/29.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Summary
• Iterate on feedback
• Effectiveness of feedback
• Linear models go a long way
• Re-evaluate your models!
![Page 30: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/30.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
References
• P. Janert, Feedback Control for Computer Systems, Sebastapol, Ca: O’Reilly Media, 2014.
• J. Hellerstein, et. al., Feedback Control of Computing Systems, Hoboken, NJ: John Wiley & Sons, 2004.
![Page 31: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/31.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Demo
![Page 32: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/32.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
Allan Espinosa
http://github.com/aespinosa/control-theory
Thank You!!
![Page 33: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…](https://reader034.vdocument.in/reader034/viewer/2022042310/5ed7df02be7cd466191cece1/html5/thumbnails/33.jpg)
© 2018 Bloomberg Finance L.P. All rights reserved.
https://www.bloomberg.com/careers/technology/
Demo link: http://github.com/aespinosa/control-theory
We’re Hiring!