title slide — finger touch.€¦ · © 2018 bloomberg finance l.p. all rights reserved....

33
© 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers… with Math

Upload: others

Post on 30-May-2020

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Monitorama

June 4-6, 2018

Allan Espinosa

Site Reliability Engineer

Autoscaling Containers…

with Math

Page 2: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

@me

Page 3: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Your Distributed System

Page 4: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Your Air Conditioner

Page 6: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Control Theory

• Influencing dynamic systems

• Corrections based on feedback loops

• Math describes effectiveness

Page 7: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Centrifugal Governor

R. Routledge, Discoveries & Inventions of the Nineteenth

Century, 13th Edition, 1901

Page 8: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Theory

Page 9: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Control Theory

• Target Output

• Input

• Actual Output

• Disturbances

Page 10: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Controller Loop Code

Page 11: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Sample Output

Page 12: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Linear-Time Invariant Systems

Page 13: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Desired Properties

• Stability

• Accuracy

• Settling time

• Overshoot

Page 14: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Stability – Unstable Systems

Page 15: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Unstable Systems

Page 16: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Unstable Systems

Page 17: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Stable Systems – Damping

Page 18: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Stable Systems

Page 19: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Accuracy

Page 20: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Settling time

Page 21: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Overshoots

Page 22: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Controllers

Page 23: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Proportional Control

Page 24: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Proportional Control

• Inherently inaccurate

• Proportional constant KP increases overshoot and settling time

Page 25: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Integral Control

• Reduces steady-state errors

• Increases settling times

Page 26: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Derivative Control

• Decreases settling time

• Sensitive to noise

Page 27: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Non-linearity

Page 28: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Other applications

• Response time vs. keep-alive time-outs

• Memory vs. worker connections

• Multiple input-multiple output

Auto-tuning, Auto-configuration

Page 29: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Summary

• Iterate on feedback

• Effectiveness of feedback

• Linear models go a long way

• Re-evaluate your models!

Page 30: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

References

• P. Janert, Feedback Control for Computer Systems, Sebastapol, Ca: O’Reilly Media, 2014.

• J. Hellerstein, et. al., Feedback Control of Computing Systems, Hoboken, NJ: John Wiley & Sons, 2004.

Page 31: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Demo

Page 32: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

Allan Espinosa

[email protected]

http://github.com/aespinosa/control-theory

Thank You!!

Page 33: Title slide — finger touch.€¦ · © 2018 Bloomberg Finance L.P. All rights reserved. Monitorama June 4-6, 2018 Allan Espinosa Site Reliability Engineer Autoscaling Containers…

© 2018 Bloomberg Finance L.P. All rights reserved.

https://www.bloomberg.com/careers/technology/

Demo link: http://github.com/aespinosa/control-theory

We’re Hiring!