Download - Prometheus (Monitorama 2016)
![Page 1: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/1.jpg)
Brian BrazilFounder
Prometheus
![Page 2: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/2.jpg)
Who am I?
Engineer passionate about running software reliably in production.
Studied Computer Science in Trinity College Dublin.
Google SRE for 7 years, working on high-scale reliable systems.
Contributor to many open source projects, including Prometheus, Ansible, Python, Aurora and Zookeeper.
Founder of Robust Perception, provider of commercial support and consulting for Prometheus.
![Page 3: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/3.jpg)
What is Prometheus?
Prometheus is a metrics-based time series database, designed for whitebox monitoring.
It supports labels (dimensions/tags).
Alerting and graphing are unified, using the same language.
![Page 4: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/4.jpg)
Development History
Inspired by Google’s Borgmon monitoring system.
Started in 2012 by ex-Googlers working in Soundcloud as an open source project, mainly written in Go. Publically launched in early 2015, and continues to be independent of any one company. Incubating with the CNCF.
Over 250 people have contributed to official repos. 50+ 3rd party integrations.
Hundreds of companies rely on it since then.
![Page 5: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/5.jpg)
Architecture
![Page 6: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/6.jpg)
Your Services have Internals
![Page 7: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/7.jpg)
Monitor the Internals
![Page 8: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/8.jpg)
Instrumentation Made Easypip install prometheus_client
from prometheus_client import Summary, start_http_serverREQUEST_DURATION = Summary('request_duration_seconds', 'Request duration in seconds')
REQUEST_DURATION.observe(7)
@REQUEST_DURATION.time()def my_handler(request): pass # Your code here
start_http_server(8000)
![Page 9: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/9.jpg)
Open ecosystem
Prometheus client libraries don’t tie you into Prometheus.
# Expose with Prometheus text format over HTTPstart_http_server(8000)
# Push to Graphite every 10 secondsgb = GraphiteBridge(('graphite.your.org', 2003))gb.start(10.0)
![Page 10: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/10.jpg)
Prometheus Clients as a Clearinghouse
![Page 11: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/11.jpg)
Power of Labels
Prometheus doesn’t use dotted.strings like metric.monitorama.portland.
We have metrics like:
metric{conference=”monitorama”,city=”portland”}
Full UTF-8 support - don't need to worry about values containing dots.
Can aggregate, cut, and slice along them.
![Page 12: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/12.jpg)
Great for Aggregation
topk(5, sum by (image)(
rate(container_cpu_usage_seconds_total{id=~"/system.slice/docker.*"}[5m] ) ))
![Page 13: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/13.jpg)
Great for Alerting
Alert on any machine being down:
ALERT InstanceDown IF up{job="node"} == 0 FOR 10m
Alert on 25% of machines being down:
ALERT ManyInstancesDown IF avg by(job)(up{job="node"}) < .75 FOR 10m
![Page 14: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/14.jpg)
Monitor as a Service, not as Machines
![Page 15: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/15.jpg)
Efficient
A single Prometheus server can handle 800K samples/s
New varbit encoding uses only 1.3 bytes/sample
Node exporter produces ~700 time series, so even with a 10s scrape interval a single Prometheus could handle over 10,000 machines!
This efficiency means that the vast majority of users never need to worry about scaling.
![Page 16: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/16.jpg)
Decentralised
Pull based, so easy to on run a workstation for testing and rogue servers can’t push bad metrics.
Each team can run their own Prometheus, no need for central management or talking to an operations team.
Perfect for microservices!
![Page 17: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/17.jpg)
Per-team Hierarchy of Targets
![Page 18: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/18.jpg)
Reliable
A monitoring system should be something you trust when your network and systems are falling apart.
Prometheus has no external dependencies. It has no complex CP clustering to go wrong.
For HA we keep it simple.
![Page 19: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/19.jpg)
Alerting Architecture
![Page 20: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/20.jpg)
Opinionated
As a project, we recognise our limitations.
We try avoid reinventing the wheel when others have already solved a problem, e.g. Grafana over Promdash.
We encourage good practices and using the right tool for the job: You could trigger a script to restart a failed process via the Alertmanager, but a supervisor such as daemontools or monit is probably a better approach.
![Page 21: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/21.jpg)
Demo
![Page 22: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/22.jpg)
What defines Prometheus?
Key attributes of the Prometheus monitoring system:
Open Source
Instrumentation Made Easy
Powerful
Efficient
Decentralized
Reliable
Opinionated
![Page 23: Prometheus (Monitorama 2016)](https://reader035.vdocument.in/reader035/viewer/2022062823/58762d4d1a28ab8b7b8b6f57/html5/thumbnails/23.jpg)
Resources
Official Project Website: prometheus.io
Official Mailing List: [email protected]
Demo: demo.robustperception.io
Robust Perception Blog: www.robustperception.io/blog
Queries: [email protected]