prometheus: from berlin to bonanza (keynote cloudnativecon+kubecon europe 2017)

Brian BrazilFounder

PrometheusFrom Berlin to Bonanza

Who am I?

One of the four core developers of Prometheus

Founder of Robust Perception

Contributor to many open source projects

Ex-Googler, after 7 years in the Dublin office

A Little HistoryPrometheus started in 2012 by Matt Proud and Julius Volz in Berlin.

In 2013 developed within SoundCloud, expanded to support Bazooka (cluster manager/scheduler), Go, Java and Ruby clients.

A Growing CommunityIn 2014 other companies start using it, myself working at Boxever and Johannes at Docker.

Project matures: new storage, new text format.

In 2015 we "publicly release", adoption increases.

The Open Source BonanzaToday there are 300+ contributors to the core repositories, and 100+ 3rd party integrations.

There are 600+ subscribers on our mailing lists, 400+ people in IRC and an estimated 500+ companies using Prometheus in production.

Many companies funding Prometheus development.

What is Prometheus?Metrics monitoring system (not logs).

A time series database. A query language.

Client libraries. An Ecosystem.

A Cloud Native approach to monitoring services.

Architecture

Monitoring ApproachService management went from manual to Chef to Kubernetes. Need to do the same for monitoring.

Care about what matters to end users, such as latency and error rates.

Distracting a human with alerts for everything that's vaguely off only leads to burnout.

Kubernetes Service DiscoveryPrometheus can discover all the pods, services, containers, and nodes of a Kubernetes cluster.

Your Kubernetes labels and annotations can become Prometheus labels.

Prometheus automatically picks up changes.

Client LibrariesInstrument your code to capture the metrics that matter to you.

If upstream libraries are instrumented, you get that for free!

Also many exporters, e.g. cAdvisor, MySQL, SNMP, Consul, JMX, HAProxy, Minecraft, Factorio.

Instrumentation in CNCF projectsKubernetes is instrumented with Prometheus, allowing you to monitor the health of the cluster itself.

Similarly Linkerd exposes metrics in our format.

Interceptors for GRPC and plugins for Fluentd can feed data to Prometheus.

The PromQL Query LanguageArbitrary aggregation, joins and slicing all possible.

Can calculate how close you'll be to your quota in 4 hours, or the 95th percentile latency across an entire datacenter.

If you can graph it, you can alert on it!

Analytics: Top 5 Docker images by CPU

topk(5, sum by (image)(

rate(container_cpu_usage_seconds_total{id=~"/system.slice/docker.*"}[5m] ) ))

Alert managementNot every alert results in a page.

Group similar alerts together, route them to the right team and throttle notifications.

Designed to work reliably during network partitions.

What is Prometheus?Metrics monitoring system (not logs).

A time series database. A query language.

Client libraries. An Ecosystem.

A Cloud Native approach to monitoring services.

Suggested Prometheus Talks, In A06 Today

12:35 Counting with Prometheus - Brian

14:40 Alerting in Cloud Native Environments - Fabian

16:20 Grafana is Not Enough: DIY User Interfaces for Prometheus - David Kaltschmidt

Suggested Prometheus Talks, In A06 Tomorrow

11:30 Integrating Long-Term Storage with Prometheus - Julius

12:15 Prometheus: The Unsung Heroes - Alejandro Ramirez

14:00 Configuring Prometheus for High Performance - Björn

14:45 Workshop: Instrumenting Own Applications for Prometheus - Alexandru Somesan & Frederic Branczyk - in C04, registration required

ResourcesOfficial Project Website: prometheus.io

User Mailing List: prometheus-users@googlegroups.com

Dev Mailing List: prometheus-developers@googlegroups.com

IRC: #prometheus on chat.freenode.net

Robust Perception Blog: www.robustperception.io/blog