kubernetes and prometheus
TRANSCRIPT
Prometheus & Kubernetes
Lessons Learnt
Tom Wilkie, Weaveworks 8th July 2016
Deploying on Kubernetes
Service-oriented monitoring
Alerting on differences
Deploying on Kubernetes
Not a new topic!
“Monitoring Kubernetes with Prometheus” - Brian Brazil
“Prometheus and Kubernetes up and running” - Fabian Reinartz
Even example config upstream
Kubernetes concepts
Pod
Container
Container
Container
Pod
Container
Container
Container
Pod
Container
Container
Container
Replication Controller / DeploymentService
Monitoring Pod
Promtheus Container
AlertManager Container
Grafana Container
Monitoring Pod
Prometheus Container
AlertManager Container
Grafana Pod
Grafana Container
Prometheus ConfigMap
Grafana ConfigMap
Grafana Pod
Grafana Container
Grafana ConfigMap
Grafana Pod
Grafana Container
SOCKS Proxy
SOCKS Proxy
Grafana Pod
Grafana Container
AuthFE
AuthFE Container
Demo
Deploying on Kubernetes
Service-oriented monitoring
When configuring Prometheus: • map Kubernetes Services to Prometheus Jobs • map Kubernetes Pods to Prometheus Instances
Using relabelling:
relabel_configs:
- source_labels: [__meta_kubernetes_service_namespace]
action: replace
regex: ^(.+)$
target_label: namespace
replacement: $1
- source_labels: [__meta_kubernetes_service_name]
regex: ^(.+)$
target_label: job
replacement: $1
If you get this
- source_labels: [__meta_kubernetes_role]
action: keep
regex: endpoint
But not all my services export metrics!
- source_labels:[__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: drop
regex: false
And add this to your service definition:
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io.scrape: "false"
What about exporters?
Memcached Pod
memcached Container
memcached-exporter Container
What about exporters?
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ^(.+)(?::\d+);(\d+)$
replacement: $1:$2
And add this to your service definition:
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io.port: "9150"
Service-oriented monitoring
Alerting on differences
What I really came here to talk about…
Kubernetes best-practice is to version control your
config.
Whats running vs whats checked in with drift unless we monitor it.
$ ./kubediff Usage: kubediff [options] <dir/file>...
Compare yaml files in <dir> to running state in kubernetes and print the differences. This is useful to ensure you have applied all your changes to the appropriate environment.
https://github.com/weaveworks/kubediff
Run it as a service:
This will export the exit code as a Prometheus metric
Get alerts from it:
https://github.com/weaveworks/kubediff
ALERT Kubediff IF max(command_exit_code{job="kubediff"}) != 0 FOR 2h
Alerting on differences
Deploying on Kubernetes
Service-oriented monitoring
Alerting on differences
Thanks! Questions? @tom_wilkie