expand your monitoring with graphite/grafana alec sheperd ......grafana - graphing and visualization...
TRANSCRIPT
![Page 1: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/1.jpg)
Expand your monitoring with Graphite/Grafana
IT Professionals Conf. 2019Alec Sheperd
![Page 2: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/2.jpg)
Motivation● Find a different way to monitor our Condor cluster● Move away from Ganglia for custom metrics● Easily create custom dashboard and views of different graphs● Use multiple datasource backends● Spoiler: Graphite and Elasticsearch
■ Running job metrics in Graphite■ Condor history in Elasticsearch
![Page 3: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/3.jpg)
What is it?
● Graphite - Time series data collection○ Carbon: collector/aggregator daemon○ Whisper: Time-series database○ Graphite-Web/API: HTTP endpoint for querying metrics and rendering
graphs
● Grafana - Graphing and visualization○ Front facing web application○ Proxy requests from time-series data sources
■ (Graphite, Elasticsearch, Prometheus…)○ Build graphs and other displays for visualization
![Page 5: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/5.jpg)
Graphite - Daemons
● Carbon-cache○ Accepts and writes metrics to disk
● Carbon-relay○ Forward metrics to multiple carbon-caches via rules or
consistent hashing● Carbon-aggregator
○ Buffer metrics in front of a carbon-cache to reduce disk IO or aggregate a matching namespace together
![Page 6: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/6.jpg)
Graphite - Pushing in metrics
Plain-text: PORT=2003
SERVER=carbon-server.icecube.wisc.edu
echo "my.namespace.metric 4 `date +%s`" | nc ${SERVER} ${PORT}
Python pickle:data = [("my.namespace.metric", (timestamp, v)), ...]
payload = cPickle.dumps(data, protocol=2)header = struct.pack("!L", len(payload))message = header + payload
![Page 7: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/7.jpg)
Graphite - Namespaces ● Metric identifier that corresponds to a POSIX file path
○ my.namespace.metric to $WHISPER/my/namespace/metric
● Group related metrics under similar name spaces○ File retention ○ Ease of querying
● Best not to get too “married” to a namespace○ Run into organization issues○ Keep dynamic paths deeper in the namespace
![Page 8: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/8.jpg)
Graphite - Retentions● Set different retention policies for different namespaces
○ Aggregation methods (average, min, max, last...)
● Lower precision for historical data ○ Save disk space and query latency at the cost of CPU
[title]pattern = ^my.namespace.*$retentions = 60s:1d,300s:1yr
● File retentions determines total disk usage
![Page 9: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/9.jpg)
Graphite - Pros
● Highly customizable○ Tailor to fit your needs○ Can scale well
● Lots of documentation and tools○ Many other technologies and services have functions
to export to graphite ● Simple design
![Page 10: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/10.jpg)
Graphite - Problems & Limitations
● Double-edged: Generally easy to stand up; tweaks required for extensive use○ Ran into performance issues when you get move
toward 200k updates/minute● No standard data collection service
○ Collectd, Collectl, Ganglia-graphite https://graphite.readthedocs.io/en/latest/tools.html
![Page 11: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/11.jpg)
Graphite - Clustering● Relay metrics to
multiple carbon-caches
● Carbon-cache nodes can be either distinct machines, or a single machine with multiple disks
● Helm charts for running cluster in kubernetes
![Page 12: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/12.jpg)
Grafana
● Web app for organizing and displaying dashboards of graphs
● Authentication via OAuth, LDAP, or custom Auth Proxy
● Collection of community built dashboards and plugins
![Page 13: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/13.jpg)
Grafana - Overview
● Dashboards are a collection of multiple panels○ Histograms, line graphs, tables, single stats…
● Each panel can have multiple queries to data sources○ HTTP-API calls to a backend
● Template variables allow dynamic changes to be made to queries
![Page 14: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/14.jpg)
Grafana - Creating a new dashboard
![Page 15: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/15.jpg)
Grafana - Simple query
![Page 16: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/16.jpg)
Grafana - Building a panel
![Page 17: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/17.jpg)
Grafana - Template variables
![Page 18: Expand your monitoring with Graphite/Grafana Alec Sheperd ......Grafana - Graphing and visualization Front facing web application Proxy requests from time-series data sources (Graphite,](https://reader033.vdocument.in/reader033/viewer/2022051408/600a6cf4621cab2966756932/html5/thumbnails/18.jpg)
Questions!
● Try out a Grafana dashboard at https://play.grafana.org/d/000000056/graphite-templated-nested?orgId=1