three pilars of observability kubernetes with elastic stack · docker • kubernetes ... apm adds...
TRANSCRIPT
![Page 1: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/1.jpg)
1
Massimo BrignoliPrincipal Solutions Architect, Elastic
Three Pilars of Observability Kuberneteswith Elastic Stack
![Page 2: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/2.jpg)
2
• Custom on-prem & cloud deployments
• Public cloud fully-managed deployments
– Google Kubernetes Engine (GKE)
– Amazon Elastic Container Service for Kubernetes (EKS)
– Azure Kubernetes Service (AKE)
• Pivotal Container Service (PKS)
• Red Hat OpenShift
Kubernetes is Taking Over the Enterprise
![Page 3: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/3.jpg)
3
Kubernetes is Complicated
Container Runtime
![Page 4: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/4.jpg)
4
Kubernetes Visibility Challenges
![Page 5: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/5.jpg)
55
Observable Kubernetes
Elastic Stack: Three Pillars of Observability in One Platform
● Logging
● Metrics
● APM Tracing
![Page 6: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/6.jpg)
6
It Comes Down to The Three Pillars of Observability
Twitter:https://blog.twitter.com/engineering/en_us/a/2013/observability-at-twitter.htmlPeter Bourgonhttps://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html
![Page 7: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/7.jpg)
7
Elastic at the Center Stage
![Page 8: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/8.jpg)
8
Elastic Stack for logs
![Page 9: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/9.jpg)
64.242.88.10 - - [07/Jan/2019:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291
64.242.88.10 - - [07/Jan/2019:16:11:58 -0800] "POST /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 404 7352
64.242.88.10 - - [07/Jan/2019:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253
For each event, print out what happened.
Metrics vs LogsLogs are chronological records of events
![Page 10: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/10.jpg)
•Turnkey experience for specific data types
•Data to dashboard in just one step
•Automated parsing and enrichment
•Default dashboards, alerts, ML jobs
Making logging more turnkey with modulesLogging Metrics Security
![Page 11: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/11.jpg)
Logging Modules
11
System
•Linux / MacOS
•Windows Events
Containers
•Docker
•Kubernetes
Databases
•MySQL
•PostgreSQL
Queues
•Kafka
•Redis
Web servers
•Apache
•Nginx
Audit data
•Filesystem
•System calls
Infrastructure Applications
WINLOGBEATFILEBEATAUDITBEAT
![Page 12: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/12.jpg)
Log File Import
12
Automatic Structure Discovery
![Page 13: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/13.jpg)
Ad-hoc log search and visualization Kibana Discover, Visualize, Dashboard
![Page 14: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/14.jpg)
14
Elastic Stack for metrics
![Page 15: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/15.jpg)
Elasticsearch beginnings
15
Primarily used for application searchSearch engineInverted index primary data structure, and is great for search
2010
![Page 16: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/16.jpg)
2012 Columnar storage Structured data storage, resulting in compact storage and faster analytics
Elasticsearch evolves to support analytics
https://www.elastic.co/blog/elasticsearch-as-a-column-store
Columnar Store, Built on Lucene "doc values"Search engineInverted index primary data structure, and is great for search
2010
![Page 17: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/17.jpg)
2014 Aggregation Framework Analytics features to slice and dice data along various dimensions
Aggregation Framework
17
Out-of-this-world aggregations
https://www.elastic.co/blog/out-of-this-world-aggregations
Search engineInverted index primary data structure, and is great for search
2010
2012 Columnar storage Structured data storage, resulting in compact storage and faster analytics
![Page 18: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/18.jpg)
BKD trees and sparse fieldsData structures optimized for numbers. Faster analytics, lower storage footprint
2016
2014 Aggregation Framework Analytics features to slice and dice data along various dimensions
Elasticsearch storage efficiencies
18
BKD Trees & Sparse Fields
https://www.elastic.co/blog/searching-numb3rs-in-5.0
1-Dimension
2-Dimensions
Sparse Data
Search engineInverted index primary data structure, and is great for search
2010
2012 Columnar storage Structured data storage, resulting in compact storage and faster analytics
![Page 19: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/19.jpg)
RollupsRoll up or aggregate older data into bigger time buckets and save on disk space
2018
Rollup support for long-term retention
Added in Elasticsearch 6.3
https://www.elastic.co/blog/data-rollups-in-elasticsearch-you-know-for-saving-space
Search engineInverted index primary data structure, and is great for search
2010
BKD trees and sparse fieldsData structures optimized for numbers. Faster analytics, lower storage footprint
2016
2014 Aggregation Framework Analytics features to slice and dice data along various dimensions
2012 Columnar storage Structured data storage, resulting in compact storage and faster analytics
![Page 20: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/20.jpg)
Elasticsearch for search and numerical analytics
20
Inverted Index for full-text search Columnar store for structured data
BKD Trees for numerical operations Rollups save space
![Page 21: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/21.jpg)
Metrics Modules
21
Infrastructure
System
•Linux
•MacOS
•Windows
•Perfmon
Cloud
•AWS
•GCP
•Azure
•DigitalOcean
•Alibaba
Containers
•Docker
•Kubernetes
Virtualization
•vSphere
Network
•Netflow
•Packets
•TLS Envelope
Storage
•Ceph
PACKETBEATMETRICBEATHEARTBEAT
Infrastructure
![Page 22: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/22.jpg)
22
Metrics Modules
Infrastructure
PACKETBEATMETRICBEATHEARTBEAT
Uptime
•Heartbeat
Custom apps
•JMX/Jolokia
•PHP-FPM
•Golang
Datastores
•MySQL
•PostgreSQL
•MongoDB
•Couchbase
•Aerospike
•Graphite
Queues
•Kafka
•Redis
•RabbitMQ
Caches
•Memcached
Web servers
•Apache
•Nginx
Other
•HAProxy
•Zookeeper
Applications
![Page 23: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/23.jpg)
Heartbeat: Uptime Monitoring
![Page 24: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/24.jpg)
Heartbeat: Uptime Monitoring
![Page 25: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/25.jpg)
Functionbeat: Serverless data shipper
Cloudwatch Cloudwatch Logs
![Page 26: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/26.jpg)
Functionbeat: Serverless data shipper
![Page 27: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/27.jpg)
Visualizing time series dataTime Series Visual Builder
![Page 28: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/28.jpg)
28
Elastic Stack for APM
![Page 29: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/29.jpg)
Example: Slow response or load times
Why APM?
03:43:45 Request "GET cyclops.ESProductDetailView"
03:43:57 Response "cyclops.ESProductDetailView 200 OK"
12 seconds - zZzzZZz
![Page 30: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/30.jpg)
Example: Errors & Exceptions
Why APM?
03:43:59 Request "POST /api/checkout"
03:43:59 Response "/api/checkout 500 ERROR"
![Page 31: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/31.jpg)
Agents, API, and APM Server
How APM works
Data processorapm-server
Data storageElasticsearch
BrowserAgent
Web server
Agent
Web server
Agent
UIKibana
BrowserAgent
BrowserAgent
Web server
Agent
![Page 32: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/32.jpg)
APM adds end-user experience and application-level monitoring to the stack
Elastic APM
● Python
● Node.js
● Ruby
● RUM (Real User Monitoring)
Language Support
● Java
● Go
● .NET (in dev)
•Focuses on search experience on top of APM data
•Just another index in Elastic Stack
•Active roadmap to expand programming languages
![Page 33: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/33.jpg)
Great overview and drill-down with industry-standard visualizations
Dedicated APM UI
![Page 34: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/34.jpg)
Single transaction
Distributed Tracing
Transaction 1
SpanSpan
Span
HTTP request Response
![Page 35: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/35.jpg)
Multiple Services
Distributed Tracing
Trace A
Transaction 1
SpanSpan
Transaction 2
Span
Transaction 3
SpanSpan
Span
![Page 36: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/36.jpg)
Combine a custom workflow with the freedom of search
Ad-hoc search in a curated UI
![Page 37: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/37.jpg)
Need another visualization? Build a dashboard, no need to wait for your vendor
APM is just another index in Elasticsearch
![Page 38: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/38.jpg)
Correlate data from different sourcesAbility to re-use analysis content Ability to re-use Elastic-provided content
Correlation between logs, metrics, and APM Elastic Common Schema
Benefits
Version 0.1 published: github.com/elastic/ecsWorking with internal groups to validateCommunity feedback welcome!
Status
![Page 39: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/39.jpg)
39
Metadata processorsEnrich events with useful metadata to correlate logs, metrics & traces
• cloud.availability_zone
• cloud.region
• cloud.instance_id
• cloud.machine_type
• cloud.project_id
• cloud.provider
• docker.container.id
• docker.container.image
• docker.container.name
• docker.container.labels
• kubernetes.pod.name
• kubernetes.namespace
• kubernetes.labels
• kubernetes.annotations
• kubernetes.container.name
• kubernetes.container.image
add_cloud_metadata add_docker_metadata add_kubernetes_metadata
![Page 40: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/40.jpg)
40
Kubernetes deployment
Node 1
Metricbeat
Filebeat
Node 2
Metricbeat
Filebeat
Node n
Metricbeat
Filebeat
Filebeat DaemonSet
Metricbeat DaemonSet
![Page 41: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/41.jpg)
4141
Logging
● Cluster level logging
● Services logging (eg. nginx, mysql)
● Custom application logging
![Page 42: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/42.jpg)
42
Kubernetes Logging
• Need for a logging solution– Kubernetes does not have a native solution
– kubectl logs is too hard for large clusters
• Cluster-level logging– Logs have separate storage and lifecycle independent of nodes, pods and containers
– Kubernetes provides no native storage solution for log data
• Application-level logging– Complicated
– Packaged applications (eg. nginx)
– Custom applications
![Page 43: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/43.jpg)
43
Two Packaged Solutions
• Fluentd DamonSet– Log collection, parsing and distribution
• Fluentd + Stackdriver for GCP
• Fluentd + Elasticsearch
![Page 44: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/44.jpg)
44
Better Log Collection with Filebeat
kubectl create -f filebeat-kubernetes.yaml
![Page 45: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/45.jpg)
45
Filebeat Auto-Discovery
filebeat.autodiscover: providers: - type: kubernetes templates: - condition: contains: kubernetes.container.image: " nginx" config: - module: nginx access: # For nginx access log prospector: type: docker containers.ids: - "${data.kubernetes.container.id}"
• A module contains
– Log file path
– Ingest pipeline
– Fields definitions
– Sample dashboards
![Page 46: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/46.jpg)
46
• Apache2 module
• Auditd module
• Icinga module
• IIS module
• Kafka module
• Logstash module
• MongoDB module
Filebeat ModulesSimplify collection, parsing and visualization of common log formats
• MySQL module
• Nginx module
• Osquery module
• PostgreSQL module
• Redis module
• System module
• Traefik module
![Page 47: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/47.jpg)
4747
Metrics
● Metrics data sources
● Popular solutions
● Metricbeat
![Page 48: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/48.jpg)
48
Kubernetes Monitoring
• What to monitor– Cluster monitoring– Pod monitoring– Application monitoring
• Metrics sources– cAdvisor & Heapster– Kube-state-metrics– Prometheus– APM
• Solutions– Heapster/InfluxDB/Grafana– Heapster/Elasticsearch– Prometheus/Grafana– APM - Datadog, Dynatrace– Metricbeat with Autodiscovery
Collect Store Analyze
ElasticsearchInfluxDB...
KibanaGrafana...
MetricbeatHeapsterPrometheus...
SearchDashboardAlerts...
Data ModelMetrics Sources
![Page 49: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/49.jpg)
49
Comprehensive Metrics Collection Metricbeat
• Kubernetes module• Monitors pods and services
– Cluster, pod & container metrics– Application metrics through auto-discovery
(eg. Nginx)
• Metrics sources - Cover them ALL– Kubelet (heapster, cAdvisor)– kube-state-metric– Kubernetes events– Prometheus module (beta)
• Curated infra UI • Dedicated Kibana app
![Page 50: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/50.jpg)
50
Out-of-the-box Dashboards
![Page 51: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/51.jpg)
51
Curated UI for KubernetesVisualize the cluster and group by nodes or namespaces or pods
![Page 52: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/52.jpg)
52
Monitor Services inside Containers with Auto-Discovery
Metricbeat Filebeat
Node n
Logs
MetricsNginx
metricbeat.autodiscover:
providers:
- type: kubernetes
host: ${HOSTNAME}
templates:
- condition.contains:
kubernetes.container.name: nginx
config:
- module: nginx
period: 10s
metricsets: [" stubstatus"]
hosts: ["${data.host}:8080"]
![Page 53: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/53.jpg)
53
Metricbeat ModulesSimplify collection and visualization of common metrics
● Aerospike module● Apache module● Ceph module● Couchbase module● Docker module● Dropwizard module● Elasticsearch module● Etcd module● Golang module● Graphite module● HAProxy module● HTTP module
● Jolokia module● Kafka module● Kibana module● Kubernetes module● kvm module● Logstash module● Memcached module● MongoDB module● Munin module● MySQL module● Nginx module
● PHP_FPM module● PostgreSQL module● Prometheus module● RabbitMQ module● Redis module● System module● uwsgi module● vSphere module● Windows module● ZooKeeper module
![Page 54: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/54.jpg)
5454
Tracing
● Elastic APM
![Page 55: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/55.jpg)
55
Microservices Can Be ComplicatedMicroservice Architecture of Uber
https://dzone.com/articles/microservice-architecture-learn-build-and-deploy-a
![Page 56: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/56.jpg)
56
First Major Open Source APM SolutionAgents, Server, Dashboards
![Page 57: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/57.jpg)
57
APM Tracing - Transaction Waterfall View
![Page 58: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/58.jpg)
58
You can do MORE ...
• Enforce access policies with X-Pack Security
• Be notified about changes & problems with X-Pack Alerting
• Be smarter with X-Pack Machine Learning
• ...
Be Creative, the Sky is NOT even the Limit with Elastic!
![Page 59: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/59.jpg)
59
Cloud Native Computing Foundation
• https://www.cncf.io/projects/
Resource Monitoring solutions
• https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/
Log monitoring:
https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver/
https://kubernetes.io/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana/
Kubernetes Resources
![Page 60: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/60.jpg)
60
Questions you may ask
• How long time do you need to resolve performance issue with
your application?
• How easy is it to get, find and combine logs, metric and APM
data on your current solution?
• How many monitoring systems you need to maintain?
• Do you keep data in silos?
![Page 61: Three Pilars of Observability Kubernetes with Elastic Stack · Docker • Kubernetes ... APM adds end-user experience and application-level monitoring to the stack ... • Fluentd](https://reader035.vdocument.in/reader035/viewer/2022062505/5ec55e9d419eb03a82219676/html5/thumbnails/61.jpg)
Questions?