Download - Анатолий Кулаков «The Metrix has you…»
![Page 1: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/1.jpg)
Anatoly Kulakov
![Page 2: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/2.jpg)
1
![Page 3: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/3.jpg)
2
![Page 4: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/4.jpg)
Troubleshooting & Remediation- Where did the problem occur?
Performance & Cost- How my changes impact overall performance?
Learning & Improvement- Can I detect or prevent this problem in the future?
Trends- Do I need to scale?
Customer Experience- Are my customers getting a good experience?
3
![Page 6: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/6.jpg)
5
![Page 7: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/7.jpg)
6
![Page 8: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/8.jpg)
7
100 measurements200 hosts every 10 sec
× 86 400seconds in a day
172 800 000points per day
![Page 10: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/10.jpg)
9
MetricsLogs
![Page 12: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/12.jpg)
11
![Page 13: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/13.jpg)
12
![Page 14: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/14.jpg)
13
![Page 15: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/15.jpg)
14
![Page 16: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/16.jpg)
15
![Page 17: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/17.jpg)
16
![Page 18: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/18.jpg)
17
![Page 19: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/19.jpg)
18
![Page 20: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/20.jpg)
19
Timestamp
2017-11-12T06:42:17
2017-11-12T06:43:18
Fields
rx = 42tx = 10
rx = 50tx = 88
Tags
host = devif = eth1
host = devif = wlan1
Network
![Page 21: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/21.jpg)
20
Timestamp Tags Fields
Network
Primary Key Indexed Column Not Indexed Column
![Page 22: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/22.jpg)
21
Timestamp Tags Fields
2017-11-12T06:42:17 42.0173, 1.0, …dev, eth1, …
Network
DateTime string[] double
8 bytes ≈ 24 bytes 8 bytes
![Page 23: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/23.jpg)
22
![Page 24: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/24.jpg)
23
![Page 25: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/25.jpg)
24
Network
Tags host = devif = eth1
host = devif = wlan1
network,host=dev,if=eth1
network,host=dev,if=wlan1
![Page 26: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/26.jpg)
25
2017-11-12T06:00:002017-11-12T06:00:052017-11-12T06:00:102017-11-12T06:00:15
-050505
Delta
--00
Delta 2
«We have found that about 96% of all time stampscan be compressed to a single bit.»
http://www.vldb.org/pvldb/vol8/p1816-teller.pdf
![Page 27: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/27.jpg)
26
Decimal
15.5
14.0625
3.25
8.625
![Page 28: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/28.jpg)
27
Decimal Double Representation
15.5 0x402f000000000000
14.0625 0x402c200000000000
3.25 0x400a000000000000
8.625 0x4021400000000000
![Page 29: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/29.jpg)
28
Decimal Double Representation XOR with previous
15.5 0x402f000000000000
14.0625 0x402c200000000000 0x0003200000000000
3.25 0x400a000000000000 0x0026200000000000
8.625 0x4021400000000000 0x002b400000000000
«Roughly 51% of all values are compressed to a single bit»
http://www.vldb.org/pvldb/vol8/p1816-teller.pdf
«… compress time series to an average of 1.37 bytes per point»
![Page 30: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/30.jpg)
Performance Counters Third party statistics API Event Tracing for Windows Application measurements
29
![Page 31: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/31.jpg)
30
![Page 32: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/32.jpg)
31
![Page 33: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/33.jpg)
32
![Page 34: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/34.jpg)
33
App
Telegraf
App
App
AppTelegraf
AppTelegraf
![Page 35: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/35.jpg)
34
![Page 36: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/36.jpg)
35https://db-engines.com/en/ranking_trend/time+series+dbms
![Page 37: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/37.jpg)
Billions of individual data points High write throughput High read throughput Large deletes (data expiration) Mostly an insert/append workload, very few
updates
36
![Page 38: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/38.jpg)
37
![Page 39: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/39.jpg)
38
SELECT median(rx), mean(tx)FROM networkWHERE time > now() - 15mAND host = 'dev'
GROUP BY time(10s)
![Page 40: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/40.jpg)
39
Real time
10 sec
1 min
1 hour
![Page 41: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/41.jpg)
40
CPU: 4-6 coresRAM: 8-32 GBIOPS: 500-1000
![Page 42: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/42.jpg)
41
Load Field writes per second
Queries per second
Unique series
Low < 5 thousand < 5 < 100 thousand
Moderate < 250 thousand < 25 < 1 million
High > 250 thousand > 25 > 1 million
Infeasible > 750 thousand > 100 > 10 million
CPU: 4-6 coresRAM: 8-32 GBIOPS: 500-1000
https://docs.influxdata.com/influxdb/v1.3/guides/hardware_sizing/
![Page 43: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/43.jpg)
42
![Page 44: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/44.jpg)
Write Performance
InfluxDB outperformed:
• MongoDB by 27x• Cassandra by 5x• Elasticsearch by 8x• OpenTSDB by 5x
43
InfluxDB MongoDB Cassandra Elasticsearch OpenTSDB
https://www.influxdata.com/_resources/
![Page 45: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/45.jpg)
Compression
InfluxDB outperformed:
• MongoDB by 84x• Cassandra by 9x• Elasticsearch by 16x• OpenTSDB by 16x
44
InfluxDB MongoDB Cassandra Elasticsearch OpenTSDB
https://www.influxdata.com/_resources/
![Page 46: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/46.jpg)
Query Performance
InfluxDB outperformed:
• MongoDB similarly• Cassandra by 168x• Elasticsearch by 10x• OpenTSDB by 4x
45
InfluxDB MongoDB Cassandra Elasticsearch OpenTSDB
https://www.influxdata.com/_resources/
![Page 47: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/47.jpg)
46
![Page 48: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/48.jpg)
47
![Page 49: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/49.jpg)
Install Telegraf and Dashboard Install AppMetrics and Dashboard Use it Remove unnecessary metrics Add new application-specific metrics
48
![Page 50: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/50.jpg)
49
Demo powered by
![Page 51: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/51.jpg)
50
![Page 52: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/52.jpg)
51http://sarahwooders.blogspot.ru/2015/02/my-first-week-at-metropia-has-been.html
![Page 53: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/53.jpg)
52
![Page 54: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/54.jpg)
53
![Page 55: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/55.jpg)
54https://www.slideshare.net/nagarajc007/mobile-drunk-driver-detection
![Page 56: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/56.jpg)
55
Agent for collecting and reporting metrics
![Page 57: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/57.jpg)
56
Time series database
![Page 58: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/58.jpg)
57
User interface for:• monitoring• alert management• data visualization• db management
![Page 59: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/59.jpg)
58
Data processing framework for:• create alerts• run ETL jobs• detect anomalies
![Page 60: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/60.jpg)
59
![Page 62: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/62.jpg)
61
Query and Write performance
Compression
Realtime Analysis
Statistics and Aggregation
Retention Policy
Continuous Queries
Downsampling
High Loads
High Throughput
![Page 63: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/63.jpg)
62
Query and Write performance
Compression
Realtime Analysis
Statistics and Aggregation
Retention Policy
Continuous Queries
Downsampling
High Loads
High Throughput
![Page 64: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/64.jpg)
63
Compression
Realtime Analysis
Statistics and Aggregation
Retention Policy
Continuous Queries
Downsampling
High Loads
High Throughput
![Page 65: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/65.jpg)
64
Compression
Realtime Analysis
Statistics and Aggregation
Retention Policy
Continuous Queries
Downsampling
High Loads
![Page 66: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/66.jpg)
65
Compression
Realtime Analysis
Statistics and Aggregation
Retention Policy
Continuous Queries
High Loads
![Page 67: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/67.jpg)
66
Compression
Realtime Analysis
Statistics and Aggregation
Retention Policy
High Loads
![Page 68: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/68.jpg)
67
Compression
Realtime Analysis
Retention Policy
High Loads
![Page 69: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/69.jpg)
68
Compression
Realtime Analysis
High Loads
![Page 70: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/70.jpg)
69
Realtime Analysis
High Loads
![Page 71: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/71.jpg)
70
Realtime Analysis
![Page 72: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/72.jpg)
71
![Page 73: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/73.jpg)
72
![Page 74: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/74.jpg)
Gorilla Paper Akumuli Run-length encoding Varints, ZigZag Dynamic time warping Sketch-based change detection
73
![Page 75: Анатолий Кулаков «The Metrix has you…»](https://reader031.vdocument.in/reader031/viewer/2022021507/5a64b58b7f8b9aa6148b4629/html5/thumbnails/75.jpg)
InfluxData Docs (docs.influxdata.com)
Grafana Docs (docs.grafana.org)
App Metrics (app-metrics.io)
Non-Sucking Service Manager (nssm.cc)
74