metrics 2.0 @ monitorama pdx 2014
TRANSCRIPT
![Page 1: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/1.jpg)
![Page 2: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/2.jpg)
by niteroi @ panoramio.com
![Page 3: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/3.jpg)
vimeo.com/43800150
![Page 4: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/4.jpg)
![Page 5: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/5.jpg)
![Page 6: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/6.jpg)
![Page 7: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/7.jpg)
problems
Metrics 2.0 concepts
implementations & examples
![Page 8: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/8.jpg)
Mostly
graphite
![Page 9: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/9.jpg)
terminology
sync
![Page 10: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/10.jpg)
(1234567890, 82)
(1234567900, 123)
(1234567910, 109)
(1234567920, 77)
db15.mysql.queries_running
host=db15 mysql.queries_running
![Page 11: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/11.jpg)
Problems
![Page 12: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/12.jpg)
Vimeo.com pagerequests/s?
server X write perf?
![Page 13: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/13.jpg)
Finding metrics
Browse hierarchies
Dashboard search .. which keywords?
Search in source code/documentation?
Ask around
...
![Page 14: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/14.jpg)
stats.hits.vimeo_com
stats_counts.hits.vimeo_com
stats.*.requesthostport.
vimeo_com_80
![Page 15: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/15.jpg)
Meaning, difference
Unit?
Where and how.. hard
Prefixes
Understanding metrics
![Page 16: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/16.jpg)
collectd.db.disk.sda1.disk_time.write
![Page 17: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/17.jpg)
Terminology? Which field is where?
Total so far? From zero per datapoint?
Aggregate? Which?
Point at t=x describes which timeframe?
Understanding metrics
![Page 18: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/18.jpg)
Change agent?
![Page 19: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/19.jpg)
Unclear, inconsistent terminology, format
tightly coupled
lack information
![Page 20: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/20.jpg)
O(S*P*A) S = # Sources
P = # People
A = # Aggregators
![Page 21: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/21.jpg)
![Page 22: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/22.jpg)
![Page 23: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/23.jpg)
times
N
![Page 24: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/24.jpg)
graph definitions are redundant and a time sink.
![Page 25: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/25.jpg)
![Page 26: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/26.jpg)
http://litlquest.com/forest-trees/see-forest-trees-2
![Page 27: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/27.jpg)
metrics 2.0
concepts
![Page 28: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/28.jpg)
Self-describing
Standardized
Orthogonal dimensions
![Page 29: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/29.jpg)
stats.timers.dfs5.proxy-server.object.GET.200.
timing.upper_90
![Page 30: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/30.jpg)
{
server: dfvimeodfsproxy5,
http_method: GET,
http_code: 200,
unit: ms,
metric_type: gauge,
stat: upper_90,
swift_type: object
}
![Page 31: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/31.jpg)
allow more characters
unit: Req/s, site: vimeo.com, ...
![Page 32: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/32.jpg)
Metadata
meta: {
src: proxy.py:458,
from: diamond
}
![Page 33: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/33.jpg)
Conceptual model vs
wire protocol vs
storage
![Page 34: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/34.jpg)
metrics20.org
![Page 35: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/35.jpg)
SI + IEC
B Err Warn ConnJob File Req ...
MB/s Err/dReq/h ...
![Page 36: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/36.jpg)
Immediate understanding
of metrics
Minimize time to graphs,
alerting rules, debugging
compatibility & flexibility
in tooling
![Page 37: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/37.jpg)
Implementations & examples
![Page 38: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/38.jpg)
![Page 39: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/39.jpg)
Carbon-tagger
…stats.gauges.host.foo 125 1234567890
service=foo instance=host target_type=gauge unit=B 123 1234567890
…
![Page 40: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/40.jpg)
![Page 41: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/41.jpg)
Statsdaemon
unit=B
unit=B
...
unit=ms
unit=ms
...
unit=B/s
unit=ms stat=meanunit=ms stat=upper_90...
![Page 42: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/42.jpg)
Keep metric
tags in sync with data
![Page 43: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/43.jpg)
GraphExplorer
![Page 44: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/44.jpg)
![Page 45: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/45.jpg)
GraphExplorer queries 101
proxy-server swift server:regex unit=ms
(AND)
![Page 46: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/46.jpg)
![Page 47: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/47.jpg)
![Page 48: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/48.jpg)
![Page 49: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/49.jpg)
![Page 50: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/50.jpg)
![Page 51: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/51.jpg)
![Page 52: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/52.jpg)
![Page 53: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/53.jpg)
upper_90 (or stat=upper_90)
from <datetime>to <datetime>
avg over <timespec>(5M, 1h, 3d, ...)
![Page 54: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/54.jpg)
Compare object put/get
stack …
http_method:(PUT|GET)
swift_type=object
avg by http_code,server
![Page 55: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/55.jpg)
![Page 56: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/56.jpg)
Comparing servers
http_method:(PUT|GET)
group by unit,target_type
avg by http_code,swift_type,http_method
![Page 57: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/57.jpg)
![Page 58: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/58.jpg)
transcode unit=Job/savg over <time>
from <datetime> to <datetime>
![Page 59: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/59.jpg)
Note: data is obfuscated
![Page 60: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/60.jpg)
Bucketing
sum by zone:eu-west|us-east|ap-southeast|us-west|
sa-east|vimeo-df|vimeo-lv
group by state
![Page 61: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/61.jpg)
Note: data is obfuscated
![Page 62: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/62.jpg)
Compare job states per region (zones bucket)
group by zone
![Page 63: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/63.jpg)
Note: data is obfuscated
![Page 64: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/64.jpg)
Unit conversion
unit=Mb/s network server:regexsum by server
![Page 65: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/65.jpg)
![Page 66: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/66.jpg)
![Page 67: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/67.jpg)
Integration
Metric unit=B/s Query unit=TB
![Page 68: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/68.jpg)
![Page 69: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/69.jpg)
Deriving
Metric unit=BQuery unit=GB/d
![Page 70: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/70.jpg)
![Page 71: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/71.jpg)
Bonus round
![Page 72: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/72.jpg)
![Page 73: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/73.jpg)
![Page 74: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/74.jpg)
![Page 75: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/75.jpg)
![Page 76: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/76.jpg)
![Page 77: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/77.jpg)
![Page 78: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/78.jpg)
![Page 79: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/79.jpg)
![Page 80: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/80.jpg)
Dashboard definition
queries = [
'cpu usage sum by core',
'mem unit=B !total group by type:swap',
'stack network unit=Mb/s',
'unit=B (free|used) group by =mountpoint'
]
![Page 81: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/81.jpg)
![Page 82: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/82.jpg)
![Page 83: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/83.jpg)
![Page 84: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/84.jpg)
Future Work
![Page 85: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/85.jpg)
● Storage aggregation rules
● graphite API functions such as cumulative, summarize and smartSummarize
●consolidateBy & Graph renderers
![Page 86: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/86.jpg)
![Page 87: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/87.jpg)
Self-describing & standardized
stat=upper/lower/mean/...target_type=counter..
![Page 88: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/88.jpg)
Select your view
![Page 89: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/89.jpg)
From: dygraphs.com
![Page 90: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/90.jpg)
Facet based suggestions
![Page 91: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/91.jpg)
unit=Err/s
![Page 92: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/92.jpg)
Conclusion
structuredselfdescribing standardized
metrics = enabler
![Page 93: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/93.jpg)
Conclusion
Manual composing should be last resort, not default
![Page 94: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/94.jpg)
Conclusion
This sucks– Tell me why– What should we do instead?
This is neat!– Help me make it better– Adopt native metrics 2.0, structured_metrics
![Page 95: Metrics 2.0 @ Monitorama PDX 2014](https://reader034.vdocument.in/reader034/viewer/2022050817/554f92f5b4c9052a518b54a7/html5/thumbnails/95.jpg)
Seen in this presentation:metrics20.org
vimeo.github.io/graph-explorer
github.com/vimeo/timeserieswidget
github.com/vimeo/carbon-tagger
github.com/vimeo/statsdaemon
github.com/Dieterbe/anthracite
github.com/graphite-ng
github.com/vimeo/graphite-influxdbgithub.com/vimeo/smoketcp
github.com/vimeo/tailgate
twitter.com/Dieter_bedieter.plaetinck.be
You might also like: