beyond averages
DESCRIPTION
When raw data becomes overwhelming, we turn to abstraction to understand our world. In our systems, the data is always overwhelming. Solutions like summary statistics have come to our rescue, and they are good—up to a point. In order to truly understand our systems, we need to know when and how to sidestep those abstractions,to get deep, detailed performance insight. In this brief diatribe inspired by John Rauser’s 2011 Velocity keynote “Look at Your Data”, I’ll explore techniques for visualizing the underlying structure of performance data and how this empowers drilling down to populations and individual samples in the data set. Video: http://www.youtube.com/watch?v=InyHBnd_chwTRANSCRIPT
BEYOND AVERAGESDan Kuebrich / appneta.com
A few of my favorite abstractions
•Abstraction lets us trade information for actionability
•Min, max, average, quantiles, stdev
•That’s a great trade!• ... right?
Averages: average at best
Averages: average at best
Averages: average at best
Averages: average at best
Percentiles: 1 of 100 slices
95%
Percentiles: 2 of 100 slices
95%
10%
Percentiles: 2 of 100 slices
95%
10%
Percentiles: 2 of 100 slices
95%
10%
Percentiles: 2 of 100 slices
95%
10%
Computers are hard
• Rarely do we have a single normal distribution underlying the data
• Different users, different requests, different resources, different instances, different times
Is there a place between Averageland and “A Beautiful Mind”?
http://now-here-this.timeout.com/2012/10/07/crazy-walls-of-clues-from-tv-film-reviewed-by-carrie-from-homeland/
HistogramsFr
eque
ncy
(eg.
# o
f cal
ls)
Value(eg. latency)
Populations revisited
95%
10%
HistogramsFr
eque
ncy
(eg.
# o
f cal
ls)
Value(eg. latency)
Populations re-revisited
95%
10%?
3d Histograms?Fr
eque
ncy
(eg.
# o
f cal
ls)
Value(eg. latency)
3d Histograms?Fr
eque
ncy
(eg.
# o
f cal
ls)
Value(eg. latency)
Time
HeatmapsFr
eque
ncy
(eg.
# o
f cal
ls)
Value(eg. latency)
HeatmapsFr
eque
ncy
(eg.
# o
f cal
ls)
Value(eg. latency)
HeatmapsFr
eque
ncy
(eg.
# o
f cal
ls)
Value(eg. latency)
HeatmapsVa
lue
(eg.
late
ncy)
Time
OK, but what about the real world?
http://www.justincarmony.com/blog/2012/06/05/customizing-graphite-charts-for-clearer-results/
Mystery #1
Mystery #1
Mystery #1
Mystery #1
Mystery #1
Mystery #2
Mystery #2
bottom 98%
Mystery #2
all of it
Mystery #3
Mystery #3: UNSOLVED