the statistics of web performance analysis

• Philip Tellis

• .com• [email protected]

• @bluesmoon• geek paranoid speedfreak• http://bluesmoon.info/

Boston #WebPerf Meetup / 2012-08-14 The Statistics of Web Performance Analysis 1

http://www.lognormal.com/

http://twitter.com/bluesmoon

http://bluesmoon.info/

http://www.meetup.com/Web-Performance-Boston/events/75714302/

I’m a Web Speedfreak



We measure real user website performance



This talk is about the Statistics we learned while building it



The Statistics of Web Performance Analysis

Philip Tellis / [email protected]

Boston #WebPerf Meetup / 2012-08-14




0Numbers



Accurately measure page performance∗



Be unintrusive

If you try to measure something accurately, you will changesomething related

– Heisenberg’s uncertainty principle



And one number to rule them all



What do we measure?

• Network Throughput• Network Latency• User perceived page load time



We measure real user data



Which is noisy



1Statistics - 1



Disclaimer

I am not a statistician



1-1Random Sampling



Population

All possible users of your system



Sample

Representative subset of the population



Bad sample

Sometimes it’s not



How to randomize?

http://xkcd.com/221/


http://xkcd.com/221/


How to randomize?

• Pick 10% of users at random and always test them

OR

• For each user, decide at random if they should be tested

http://tech.bluesmoon.info/2010/01/statistics-of-performance-measurement.html


http://tech.bluesmoon.info/2010/01/statistics-of-performance-measurement.html


Select 10% of users - I

if($sessionid % 10 === 0) {// instrument code for measurement

}

• Once a user enters the measurement bucket, they staythere until they log out

• Fixed set of users, so tests may be more consistent• Error in the sample results in positive feedback



Select 10% of users - II

if(rand() < 0.1 * getrandmax()) {// instrument code for measurement

}

• For every request, a user has a 10% chance of beingtested

• Gets rid of positive feedback errors, but sample size !=10% of population



How big a sample is representative?

Select n such that∣∣∣1.96 σ√n

∣∣∣ ≤ 5%µ



1-2Margin of Error



Standard Deviation

• Standard deviation tells you the spread of the curve• The narrower the curve, the more confident you can be



MoE at 95% confidence

±1.96 σ√n



MoE & Sample size

There is an inverse square root correlation between sample sizeand margin of error



1-3Central Tendency



One number

• Mean (Arithmetic)• Good for symmetric curves• Affected by outliers

Mean(10, 11, 12, 11, 109) = 30



One number

• Median• Middle value measures central tendency well• Not trivial to pull out of a DB

Median(10, 11, 12, 11, 109) = 11



One number

• Mode• Not often used• Multi-modal distributions suggest problems

Mode(10, 11, 12, 11, 109) = 11



Other numbers

• A percentile point in the distribution: 95th, 98.5th or 99th

• Used to find out the worst user experience• Makes more sense if you filter data first

P95th(10, 11, 12, 11, 109) = 12



Other means

• Geometric mean• Good if your data is exponential in nature

(with the tail on the right)

GMean(10, 11, 12, 11, 109) = 16.68



Wait... how did I get that?

N

√ΠN

i=1xi — could lead to overflow

e

(ΣN

i=1 loge(xi )N

)— computationally simpler



Other means

And there is also the Harmonic mean, but forget about that



...though consequently

We have other margins of error• Geometric margin of error

• Uses geometric standard deviation• Median margin of error

• Uses ranges of actual values from data set

• Stick to the arithmetic MoE– simpler to calculate, simpler to read and not incorrect



2Statistics - 2



2-1Distributions



Let’s look at some real charts



Sparse Distribution



Log-normal distribution



Bimodal distribution



What does all of this mean?



Distributions

• Sparse distribution suggests that you don’t have enoughdata points

• Log-normal distribution is typical• Bi-modal distribution suggests two (or more) distributions

combined



In practice, a bi-modal distribution is not uncommon



Hint: Does your site do a lot of back-end caching?



2-2Filtering



Outliers

• Out of range data points• Nothing you can fix here• There’s even a book about

them



DNS problems can cause outliers

• 2 or 3 DNS servers for an ISP• 30 second timeout if first fails• ... 30 second increase in page load time• Maybe measure both and fix what you can• http://nms.lcs.mit.edu/papers/dns-ton2002.pdf


http://nms.lcs.mit.edu/papers/dns-ton2002.pdf


Band-pass filtering



Band-pass filtering

• Strip everything outside a reasonable range• Bandwidth range: 4kbps - 4Gbps• Page load time: 50ms - 120s

• You may need to relook at the ranges all the time



IQR filtering



IQR filtering

Here, we derive the range from the data



Further Reading

lognormal.com/blog/2012/08/13/analysing-performance-data/


http://www.lognormal.com/blog/2012/08/13/analysing-performance-data/


Summary

• Choose a reasonable sample size and sampling factor• Tune sample size for minimal margin of error• Decide based on your data whether to use mode, median

or one of the means• Figure out whether your data is Normal, Log-Normal or

something else• Filter out anomalous outliers



• Philip Tellis

• .com• [email protected]

• @bluesmoon• geek paranoid speedfreak• http://bluesmoon.info/


http://www.lognormal.com/

http://twitter.com/bluesmoon

http://bluesmoon.info/


Thank you



Photo credits

• http://www.flickr.com/photos/leoffreitas/332360959/ by leoffreitas

• http://www.flickr.com/photos/cobalt/56500295/ by cobalt123

• http://www.flickr.com/photos/sophistechate/4264466015/ by LisaBrewster


http://www.flickr.com/photos/leoffreitas/332360959/

http://www.flickr.com/photos/cobalt/56500295/

http://www.flickr.com/photos/sophistechate/4264466015/


List of figures

• http://en.wikipedia.org/wiki/File:Standard_deviation_diagram.svg

• http://en.wikipedia.org/wiki/File:Normal_Distribution_PDF.svg

• http://en.wikipedia.org/wiki/File:KilroySchematic.svg

• http://en.wikipedia.org/wiki/File:Boxplot_vs_PDF.png


http://en.wikipedia.org/wiki/File:Standard_deviation_diagram.svg

http://en.wikipedia.org/wiki/File:Normal_Distribution_PDF.svg

http://en.wikipedia.org/wiki/File:KilroySchematic.svg

http://en.wikipedia.org/wiki/File:Boxplot_vs_PDF.png


the statistics of web performance analysis

Technology

webperf meetup

web speedfreakboston

performancemeasurem

population boston

system boston

positive feedback boston

margin of error boston

moe sample size