the statistics of web performance

IntroductionStatistics - IStatistics - II

The Statistics of Web Performance

Philip Tellis / [email protected]

ConFoo / 2010-03-12

ConFoo / 2010-03-12 The Statistics of Web Performance

http://confoo.ca/

http://confoo.ca/


$ finger philip

Philip [email protected]

@bluesmoonyahoogeek


http://confoo.ca/


The goalPerformance Measurement

Introduction


http://confoo.ca/



Accurately measure page performanceAt least, as accurately as possible


http://confoo.ca/



Be unintrusive

If you try to measure something accurately, you will changesomething related

– Heisenberg’s uncertainty principle


http://confoo.ca/



And one number to rule them all


http://confoo.ca/



Bandwidth

Real bandwidth v/s advertised bandwidthBandwidth to your server, not to the ISPBandwidth during normal internet usage

If the user’s always watching movies, you’re not winning


http://confoo.ca/



Latency

How long does it take a byte to get to the user?Wired, wireless, mobile, satellite?How many hops in between?Speed of light is constant

This is not a battle we will soon win.When was the last time you heard latency mentioned in aTV ad?

http://www.stuartcheshire.org/rants/Latency.html


http://www.stuartcheshire.org/rants/Latency.html

http://confoo.ca/



User perceived page load time

Time from “click on a link” to “spinner stops spinning”This is what users notice

Depends on how long your page takes to buildDepends on what’s in your pageDepends on how long components take to loadDepends on how long the browser takes to execute andrender


http://confoo.ca/



We need to measure real user data


http://confoo.ca/



The statistics apply to any kind of performance data though


http://confoo.ca/


Random SamplingMargin of ErrorCentral Tendency

Statistics - I


http://confoo.ca/



Disclaimer

I am not a statistician


http://confoo.ca/



Population

All possible users of your system


http://confoo.ca/



Sample

Representative subset of the population


http://confoo.ca/



Bad sample

Sometimes it’s not


http://confoo.ca/



How to randomize?

Pick 10% of users at random and always test them

OR

For each user, decide at random if they should be tested

http://tech.bluesmoon.info/2010/01/statistics-of-performance-measurement.html



http://confoo.ca/



Select 10% of users - I

if($sessionid % 10 === 0) {// instrument code for measurement

}

Once a user enters the measurement bucket, they staythere until they log outFixed set of users, so tests may be more consistentError in the sample results in positive feedback


http://confoo.ca/



Select 10% of users - II

if(rand() < 0.1 * getrandmax()) {// instrument code for measurement

}

For every request, a user has a 10% chance of beingtestedGets rid of positive feedback errors, but sample size !=10% of population


http://confoo.ca/



How big a sample is representative?

Select n such that∣∣∣1.96 σ√n

∣∣∣ ≤ 5%µ


http://confoo.ca/



Standard Deviation

Standard deviation tells you the spread of the curveThe narrower the curve, the more confident you can be


http://confoo.ca/



MoE at 95% confidence

±1.96 σ√n


http://confoo.ca/



MoE & Sample size

There is an inverse square root correlation between samplesize and margin of error


http://confoo.ca/



But wait... it’s not complicated enough.We have different types of margins of error...more about that later


http://confoo.ca/



Ding dong


http://confoo.ca/



One number

Mean (Arithmetic)Good for symmetric curvesAffected by outliers

Mean(10, 11, 12, 11, 109) = 30


http://confoo.ca/



One number

MedianMiddle value measures central tendency wellNot trivial to pull out of a DB

Median(10, 11, 12, 11, 109) = 11


http://confoo.ca/



One number

ModeNot often usedMulti-modal distributions suggest problems

Mode(10, 11, 12, 11, 109) = 11


http://confoo.ca/



Other numbers

A percentile point in the distribution: 95th, 98.5th or 99th

Used to find out the worst user experienceMakes more sense if you filter data first

P95th(10, 11, 12, 11, 109) = 12


http://confoo.ca/



Other means

Geometric meanGood if your data is exponential in nature(with the tail on the right)

GMean(10, 11, 12, 11, 109) = 16.68


http://confoo.ca/



Wait... how did I get that?

N

√ΠN

i=1xi — could lead to overflow

e

(ΣN

i=1 loge(xi )N

)— computationally simpler


http://confoo.ca/



Other means

And there is also the Harmonic mean, but forget about that


http://confoo.ca/



...though consequently

We have other margins of errorGeometric margin of error

Uses geometric standard deviationMedian margin of error

Uses ranges of actual values from data set

Stick to the arithmetic MoE– simpler to calculate, simpler to read and not incorrect


http://confoo.ca/


FilteringThe Log-Normal distribution

Statistics - II


http://confoo.ca/



Outliers

Out of range data pointsNothing you can fix hereThere’s even a book aboutthem


http://confoo.ca/



DNS problems can cause outliers

2 or 3 DNS servers for an ISP30 second timeout if first fails... 30 second increase in page load timeMaybe measure both and fix what you canhttp://nms.lcs.mit.edu/papers/dns-ton2002.pdf


http://nms.lcs.mit.edu/papers/dns-ton2002.pdf

http://confoo.ca/



Band-pass filtering


http://confoo.ca/



Band-pass filtering

Strip everything outside a reasonable rangeBandwidth range: 4kbps - 4GbpsPage load time: 50ms - 120s

You may need to relook at the ranges all the time


http://confoo.ca/



IQR filtering


http://confoo.ca/



IQR filtering

Here, we derive the range from the data


http://confoo.ca/



Let’s look at some real charts


http://confoo.ca/



Bandwidth distribution for web devs

x-axis is linear


http://confoo.ca/



Now let’s use log(kbps) instead of kbps

x-axis is exponential


http://confoo.ca/



Exponential == Geometric

Categories/Buckets grow exponentiallyData is related geometricallyUse the geometric mean and geometric margin of error

Error_range =[

gmean/gmoe, gmean ∗ gmoe]

Non-linear ranges are hard for humans to grok


http://confoo.ca/


So...


http://confoo.ca/


Further reading

Web Performance - Not a Simple Numberhttp://www.netforecast.com/Articles/BCR+C25+Web+Performance+-+Not+A+Simple+Number.pdf

Revisiting statistics for web performance (introduction toLog-Normal)http://home.pacbell.net/ciemo/statistics/WhatDoYouMean.pdf

Random Samplinghttp://tech.bluesmoon.info/2010/01/statistics-of-performance-measurement.html

Khan Academy’s tutorials on statisticshttp://khanacademy.com/

Learning about Statistical Learninghttp://measuringmeasures.blogspot.com/2010/01/learning-about-statistical-learning.html

Wikipedia articles on Random Sampling, Central Tendency,Standard Error, Confounding, Means and IQR


http://www.netforecast.com/Articles/BCR+C25+Web+Performance+-+Not+A+Simple+Number.pdf

http://home.pacbell.net/ciemo/statistics/WhatDoYouMean.pdf


http://khanacademy.com/

http://measuringmeasures.blogspot.com/2010/01/learning-about-statistical-learning.html

http://en.wikipedia.org/wiki/Random_sample

http://en.wikipedia.org/wiki/Central_tendency

http://en.wikipedia.org/wiki/Standard_error_%28statistics%29

http://en.wikipedia.org/wiki/Confounding

http://en.wikipedia.org/wiki/Mean

http://en.wikipedia.org/wiki/IQR

http://confoo.ca/


Summary

Choose a reasonable sample size and sampling factorTune sample size for minimal margin of errorDecide based on your data whether to use mode, medianor one of the meansFigure out whether your data is Normal, Log-Normal orsomething elseFilter out anomalous outliers


http://confoo.ca/


contact me

Philip [email protected]

bluesmoon.info@bluesmoon


http://bluesmoon.info/

http://confoo.ca/


Photo credits

http://www.flickr.com/photos/leoffreitas/332360959/ by leoffreitas

http://www.flickr.com/photos/cobalt/56500295/ by cobalt123

http://www.flickr.com/photos/sophistechate/4264466015/ by LisaBrewster

http://www.flickr.com/photos/nchoz/243216008/ by nchoz


http://www.flickr.com/photos/leoffreitas/332360959/

http://www.flickr.com/photos/cobalt/56500295/

http://www.flickr.com/photos/sophistechate/4264466015/

http://www.flickr.com/photos/nchoz/243216008/

http://confoo.ca/


List of figures

http://en.wikipedia.org/wiki/File:Standard_deviation_diagram.svg

http://en.wikipedia.org/wiki/File:Normal_Distribution_PDF.svg

http://en.wikipedia.org/wiki/File:KilroySchematic.svg

http://en.wikipedia.org/wiki/File:Boxplot_vs_PDF.png


http://en.wikipedia.org/wiki/File:Standard_deviation_diagram.svg

http://en.wikipedia.org/wiki/File:Normal_Distribution_PDF.svg

http://en.wikipedia.org/wiki/File:KilroySchematic.svg

http://en.wikipedia.org/wiki/File:Boxplot_vs_PDF.png

http://confoo.ca/

the statistics of web performance

Technology

introduction statistics

goal statistics

error statistics

central tendency statistics

page performance

kind of performance

introduction confoo

possible confoo