edge 2016 measuring what matters

Measuring What Matters

© AKAMAI - EDGE 2016

Introductions

Cliff CrockerAkamai

Buddy BrewerSOASTA


Three eras of modern performance measurement

2001-2006 Basically one browser

2007-2010 Transitional period

2011-2016 WTF just happened


2001-2006: Internet Explorer 6

2001 - 2006


2007-2010: Transitional period

2007

IE7iPhoneFirefox 2

2008 2009 2010



2007

IE7iPhoneFirefox 2

2008

IE7iPhone 3GFirefox 3AndroidChrome 1

2009 2010



2007

IE7iPhoneFirefox 2

2008


2009

IE8iPhone 3GSFirefox 3Android 2.1Chrome 1

2010



2007

IE7iPhoneFirefox 2

2008


2009

IE8iPhone 3GSFirefox 3Android 2.1Chrome 1

2010

IE8iPhone 3GSFirefox 3Android 2.1Chrome 1iPadNav Timing


2011-2016: WTF just happened

?How many browsers in a typical Thanksgiving weekend?


2011-2016: WTF just happened

572How many browsers in a typical Thanksgiving weekend?


Meanwhile...


So what can I measure?


What is available today - Synthetic Tooling

ProsCPU metricsTest environments & other places without real usersAccess to everything including headers

ConsGaps in: Geos, browsers, pagesImpossible to precisely match user demo- & technographicsNo way to realistically measure browser cache impact


What is available today - RUM

ProsNear perfect coverage of browsers, geos, pagesAbility to understand impact to performanceRealistic measurement of browser cache impact

ConsGaps in: Geos, browsers, pagesImpossible to precisely match user demo- & technographicsNo access to HTTP header data


EngagementRevenueCoverageCaching

CPUPre-prod

Availability

Page detailsResourcesHeaders

SyntheticReal


https://www.w3.org/wiki/Web_Performance/Publications

W3C - Performance Working Group


Navigation Timing API


Resource Timing API


Coming soon to a browser near you! Measure the size of

assets in addition to performance.

Resource Timing (2.0) - Transfer Size


● Measure performance using high-precision timestamps

● Consists of “marks” and “measures”● PerformanceMark: Timestamp● PerformanceMeasure: Duration between two given

marks

Browser Support - User Timing API


Browser Support - Navigation Timing API


Browser Support - Resource Timing API


Browser Support - User Timing API


So what should I measure?


What matters depends on you


What matters to an ops engineer?

● Preference for binary metrics● Looks for indicators of system performance● Affinity to network and back-end metrics

MTTR



Network

Server



Network

Server

function getPerf() { var timing = window.performance.timing; return { dns: timing.domainLookupEnd - timing.domainLookupStart, connect: timing.connectEnd - timing.connectStart };}

ttfb: timing.responseStart - timing.connectEnd};


Help answers questions like...

● Do I have an issue with an upstream DNS resolver?● Am I having Tier1 connectivity issues?● How fast is my application server responding?



Don’t be misled by “super fast” responses● DNS Caching● Prefetching● Connection reuse● Pages served from browser cache



Object Level Timing

● Resource timing API● Binary metric for testing infrastructure● Measure performance of a third-party● CDN healthcheck



Object Timing

var rUrl = ‘https://edge.akamai.com/ec/us/multimedia/images/edge/akamai-edge-conference-speaker-buddy-brewer.jpg’;var me = performance.getEntriesByName(rUrl)[0];var timings = { loadTime: me.duration, dns: me.domainLookupEnd - me.domainLookupStart, tcp: me.connectEnd - me.connectStart, waiting: me.responseStart - me.requestStart, fetch: me.responseEnd - me.responseStart}


What matters to a front-end developer?

Optimized Load Times

● “Do no harm”● Looks for indicators of application performance● Looks to measure a “fully loaded” page● Affinity for front-end metrics



Client



function getPerf() { var timing = window.performance.timing; return { Basepage: timing.responseEnd - timing.responseStart, DLoading: timing.domLoading – timing.navigationStart, DInt: timing.domIneractive – timing.navigationStart, DContLoaded: timing.domContentLoadedEventEnd – timing.navigationStart, DContLoadTime: timing.domContentLoadedEventEnd – timing.domContentLoadedEventStart, DComplete: timing.domComplete - timing.navigationStart, PLoad: timing.loadEventStart - timing.navigationStart

}; }

Client


Be wary of onload...

~1s OnLoad ~3.5s visibly complete


What matters to a designer?

UX

● Performance by design (D4P)● Looks for indicators of human perception● How quickly the page renders● Affinity for custom metrics and render metrics


RUM first paint and start render

Chrome - “First Paint” Visual Start Render


User perception

Courtesy of the filament group: https://www.filamentgroup.com/lab/weight-wait.html


Custom Metrics


Custom Metrics - User Timing API

performance.mark(“startTask”);//Some stuff you want to measure happens hereperformance.mark(“stopTask”);

//Measure the duration between the two marksperformance.measure(“taskDuration”,“startTask”,“stopTask”);

//Measure the load of an image<img src=”image1.gif” onload=”performance.mark(‘image1’)”>


What matters to a business executive?

$$$$$$$● Business impact analysis● Simplified/lean metrics● Actionable data


What matters to a business executive?


Don’t create unintended silos

Biz

Ops

Dev

Design


Don’t create unintended silosThe success of my business is a reflection of...

The digital experience I provide, which depends on...

how quickly I can render content in the browser, which is impacted by...

the performance and stability of my infrastructure.

1

2

3

4

Biz

Design

Ops

Dev


“I just want to buy my Mom’s birthday present.”

“I just want to know if it will rain today.”

“I just want to share pictures of my vacation”


How do I collect it?


Real users are not normal

What do I do with it?


2.9s - p50 (median)

4.4s - Average

10.4s - p95



What should I throw out?


Load Time = 3500 years


Load Time = -(10years)


What should I keep?


Other opportunities for bots...

1. How does performance impact bot behavior?2. Are bots skewing the numbers I’m reporting?3. Am I providing an optimal experience for search bots?4. Does any of this impact SEO?


Summary

● We have amazing tools at our disposal - your users are wired for sound!

● Measuring what matters is situation independent● Important to look at all metrics from the same context● Use robust statistics when looking at your data

edge 2016 measuring what matters

Technology