edge 2016 measuring what matters

Measuring What Matters

Introductions

Cliff CrockerAkamai

Buddy BrewerSOASTA

Three eras of modern performance measurement

2001-2006 Basically one browser

2007-2010 Transitional period

2011-2016 WTF just happened

2001-2006: Internet Explorer 6

2001 - 2006

2007-2010: Transitional period

IE7iPhoneFirefox 2

2008 2009 2010

IE7iPhoneFirefox 2

IE7iPhone 3GFirefox 3AndroidChrome 1

2009 2010

IE7iPhoneFirefox 2

IE8iPhone 3GSFirefox 3Android 2.1Chrome 1

IE7iPhoneFirefox 2

IE8iPhone 3GSFirefox 3Android 2.1Chrome 1

IE8iPhone 3GSFirefox 3Android 2.1Chrome 1iPadNav Timing

2011-2016: WTF just happened

?How many browsers in a typical Thanksgiving weekend?

2011-2016: WTF just happened

572How many browsers in a typical Thanksgiving weekend?

Meanwhile...

So what can I measure?

What is available today - Synthetic Tooling

ProsCPU metricsTest environments & other places without real usersAccess to everything including headers

ConsGaps in: Geos, browsers, pagesImpossible to precisely match user demo- & technographicsNo way to realistically measure browser cache impact

What is available today - RUM

ProsNear perfect coverage of browsers, geos, pagesAbility to understand impact to performanceRealistic measurement of browser cache impact

ConsGaps in: Geos, browsers, pagesImpossible to precisely match user demo- & technographicsNo access to HTTP header data

EngagementRevenueCoverageCaching

CPUPre-prod

Availability

Page detailsResourcesHeaders

SyntheticReal

https://www.w3.org/wiki/Web_Performance/Publications

W3C - Performance Working Group

Navigation Timing API

Resource Timing API

Coming soon to a browser near you! Measure the size of

assets in addition to performance.

Resource Timing (2.0) - Transfer Size

● Measure performance using high-precision timestamps

● Consists of “marks” and “measures”● PerformanceMark: Timestamp● PerformanceMeasure: Duration between two given

Browser Support - User Timing API

Browser Support - Navigation Timing API

Browser Support - Resource Timing API

Browser Support - User Timing API

So what should I measure?

What matters depends on you

What matters to an ops engineer?

● Preference for binary metrics● Looks for indicators of system performance● Affinity to network and back-end metrics

Network

Server

Network

Server

function getPerf() { var timing = window.performance.timing; return { dns: timing.domainLookupEnd - timing.domainLookupStart, connect: timing.connectEnd - timing.connectStart };}

ttfb: timing.responseStart - timing.connectEnd};

Help answers questions like...

● Do I have an issue with an upstream DNS resolver?● Am I having Tier1 connectivity issues?● How fast is my application server responding?

Don’t be misled by “super fast” responses● DNS Caching● Prefetching● Connection reuse● Pages served from browser cache

Object Level Timing

● Resource timing API● Binary metric for testing infrastructure● Measure performance of a third-party● CDN healthcheck

Object Timing

var rUrl = ‘https://edge.akamai.com/ec/us/multimedia/images/edge/akamai-edge-conference-speaker-buddy-brewer.jpg’;var me = performance.getEntriesByName(rUrl)[0];var timings = { loadTime: me.duration, dns: me.domainLookupEnd - me.domainLookupStart, tcp: me.connectEnd - me.connectStart, waiting: me.responseStart - me.requestStart, fetch: me.responseEnd - me.responseStart}

What matters to a front-end developer?

Optimized Load Times

● “Do no harm”● Looks for indicators of application performance● Looks to measure a “fully loaded” page● Affinity for front-end metrics

Client

function getPerf() { var timing = window.performance.timing; return { Basepage: timing.responseEnd - timing.responseStart, DLoading: timing.domLoading – timing.navigationStart, DInt: timing.domIneractive – timing.navigationStart, DContLoaded: timing.domContentLoadedEventEnd – timing.navigationStart, DContLoadTime: timing.domContentLoadedEventEnd – timing.domContentLoadedEventStart, DComplete: timing.domComplete - timing.navigationStart, PLoad: timing.loadEventStart - timing.navigationStart

Client

Be wary of onload...

~1s OnLoad ~3.5s visibly complete

What matters to a designer?

● Performance by design (D4P)● Looks for indicators of human perception● How quickly the page renders● Affinity for custom metrics and render metrics

RUM first paint and start render

Chrome - “First Paint” Visual Start Render

User perception

Courtesy of the filament group: https://www.filamentgroup.com/lab/weight-wait.html

Custom Metrics

Custom Metrics - User Timing API

performance.mark(“startTask”);//Some stuff you want to measure happens hereperformance.mark(“stopTask”);

//Measure the duration between the two marksperformance.measure(“taskDuration”,“startTask”,“stopTask”);

//Measure the load of an image<img src=”image1.gif” onload=”performance.mark(‘image1’)”>

What matters to a business executive?

$$$$$$$● Business impact analysis● Simplified/lean metrics● Actionable data

Don’t create unintended silos

Design

Don’t create unintended silosThe success of my business is a reflection of...

The digital experience I provide, which depends on...

how quickly I can render content in the browser, which is impacted by...

the performance and stability of my infrastructure.

Design

“I just want to buy my Mom’s birthday present.”

“I just want to know if it will rain today.”

“I just want to share pictures of my vacation”

How do I collect it?

Real users are not normal

What do I do with it?

2.9s - p50 (median)

4.4s - Average

10.4s - p95

What should I throw out?

Load Time = 3500 years

Load Time = -(10years)

What should I keep?

Other opportunities for bots...

1. How does performance impact bot behavior?2. Are bots skewing the numbers I’m reporting?3. Am I providing an optimal experience for search bots?4. Does any of this impact SEO?

Summary

● We have amazing tools at our disposal - your users are wired for sound!

● Measuring what matters is situation independent● Important to look at all metrics from the same context● Use robust statistics when looking at your data

edge 2016 measuring what matters

Technology

measuring what matters -- t ’s new approach...

measuring what matters presentation svhr - …€¦ ·...

measuring what matters workshop report

saas metrics – measuring and improving what matters

miami matters: measuring what matters in miami-dade county

measuring what matters

measuring what matters: noncognitive skills - grit

sustainable agriculture: measuring what matters

nef measuring what matters

measuring what matters in social media

measuring what matters handbook

measuring what matters: effective performance...

measuring what matters - palm beach state

measuring what matters: -

whitepaper: start measuring what matters - happiest minds

measuring what matters - cjr.ufv.ca

new metrics, new opportunities - measuring what matters

socialbakers: measuring what matters, social matters hk 2014

national environmental indicators: measuring what matters?

measuring what matters for kaua`i