Jon Fox@jfox85

Performance Monitoring:

Synthetic vs

RUM

Velocity 2013Friday, June 28, 13

Performance Monitoring

Friday, June 28, 13

Performance Monitoring

Two primary methods:

- Synthetic (Active)

- Real User Measurement - RUM (Passive)

Friday, June 28, 13

Synthetic Overview

- Browser in the cloud

- Usually has extra instrumentation

Friday, June 28, 13

Synthetic Overview

- Often in controlled environments - Reliable bandwidth - Fixed specs - Dedicated hardware

Friday, June 28, 13

Synthetic Overview

- Generally in several geographic locations

- Offer various browsers / rendering engines

Friday, June 28, 13

Synthetic - How It Works

Provider Your Website

Cloud Agent

Friday, June 28, 13

Synthetic - How It Works

Provider Your Website

Tell provider to run a test

Friday, June 28, 13

Synthetic - How It Works

Provider Your Website

Provider sends the test to an agent

Friday, June 28, 13

Synthetic - How It Works

Provider Your Website

The agent requests your website

Friday, June 28, 13

Synthetic - How It Works

Provider Your Website

The agent receives the page and renders the responsewhile measuring load times

Friday, June 28, 13

Synthetic - How It Works

Provider Your Website

The results are sent back to the provider to be viewedand possibly aggregated with other results

Friday, June 28, 13

Synthetic - What kind of data do we get?

Source: http://www.webpagetest.org/

Friday, June 28, 13

Synthetic - What kind of data do we get?

Source: http://www.webpagetest.org/

Friday, June 28, 13

Synthetic - What kind of data do we get?

Source: http://www.webpagetest.org/

Friday, June 28, 13

Synthetic - What kind of data do we get?

Source: http://www.webpagetest.org/

Friday, June 28, 13

Synthetic - What kind of data do we get?

Source: http://www.webpagetest.org/

Friday, June 28, 13

Synthetic - What kind of data do we get?

Source: http://www.webpagetest.org/

Friday, June 28, 13

Real User Measurement (RUM)

On the other hand we have RUM

Friday, June 28, 13

RUM Overview

- Add JavaScript to the page

- Gather timing data from the browser

<script type='text/javascript'>var TBRUM=TBRUM||{};TBRUM.q=TBRUM.q||[];TBRUM.q.push(['mark','firstbyte',(new Date).getTime()]);</script>

Friday, June 28, 13

RUM Overview

- Every page and every visitor

- Actual visitor timings

Friday, June 28, 13

RUM Overview

- Covers every device and browser that visits your site

Friday, June 28, 13

RUM - How It Works

Your Website

JavaScript

Beacon Aggregation /Processing

Site Visitor

Friday, June 28, 13

RUM - How It Works

Your Website

JavaScript

Beacon Aggregation /Processing

Site Visitor

The site visitor decides to come to your site

Friday, June 28, 13

RUM - How It Works

Your Website

JavaScript

Beacon Aggregation /Processing

Site Visitor

They request your page and view it as normal

Friday, June 28, 13

RUM - How It Works

Your Website

JavaScript

Beacon Aggregation /Processing

Site Visitor

The page includes the RUM Javascript, which collects timing data from the browser

Site Visitor

Friday, June 28, 13

RUM - How It Works

Your Website

JavaScript

Beacon Aggregation /Processing

Site Visitor

The timing data is then sent to the collection service via a beacon image request

Site Visitor

Friday, June 28, 13

RUM - How It Works

Your Website

JavaScript

Beacon Aggregation /Processing

Site Visitor

The data is then aggregated and processed to generate pretty charts and graphs

Site Visitor

Friday, June 28, 13

RUM - Navigation Timing Spec

Source: https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html

Friday, June 28, 13

RUM - Navigation Timing Availability

Source: http://caniuse.com/nav-timing

Friday, June 28, 13

RUM - Other Timing Methods

- Google Toolbar

Source: http://www.google.com/toolbar/ie/index.html

Friday, June 28, 13

RUM - Other Timing Methods

- Page unload cookie

Friday, June 28, 13

RUM - Other Timing Methods

- Start of document timer

Friday, June 28, 13

RUM - Other Data

- Estimated Bandwidth- IP Address- URL- Referer- Cookies- Local Storage support- Number of images, JavaScript, CSS, etc - User actions on the page- Memory Usage (very limited availability)- Anything else you can get from JavaScript!

Friday, June 28, 13

RUM - Resource Timing Spec

Source: http://www.w3.org/TR/resource-timing/

Friday, June 28, 13

State of Resource Timing Spec

- IE10, Chrome, Chrome Mobile

- Various degrees of support / compatibility

- Same origin policy by default

- Override with Timing-Allow-Origin HTTP response header

Friday, June 28, 13

RUM - What kind of data do we get?

Friday, June 28, 13

RUM - What kind of data do we get?

Friday, June 28, 13

RUM - What kind of data do we get?

Friday, June 28, 13

RUM - What kind of data do we get?

Friday, June 28, 13

Comparison

So how do these methods compare?

Friday, June 28, 13

Methodology

Ran both on velocityconf.com

Torbit used for RUM

Webpagetest for Synthetic

Approximately 3 months (3/18/13 - 6/14/13)

Friday, June 28, 13

Methodology

Synthetic hit the following URLs once an hour:

http://velocityconf.com/velocity2013/http://velocityconf.com/velocity2013/public/schedule/topic/935http://velocityconf.com/velocity2013/public/content/exhibitorshttp://velocityconf.com/velocity2013/public/schedule/gridhttp://velocityconf.com/velocity2013/public/content/abouthttp://velocityconf.com/velocity2013/public/schedule/speakers

Friday, June 28, 13

Methodology

From the following synthetic agents:Dulles - Chrome - IE8 - IE9 - IE10 - Firefox - Safari

Chicago - Chrome

San Jose - IE9

Los Angeles - Chrome

Dublin - IE8

Sao Paulo - IE8

Toronto - IE8

Amsterdam - IE9 - Chrome

Frankfurt - IE9 - Firefox

Moscow - Chrome - IE8

Stockholm - Safari

Delhi - IE8

China - IE 8

Tokyo - IE 8

Sydney - Chrome

Friday, June 28, 13

Methodology

Real User Measurement:

RUM was run on every page (as far as I know)

Friday, June 28, 13

Results - Data Overview

Synthetic Load Times

RUM Load Times

Friday, June 28, 13

Results - Data Overview

Synthetic Load Times

RUM Load Times

Friday, June 28, 13

Results - Browser Coverage

Synthetic RUM

- Chrome - IE 10 - IE 9 - IE 8 - Firefox - Safari

- Chrome - IE 10 - IE 9 - IE 8 - IE 7 - Firefox - Safari - Opera - Android - iPhone - And more...

Friday, June 28, 13

Results - Geographic Coverage

Synthetic:

Friday, June 28, 13

Results - Geographic Coverage

RUM:

Friday, June 28, 13

Results - Data Insights

RUM:

Synthetic:

0"5"10"15"20"25"30"35"40"45"50"

1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 16" 17" 18" 19" 20" 21" 22" 23" 24"

Load

%Tim

e%(sec)%

Time%(hrs)%

RUM"Avg"

90th"

Syn"Avg"

Friday, June 28, 13

Results - Data Insights

0"1"2"3"4"5"6"7"8"9"10"

1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 16" 17" 18" 19" 20" 21" 22" 23" 24" 25" 26" 27" 28" 29"

Load

%Tim

es%(sec)%

Time%(hrs)%

RUM"Avg"

RUM"90th"

Syn"Avg"

0"

200"

400"

600"

800"

1000"

1200"

1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 16" 17" 18" 19" 20" 21" 22" 23" 24" 25" 26" 27" 28" 29"

Pageview

s%

Pageviews"

Friday, June 28, 13

Results - Data Insights

Friday, June 28, 13

Results - Data Insights

Friday, June 28, 13

When To Use?

When should you use one over the other?

Friday, June 28, 13

When to use Synthetic

- Pre-production

- Low / no traffic pages

- Going deep on a problem

- Critical pages and paths

- Monitoring competitors

Friday, June 28, 13

When to use RUM

- Ultimate source of truth

- Finding out you have a problem

- Monitoring trends over time

- Vendor or departmental SLAs

Friday, June 28, 13

Build or Buy? - Synthetic

Should you build it yourself or buy a solution?

Friday, June 28, 13

Build - Synthetic

Open source

Self-host options

Friday, June 28, 13

Reasons to Buy - Synthetic

Need lots of agents

Friday, June 28, 13

Reasons to Buy - Synthetic

Need lots of locations

Friday, June 28, 13

Reasons to Buy - Synthetic

Simpler aggregation and reporting

Friday, June 28, 13

Build or Buy? - RUM

What about building your own RUM?

Friday, June 28, 13

Build - RUM

Collection is pretty easy...

Episodes - Steve Soudershttp://stevesouders.com/episodes/

Boomerang - Philip Tellishttps://github.com/lognormal/boomerang

Friday, June 28, 13

Reasons to Buy - RUM

Storage and processing are hard...

Friday, June 28, 13

Reasons to Buy - RUM

Dealing with outliers & bad data

Friday, June 28, 13

Conclusions

So which method should you use?

Friday, June 28, 13

Conclusions

A mix of both is best

Friday, June 28, 13

Why do we care?

And in case you need a reminder...

Friday, June 28, 13

Performance Matters

Aggregated data from 1,000+ Torbit Sites

Friday, June 28, 13

Performance Matters

Aggregated data from 1,000+ Torbit Sites

Friday, June 28, 13

Conclusions

Free options are available:

Synthetic RUM

Friday, June 28, 13

Conclusions

Start collecting data now!

Friday, June 28, 13

Questions?

Jon Foxjon@torbit.com

@jfox85

Friday, June 28, 13