1 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

DO ALL USERS BENEFIT EQUALLY FROM

WEB PERFORMANCE OPTIMIZATIONS?

Arnaud Becart – @arnaud_beip-label

2 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2REAL-USER MONITORING ≠

SYNTHETIC TESTING

Previously, at Velocity Berlin…

3 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

Synthetic Testing

4 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

PROOF BY EXAMPLE

Mobile + 0 sTablet + 4 s

PC + 4 s Synthetic

Tests+ 0 s

Real Users

Synthetic Tests don’t see everything

5 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

SOME DIFFERENCES BETWEEN SYNTHETIC TESTS & RUM

Browser

Browser version

OS

Device

CPUMemory

OS version

Fixed location

Fixed last mile

Bandwidth

Provider Browser add-ons

Antivirus

Mobilelast mile

2. No synthetic testing

Networkaccess

3. Variable, depending on users’ behaviour

Nomadic usage

1. Similar or different between synthetic & real-user testing

Browser cache

Multiple tabs

6 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

BACK TO THE TITLE

DO ALL USERS BENEFIT EQUALLY FROMWEB PERFORMANCE

OPTIMIZATIONS?

Fewer HTTP

requests

Expires header

CSS at the top Accelerate

rendering

CDNGzip

JS at the bottom

Reduce size

…(thanks Steve)

Defer third-party

content

Avoid redirects

7 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

REAL LIFE EXAMPLE

E-RETAILWEB SITE

8 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

METHODOLOGY

synthetic_monitoring()optimization ()

while not improved(synthetic_monitoring()):optimization()

//responsetime is better!real_user_monitoring()

9 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

SYNTHETIC TREND

Fewer requests

& Compression

35%Reduction

in page size

30% quicker start rendering

23% quicker page load time

Page Size

Page Load Time

2.5 Mb

1.6 Mb

7.4 s

5.7 s

1st Step: Confirm trend with synthetic testing

10 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

Real Users’ Performance gain: 11%

WHAT ABOUT REAL USERS ?

2nd Step: Align Real Users & Synthetic tests

URL, Country, ISP, Metric (onload event), Period (2 months)

Page load time: 9.2 s 8.2 s

3rd Step: Analyze 20M real-user measurements

11 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

BACK TO THE TITLE AGAIN

DO ALL USERS BENEFIT EQUALLY FROM WEB PERFORMANCE

OPTIMIZATIONS?

12 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

USER SATISFACTION

% of users loading the page in less than 9 seconds

Mobile: 43% 55%Tablet: 52% 60%

+12

+8

13 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

THE DEVICE EFFECT

Smartphones = High BenefitTablets = Small Benefit

Mobile

PC

Probe

Tablet

0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0%

+24.6%

+10.6%

+22.7%

+6.7%

Response time gain (%) per device

12.1 11.3 s

7.4 5.7 s

9.1 8.1 s

21.7 16.4 s

14 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

THE NETWORK EFFECT

Mobile

Tablet

PC

0.0% 5.0% 10.0%15.0%20.0%25.0%30.0%

20.1%

6.9%

10.4%

29.7%

17.6%

15.0%

Mobile NetworkFixed Network

Response time gain (%) per type of Network connection

Slow bandwidth + fewer bytes to load = High Benefit

15 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

THE ‘CPU’ EFFECT

*Not just CPU: processing capacity & available memoryDevices with Very Good or Very Bad Performance

(old?)benefit less from size optimization

Very Old

Average

Very Good

+12.1%

+18.0%

+15.2%

+7.9%

+4.2%

+2.9%Dual Core 2.2 GHz 4G RAM

iPad 1

iPhone 4

5.5 5.3 s

7.5 7 s

8.5 8 s

10 8.5 s

14.5 12

s

20s 17 s

iPhone 5

Response time gain (%) per device capacity*

16 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

THE BROWSER EFFECT

No improvement for SafariHigher benefit for IE

-5.0% 0.0% 5.0% 10.0% 15.0% 20.0%

+16.0%

+10.6%

+6.4%

-0.9%8.5 8.6 s

9.9 9.3 s

8.3 7.5 s

9 7.6 s

Response time gain (%) per browser on fixed devices

17 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

THE BROWSER EFFECT

Very Old

Old

Average

Good

Very Good

Premium Devices!

-10.0% -5.0% 0.0% 5.0% 10.0% 15.0% 20.0%

SafariChromeFirefoxIE

Chrome: 22.8 23.3 s

Safari: 7.8 8.4 sSafari: 8.3 8.4 s

Response time gain (%) per browser & processing (PC)

Are Firefox users geeks?Is Safari not made for new devices?

Is Chrome not made for very old devices?

Firefox: 6 5.3 s

18 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

CONCLUSIONDO ALL USERS BENEFIT EQUALLY FROM

WEB PERFORMANCE OPTIMIZATIONS?

19 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

CONCLUSIONDO ALL USERS BENEFIT EQUALLY FROM

WEB PERFORMANCE OPTIMIZATIONS ?Synthetic

Tests+23%

Real Users+11%

PC+10.6

%

IE +18%

Smartphone

+25%

Old PC+18%

Tablet+7%

New PC+4%

Smartphone 3G Network

+30%Tablet

3G Network+18%

Chrome +8%

20 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

Don’t trust only Synthetic Tests to justify web optimizations

Not all devices are equal – especially with the network factorNot all browsers are equal – especially with “CPU”/”Old device” factor

Don’t be too quick to draw conclusions about the actual effectiveness of optimizations for your real users

21 Rep

rod

uct

ion inte

rdit

e. ©

ip

-lab

el 2

01

2

THANK YOU!

www.ip-label.com

Arnaud Becart@arnaud_be