adaptive static resource optimization

(c) 2009 Facebook, Inc. or its licensors.  "Facebook" is a registered trademark of Facebook, Inc.. All rights reserved. 1.0

Adaptive Static Resource Optimization

The AJAX Experience 2009Sep 15, 2009 Boston, MA

David Wei and Changhao Jiang

Best practice: from theory to operation?

“Reduce # of HTTP Requests”

Package JS/CSS; Sprite Image

Best practice: from theory to operation?

“Reduce # of HTTP Requests”

Package JS/CSS; Sprite Image

Interface & Infrastructure

Operation

1 Optimization must be adaptive

2 Adaptive Packager: Designs

3 Adaptive Packager: Operations

4 Lessons learnt

Agenda

Optimization must be adaptive

Facebook’s day-to-day challenges

• Deep Integration

• Viral Adoption

• Agile Development

Performance optimization has to be adaptive!

Perf optimization must be adaptive• Day 1: Some smart engineers start a project!<Print css tag for feature A>

<Print css tag for feature B>

<Print css tag for feature C>

<print HTML of feature A>

<print HTML of feature B>

<print HTML of feature C>

…

“Let’s write a new page with features A, B and C!”

• Day 2: Some smart engineers run YSlow and says…<Print css tag for feature A>

<Print css tag for feature B>

<Print css tag for feature C>

<print HTML of feature A>

<print HTML of feature B>

<print HTML of feature C>

…

“A & B & C are always used; let’s package them together!”

• Day 2: Awesome! <Print css tag for feature

A&B&C>

<print HTML of feature A>

<print HTML of feature B>

<print HTML of feature C>

…

• Day 3: feature C evolves…<Print css tag for feature A & B & C>

<print HTML of feature A>

<print HTML of feature B>

If (users_signup_for_C()) { <print HTML of feature C>}

…

• Day 3:<Print css tag for feature A & B & C>

<print HTML of feature A>

<print HTML of feature B>

If (users_signup_for_C()) { <print HTML of feature C>}

…

A&B are always used, while C is not. ..

• Day 4: feature C is deprecated<Print css tag for feature A & B & C>

<print HTML of feature A>

<print HTML of feature B>

// no one uses C { <print HTML of feature C>}

…

• Day 4: we start to send unused bits<Print css tag for feature A & B & C>

<print HTML of feature A>

<print HTML of feature B>

// no one uses C { <print HTML of feature C>}

…

It is hard to remember we should remove C here.

• One months later…<Print css tag for feature A & B & C & D & E & F & G…>

if (F is used) <print HTML of feature F>

<print HTML of feature G>

if (F is not used) { <print HTML of feature E>}

…

Thousands of dead CSS rules in the package.

• One months later…<Print css tag for feature A & B & C & D & E & F & G…>

if (F is used) <print HTML of feature F>

<print HTML of feature G>

if (F is not used) { <print HTML of feature E>}

…

Performance optimization has to be adaptive!

Thousands of dead CSS rules in the package.

Adaptive Static Resource OptimizationChallenges:• Deep Integration

• Viral Adoption

• Agile Development

Responses:• Build interface to separate

requirement declaration and delivery of static resources

• Requirement declaration: lives with HTML generation, logged

• Delivery: Globally optimized by analyzing the logged trace

Adaptive Packager: Designs

Interfaces

• Back to Day 1: require_static(A_css); <render HTML of feature

A>

require_static(B_css); <render HTML of feature B>

require_static(C_css);<render HTML of feature C>

<deliver all required CSS>

<print all rendered HTML>

Separate Declaration from actual Delivery

Global Optimization on Delivery

Requirement Declaration lives with HTML

Offline analysisPackager: Global JS/CSS Optimization

Online processrequire_static(A_css); <render HTML of

feature A>

require_static(B_css); <render HTML of feature B>

require_static(C_css); <render HTML of feature C>

<deliver all required CSS>

<print all rendered HTML>

Usage Pattern logs

Packaging algorithm

“Optimal” packages

Packager: Usage Pattern Logs

Usage Pattern logs

Page Load

1 2 3 4 5 … 100000

Page Count

10 M

1 M

100 K

20 K

10 K

1K

A.css (1 KB)

1 1 1 1 1 1

B.css (1 KB)

1 1 1 1 1 1

C.css (300B)

1 1 1

D.css (2 KB)

1 1

E.css (700B)

1 1 1

F.css (400B)

1 1

G.css (600B)

1 1 1 1 1

Packager: Cost/Benefit Model▪ To package two files A & B:

▪ “Cost”: for page requests that only uses A, we waste the bytes of B, vice versa

▪ “Benefit”: for page requests that uses both A and B: we save one round trip

▪ Bytes / Bandwidth ~ Latency

▪ “Profit” to be maximized: Benefit

– Cost

Page Load

1 2 3 4 5 … 100000

Page Count

10 M

1 M

100 K

20 K

10 K

1K

A.css (1 KB)

1 1 1 1 1 1

B.css (1 KB)

1 1 1 1 1 1

C.css (300B)

1 1 1

D.css (2 KB)

1 1

E.css (700B)

1 1 1

F.css (400B)

1 1

G.css (600B)

1 1 1 1 1

Page Load

1 2 3 4 5 … 100000

Page Count

10 M

1 M

100 K

20 K

10 K

1K

A.css (1 KB)

1 1 1 1 1 1

B.css (1 KB)

1 1 1 1 1 1

C.css (300B)

1 1 1

D.css (2 KB)

1 1

E.css (700B)

1 1 1

F.css (400B)

1 1

G.css (600B)

1 1 1 1 1

Packager: Cost/Benefit Model▪ Assume: latency = 40ms, and bandwidth = 1 Mbps▪ A+B: 40ms * 11.131M

No cost, pure gain. Definitely package

Page Load

1 2 3 4 5 … 100000

Page Count

10 M

1 M

100 K

20 K

10 K

1K

A.css (1 KB)

1 1 1 1 1 1

B.css (1 KB)

1 1 1 1 1 1

C.css (300B)

1 1 1

D.css (2 KB)

1

E.css (700B)

1 1 1

F.css (400B)

1 1

G.css (600B)

1 1 1 1 1

Packager: Cost/Benefit Model▪ Assume: latency = 40ms, and bandwidth = 1 Mbps▪ B+C: 40ms * 11.1M – 300B / 1Mbps *

0.031M

Benefit larger than cost OK to package

Page Load

1 2 3 4 5 … 100000

Page Count

10 M

1 M

100 K

20 K

10 K

1K

A.css (1 KB)

1 1 1 1 1 1

B.css (1 KB)

1 1 1 1 1 1

C.css (300B)

1 1 1

D.css (2 KB)

1

E.css (700B)

1 1 1

F.css (400B)

1 1

G.css (600B)

1 1 1 1 1

Packager: Cost/Benefit Model▪ Assume: latency = 40ms, and bandwidth = 1 Mbps▪ B+D: 40ms * 1K – 2K / 1Mbps * 11.13M

Cost larger than benefit Don’t package

Packager: Algorithms

Packaging algorithm

Page Load

1 2 3 4 5 … 100000

Page Count

10 M

1 M

100 K

20 K

10 K

1K

A.css (1 KB)

1 1 1 1 1 1

B.css (1 KB)

1 1 1 1 1 1

C.css (300B)

1 1 1

D.css (2 KB)

1

E.css (700B)

1 1 1

F.css (400B)

1 1

G.css (600B)

1 1 1 1 1

▪ Greedy algorithm▪ Simulated

Annealing▪ Clustering

algorithms▪ …

Packager: Optimal packages

Usage Pattern logs

Packaging algorithm

“Optimal” packages

Page Load

1 2 3 4 5 … 100000

Page Count

10 M

1 M

100 K

20 K

10 K

1K

A.css (1 KB)

1 1 1 1 1 1

B.css (1 KB)

1 1 1 1 1 1

C.css (300B)

1 1 1

D.css (2 KB)

1

E.css (700B)

1 1 1

F.css (400B)

1 1

G.css (50B)

1 1 1 1 1

▪ Pkg 1: A, B, C▪ Pkg 2: E, F, G

Adaptive Static Resource OptimizationAdaptive Packaging / Spriting• Cross-feature optimizations (e.g. search + ads)

• Adaptive to change of user behaviors and code developments

• Same methodology works for image spriting

Adaptive Static Resource OptimizationAdaptive progressive rendering• flush the commonly used JS/CSS/Images to the browsers,

before the page is generated at the server

• Browser downloads JS/CSS in parallel with page generation at server side

Adaptive Progressive RenderingAdaptive progressive rendering• Progressive rendering: flush the commonly used

JS/CSS/Images to the browsers, before the page is generated at the server

Adaptive progressive renderingWhat are the common JS / CSS should we flush?

Usage Pattern logs

Page Load

home

profile

home

profile

photo

…

Page Count

10 M 1 M 100 K

20 K 10 K

A.css (1 KB)

1 1 1 1 1

B.css (1 KB)

1 1 1 1 1

C.css (300B)

1 1 1

D.css (2 KB)

1

E.css (700B)

1 1

F.css (400B)

1 1

G.css (600B)

1 1 1

Adaptive progressive renderingSum on the page name:

Page Load

home

profile

home

profile

photo

…

Page Count

10 M 1 M 100 K

20 K 10 K

A.css (1 KB)

1 1 1 1 1

B.css (1 KB)

1 1 1 1 1

C.css (300B)

1 1 1

D.css (2 KB)

1

E.css (700B)

1 1

F.css (400B)

1 1

G.css (600B)

1 1 1

Page Load

home

profile

photo

…

Page Count

10.1 M

1.02 M

10 K

A.css (1 KB)

10.1M

1.02M

10K

B.css (1 KB)

10.1M

1.02M

10K

C.css (300B)

10.1M

1M

D.css (2 KB)

10M

E.css (700B)

0.02M

10K

F.css (400B)

0.02M

10K

G.css (600B)

10M 0.02M

10K

Adaptive progressive renderingRemove the rare usages

Page Load

home

profile

photo

…

Page Count

10.1 M

1.02 M

10 K

A.css (1 KB)

10.1M

1.02M

10K

B.css (1 KB)

10.1M

1.02M

10K

C.css (300B)

10.1M

1M

D.css (2 KB)

10M

E.css (700B)

0.02M

10K

F.css (400B)

0.02M

10K

G.css (600B)

10M 0.02M

10K

Page Load

home

profile

photo

…

A.css (1 KB)

Y Y Y

B.css (1 KB)

Y Y Y

C.css (300B)

Y Y

D.css (2 KB)

Y

E.css (700B)

Y

F.css (400B)

Y

G.css (600B)

Y Y

Adaptive Static Resource OptimizationAdaptive progressive rendering• More than 500 msec gain for home page end-to-end time

• Adaptive to the change of popular resources

• Need to solve issues with redirections, HTTP-only cookie and buffers

Adaptive Packager: Operations

Update frequency

• Too slow: fail to adapt to usage changes• Too fast: invalidate caches• Usually weekly• Can be smarter

Resources for new features

• No usage logs available• Developers to estimate the initial usage

pattern• Update packages one day after release, if

necessary

Deployment statussince Nov 2008• Rebalance weekly based on previous week’s

usage pattern (>100K unique usage patterns)• Javascript and CSS packaging only (image soon)

Scalable with developmentJS/CSS Code base: Nov 2008 -> Sep 2009

Date # of JS files

# of JS bytes

# of pkg at a

home.php

# of bytes at a

home.php

Nov 2008 461 4.4 MB 29 629 KB

Sep 2009 1045 6.8 MB 19 487 KB

Date # CSS files # of CSS bytes

# of pkg at a

home.php

# of bytes at a

home.php

Nov 2008 487 1.7 MB 24 69 KB

Sep 2009 938 1.8 MB 14 48 KB

Adaptive to new usages# of JS/CSS Packages served in the past month

Experiment: Adaptive Image SpritingThe puzzle of image spriting:• Thousands of virtual gifts with static images, which to

sprite?

Experiment: Adaptive Image SpritingThe puzzle of image spriting:• The answer is…

Lessons learnt

Regression detection can be trickier

“The page has 5KB more Javascripts!”

• Regression introduced by Packager?

• Or, by the code?

When data go wrong

“All models are wrong.Some models are useful.”• Not all users have 1Mbps and 40ms RTT

connections

• A/B Testing to find an optimal model

• Customized packaging for user groups• Per browser groups: ~5% improvements• Locale / IP Prefix might yield better performance

Human errors are possibleAutomatic analysis is preferable:require_static(A_css); //forgot to remove the

require_static

require_static(B_css); <render HTML of feature B>

require_static(C_css); <render HTML of feature C>

<deliver all required CSS>

<print all rendered HTML>

Summary▪ Performance optimization must be adaptive▪ Good interface helps optimization▪ Trace-based analysis is powerful▪ Always monitor the optimization effect

Thank you!