Optimization Best Practices

Jessie BeckerCMO, Optimizely

What is Optimizely?

Powering the ideal experience for every customer at each touchpoint

Powerfully Simple Technology

•Code-free WYSIWYG visual editor makes it easy to create and launch new experiments

•Get started in minutes with easy implementation

•Iterate faster and bypass bottlenecks, such as code freezes, resource restraints and AppStore approvals.

Optimize websites or mobile apps

Real-Time, Actionable Insights

•Real-time analytics provide valuable insights into performance so you can take action

•Launch statistically significant winning variations with one click

•Create and share reports for greater visibility, highlighting key metrics and insights for your business

Unlocking the Power of Optimization

“As Google does, most of our work is proven or disproven by A/B testing.

Only that which leads to an improved experience gets rolled out to everyone.”

- Netflix Investor Relations

“It’s really important to get it out in the market and test it with people to see what they think.”

- Mike Schroepfer, CTO of Facebook

“If you double the number of experiments you do per year, you’re going to double your inventiveness.”

- Jeff Bezos, CEO of Amazon

Unlocking the Power of Optimization

Experimentation at Netflix

Experimentation at Netflix

Best Practice #1:

Use optimization

to transform traditional

processes.

Testing & Optimization are Related

Continuous

Optimization

One-Off

Testing

Optimization Encourages Constant Iteration and Discovery

• Data is actionable,

improvement measurable,

and risks calculated

• Iteration through testing is

the first step

• Dynamic experiments enable

new discoveries

Iron MountainEnterprise information management services

$7B market capitalization

Target High-Performing Experiences

Target High-Performing Experiences

+123%Engagement

Generic Control Healthcare Audience Variation

Industry-specific information delivered based on IP address

Transform Marketing Campaigns

Code.orgNonprofit backed by Bill Gates and Mark Zuckerberg

91M participants to date

Transform Marketing Campaigns

Simple copy experiments garnered 12 million new participants

+40%Participants

Headline and CTA Control Winning Headline and CTA Variation

Transform Marketing Campaigns

• Campaigns are dynamic

• Tremendous performance upside

• Closing the loop from execution to

measurement prompts new action

• Improves the ROI of many digital

marketing channels• Social/PR

• Paid Advertising

• Email Marketing

Best Practice #2:

Swing for the fences;

then scale to

what’s feasible.

Experimentation is More than Iteration

• Generate experiment data to qualify strategic choices

• Quantify the impact of investments and future improvements

Incentive Experiment: Netflix Referrals

Incentive Experiment: Netflix Referrals

Hypothesis: Incentives to invite friends will

generate cost-effective new users.

Variable: What value of offer will elicit a

response?

Lessons Learned:

• Test at the limits to determine effect

• Scale back to a reasonable level once

viable marketing channel has been

established

Organic Referral

Control

Invite a friend to Netflix!

Give One, Get One

Variation

Invite a friend for $100 credit!

$100 Credit

Variation

Incentive Experiment: Netflix Referrals

New Feature: Payment Options

Painted Door ExperimentAnswer the question:

Is a new payment option attractive to customers?

New Feature: Payment Options

Hypothesis: Offering a new payment

method to users will increase signups.

Variable: Does the existence of this

feature matter to visitors?

Lessons Learned:

• Use a ‘painted door’ to determine if

development resources are needed to

build the option

• Quantify estimated value in advance

Best Practice #3:

Leverage the collective

creativity of your team.

Leadership Leadership

Individual Contributors Individual Contributors

Optimization Fosters Innovation

Leadership Leadership

Individual Contributors Individual Contributors

• Flow of ideas is multi-directional and

meritocratic

• Evaluate and execute on more ideas

• Uncover talent for both creativity and

data intelligence

• Diversity of skills and experiences is

a key strength of an optimization

program

Optimization Fosters Innovation

E-Commerce Signup Experiment

Trunk ClubSubscription clothing retailer

Recently acquired by Nordstrom for $350M

E-Commerce Signup Experiment

Counterintuitive concept immediately coded to production

+150%New customer

acquisition

Onboarding Experience Variation

Best Practice #2:

Swing for the fences; then scale to

what’s feasible.

Best Practice #1:

Use optimization to transform

traditional processes.

Best Practice #3:

Leverage the collective creativity of

your team.

Questions?