state farm presentation at the chief analytics officer, fall 2016

Contains CONFIDENTIAL information which may not be disclosed without express written authorization

Do You Know or Do You Think You Know?

Incorporating Business Experiments Into

Strategy Development

Andy Pulkstenis

Director, Advanced Analytics

[email protected]

Contains CONFIDENTIAL information which may not be disclosed without express written authorization 1

• Setting the context • State Farm examples, from simple to sophisticated • Blueprint for getting testing started (or growing testing) at your firm • Responding to resistance


What are we doing today?


What is a “designed experiment?”

Designed testing is a way of treating the business environment like a controlled experimental setting, randomly assigning strategies to customers as in a clinical drug study or lab

Benefits

• Clear, accurate insights about causation • Ultra-clean data for model development = more accurate

prediction • Enables strategy optimization at the customer level


Innovators, on Testing

“By thorough experimentation and trial-and-error in the market, [successful companies] iterate towards a strategy and a business model that really is viable.”

- Harvard Business School Professor Clayton Christensen, a recognized authority on “disruptive innovation”


- Rich Fairbank, Chairman & CEO Capital One

“What we’ve done is to create an innovation machine. This is not just a credit card strategy, testing is a revolution that can be applied to many businesses.”



- Jeff Bezos, Amazon Founder

“If you double the number of experiments you do per year, you’re going to double your inventiveness.”


Fun fact: Amazon once fired a team for implementing strategic change without proper statistical testing first!


How do businesses use testing?

Strategy Optimization

Targeted Messaging

Pricing

Real-time Website Design/Optimization

Customer-level Decisioning

Retention & Save Strategies

Product & Strategy Development

Operations Improvement


Specific business examples

Sources: “How to Design Smart Business Experiments” by Tom Davenport, HBR 2/2009 “The Discipline of Business Experimentation” by Stefan Thomke and Jim Manzi, HBR 12/2014

Do visible lobster tanks increase lobster sales?

Are higher bids driven by CC payment option?

Will opening stores an hour later lead to a drop in sales?

Nearly all aspects of business.

http://www.google.com/imgres?imgurl=http://s.filife.com/be/company/images/CapitalOneLogo_2.png&imgrefurl=http://www.filife.com/reviews/capital-one-health-insurance&h=220&w=220&sz=9&tbnid=X9m_HOdMeCd-rM:&tbnh=107&tbnw=107&prev=/images?q=capital+one+logo+-bank&hl=en&usg=__n9FMka1p-W6RMC9V7K6zMOtNXLo=&ei=dc-XS4jxCI2wtgeDi_nkAQ&sa=X&oi=image_result&resnum=3&ct=image&ved=0CAoQ9QEwAg


Typical Testing Evolution

None A/B “one factor at a time”

(OFAT) Multivariate

(MVT) MVT w/

covariate analysis

Pick “one winner for all” Increased business insight while progressing left to right

Enables customer or segment-level strategy/message

optimization

Companies typically flow through an evolution when embarking on testing agendas

Continuous testing

A pervading culture of evolving, continuous, and cyclical testing provides

strategic foundation

Many stop at A/B or OFAT through lack of knowledge or lack of skillset, missing out on benefits that increase with each successive milestone.


Which banner led to the most visits to the

“Agent Search” page?

Version B:

Version A: (Click leads to “Drive

Safe & Save” page)

(Click leads to “Agent

Search” page)


Answer: Version B led more people to the Agent Search page


Which statefarm.com banner led to the most visits to

the “Drive Safe & Save” page?

Version B:

Version A: (Click leads to “Drive

Safe & Save” page)

(Click leads to “Agent

Search” page)


“Find an Agent” drove more people to the “Drive Safe &

Save” page than the “Drive Safe & Save” experience did!


Version A: Version B:

Which mailer generated the highest lift in

Life insurance sales?


Claims Contact Center

Claims Contact Center – “Out of Scope” Calls

Can proactive sharing of contact info reduce misdirected or “out of scope” Claims

calls?


Multivariate Discount Double Check Test

• Explored 7 different page features as tested ‘factors’ (text, graphics, layout,

images, colors, etc.)

• Results in 384 distinct web page variations (with a subset of 200+ valid

combinations)

• Only needed to physically test 58 combinations

• Improved quote starts by over 50% and quote completes by 10%


My roadmap for launching testing

Be Persistent It will take legwork and determination to find the partner in step two

Be Ambitious Aim for moderate or greater business impact

Be Careful Carefully execute the approach with business partners

Be Ready Build the “success story” deck before you need it

Be A Teacher There is a significant initial education component

Be Opportunistic Follow the path of least resistance


Best Practices: Set Up for Success

Assemble the

right structure Identify key roles at the outset (business, stats/DOE, execution, structural or creative design)

Assemble the

right players

Don’t just fill seats with “good people.” If you need an SME in a certain area, then staff that role with an SME

Assemble the

right questions

A surprising amount of time in early DOE engagements is spent helping the business partner define the problem as a quantifiable hypothesis

Assemble the

right scope

Too ambitious and you fail spectacularly Not ambitious enough and you really don’t move the needle

Assemble the

right controls

Make sure to monitor execution and data collection from Day 1 and throughout the process


Common Arguments

Regulation

Initial resistance to statistical testing typically falls into

three categories:

Complication

Organizational

Tradition


Regulation

Our industry is too heavily regulated for this

Didn’t stop Finance, Pharma, and others from

successful transition to DOE

Even if allowed, the regulators would never

understand it

It’s not “fair” to randomly assign strategies to

customers in a test

Successful testing industries proactively educate

regulators on key concepts

• Current strategy isn’t “fair” either

• Jonas Salk & polio

• “Beautiful experiment over which the

epidemiologist could become quite ecstatic but

would make the humanitarian shudder”

Perceived Barrier Response


Complication

Too complicated Nonsense – no more complicated than

predictive modeling, just new to your firm

Ignoring complex relationships doesn’t mean

they aren’t there

Start small, manageable

• Design according to cost constraints

• Much cheaper than hoping we guessed

correctly (think of it as “buying data”)

This can be handled by applying exclusions

to the design

Complexity of analysis

Too difficult to execute

Some combinations don’t make

business sense

Too expensive

Perceived Barrier Response


“We tried it once and we saw no

improvement” (i.e. learned nothing)

“We intuitively know our business”

Perceived Barrier

“Our external consulting firms tell us

all we need to know about course

correcting”

Organizational Tradition

• You don’t hit a home run every time

• You learned that those strategies were

equivalent

• Often gain fresh, real insights that defy

historical logic or perceived wisdom

• “The greatest obstacle to discovery is not

ignorance - it is the illusion of knowledge.”

- Daniel Boorstin

Response

Designed testing consistently outperforms

consultant-provided solutions

“I have not failed 10,000 times. I have succeeded in proving that those 10,000 ways will not work. When I have eliminated the ways that will not work, I will find the way that will work.” - Thomas Edison

Today WD-40 can be found in 4/5 US homes

WD-1 WD-39 “didn’t work!”


Questions/Comments

[email protected]


Appendix


Engagement Model

Business Partner

Statistical Design

Implementation/Execution

Structural and Creative Design Test

ing

Team

Ensure alignment with broader

strategic goals

Strategic Approval

“Testing Council” of SME’s ensures alignment with

statistical best practices and production constraints

Structural Approval

• Statistical DOE • Production Environment Document key hypotheses,

findings, participants, and actions taken

Test Results Repository

Execute/Monitor/Analyze/Take Action

Ensure alignment with broader strategic goals

Strategic Approval

Execute/Monitor/Analyze/Take Action “Testing Council” of SME’s

ensures alignment with statistical best practices and

production constraints

Structural Approval


What do I mean by “test-and-learn culture?”

• Designed experiments inform (and determine) strategy development

• Understanding of when to settle for simple “A/B” testing and when to use more sophisticated multifactor tests

• Internal standards/best practices for test design • Formal internal governance

A true testing culture demonstrates the following elements:


Four “Test Factors”:

• Engagement (3)

• Primary Emphasis (2)

• Tabs (2)

• George (2)

“Quote” “Yes”

“Phone”

A Multivariate Test Example

“Simple”

“Active”

“Yes”

“No”

“Passive”

“No”

This results in 24 combinations or “test cells”

Metric of interest: quote starts


So we tested a statistically chosen subset of 12 that allows us:

• To draw conclusions about all 24 combinations

• Make inferences about the impact of individual factors

and associated interactions

Physically testing all 24 combinations is undesirable:

• Implementation complexity

I applied a “reduced” design


• We learned “Why?” not just “Which one?” • Discovered interactions that had been downplayed as unlikely • Better business result:

- 12% > than current page - 10% > than vendor-recommended combo

• Set the stage for a 7-factor 200+ combination comprehensive test

• “George”

Benefits of using multifactor testing

state farm presentation at the chief analytics officer, fall 2016

Data & Analytics