#inspire14

Test and Control:

Solution Development

Andrew Jastrzembski, Jason Thomas,

Michal Polanowski

Delhaize America

Who Are We Andy Jastrzembski

Shopper Insights Manager

where he provides the

voice of the shopper.

Started as an Operations

Researcher Officer in the

US Air Force and then

transitioned to retail. Has

BS in Applied Mathematics

and Psychology and a MS

in Operations Research.

Jason Thomas

One of the founders of the

team. Worked for the

company for 13 years.

Currently, oversees the

development of Shopper

and Basket-level

algorithms and scripts for

both ad hoc analyses and

production analytics.

Michal Polanowski

With Delhaize America

since 2012. Focus on the

test and control support

for various business

strategy, marketing, and

category initiatives. Prior,

taught at the University of

Georgia where he received

his PhD.

#inspire14

Disclaimer • The views and opinions expressed in this presentation are solely those of the

authors of the presentation. These views and opinions do not necessarily

represent those of Delhaize America, Delhaize Group or any of our subsidiaries.

• This case study utilizes financial and geographical references for educational

purposes only. The values and results presented are fictional. Any numerical,

geographical, or program similarities to actual Delhaize America programs are

coincidental.

#inspire14

Delhaize America • Part of Delhaize Group with food supermarkets in 6 countries

• Delhaize America consists of three banners with over 1,500 food supermarkets

#inspire14

Consumer Insights • Our mission is to deliver and drive the voice of consumer throughout the

organization by providing actionable insights

• Shopper Insights delivers insights based on transaction and customer level data

• We support the analytical needs of strategy, marketing, pricing, category

management, merchandising, finance, and legal

#inspire14

Agenda

• Why are we at this conference?

• What is Test and Control?

• Research Problem

• Test and Control Methodology

• Meeting with Business Owners

• Identify Stores to Be Excluded

• Match Test and Control

• Perform the Test and Evaluate the Results

• Present Findings to Business Owners

#inspire14

First comprehensive test and control environment in Alteryx

We want to share our methodology with the rest of the industry

Why we decided to speak at #inspire14?

#inspire14

Test and Control Overview

Total Population Test and Control Match

Not Suitable

Stores

• Estimates the effectiveness of an initiative/change

• Matches stores/households in the pre period (before the initiative)

• Analyzes the behavior after the initiative

#inspire14

-25%

0%

25%

04/24/12 05/22/12 06/19/12 07/17/12 08/28/12

Pe

rfo

rman

ce

Store Performance Lift: 15%, Impact: $5,600

Test Stores Control Stores

Lift is the gap between the test and control store performance

Test and Control Overview

Lift

Pre Period Test Period

#inspire14

-25%

0%

25%

04/24/12 05/22/12 06/19/12 07/17/12 08/28/12

Pe

rfo

rman

ce

Store Performance Lift: 15%, Impact: $5,600

Test Stores Control Stores

Lift is the gap between the test and control store performance

Test and Control Overview

Lift

Pre Period Test Period

#inspire14

Lift Results Can Be Tricky

Test and Control Overview

Test Store

Performance

Control Store

Performance

+ Lift - Lift

+ Lift

#inspire14

Food Lion has a Sunday flyer in small local newspapers

Research: Effectiveness of the Sunday Flyers

• Food Lion wanted to test the effectiveness of the flyer

• Currently we spend $1.5M/week on Sunday flyers

• A test market representative of all Food Lion stores

• The test market will not receive a Sunday flyer

#inspire14

Test and Control Process

Meet with Business Owner

Performed in Alteryx

Identify Stores to be

Excluded

Match Test and Control

Stores

Perform the Test

Meet with Business Owner

#inspire14

Understand the business reasons for the test

Meeting with Business Owners

• Define criteria for success and failure

• Success = an impact of at least $1.4K per store per week ($1.5M total)

• Align on dates, stores, products that are involved

• All stores belong to the same region and flyer covers products in all departments

• Identify blackout periods and other business initiatives that can potentially

impact the test or control results

• No blackout periods and no other Food Lion activity

#inspire14

Test and Control Process

Meet with Business Owner

Performed in Alteryx

Identify Stores to be

Excluded

Match Test and Control

Stores

Perform the Test

Meet with Business Owner

#inspire14

Numerous events can skew the results

Identify Stores to be Excluded

• Competitive events

• Store openings, closings, remodels

• Seasonal stores with high sales volatility

• Natural events

High Volatility Low Volatility

#inspire14

We automated the process

Competitive Events and Store Volatility Module

Volatility= St.Dev(Sales)

Mean(Sales)

#inspire14

Review stores with high volatility and competitive activity

Volatile Stores Should Be Removed

Remove

These

Stores

Food Lion

Store # Event Date

Estimated

Impact Volatility

5043 Lowes Foods 8/7/13 ($47,500) 0.65

5323 0.44

5011 Publix 4/11/13 ($22,500) 0.43

5554 Kroger 6/14/13 ($75,000) 0.09

5832 Wal-Mart 7/26/13 ($1,000) 0.08

5934 0.04

5907 Road Work 6/15/13 ($1,000) 0.02

5123 0.01

#inspire14

Test and Control Process

Meet with Business Owner

Performed in Alteryx

Identify Stores to be

Excluded

Match Test and Control

Stores

Perform the Test

Meet with Business Owner

#inspire14

Match similar test and control stores

Test and Control Stores Match

• A good match controls any store fluctuations in the match variables

• Match on a number of metrics

• Sales, unit movement, margin, number of customers

• We matched on total store sales and unit movement

• Match on up to 6 metrics per test (recommended 2 – 3)

• Match within specific clusters:

• Region, urbanicity, strategy initiatives

#inspire14

Test and Control Match Module

#inspire14

The goal is to have test and control stores close to each other

Evaluate Test and Control Match

Poor

Match

Remove

These

Stores

Good

Match

Test and Control Process

Meet with Business Owner

Performed in Alteryx

Identify Stores to be

Excluded

Match Test and Control

Stores

Perform the Test

Meet with Business Owner

#inspire14

Test and Control Module

#inspire14

Test and Control Analysis Module

You better

know what you

are doing !!!

#inspire14

Perform the Test and Evaluate the Results

5921 5122 5993 5984 5097 5789 5121 5432 5162 5427

Statistic Treatment Control

Average 2.4% 3.9%

Minimum -1.4% -13.2%

Maximum 6.5% 16.2%

Standard Deviation 2.2% 3.2%

Some Control Stores May

Need Additional Inspection

Alteryx generates individual reports for each metric analyzed

Sometimes, additional data investigation may be required

#inspire14

Test and Control Results

Lift Expected Impact Significance Level

-1.5% -$2.7K 99.9%

Test t-Statistic Degrees of Freedom p-Value

Control vs. Treatment 3.55 110.6 0.001

-25%

0%

25%

04/24/12 05/22/12 06/19/12 07/17/12 08/28/12

Pe

rfo

rman

ce

Store Performance Lift: -1.5%, Impact: -$2.7K

Test Control

Pre Period Test Period

#inspire14

Test and Control Process

Meet with Business Owner

Performed in Alteryx

Identify Stores to be

Excluded

Match Test and Control

Stores

Perform the Test

Meet with Business Owner

#inspire14

Success Criteria: An impact of at least $1.4K per store per week

Recommendation: Keep the Sunday flyer in rural and suburban areas;

discontinue in urban areas

Meeting with Business Owners

Stores Metric Lift Expected Impact

(Per Store Per Week)

Significance

Level Decision

All Stores Total Store Sales -1.5% -$2.7K 99.9% Success

Rural Total Store Sales -2.8% -$5.2K 100.0% Success

Suburban Total Store Sales -0.9% -$1.4K 99.0% Success

Urban Total Store Sales -0.3% -$0.5K 97.7% Fail

#inspire14

Questions

#inspire14

THANK YOU!