Experimental Design: Part I

MAR 6648: Marketing ResearchFebruary 3, 2010

Overview

• What are the basic features of an experiment?• How do those features get implemented in a

real experiment?• How do we adapt experiments to meet our

goals and resources?

1. Experimentation is the conscious manipulation of one or more variables by the experimenter in such a way that its effect on one or more variables can be measured.

2. The variable being manipulated is called the independent variable (a.k.a. cause).3. The variable being measured is called the dependent variable (a.k.a. effect).4. Elimination of other possible causal factors: i.e., the research design should rule

out the other factors (exogenous variables) as potentially causal ones. 5. This is typically done through random assignment to condition

An example of an experiment

• Suppose you want to know whether commercials make people enjoy TV shows less

• This means you’ll want to have some shows without commercials, and some shows with them– Therefore, commercials (or not) is the independent

variable

• And you’ll want to measure enjoyment of the TV shows they watch– Therefore, enjoyment is the dependent variable

Conditions

• Not in terms of what you can and can’t do…• Each independent variable (or combination of

IVs) is called a condition

Condition 1 Condition 2

Hypotheses

• Experimentation is essentially the process of trying to determine which of two hypotheses is not false

• The null hypothesis:– H0: Usually that there are no differences between

conditions• The alternative hypothesis:– H1: usually that there is a difference between conditions

• P-values in stats essentially represent the likelihood that we found evidence for H1 by chance alone

Confirmation Bias

• We are inclined to confirm our beliefs but less inclined (or able) to disconfirm them

• A real world example:– Business managers don’t keep track of those they

don’t hire

• Why?– Theories lead to unwarranted confidence– Inability to search out disconfirmation– Fixation or mental set

Control condition

• Control conditions allow us to see that our manipulations caused (or didn’t cause) a change in the dependent variable

• Usually a control condition is just no manipulation– This is sometimes done by adjusting when you run

your manipulation• Sometimes, though, you want to compare your

new manipulation to what’s typically done now– The control condition may be the standard or default

Random Assignment

• This essentially means that any one participant is equally likely to be in any condition– Usually you put your conditions in random order,

and assign participants in the order that they “arrive”

– Computers now allow you to assign people on the spot• Randomizer.org or random.org are good sources

An example of an experiment• The hallmark of an experiment is random

assignment to conditions– Let’s say the groups (the commercial watchers and

the people who watch it straight through) now look different!

– Random assignment means that the two groups should not have differed systematically at the start

– It also means that only your independent variable was different between groups

• Random assignment and manipulation of the IV mean that you can infer that the IV causes a change in the DV

An example of an experiment

• Question: do commercials make you enjoy a TV show less? Do people correctly predict this?

• Randomly assign your participants to groups– Half will predict how they enjoy a TV show with or

without them, half will actually experience it and report how they feel

– Half will watch a TV show with commercials, half will watch the same show without them

• Measure enjoyment or predicted enjoyment

An example of an experiment

Nelson, Meyvis, & Galak, 2009

Example

• Objective: GAP wishes to gauge whether new more aggressive sales techniques employed by store assistants increase sales

• What is the best experimental design?

Experiment 1

• Design: – 50 stores are sampled at random and assistants

are trained in the new approach

Metric = Metric =

MINUS

Average sales for the 50 storesin the next six months

Average sales for the 50 storesin the next six months

Average sales for the 50 storesin the prior six months

Average sales for the 50 storesin the prior six months

Notation• X = Exposure of a sample to the independent

variable (i.e., what we manipulate – “treatment”)

• O = Observation of measurement of the dependent variable (i.e., what we measure / want to affect)

• Movement through time is represented by the horizontal arrangement of Xs and Os from left to right.

Experiment 1: One group – before after

Causal Effect of X = O2-O1

Problems with this design?

• History or maturation• Defensiveness• Mortality• Instrumentation

Experiment 2

• Design: – 50 stores are sampled at random and the

assistants are trained in the new approach– Another 50 stores are sampled at random as

control

Metric = Metric =

MINUS

Average sales for the 50 test storesin the next six months

Average sales for the 50 test storesin the next six months

Average sales for the 50 control storesin the next six months

Average sales for the 50 control storesin the next six months

Experiment 2: Two group – only after

Causal Effect of X = O2-O1

Problems with this design?

Experiment 3

• Design: – 50 stores are sampled at random and assistants

are trained in the new approach– Another 50 stores are sampled at random as

controlMetric = Metric =

MINUS

MINUS

MINUS

Average sales for the 50 test stores in the next six monthsAverage sales for the 50 test stores in the next six months

Average sales for the 50 test stores in the prior six monthsAverage sales for the 50 test stores in the prior six months

Average sales for the 50 control stores in the next six monthsAverage sales for the 50 control stores in the next six months

Average sales for the 50 control stores in the prior six monthsAverage sales for the 50 control stores in the prior six months

Experiment 3: Two group – before after

Causal Effect of X = O4-O3 – (O2-O1)

More Advanced Experiments

• We have so far mainly looked at simple experiments

• But often we need to test several variables • When deciding on a marketing plan for a new

product there are many factors involved

Factorial Design

• Suppose we wish to test both product price and web-design for an e-business

$9.99 $14.99 $19.99

Design 1

Design 2

Price

Des

ign

Full FactorialDesign!

Full FactorialDesign!

Interactions and main effects

Factorial Design

• What do we do if we have many factors and levels?

• Example: – 5 prices, 4 product designs, 3 ad-copies 5*4*3

= 60 experimental cells!

• Solution: Use a fractional factorial design– Only use a subset of all 60 cells in experiment – Rely on regression analysis to extrapolate

Latin Squares

1st ad 2nd ad 3rd ad 4th ad

Group #1 Positive ad, Male speaker

Positive ad, Female speaker

Negative ad, Female speaker

Negative ad, Male speaker

Group #2Positive ad,

Female speaker

Negative ad, Male speaker

Positive ad, Male speaker

Negative ad, Female speaker

Group #3 Negative ad, Male speaker

Negative ad, Female speaker

Positive ad, Female speaker

Positive ad, Male speaker

Group #4Negative ad,

Female speaker

Positive ad, Male speaker

Negative ad, Male speaker

Positive ad, Female speaker

Latin Squares

1st ad 2nd ad 3rd ad 4th ad

Group #1 α β δ γ

Group #2 β γ α δ

Group #3 γ δ β α

Group #4δ α γ β

An experiment?

• Steve was interested to see how much labels on wine bottles affect how much people enjoy the wine inside them. At a party, he served the wines like normal, leaving the bottles out for people to pour from, labels still on. He asked everyone to indicate which wine they liked the best. At the next party he threw, he poured the wine into decanters, so that his guests couldn’t see the labels when they poured the wine. They again indicate which wine they liked best, and they had different preferences from the last party.

An experiment?

• The owner of two McDonalds franchises here in Gainesville wants to see if transactions run more quickly if he uses both drive-thru windows or only one. He picks one restaurant to use both windows at all times for a month, and the other he has closed at all times for a month. He finds that the drive-thru that uses both windows has notably faster service times.

An experiment?

An experiment?

Summary

• Experiments are very useful for determining causality– The main hallmarks of experiments are random

assignment to condition, manipulation of the independent variable, and a control group

– There are many different types of experiments, which vary largely on whether they are run within or between subjects (or both), when the manipulation is run, and how many conditions are used