Causal Design and Marketing Experiments!

Chapter 4

Causal Design and Marketing Experiments!

1.  The Quest of Causality 2.  Experimentation 3.  Quasi-Experimentation 4.  Managerial Aspects of Experimentation and

Quasi Experimentation 5.  Four Experimental Design Procedures

14/04/2014   AMA   2  

1 – The Quest of Causality!•  “ I know quite certainly that I have no special gift.

Curiosity, obsession, and dogged endurance have brought me my ideas. (Albert Einstein)

•  The new advertising campaign caused 10% increased in the sales.

•  The new sales training program has resulted in lower sales force turnover.

•  Are there other possible factors or events that could have led to the observed changes?

•  Rigorously and honestly answering the question lies at the heart of investigations of causality and forms a major part of conclusive marketing research.

14/04/2014   AMA   3  

1 – The Quest of Causality!•  Managers and researchers must be able to

discern the conditions under which proper causal statements can be formulated and claimed to hold.

•  In marketing, not all conditions allow for tight, accurate causal statements.

14/04/2014   AMA   4  

1 – The Quest of Causality!•  Deterministic Causation: The ordinary concept

of causality that presumes that an effect always follows a cause; differs from the scientific concept of probabilistic causation conceptualizing effects in terms of their statistical probability.

•  Probabilistic Causation: The notion, common in the philosophy and science, that research can never truly prove causality, only infer with some degree of confidence.

14/04/2014   AMA   5  

1 – The Quest of Causality!•  Necessary Conditions for Causality •  The scientific concept is different and complex

as held by a common person. There is difference between the so-called common sense concepts of causality and scientific concept.

•  The commonsense view hold that a single event (the cause) always results in another event (the effect).

14/04/2014   AMA   6  

1 – The Quest of Causality!•  In the commonsense causality the effect always

follows whenever the cause take place. We refer to this as deterministic causation (X causes Y).

•  In contrast scientific notion specifies the effect only as being probable. This is called probabilistic causation (we can infer but we cannot prove this).

•  Statisticians also state that there is always a chance of error because data is never perfect (e.g., biases, coding errors, missing values, or entire variable). Such phenomena is even true in ‘hard’ sciences.

14/04/2014   AMA   7  

1 – The Quest of Causality!•  Marketing effects are caused by multiple forces

acting at once and, at best, we can only infer a causal relationship; we can never demonstrate it definitively.

•  Under the following three conditions researchers can claim to have made causal inferences, and all should be met. 1.  Concomitant Variation 2.  Time Occurrence of Variables 3.  Elimination of other Possible Causal Factors

14/04/2014   AMA   8  

1 – The Quest of Causality!•  Concomitant Variation is the extent to which a

cause, X, and an effect, Y, occur together or vary together in the way predicted by a hypothesis under consideration.

•  For example if a company does advertising to improve the attitude of people that would result in rising sales. We can say that there is a concomitant variation between the attitude and sales (improved attitude causes the sales to increase)

14/04/2014   AMA   9  

1 – The Quest of Causality!•  There can some issues here to understand the

probabilistic causation. 1.  Reverse Causation (people becoming more

experienced with the cars) 2.  Omitted Variable (the company may have

improved the quality of the cars) 3.  Insufficient Variation (advertising may have

increased the in-store visits )

14/04/2014   AMA   10  

1 – The Quest of Causality!•  Time Occurrence of Variables is that the one

event cannot cause another event if it occurs after the other event. The causing event must occur either before or simultaneously with the effect.

•  We can also do some research about the changing attitude of people towards the company before and after and if there is an improvement then we can say that hypothesis would be tenable.

•  And if sales only improved after the car purchase then hypothesis is untenable.

14/04/2014   AMA   11  

1 – The Quest of Causality!•  Elimination of other Possible Causal Factors should

be done as there can be a case where other possible causes were not systematically eliminated, and the research design did not allow for the identification of the true causal relationship.

•  Consider a case where a scientist wants to check the effect of artificial sweetener on sugar level of a person with a usual soft drink. She infers that there is causality present and artificial sweetener is not effective. Is the scientist right at this conclusion.

14/04/2014   AMA   12  

2 – Experimentation!A.   Basic Definitions in Experimentation

1)  Experiments 2)  Treatments 3)  Test Units 4)  Dependent Variables 5)  Extraneous Variables (Types : History, Maturation, Testing,

Instrumentation, Statistical Regression, Selection Bias, Test Unit Mortality)

6)  Experimental Design 7)  Validity in Experimentation

B.  Rigorously Defining Experiments – X-O-R Syntax C.  Three Pre-Experimental Design (One-Shot Case Study, One Group

Pre-Test-Post-Test Design, Static Group Comparison) D.   Three True-Experimental Design (Pre-Test-Post-Test Control

Group Design)

14/04/2014   AMA   13  

2 – Experimentation!•  Experimentation is the fundamental tool to

help identify causal relationships. •  The objective of an experiment is to measure

the effect of explanatory (independent) variables on another (dependent) variable of interest while controlling for other variables that might confuse the ability to make causal inferences.

14/04/2014   AMA   14  

2 – Experimentation!•  Experiment: A rigorous investigation in which the

researcher controls several (independent) variables and measures or observe their effects on one or more dependent variables.

•  Treatment: In experimental design, the manipulated independent variable.

•  Test Unit: In experimental design, the entity to whom (or to which) treatments are presented and whose responses to the treatments are measured.

•  Extraneous variables: Any variable, other than specific to the treatment administered to test units, that may affect the response of the test units to the treatments, including history effect, maturation effect, testing effect, instrumentation effect, statistical regression effect, selection bias, and test unit mortality; also referred to as confound.

14/04/2014   AMA   15  

2 – Experimentation!•  Confounding Variable (Confound): Any variable other than those

specific to the treatment, that may effect how test units respond to the treatments; makes it difficult to attribute effects to variables controlled or manipulated by the experimenter.

•  Experimental Design: A form of investigation in which the researcher can directly control one of more (independent) variables to study their effects on other (dependent) variable.

•  Internal validity: A way to evaluate experiments by verifying that changes in the dependent or criterion variable truly arose from changes in the independent or treatment variables alone.

•  External Validity: A method of evaluating the results of experiments; formally, the degree to which conclusions would hold under other, presumably identical circumstances.

•  History Effect: The occurrence of events outside of, but taking place at the same time as, the experiment that can affect the dependent variable.

14/04/2014   AMA   16  

2 – Experimentation!•  Maturation Effect: An effect similar to the history effect except that it

pertains to the changes in the experimental units themselves over time. •  Testing Effect: An effect that can come about when the pretest itself exerts

an influence on how participants perform on the post-test. •  Direct (main) Testing Effect: Occurs when the first of two observations

affects the second. •  Interactive (reactive) Testing Effect: In an experiment, the effect of the

test unit’s pretreatment measurement on the reaction to the treatment. •  Instrumentation Effect: In statistical models, an effect that arises when

instruments, observers, or scorers change over the course of an experiment.

•  Statistical Regression Effect: When, in an experiment, test units are selected (for exposure to the treatment) based on an extreme pretreatment score; usually considered poor research practice.

•  Test Unit Mortality: A serious problem that occurs when test units withdraw from an experiment before their role is complete.

14/04/2014   AMA   17  

2 – Experimentation!•  One-shot Case Study Design: A simple form of pre-experimental design

where a single group of test units is first exposed to a treatment and a measurement is then taken on the dependent variables without random assignment of test units to the treatment group.

•  One Group Pre-test-post-test Design: A simple form of pre-experimental design, equivalent to a one-shot case study design but where an additional pre-test measurement is taken before the treatment.

•  Static Group Comparison Design: An experimental design in which two treatment groups, one that has been exposed to the treatment and one that has not, are observed after the treatment has been presented.

•  Control Group: A group that is comparable to the treatment group in terms of measurable characteristics but did not receive the treatment.

•  Baseline: A figure that can be used to provide context for a comparison with another figure.

14/04/2014   AMA   18  

2 – Experimentation!•  Generally experiments help to answer following question

types: a)  Can we increase profits by servicing small accounts

through mail rather than walk-ins. b)  Can we increase sales by acquiring more shelf space. c)  Will the addition of some chemical (stannous fluoride) to

our toothpaste reduce user’ cavities. d)  Does the frequency of a sales call in a specific time period

help increase sales of a specific account. e)  What is better color or black and white in a print ad. f)  What is the optimal promotional techniques. g)  Is it necessary to change the attitude of subject to increase

sales.

14/04/2014   AMA   19  

2 – Experimentation!•  An Experiment is carried out when one of more

independent variables is deliberately manipulated or controlled by an experimenter in a planned fashion and the effects on the dependent variable (or variables) is measured.

•  In surveys or observational studies there is no manipulation of independent variables by the researcher. This is the fundamental difference between experimental and non-experimental research.

14/04/2014   AMA   20  

2 – Experimentation!•  In searching for causal relationships in non-

experimental situation the researcher must proceed in reverse: observe the effect and then search for a cause.

•  In this situation one can never be sure of the proper time order of occurrence of variables and effects of other possible independent variables that have been excluded from consideration.

14/04/2014   AMA   21  

2 – Experimentation!•  Treatments are the manipulated alternatives or

independent variables whose effects are then measured.

•  Examples in marketing are plentiful including product design, packaging or name, advertising themes, price levels, distribution strategies, and promotional incentives.

•  Treatments can be a nominal scale based which means that ordinal, interval, and ratio are also acceptable.

14/04/2014   AMA   22  

2 – Experimentation!•  For example if we are interested in sales

response the treatment can range from different ad designs (nominal) to various promotional price-reduction percentages.

•  Whatever the data type, researcher should clearly define treatments to be used in the research.

14/04/2014   AMA   23  

2 – Experimentation!•  The Test Units are the entities to whom (or to

which) the treatments are presented and whose response to the treatments is measured.

•  It is common in marketing for both people and physical locations such as stores, geographical areas to be used as test units.

•  For example people (test units) may be asked to try a product (the treatment) and then have their attitudes and opinions towards it measured (dependent variables).

14/04/2014   AMA   24  

2 – Experimentation!•  Alternatively, different aisle display

(treatments) may be set up in supermarkets (test units) and sales levels (dependent variables) cab be measured.

•  Conceptualizing a marketing experiment in these terms (test units, treatments, and dependent measures) during the planning phase helps avoid conceptual and analytical complications later.

14/04/2014   AMA   25  

2 – Experimentation!•  The Dependent Variables are the measures taken

on the test units. •  Typical marketing examples are sales, preference,

awareness, willingness to purchase, attitudes, and a host of related concepts.

•  It is desirable for dependent variable to be measured at interval level to allow for ease of analysis, so that regression analysis can be used.

14/04/2014   AMA   26  

2 – Experimentation!•  The Extraneous Variables are all variables

other than the treatments that potentially affect the response of the test units to the treatments. These variables can distort the dependent variable measures in such a way as to weaken or or entirely invalidate a researcher’s ability to make causal inferences, they are also called ‘nuisance factors’

14/04/2014   AMA   27  

2 – Experimentation!•  For example, a book publisher attempting to

measure the responses of buyers to two different cover designs would want to keep other aspects of the book the same for each buyer group. If the publisher allowed extraneous variables like price, book dimensions, title, or paper quality to vary between buyer groups, she could not be sure that she was measuring the effects of cover.

•  Those other variables are said to ‘confound’ the experiment and are often referred to as confounds.

14/04/2014   AMA   28  

2 – Experimentation!•  Researcher has three possible courses of action

with respect to extraneous variables. 1.  When practicable, an extraneous variable can be

controlled, price or dimensions of book can be held constant.

2.  If physical control is not possible, the assignment of treatments to test units may be randomized. The book publisher could randomly different prices to all buyers. In experiments with human test units, this usually takes the form of randomly assigning the test units to different treatments.

14/04/2014   AMA   29  

2 – Experimentation!–  In this way it is hoped that extraneous factors that

could plausibly affect the outcome (such as education, age, or product experience) are equally represented in each treatment group. In marketing research we can hardly ‘control’ human behavior and we have to rely on randomization. Control and randomization are not either-or. If we like to test a book cover with group of kids and another group of adults, we have to randomize to get the valid inference.

14/04/2014   AMA   30  

2 – Experimentation!3.  The third way to control the effects of extraneous

variables is through the use of specific experimental design (on following slides) that accomplish this purpose.

•  If physical control, randomization, and design features do not eliminate the differential effects of extraneous variables among treatment groups, the experiment has been confounded, and no causal statements are possible.

14/04/2014   AMA   31  

2 – Experimentation!•  As mentioned earlier we call such an

extraneous variable a confounding variable or simply a confound.

•  For example, suppose we are using two cities as our test units and it rains in one city and not in the other. If rain affects the dependent variable (say, the no. of car washes), the experiment has been confounded and rain was the confounding variable.

14/04/2014   AMA   32  

2 – Experimentation!•  An Experimental Design involves the

specification of treatments to be manipulated, test units to be used, dependent variables to be measured, and procedures for dealing with extraneous variables.

•  In Validity in Experimentation two concepts of validity are relevant in experimentation: internal validity and external validity.

14/04/2014   AMA   33  

2 – Experimentation!•  Internal Validity is the basic minimum validity

that must be present in an experiment before any conclusion about treatment effects can be made. It relates to whether the observed effects on the test units could have been caused by variables other than the treatment, that is, by extraneous variables.

•  Without internal validity the experiment is confounded and largely useless for purpose of causal inference.

14/04/2014   AMA   34  

2 – Experimentation!•  External Validity is concerned with the extent to

which the experimental results can be generalized beyond the experiment.

•  To what populations, geographic areas, treatment variables, and measurement variables can the measured effect be protected.

•  For example student researcher often rely on other students or stores near campus. Even if their experiments are otherwise flawless, it would be premature to claim that results extend outside the realm of their fellow students or vicinity.

14/04/2014   AMA   35  

2 – Experimentation!•  Ideally an experimental design should be

strong on both types of validity; unfortunately one type must often be traded against the other.

•  For example, an advertiser may ask respondents to view advertisements in a lab or rented space in a mall. Can any effect in such environments be generalized to home viewing environments?

14/04/2014   AMA   36  

2 – Experimentation!B. Rigorously Defining Experiments •  For researchers to collaborate and understand

the results of one another’s experiments, a common language is required.

•  To facilitate our discussion of specific experimental designs, we use a set of symbols common across social science in general and marketing research applications in particular, referred to as X-O-R Syntax.

14/04/2014   AMA   37  

2 – Experimentation!•  X represents the exposure of a test group to an

experimental treatment, the effects of which are to be determined. This is often referred to as a ‘treatment’.

•  O refers to process of observation or measurement of the dependent variable on test units (i.e., within the test group).

•  R indicates that individuals have been assigned at random to separate the treatment groups or that groups themselves have been allocated at random to separate treatments.

14/04/2014   AMA   38  

2 – Experimentation!•  Types of Extraneous Variables •  Extraneous variables needed to be controlled

to ensure that the experiment has not been confounded or experiment is internally valid.

•  History Effect refers to the occurrence of specific events that are external to the experience but that take place during the course of the experiment.

14/04/2014   AMA   39  

2 – Experimentation!•  For example, consider a design

O1 X1 O2 •  Where O1 and O2 measures the dollar sales of

personnel and X1 represents a new sales training program. The difference O1- O2 is the measurement of the treatment effect.

•  However, there can be some other reasons for it; improvement in general business conditions.

•  The greater the length of time, the greater the chance of history effect confounding an experiment of this type.

14/04/2014   AMA   40  

2 – Experimentation!•  Maturation Effect is similar to history except

that it is concerned with changes in the experimental units themselves that occur with the passage of time.

•  For example, getting older, getting hungrier, developing fatigue, and under going many types of learning or experience.

14/04/2014   AMA   41  

2 – Experimentation!•  Testing Effect is concerned with the possible

effects on the experiment of taking a measure on the dependent variable before presentation on the treatment. There are two kinds of testing effect. – Direct or Main Testing Effect, it occurs when the first

observation affects the second observation. For example if we ask respondents to fill the same questionnaire before and after the treatment, they may respond differently as now they are ‘expert’ with it, or just to express variety of opinion. The internal validity of the experiment is compromised.

14/04/2014   AMA   42  

2 – Experimentation!– Reactive or Interactive Testing Effect is a situation

test unit’s permanent measurement affects the reaction to the treatment. It influences external validity.

–  For example a pretreatment questionnaire that asks questions about shampoo may sensitize the respondent to the shampoo market and distort the awareness levels of a new introduction (the treatment). Hence, the measured effect cannot be generalized to other potential customers who were not similarly sensitized. There should not be even subtle cues from the researcher or from the environment to the respondents.

14/04/2014   AMA   43  

2 – Experimentation!•  Instruments Effect refers to changes in the

calibration of the measuring instrument used or changes in the observers or scorers.

•  If sales were measured in a different store or geographic location after the study, or if the accounting calculated net sales differently, the difference O1-O2 could be explained by this change in instrumentation.

14/04/2014   AMA   44  

2 – Experimentation!•  Statistical Regression Effects occur where test units have

been selected for exposure to the treatment on the basis of an extreme pre-treatment score.

•  For example, suppose that in the sales training example only poorly performing salespeople had been given the new training program. Subsequent sales increases might be attributed to the regression effect, because random occurrences such as weather, family problems, or luck helped define good or poor performance of the salespeople in the pretreatment measures. Subsequent random occurrences will make some of the poor performers do better the following year, thus confounding the experiment.

14/04/2014   AMA   45  

2 – Experimentation!•  Selection Bias refers to the assigning of test units

to treatment groups in such a way that the groups differ on the dependent variable before the presentation of the treatments.

•  Test units should be randomly assigned to treatment groups.

•  If test units self-select their own groups or are assigned to groups on the basis of researcher judgment, the possibility of selection bias exists.

14/04/2014   AMA   46  

2 – Experimentation!•  Test Unit Mortality refers to test units

withdrawing from experiment while it is in progress.

14/04/2014   AMA   47  

2 – Experimentation!•  Three Pre-Experimental Designs questions

the internal experimental validity. •  These can be used in the exploratory research

but should be avoided in the actual experiment.

14/04/2014   AMA   48  

2 – Experimentation!•  One Shot Case Study design is presented

symbolically as follows: X1 O1

•  A single group of test units is first exposed to treatment X1, and then a measurement is taken on the dependent variable.

•  The test units were self selected or selected by the experimenter using some criterion, hopefully reflecting representativeness, and not convenience sample.

14/04/2014   AMA   49  

2 – Experimentation!•  For example, a manager requests volunteer to take part

in a new sales training program, and a measure of their new sales performance is taken sometime after the program is completed.

•  The impossibility of drawing meaningful conclusions from such design should be apparent.

•  The level of O1 is the result of many uncontrolled factors. Thus, history, maturation, selection, and mortality problems all render this design internally invalid.

14/04/2014   AMA   50  

2 – Experimentation!•  One Group Pre-Test-Post-Test-Design is

represented symbolically as follows. O1 X1 O2

•  There is a pre-test measurement of sales performance to the one-shot study design and it provides a baseline O1.

14/04/2014   AMA   51  

2 – Experimentation!•  There can be problems like a)   maturation (team getting experienced through time), b)   Testing (pre-test could have affected performance), c)   Instrumentation (prices of goods could have changed), d)   Selection Bias (test units could have self-selected), e)   Mortality (some test could have dropped out), and f)  Regression (test units may have selected at a

particularly bad year). •  Even if the design is

R O1 X1 O2 •  we would be able to get rid of selection bias

14/04/2014   AMA   52  

2 – Experimentation!•  Static Group Comparison Design uses two

treatment groups. One that has been exposed to the treatment and the other one has not.

•  Both groups are observed only after the treatment has been presented, and test units are not randomly assigned to the groups.

•  Symbolically, the design is as follows

14/04/2014   AMA   53  

2 – Experimentation!Group 1: X1 O1 Group 2: O2

•  Group 2 is called a control group because it has not received the treatment and so may serve as a baseline for comparison.

•  In marketing we often define the control group treatment as the current level of marketing activity. This can be shown symbolically

Experimental Group 1: X1 O1 Control Group 2: X2 O2

14/04/2014   AMA   54  

2 – Experimentation!•  Here X2 is the baseline marketing program

with which we wish to compare X1. •  For example, in trying the new sales training

program on some people while keeping the old one intact for some of them.

•  There are few problems here also e.g. test units have been selected randomly, mortality issues.

14/04/2014   AMA   55  

2 – Experimentation!•  Three True Experimental Designs •  A true experimental design is one where

researcher is able to eliminate all extraneous variables as competitive hypotheses to the treatment, at least in theory.

14/04/2014   AMA   56  

2 – Experimentation!•  Pre-test-Post-test Control Group Design is

presented symbolically as follows. Experimental Group: R O1 X1 O2

Control Group: R O3 O4

•  Here, X1 is the treatment of interest (if there is a baseline then there could be a possible X2.

14/04/2014   AMA   57  

2 – Experimentation!•  Random assignment of the groups eliminates selection

bias as a potential confounding variable. •  The premise here is that all extraneous variables

operate equally on the both the experimental group and the control group. The only difference between the groups is the presentation of the treatment to the experimental group.

•  Therefore, O2 – O1 is the sum of the treatment effect and the effects of the extraneous variables, whereas O4 – O3 accounts for the extraneous variables only.

14/04/2014   AMA   58  

2 – Experimentation!•  In symbols, O2- O1 = EXT+TE

O3- O4 = EXT •  Where TE is the treatment effect and EXT is

the sum of all extraneous effect (history, maturation, testing, instrumentation, regression, test unit mortality). (O2- O1) – (O4 – O3) = TE

•  We have found out the effect of independent variable.

14/04/2014   AMA   59  

2 – Experimentation!•  All threats to the internal validity are

addressed in this design. But there is a lacking of not mentioning the extraneous variables and their effects are assumed to be common.

•  If the extraneous variable is acting different on one of the group then it would be confounding.

14/04/2014   AMA   60  

3 – Quasi Experimentation!•  In designing a research, the researcher often

creates artificial environments to have control over independent and extraneous variables.

•  In this case, there would be higher internal validity and less external validity.

•  One response to this problem is the quasi-experimental design.

14/04/2014   AMA   61  

3 – Quasi Experimentation!•  To confer internal validity the researcher tries

to control for extraneous and independent variables, but this would raise the questions regarding external validity.

•  Control over ‘when’ and ‘to whom’ of measurement but lacks control of ‘when’ and ‘to whom’ of exposure.

14/04/2014   AMA   62  

3 – Quasi Experimentation!•  Time Series Experiment is the periodic

measurement on the dependent variables for some test units.

•  Consumer purchase panels providing periodic activity (Os). A marketer may undertake new advertising campaign (X).

•  Here the marketer has control over advertising timing but not advertising exposure.

14/04/2014   AMA   63  

3 – Quasi Experimentation!•  A: the campaign has

both short and long run positive effects.

•  B: there is a short-run positive effect.

•  C: delayed but seems a long run effect.

•  D, E, F: cannot be inferred as changes already occuring.

14/04/2014   AMA   64  

4 – Managerial Aspects!•  Comparison with other procedures: Most the

techniques used in marketing are descriptive ones (secondary data, observation, surveys, panels and web log analysis and simulation)

•  Constraints regarding time, money and other resources may limit the feasibility of causal research.

•  Researcher can be right inferring the causal relationship but not 100% sure.

14/04/2014   AMA   65  

4 – Managerial Aspects!•  Laboratory vs. Field Experiments •  Laboratory would be showing off TV

commercials to test units while field would running comm. during actual TV programs.

•  There are issues of validity, accuracy, cost, control variables, and others.

•  Laboratory Experiments would have lower external validity, low cost, and require less time.

14/04/2014   AMA   66  

4 – Managerial Aspects!•  Limitations of Experimentation: It is not possible

to control for extraneous variables. •  In field experiments lack of cooperation can be an

issue. •  Lack of knowledge of a experiment procedures. •  Experiments are costly, time consuming, and

requires large amount of data for expert analysis. •  People would be test units most of the time and

care should be taken to avoid biasedness.

14/04/2014   AMA   67  

4 – Managerial Aspects!•  Stages in Conducting an Experiment 1.  State the problem 2.  Formulate hypothesis (es). 3.  Construct experimental design. 4.  Be sure that design answers the question. 5.  Analysis of data (before actually conducting the

experiments) 6.  Perform the experiment. 7.  Apply statistical procedures (whether effects are real or

there is noise) 8.  Draw conclusion while paying attention to the internal

and external validity.

14/04/2014   AMA   68