fundamentals of program impact evaluation

Peter M. Lance, PhDMEASURE Evaluation University of North Carolina at Chapel Hill

MARCH 31, 2016

Fundamentals of Program Impact Evaluation

Global, five-year, $180M cooperative agreement

Strategic objective:

To strengthen health information systems – the capacity to gather, interpret, and use data – so countries can make better decisions and sustain good health outcomes over time.

Project overview

Improved country capacity to manage health information systems, resources, and staff

Strengthened collection, analysis, and use of routine health data

Methods, tools, and approaches improved and applied to address health information challenges and gapsIncreased capacity for rigorous evaluation

Phase IV Results Framework

Global footprint (more than 25 countries)

How Do We Know If A Program Made A Difference? A Brief Helicopter Tour of Methods for Estimating Program

Impact

• The Program Impact Evaluation Challenge

• Randomization

• Selection on observables

• Within estimators

• Instrumental variables

• The Program Impact Evaluation Challenge

• Randomization

• Selection on observables

• Within estimators

• Instrumental variables

• The Program Impact Evaluation Challenge

• Randomization

• Selection on observables

• Within estimators

• Instrumental variables

• The Program Impact Evaluation Challenge

• Randomization

• Selection on observables

• Within estimators

• Instrumental variables

• The Program Impact Evaluation Challenge

• Randomization

• Selection on observables

• Within estimators

• Instrumental variables

• The Program Impact Evaluation Challenge

• Randomization

• Selection on observables

• Within estimators

• Instrumental variables

• The Program Impact Evaluation Challenge

• Randomization

• Selection on observables

• Within estimators

• Instrumental variables

Newton’s “Laws” of Motion

Did the program make a difference?

Did the program cause a change in an outcome of interest Y ?

(Causality)

Our outcome of Interest

What happens if an individual does not participate in a program

What happens if that individual does participate in a program

Potential Outcomes

:

:

:

Our outcome of interest

What happens if an individual does not participate in a program

What happens if that individual does participate in a program

Potential Outcomes

:

:

:

Our outcome of interest

What happens if an individual does not participate in a program

What happens if that individual does participate in a program

Potential Outcomes

:

:

:

Our outcome of interest

What happens if an individual does not participate in a program

What happens if that individual does participate in a program

Potential Outcomes

:

:

:

What happens if

the individual participates

{Causal} Program Impact

Program Impact

What happens if

the individual does not

participate

What happens if

the individual participates

{Causal} Program Impact

Program Impact

What happens if

the individual does not

participate

What happens if

the individual participates

{Causal} Program Impact

Program Impact

What happens if

the individual does not

participate

What happens if

the individual participates

{Causal} Program Impact

Program Impact

What happens if

the individual does not

participate

What happens if

the individual participates

{Causal} Program Impact

Program Impact

What happens if

the individual does not

participate

𝑃 𝑖={1 if   individual   𝑖participates                  ¿0 if   individual   𝑖does  not   participate

Program Participation

𝑌 𝑖=𝑃 𝑖∙𝑌 𝑖1+(1−𝑃 𝑖 ) ∙𝑌 𝑖

0

Observed Outcome

𝑌 𝑖=𝑃 𝑖∙𝑌 𝑖1+(1−𝑃 𝑖 ) ∙𝑌 𝑖

0

Observed Outcome

𝑃 𝑖=1

𝑌 𝑖=1∙𝑌 𝑖1+ (1−1 ) ∙𝑌 𝑖

0

Observed Outcome

𝑃 𝑖=1

𝑌 𝑖=𝑌 𝑖1+0 ∙𝑌 𝑖

0

Observed Outcome

𝑃 𝑖=1

𝑌 𝑖=𝑌 𝑖1

Observed Outcome

𝑃 𝑖=1

𝑌 𝑖=𝑃 𝑖∙𝑌 𝑖1+(1−𝑃 𝑖 ) ∙𝑌 𝑖

0

Observed Outcome

𝑌 𝑖=𝑃 𝑖∙𝑌 𝑖1+(1−𝑃 𝑖 ) ∙𝑌 𝑖

0

Observed Outcome

𝑃 𝑖=0

𝑌 𝑖=0 ∙𝑌 𝑖1+(1−0 ) ∙𝑌 𝑖

0

Observed Outcome

𝑃 𝑖=0

𝑌 𝑖=𝑌 𝑖0

Observed Outcome

𝑃 𝑖=0

{𝑌 𝑖1 ,𝑌 𝑖

0 }

Observed Outcome

{𝑌 𝑖1 ,𝑌 𝑖

0 }

Observed Outcome

{𝑌 𝑖1 ,𝑌 𝑖

0 }

Observed Outcome

{𝑌 𝑖1 ,𝑌 𝑖

0 }

Observed Outcome

{𝑌 𝑖1 ,𝑌 𝑖

0 }

Observed Outcome

Fundamental Identification Problem

of Program Impact Evaluation

{𝑌 𝑖1 ,𝑌 𝑖

0 }

Observed Outcome

Fundamental Identification Problem

of Program Impact Evaluation

Individual Population

Individual Population Hi. They call me

individual i

Individual Population ?!?

{𝑌 𝑖1 ,𝑌 𝑖

0 }

{𝑌 𝑖1 ,𝑌 𝑖

0 }

An expected value for a random variable is the average value from a large number of repetitions of the experiment that random variable represents

An expected value is the true average of a random variable across a population

Expected Value

An expected value for a random variable is the average value from a large number of repetitions of the experiment that random variable represents

An expected value is the true average of a random variable across a population

Expected Value

An expected value is the true average of a random variable across a population

Expected Value

Expectations: Properties

Expectations: Properties

Expectations: Properties

Expectations: Properties

Expectations: Properties

Expectations: Properties

Expectations: Properties

Expectations: Properties

Expectations: Properties

Expectations: Properties

Average Treatment Effect (ATE)

Average Effect of Treatment on the Treated (ATT)

Hi there

Individual Impact

𝑌 𝑖1−𝑌 𝑖

0

𝐸 (𝑌 𝑖1−𝑌 𝑖

0 )

Average Treatment Effect (ATE)

Average Effect of Treatment on the Treated (ATT)

Treatment Effects

Suppose that we have a sample of individuals….

…but for each individual we observe either or …

…but not both

So how do we estimate??

Suppose that we have a sample of individuals….

…but for each individual we observe either or …

…but not both

So how do we estimate??

Remember, however, a key property of expectations:

…but this means that in principle we could estimate and

separately

So how do we estimate??

Remember, however, a key property of expectations:

…but this means that in principle we could estimate and

separately

So how do we estimate??

For instance, suppose that in our sample we have:

participants()

and

non-participants()

(hence )

So how do we estimate??

Then an estimator of is

calculated with the participants out of the sample of individuals

So how do we estimate??

Then an estimator of is

calculated with the participants out of the sample of individuals

So how do we estimate??

Then an estimator of is

calculated with the participants out of the sample of individuals

So how do we estimate??

Then an estimator of is

calculated with the participants out of the sample of individuals

So how do we estimate??

Then an estimator of is

calculated with the participants out of the sample of individuals

So how do we estimate??

Then an estimator of is

calculated with the participants out of the sample of individuals

So how do we estimate??

Similarly, an estimator of is

calculated with the non-participants out of the sample of individuals

So how do we estimate??

So then an estimate of

is

So how do we estimate??

But is it a good estimate??

So we have two samples of size

By random chance, between the two samples we almost surely have

1. A different precise mix of individuals

2. A different number of participants () and non-participants ()

3. Different estimates and of and : 𝑌 1=

∑𝑗=1

𝑛𝑃

𝑌 𝑗

𝑛𝑃 =∑𝑗=1

𝑛𝑃

𝑌 𝑗1

𝑛𝑃

𝑌 0=∑𝑘=1

𝑛𝑁

𝑌 𝑘

𝑛𝑁 =∑𝑘=1

𝑛𝑁

𝑌 𝑘0

𝑛𝑁

So we have two samples of size

By random chance, between the two samples we almost surely have

1. A different precise mix of individuals

2. A different number of participants () and non-participants ()

3. Different estimates and of and : 𝑌 1=

∑𝑗=1

𝑛𝑃

𝑌 𝑗

𝑛𝑃 =∑𝑗=1

𝑛𝑃

𝑌 𝑗1

𝑛𝑃

𝑌 0=∑𝑘=1

𝑛𝑁

𝑌 𝑘

𝑛𝑁 =∑𝑘=1

𝑛𝑁

𝑌 𝑘0

𝑛𝑁

So we have two samples of size

By random chance, between the two samples we almost surely have

1. A different precise mix of individuals

2. A different number of participants () and non-participants ()

3. Different estimates and of and : 𝑌 1=

∑𝑗=1

𝑛𝑃

𝑌 𝑗

𝑛𝑃 =∑𝑗=1

𝑛𝑃

𝑌 𝑗1

𝑛𝑃

𝑌 0=∑𝑘=1

𝑛𝑁

𝑌 𝑘

𝑛𝑁 =∑𝑘=1

𝑛𝑁

𝑌 𝑘0

𝑛𝑁

So we have two samples of size

By random chance, between the two samples we almost surely have

1. A different precise mix of individuals

2. A different number of participants () and non-participants ()

3. Different estimates and of and : 𝑌 1=

∑𝑗=1

𝑛𝑃

𝑌 𝑗

𝑛𝑃 =∑𝑗=1

𝑛𝑃

𝑌 𝑗1

𝑛𝑃

𝑌 0=∑𝑘=1

𝑛𝑁

𝑌 𝑘

𝑛𝑁 =∑𝑘=1

𝑛𝑁

𝑌 𝑘0

𝑛𝑁

So we have two samples of size

By random chance, between the two samples we almost surely have

1. A different precise mix of individuals

2. A different number of participants () and non-participants ()

3. Different estimates and of and : 𝑌 1=

∑𝑗=1

𝑛𝑃

𝑌 𝑗

𝑛𝑃 =∑𝑗=1

𝑛𝑃

𝑌 𝑗1

𝑛𝑃

𝑌 0=∑𝑘=1

𝑛𝑁

𝑌 𝑘

𝑛𝑁 =∑𝑘=1

𝑛𝑁

𝑌 𝑘0

𝑛𝑁

So we have two samples of size

By random chance, between the two samples we almost surely have

1. A different precise mix of individuals

2. A different number of participants () and non-participants ()

3. Different estimates and of and : �̂� 𝟏=

∑𝒋=𝟏

𝒏𝑷

𝒀 𝒋

𝒏𝑷 =∑𝒋=𝟏

𝒏𝑷

𝒀 𝒋𝟏

𝒏𝑷

𝑌 0=∑𝑘=1

𝑛𝑁

𝑌 𝑘

𝑛𝑁 =∑𝑘=1

𝑛𝑁

𝑌 𝑘0

𝑛𝑁

�̂� 𝟏 𝑬 (𝒀𝟏 )

�̂� 𝟏=∑𝒋=𝟏

𝒏𝑷

𝒀 𝒋

𝒏𝑷 =∑𝒋=𝟏

𝒏𝑷

𝒀 𝒋𝟏

𝒏𝑷

�̂� 𝟏=∑𝒋=𝟏

𝒏𝑷

𝒀 𝒋

𝒏𝑷 =∑𝒋=𝟏

𝒏𝑷

𝒀 𝒋𝟏

𝒏𝑷

�̂� 𝟏=∑𝒋=𝟏

𝒏𝑷

𝒀 𝒋

𝒏𝑷 =∑𝒋=𝟏

𝒏𝑷

𝒀 𝒋𝟏

𝒏𝑷

𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )

�̂� 𝟏=∑𝒋=𝟏

𝒏𝑷

𝒀 𝒋

𝒏𝑷 =∑𝒋=𝟏

𝒏𝑷

𝒀 𝒋𝟏

𝒏𝑷

𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )

𝑬 (�̂� 𝟏)=𝑬 (∑𝒋=𝟏𝒏𝑷

𝒀 𝒋𝟏

𝒏𝑷 )𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )

𝑬 (�̂� 𝟏)=𝒏𝑷 ∙𝑬 (∑𝒋=𝟏𝒏𝑷

𝑬 (𝒀 𝒋𝟏 ))

𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝑬 (∑𝒋=𝟏

𝒏𝑷

𝑬 (𝒀 𝒋𝟏))

𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝒏

𝑷 ∙𝑬 (𝒀 𝒋𝟏 )

𝑬 (�̂� 𝟏)=𝑬 (𝒀 𝒋𝟏)

1ST Rule:

2nd Rule:

𝑬 (�̂� 𝟏)=𝑬 (∑𝒋=𝟏𝒏𝑷

𝒀 𝒋𝟏

𝒏𝑷 )𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )

𝑬 (�̂� 𝟏)=𝒏𝑷 ∙𝑬 (∑𝒋=𝟏𝒏𝑷

𝑬 (𝒀 𝒋𝟏 ))

𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝑬 (∑𝒋=𝟏

𝒏𝑷

𝑬 (𝒀 𝒋𝟏))

𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝒏

𝑷 ∙𝑬 (𝒀 𝒋𝟏 )

𝑬 (�̂� 𝟏)=𝑬 (𝒀 𝒋𝟏)

1ST Rule:

2nd Rule:

𝑬 (�̂� 𝟏)=𝑬 (∑𝒋=𝟏𝒏𝑷

𝒀 𝒋𝟏

𝒏𝑷 )𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )

1ST Rule: 1ST Rule:

2nd Rule:

𝑬 (�̂� 𝟏)=𝑬 (∑𝒋=𝟏𝒏𝑷

𝒀 𝒋𝟏

𝒏𝑷 )𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )

𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙(∑𝒋=𝟏

𝒏𝑷

𝑬 (𝒀 𝒋𝟏 ))

1ST Rule: 1ST Rule:

2nd Rule:

𝑬 (�̂� 𝟏)=𝑬 (∑𝒋=𝟏𝒏𝑷

𝒀 𝒋𝟏

𝒏𝑷 )𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )

𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙(∑𝒋=𝟏

𝒏𝑷

𝑬 (𝒀 𝒋𝟏 ))

𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝒏

𝑷 ∙𝑬 (𝒀 𝒋𝟏 )

𝑬 (�̂� 𝟏)=𝑬 (𝒀 𝒋𝟏)

1ST Rule:

𝑬 (�̂� 𝟏)=𝑬 (∑𝒋=𝟏𝒏𝑷

𝒀 𝒋𝟏

𝒏𝑷 )𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )

𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙(∑𝒋=𝟏

𝒏𝑷

𝑬 (𝒀 𝒋𝟏 ))

𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝒏

𝑷 ∙𝑬 (𝒀 𝒋𝟏 )1ST Rule:

𝑬 (�̂� 𝟏)=𝑬 (∑𝒋=𝟏𝒏𝑷

𝒀 𝒋𝟏

𝒏𝑷 )𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )

𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝑬 (∑𝒋=𝟏

𝒏𝑷

𝑬 (𝒀 𝒋𝟏))

𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝒏

𝑷 ∙𝑬 (𝒀 𝒋𝟏 )

𝑬 (�̂� 𝟏)=𝑬 (𝒀 𝒋𝟏)

𝑬 (�̂� 𝟏)=𝑬 (𝒀 𝒋𝟏)

𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )

𝑬 (𝒀 𝒋𝟏 )=𝑬 (𝒀𝟏 )

𝑬 (�̂� 𝟏)=𝑬 (𝒀 𝒋𝟏)

𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )

𝑬 (𝒀 𝒋𝟏 )=𝑬 (𝒀𝟏 )

𝑬 (�̂� 𝟏)=𝑬 (𝒀 𝒋𝟏)

𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )

𝑬 (𝒀 𝒋𝟏 )=𝑬 (𝒀𝟏 )

𝑬 (𝒀 𝒋𝟏 )=𝑬 (𝒀𝟏 )

𝑬 (𝒀𝟏 )

𝑃=0𝑃=0

𝑃=0

𝑃=1𝑃=1

𝑃=1

𝑃=0

𝑃=1𝑃=1

𝑃=1

𝑃=0

𝑃=1

𝑃=1

𝑃=0

𝑃=0

𝑃=1

𝑃=1

𝒀 𝟏

𝑃=0𝑃=0

𝑃=0

𝑃=1𝑃=1

𝑃=1

𝑃=0

𝑃=1𝑃=1

𝑃=1

𝑃=0

𝑃=1

𝑃=1

𝑃=0

𝑃=0

𝑃=1

𝑃=0

𝒀 𝟏

𝑃=1𝑃=1

𝑃=1

𝑃=1𝑃=1

𝑃=1

𝑃=1

𝑃=1

𝑃=1

𝒀 𝟏

Z W

“Z Causes W”

𝑬 (𝑾∨𝒁 )≠𝑬 (𝑾 )

Z W

“Z causes W”

𝑬 (𝑾∨𝒁 )≠𝑬 (𝑾 )

Z W

“Z causes W”

𝑬 (𝑾∨𝒁 )≠𝑬 (𝑾 )

X Y1

X

Y

P

X

Y

P

0

X

Y

P

X

Y

P

X Y1

X Y1

X Y1

X

Y

P

𝑃=1𝑃=1

𝑃=1

𝑃=1𝑃=1

𝑃=1

𝑃=1

𝑃=1

𝑃=1

Selection Bias

The estimator

of

would be biased if some individuals occurred only among participants or non-participants

Or more often among one of the two groups

X

Y

P

X

Y

P

Sir Austin Bradford Hill

Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more

effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?

Bradford Hill Criteria

Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more

effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?

Bradford Hill Criteria

Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more

effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?

Bradford Hill Criteria

Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more

effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?

Bradford Hill Criteria

Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more

effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?

Bradford Hill Criteria

Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more

effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?

Bradford Hill Criteria

Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more

effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?

Bradford Hill Criteria

Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more

effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?

Bradford Hill Criteria

Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more

effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?

Bradford Hill Criteria

Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more

effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?

Bradford Hill Criteria

Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more

effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?

Bradford Hill Criteria

We are presented with data in the form of a sample:

Causality: Our Approach

,

We are presented with data in the form of a sample:

Causality: Our Approach

,

Assumptions

ModelE(Y1-Y0),

E(Y1-Y0|P=1),Etc.

We are presented with data in the form of a sample:

Causality: Our Approach

,

Assumptions

ModelE(Y1-Y0),

E(Y1-Y0|P=1),Etc.

Conclusion

Links:

The manual:

http://www.measureevaluation.org/resources/publications/ms-14-87-en

The webinar introducing the manual:

http://www.measureevaluation.org/resources/webinars/methods-for-program-impact-evaluation

My email:pmlance@email.unc.edu

MEASURE Evaluation is funded by the U.S. Agency for International Development (USAID) under terms of Cooperative Agreement AID-OAA-L-14-00004 andimplemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International, John Snow, Inc., Management Sciences for Health, Palladium Group, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government.www.measureevaluation.org