The Intersection of Performance Measurement and

Program Evaluation:Searching for the Counterfactual

Moscow 2011

Douglas J. BesharovSchool of Public PolicyUniversity of Maryland

Performance Management

4 Efficiency studies (“outputs”)4 How much does the program cost?

4 Monetary, nonmonetary, and opportunity costs4 Could it be delivered more efficiently?

4 Effectiveness studies (“outcomes” and “impacts”)4 Does the program achieve its goals?4 Could it be more effective?

4 Both require a comparison, or a “counterfactual”

8Douglas J. Besharov, June 2011

Appearances Can Be DeceivingGiving children a “Head Start”

Douglas J. Besharov, June 2011 2

It Matters How Children Are Raised

Douglas J. Besharov, June 2011 3

Ineffective Early Childhood Education ProgramsIHDP

1985-1988

• Low-birth weight, pre-term infants and their parents

• Home visits, parenting education, and early childhood education

• $20,400 per child per year

• No significant impacts, initial IQ gains fade

CCDP1990-1995

• Poor children under age 1 and their parents

• Case management, parenting education, early childhood education, and referrals to community-based services

• $19,000 per family per year ($60 million annually)

• No significant impacts

Early Head Start1996-2008

• Poor children ages 0-2 and their parents

• Child development, parenting education, child care, and family support services

• $18,500 per child per year ($700 million annually)

• No significant impacts

Douglas J. Besharov, June 2011 4

Douglas J. Besharov (October 21, 2008)

“The closest thing to immortality on this Earth is a federal government program.”

– Ronald Reagan

Program Improvement, not Program Dismantling

Performance Management

4 Leadership,

4Management, and

4Measurement

6Douglas J. Besharov, June 2011

Performance Management

4 Leadership,

4Management, and

4Measurement

7Douglas J. Besharov, June 2011

Point #1

Counterfactuals are needed for accurate performance measurement.

9Douglas J. Besharov, June 2011

4 Head Start Impact Study (2010): 10 years and running

4 Moving to Opportunity Study (1994): 17 years and running

4 Employment Retention and Advancement evaluation (1998): 13 years and running

4 Building Strong Families Project (2002): 9 years and running

4 National Job Corps Study (1993): 15 years to complete

Impact Evaluations Take Too Long

to Manage Performance

Douglas J. Besharov, June 2011 10

Logic Model for Job Training Programs

Inputs

Training facilities

Staff

Funding

Client characteristics

Activities

Job search/job readiness training

Classroom instruction

Job skills training

Outputs

Hours of training instruction

Hours of practice

Staff admin

Skill certificates

Outcomes

Job search skills

Technical job skills

Interpersonal skills

ProximalImpacts

Earnings Employment

UI/Welfare Receipt

Crime

DistalImpacts

Higher lifetime

earnings/employment

Lower poverty

Stronger economy

Theory: If government provides job training to the unemployed, than the unemployed will receive job skills necessary for good jobs, increased earnings, and a stronger economy

Design: (1) Job search/job readiness training, (2) skills training, (3) in a classroom.

External Community and Societal Context

Problem: Some unemployed do not have the necessary skills to obtain and keep well-paying employment, leading to lower income, greater use of government benefits, and a weaker economy.

Point #2

Carefully applied, a measured outcome coupled with a logic model’s theory of change

—often buttressed by other evidence—can serve as a more timely and more

useful performance measure than a formal evaluation of long-term impacts.

12Douglas J. Besharov, June 2011

4 There is no output, so no positive outcome can be reasonably predicted.

4 The output itself is sufficiently suggestive of a likely outcome.

4 The output is produced at such a prohibitively high cost, that, regardless of its likely outcome, it does not meet cost-effectiveness or cost-benefit tests.

When “Outputs” Imply “Outcomes”

Douglas J. Besharov, June 2011 13

4 Evaluations of on-going programs4 Rolling randomized experiments

4 Pre-post studies (with embedded counterfactual)

4 Regression-discontinuity designs

4 Evaluations of specific program “improvements”4 Randomized experiments

4 Pipeline studies (or rolling implementation)

4 Interrupted time series studies

Feasible “Outcome” Evaluations

Douglas J. Besharov, June 2011 14

A Clear Interrupted Time Series

Douglas J. Besharov, June 2011 15

Circling the Wagons

Douglas J. Besharov, June 2011 16

4 Top-down administrative and funding incentives

-- together with --

4 Bottom-up voucher-like programs

Accountability Systems

Douglas J. Besharov, June 2011 17

Appearances Can Be DeceivingGiving children a “Head Start”

Douglas J. Besharov, June 2011 2

When “Outcomes” Imply “Impacts”

4When the desired impact is reasonably predicted to follow from the measured outcome

Douglas J. Besharov, June 2011 10

Teen Pregnancy in Anson County, NCTeen Pregnancy in Anson County, NC20012001--20082008Scale: 0Scale: 0--100100

0

20

40

60

80

100

2001 2002 2003 2004 2005 2006 2007 20080

20

40

60

80

100Rate per 1,000

Adolescent Parenting Program

Number of pregnancies

Ineffective Job Training ProgramsJob Corps

• Low-income youth

•

•

JTPA1987-1994

• Low-income adults, dislocated workers, and out-of school youth

• Classroom training, on-the-job training, job search assistance, adult basic education, and other services

• $2,400 per participant for 3-4 months ($60 million annually)

• Women: Small initial gains in earnings, employment, and GED receipt fade by 5 yrs

• Men: Small initial gains in earnings fade by 5 yrs, no other impacts

• Youth: On significant impacts

WIA (dislocated)2003-2005

• cts

Douglas J. Besharov, June 2011 2