next generation impact measurement | july 19, 2016

Next Generation Impact Measurement

Measuring What Matters to Nonprofits

Why Do We NOT Know If We Are Making a Difference?

What are Nonprofits Being Asked to Measure?

10% Have Some Output Data

Disconnected datasets<500 cases per dataset

No ”true” outcomes

1% have a Little Bit of Outcome Data

Disconnected datasets <250 cases per dataset

30%-50% aren’t using valid and reliable metrics

100% of 501c3s Have 990s

Shared Dataset600,000 cases

No program quality or outcome data

600,000

60,000 ….

$400M spent on filing 990s $12B $30B

What’s Stopping Us?

The MetricsChallenge #1

Lots of Outputs Little Outcomes

The Balance of What We Measure

The Unfair Economics of the Nonprofit Sector

Consumer Buyers “Bulk” Proxy Buyers

A Consumer Buys Personal Outcomes A Funder Buys Population Impact

Postpartum Doula ServicesBought by Me, for Me

Postpartum Home Visitation ServicesBought by Others, for a Population

The Act of “Bulk” Proxy Buying Creates Unrealistic Accountability


Postpartum Home Visitation ServicesBought by Others, for You

Outcome Metrics aren’t Realistic

Lower Stress Reduction in Child Abuse




Better Infant Care Skills and

Knowledge

Achievement of

Developmental Milestones




Improved Maternal Physical

Recovery

Reduction in Subsequent Pregnancies




Better Sibling Adjustment

Reduction in Risky

Behaviors




Better Communication with Providers

Kindergarten Readiness

Show Us Don’t Show Us

As a Result, Our Current Impact Metrics…

How MuchWe Are Doing

Direct Results

What Works

The MethodChallenge #2

Does a Program Work for the Average Participant?

How Does a Program Work for Targeted Segments?

Means versus Me

Did the Program Recipe Move the Average?

Were Always Above Average

Still Below Average

The Average Moved

Which Program Ingredients Work for You?

You are like me! Can you tell me how you did it?

Data AnalysisChallenge #3

Analyses Describe What Just Happened

The Unresolved Challenges of Impact MeasurementChallenge #3: Data Analysis

Analyses Describe What Just Happened

Amount, Frequencies,

Averages, Trends

Statistical Significance of

Outcomes

Predict What Might Happen Provide Insight on What Might Work

Very Few Existing Analyses…

Describes Never

As a Result, Our Current Impact Metrics…

Often, What We Just Did

Sometimes, Our Results

Diagnoses ProblemsPredicts What Might

HappenProvides Insights About

What To Do

Measuring in IsolationChallenge #4

We Don’t Share Stories, Insights or LessonsData Analytics Findings

The Solution

Step One: Come Together w/in Fields of Practice

Come Together w/in Fields of PracticeThere’s no one-size-fits-all ideal business

performance and impact model for ALL for-profit service companies

Each ”industry” has it’s own standards, and when it comes to evaluating their services, each business within an industry competes on a set of common

market-driven (customer-driven need and demand) outcomes.

Academic Achievement Consulting/Capacity Building

Employment Career Advancement

It’s time to do the same within our

sectorYouth Development

Nonprofit Capacity Building

Senior Services

Workforce Development

Environmental Policy

Etc.

Step Two: Agree on Shared Metrics

Develop Shared Metrics

Target Population Field Practices Achievable Field Outcomes

Step Three: Real-World Methods

Natural ExperimentControlled Experiment

Conduct Natural Experiments

Add Predictive Analytics

“Predictive analytics tells you what will [likely] happen”

Jeff Bertolucci, Information Week

Add Prescriptive Analytics

“…prescriptive analytics tells you what to do about

it.”Jeff Bertolucci, Information Week

Find the Positive Deviants!

Step Four: Make Meaning, Together

Everyone Gathering Stories, Finding Positive Deviants and Sharing How They Did It

Data-Driven Decision Making for Impact

next generation impact measurement | july 19, 2016

Government & Nonprofit