results-based accountability · part 2: secondary indicators part 3: data development agenda 3 to 5...

RBA Results-Based Accountability

The Fiscal Policy Studies Institute

raguide.org & resultsaccountability.com

Book & DVD Orders amazon.com

clearimpact.com

SCORECARD

SIMPLE

COMMON SENSE

PLAIN LANGUAGE

MINIMUM PAPER

USEFUL

Results-Based Accountability is made up of two parts:

Performance Accountability is about the well-being of

CUSTOMER POPULATIONS for Programs – Agencies – Service Systems

Population Accountability is about the well-being of

WHOLE POPULATIONS for Communities – Cities – Counties – States - Nations

Population Accountability

Performance Accountability

1.

2.

Results-Based Accountability

COMMON LANGUAGE

COMMON SENSE

COMMON GROUND

Common Language

THE LANGUAGE TRAP Too many terms. Too few definitions. Too little discipline

Benchmark

Target

Indicator Goal

Result

Objective

Outcome

Measure

Modifiers Measurable Core Urgent Qualitative Priority Programmatic Targeted Performance Incremental Strategic Systemic

Lewis Carroll Center for Language Disorders Measurable urgent systemic indicators Core qualitative strategic objectives Make up your own jargon.

DEFINITIONS

Children born healthy, Children ready for school Safe communities, A Healthy Green City, Prosperous Economy

Rate of low-birthweight babies, Percent ready at K entry, crime rate, air quality index, unemployment rate

6. How much did we do? 7. How well did we do it? 8. Is anyone better off?

RESULT or OUTCOME

INDICATOR or BENCHMARK

PERFORMANCE MEASURE

A condition of well-being for children, adults, families or communities.

A measure which helps quantify the achievement of a result.

A measure of how well a program, agency or service system is working. Three types:

Popu

latio

n Pe

rform

ance

Children born healthy

Rate of low-birthweight babies Percent ready at K entry

Children ready for school

crime rate

Safe communities

air quality index

A Healthy Green City

unemployment rate

Prosperous Economy

INDICATOR

RESULT

PERFORMANCE MEASURE

Popu

latio

n Pe

rform

ance

= Customer Results

3.

4.

5.

From Ends to Means

ENDS

MEANS

From Talk to Action Po

pulat

ion

Perfo

rman

ce

RESULT

INDICATOR

PERFORMANCE MEASURE

Customer result = Ends Service delivery = Means

From Talk to Action

1. Safe Communities

2. Crime rate

3. Average police response time

4. People live in safe stable affordable housing

5. % spending >30% of income on housing

6. People have living wage jobs and income

7. % of people with living wage jobs and income

8. % of participants in job training who get living wage jobs

IS IT A RESULT, INDICATOR OR PERFORMANCE MEASURE?

Pop. RESULT

Pop. INDICATOR

Perf. MEASURE

Pop. RESULT

Pop. INDICATOR

Pop. RESULT

Pop. INDICATOR

Perf. MEASURE

|

. .

.

.

.

.

.

.

.

Translation Guide/Rosetta Stone Not the Language Police

Ideas

1. A condition of well-being for children, adults, families & communities

2.

3.

etc.

Group 1 Group 2 Group 3 etc.

RESULT OUTCOME GOAL

TRANSLATION

Back to the Idea

Results – Indicators – Performance Measures in Amharic, Cambodian, Laotian, Somali, Spanish, Tigrigna, Vietnamese

The Three L’s of Success in any serious change effort

1. Leadership 2. Language

Searching for the 3rd L

Leverage Latitude Levity Lithium Luggage

Georgia Policy Council for Children and Families

RESULTS

● Healthy Children

● Children Ready for School

● Children Succeeding in School

● Strong Families

● Self Sufficient Families

VERMONT Population Outcomes (Act 186)

1. Vermont has a prosperous economy. 2. Vermonters are healthy. 3. Vermont’s environment is clean and sustainable. 4. Vermont’s communities are safe and supportive. 5. Vermont’s families are safe, nurturing, stable, and supported. 6. Vermont’s children and young people achieve their potential. 7. Vermont’s elders and people with disabilities and people with mental conditions live with dignity and independence in settings they prefer. 8. Vermont has open, effective, and inclusive government at the State and local levels.

VERMONT Population Outcomes (Act 186)

1. Vermont has a prosperous economy. 2. Vermonters are healthy. 3. Vermont’s environment is clean and sustainable. 4. Vermont’s communities are safe and supportive. 5. Vermont’s families are safe, nurturing, stable, and supported. 6. Vermont’s children and young people achieve their potential. (A) Pregnant women and young people thrive. (B) Children are ready for school. (C) Children succeed in school. (D) Youth choose healthy behaviors. (E) Youth successfully transition to adulthood. 7. Vermont’s elders and people with disabilities and people with mental conditions live with dignity and independence in settings they prefer. 8. Vermont has open, effective, and inclusive government at the State and local levels.

CARDIFF, WALES Community Outcomes

1. People in Cardiff are healthy

2. Cardiff has a clean attractive and sustainable environment

3. People in Cardiff are safe and feel safe

4. Cardiff has a thriving and prosperous economy

5. People in Cardiff achieve their full potential.

6. Cardiff is a great place to live, work and play.

7. Cardiff has a fair, just and inclusive society.

Hillsborough Co.

NEW MEXICO DOH

Getting from TALK TO ACTION

Leaking Roof (Results thinking in everyday life)

Experience

Measure

Story behind the baseline (causes)

Partners

What Works

Action Plan

Cm of Water BASELINE

? Fixed

Not OK

Turning the Curve

Action Plan # 2

Results / Outcomes

Indicators

Baseline & Story

Experience

Criteria for

Choosing Indicators as Primary vs. Secondary Measures

Communication Power

Proxy Power

Data Power

Does the indicator communicate to a broad range of audiences?

Does the indicator say something of central importance about the result? Does the indicator bring along the data HERD?

Quality data available on a timely basis.

Choosing Indicators Worksheet

Outcome or Result_______________________

Candidate Indicators Communication Power

Proxy Power

Data Power

H M L

H

Measure 1

Measure 2

Measure 3

Measure 4

Measure 5

Measure 6

Measure 7

Measure 8

HData Development Agenda

Safe Community

H M L H M L

H H H L

Three Part Indicator List for each Result

Part 1: Primary Indicators

Part 2: Secondary Indicators

Part 3: Data Development Agenda

● 3 to 5 “Headline” Indicators ● What this result “means” to the community ● Meets the Public Square Test

● Everything else that’s any good (Nothing is wasted.) ● Used later in the Story behind the Curve

● New data ● Data in need of repair (quality,timeliness etc.)

The Matter of Baselines

Baselines have two parts: history and forecast

H

M

L

History Forecast Turning the Curve Point to Point

OK?

1975

1980

1982

1990

2000

2005

0

5

10

15

20

25

30

Tho

usan

ds

Source 1982 to 2005: Actual data from the NHTSA Fatality Analysis Reporting System (FARS)Source 1975 to 1981: Estimate based on NHTSA data on % of fatality drivers with BAC of .10 or greater.

Alcohol-Related Traffic FatalitiesUnited States 1975 to 2005

45 people per day

75 people per day

2010

28 people per day

MADD

Rebound

Hull, UK "Woundings"

Alternative to Traditional Evaluation Methods: DEMONSTRATING the CONTRIBUTION

of complex change efforts… requires 3 elements:

A Curve to Turn

2 …and it had a timely relationship to….

3 …. a turn in the curve.

c FPSI

1 We tried a bunch of stuff that had a credible chance of making a

difference …

Bonus: a background trend stayed the same or got worse.

4

Rebound

- Rosell

“If I include you, you will be my partner.

If I exclude you, you will be my judge.”

“All performance measures

that have ever existed

for any program

in the history of the universe

involve answering two sets of

interlocking questions.”

How Much did we do?

( # )

How Well

did we do it? ( % )

Quantity Quality

Performance Measures

Effort How hard did we try?

Effect Is anyone better off?


Effort

Effect

How Much

How Well


How much service did we deliver?


How well did we

deliver it?

How much change / effect

did we produce?

What quality of change / effect

did we produce?

Quantity Quality

E

ffect

Effo

rt

Ou

tput

Inpu

t

How much did we do?


How well did we do it?

Is anyone better off?

Quantity Quality

E

ffect

Effo

rt

# %

How much did we do?

Education



Quantity Quality

E

ffect

Effo

rt Number of students

Student-teacher ratio

Number of high school graduates

Percent of high school graduates

How much did we do?

Education



Quantity Quality

E

ffect

Effo

rt Number of students

Student-teacher ratio

Percent of 9th graders who

graduate on time and enter college or

employment after graduation

Number of 9th graders who

graduate on time and enter college or

employment after graduation

How much did we do?

Pediatric Practice



Number of children treated

Percent of patients waiting

less than 30 min in the waiting room

Quantity Quality

E

ffect

Effo

rt

# children

fully immunized

(in the practice)

% children

fully immunized

(in the practice)

How much did we do?

Child Welfare



Quantity Quality

Effe

ct

E

ffort

# of

children in foster care

Workload ratio: No. of children per

worker

# of children in

permanent plcmt within 6 months or entering care

% of children in

permanent plcmt within 6 months of entering care

How much did we do?

Juvenile Justice



Quantity Quality

Effe

ct

E

ffort

Number of children in

custody

Percent of children in

community based (vs. institutional)

care

Number of children exiting

custody

Percent of children exiting custody with no repeat offence in

6 months

How much did we do?

Drug/Alcohol Treatment Program



Number of persons treated

Percent of staff with training/

certification

Number of clients off of alcohol & drugs - at exit - 12 months after exit

Percent of clients off of alcohol & drugs - at exit - 12 months after exit

Quantity Quality

E

ffect

Effo

rt

How much did we do?

Fire Department



Number of responses

Response Time

Quantity Quality

E

ffect

Effo

rt

# of fires kept to

room of origin

% of fires kept to

room of origin

How much did we do?

General Motors



# of production hrs # tons of steel

Employees per vehicle

produced

# of cars sold $ Amount of Profit

$ Car value after 2 years

Quantity Quality

E

ffect

Effo

rt

Source: USA Today 9/28/98

% Market share Profit per share

% Car value after 2 years

How much did we do?

Not All Performance Measures Are Created Equal



Least Important

Quantity Quality

E

ffect

Effo

rt

Most Important

Least

Most

Also Very Important

Quality

RBA Categories Account for All Performance Measures (in the history of the universe)

Quantity Quality

Efficiency, Admin overhead, Unit cost Staffing ratios, Staff turnover Staff morale, Access, Waiting time, Waiting lists, Worker safety

Customer Satisfaction (quality service delivery & customer benefit)

Cost / Benefit ratio Return on investment

Client results or client outcomes

Effectiveness Value added Productivity

Benefit value

Product Output Impact

Process Input

Effe

ct

Effo

rt

Cost Total Quality Mgmt (TQM)

Effectiveness

Efficiency Admin overhead, Unit cost

Quantity Quality






Benefit value

Process Input

Effe

ct

Effo

rt

Cost



Total Quality Mgmt (TQM)

Quantity Quality






Benefit value

Process Input

Effe

ct

Effo

rt

Cost

1. Did we treat you well?

2. Did we help you with your problems?

*



* World’s simplest complete customer satisfaction survey


Quantity Quality






Benefit value

Process Input

Effe

ct

Effo

rt

Cost


OBA Categories Account for All Performance Measures (in the history of the universe)


How much did we do?

The Matter of Control



Quantity Quality

E

ffect

Effo

rt

Least Control

PARTNERSHIPS

Most Control

The Matter of Use

1.  The first purpose of performance measurement is to IMPROVE PERFORMANCE.

2. Avoid the performance measurement equals punishment trap.

● Create a healthy organizational environment.

● Start small.

● Build bottom-up and top-down simultaneously.

CHARTS on the WALL

Advanced Baseline Display

Your Baseline

Comparison Baseline

Goal (line) Target or Standard

Instead: Count anything better than baseline as progress.

Avoid publicly declaring targets by year if possible.

●

Create targets only when they are:

FAIR & USEFUL

x

Aspiration NOT

Punishment

How much did we do?

Performance Accountability Types of Measures found in each Quadrant



# Clients/customers served

# Activities (by type of activity)

% Common measures e.g. client staff ratio, workload ratio, staff turnover rate, staff morale, % staff fully trained, % clients seen in their own language, worker safety, unit cost

% Skills / Knowledge (e.g. parenting skills)

#

% Attitude / Opinion (e.g. toward drugs)

#

% Behavior (e.g.school attendance)

#

% Circumstance (e.g. working, in stable housing)

#

% Activity-specific measures

e.g. % timely, % clients completing activity, % correct and complete, % meeting standard

Point in Time vs.

2 Point Comparison # %

How much did we do?

Service: __________________________________



Primary customers # students patients persons trained Primary activity # hours of

instruction diagnostic tests job courses alarms responded to

If your service works really well, how are your customer's better off?

# students who graduate patients who fully recover fires kept to room of origin

%

Unit cost

Workload ratio % of ___x___ that

happen on time

persons who get jobs

School Hospital Job Training Fire Department

How much did we do?

Choosing Headline Measures and the Data Development Agenda



Quantity Quality

Effe

ct

E

ffort

# Measure 1 ---------------------------- # Measure 2 ----------------------------

# Measure 3 ---------------------------- # Measure 4 ----------------------------

# Measure 5 ---------------------------- # Measure 6 ----------------------------

# Measure 7 ----------------------------

#1 Headline

#2 Headline

#3 Headline

#1 DDA

#2 DDA

#3 DDA % Measure 8 ---------------------------- % Measure 9 -----------------------------

% Measure 10 --------------------------- % Measure 11 ---------------------------

% Measure 12 --------------------------- % Measure 13 ---------------------------

% Measure 14 ---------------------------

# Measure 15 ---------------------------- # Measure 16 ----------------------------

# Measure 17 ---------------------------- # Measure 18 ----------------------------

# Measure 19 ---------------------------- # Measure 20 ----------------------------

# Measure 21 ----------------------------

% Measure 15 ---------------------------- % Measure 16 ----------------------------

% Measure 17 ---------------------------- % Measure 18 ----------------------------

% Measure 19 ---------------------------- % Measure 20 ----------------------------

% Measure 21 ----------------------------

LR

UR

Primary v. Secondary

Direct v. Indirect

Internal v. External

Baseline & Story

0

5

10

15

20

25

30

35

0

50

100

150

200

250

300

1970 1980 1990 2000 2010 2020 2030 2040

pipe

age

Watermain Breaks per Year

Main Breaks "Main Breaks without renewal Pipe age (right axis) Trend (Financial Plan forecast) Trend (Main Breaks without Renewal)

Trend without renewal

Start of Replacement Program Trend of

Financial Plan

Watermain Breaks per year London Ontario

Select 3 to 5 Performance Measures ACROSS THE BOTTOM OF THE ORG

CHART

3 - 5 3 - 5 3 - 5 3 - 5 3 - 5 3 - 5 3 - 5

3 - 5 3 - 5 3 - 5

3 - 5

20 – 60 – 20 Rule

Next Generation Contracting Contract Provisions

Provision 1. Specify the 3 to 5 most important performance measures (from the How well did we do it? and Is anyone better off? categories).

Provision 2. Specify that the contractor will use a continuous improvement process (the RBA 7 Questions).

Provision 3. Specify how the funder and contractor will work in partnership to maximize LR customer results (quarterly meetings using the 7 questions as the agenda).

Provision 4. Specify that the funder will work with the funding community to simplify and standardize contracting and performance reporting.

Provision 5: : Clear articulation of role in population/community well-being using the language of contribution not attribution.

Provision 6: 10% for quality management and administration. Provision 7: Multi-year funding using 3 year rolling contracts

Provision 8: Use of targets that are fair and useful.

Provision 9: Fund flexibility and virtual funding pool: transfer of up to 10% across line items and program lines.

Provision 10: Request for Results: Getting past the sometimes negative effects of competitive RFP contracting or tendering.


Putting Population


TOGETHER

and

Contribution relationship

Alignment of measures

Appropriate responsibility

THE LINKAGE Between POPULATION and PERFORMANCE

POPULATION ACCOUNTABILITY

Healthy Births Rate of low birth-weight babies Children Ready for School Percent fully ready per K-entry assessment Self-sufficient Families Percent of parents earning a living wage

CUSTOMER RESULTS

# persons receiving training

Unit cost per person

trained

# who get living wage

jobs

% who get living wage

jobs

PERFORMANCE ACCOUNTABILITY

POPULATION RESULTS

Job Training Program

Performance measure

Population Indicator

Every time you present

your program,

Use a two-part

approach.

Result: to which you contribute to most directly.

Indicators:

Story:

Partners:

What would it take?:

Your Role: as part of a larger strategy.


Program:

Performance measures:

Story:

Partners:

Action plan to get better:


Your Role


Indicators:

Story:

Partners:


Your Role: within the larger strategy.


Program:


Story:

Partners:



Your Role


your program,

Use a two-part

approach.


your program,

Use a two-part

approach.


Indicators:

Story:

Partners:


Your Role: as part of a larger strategy.


Program:


Story:

Partners:



Your Role

Shortcut

Different Kinds of Progress 1. Data

a. Population indicators Actual turned curves: movement for the better away from the baseline. b. Program performance measures: customer progress and better service: How much did we do?

How well did we do it? Is anyone better off?

2. Accomplishments: Positive activities, not included above.

3. Stories behind the statistics that show how individuals are better off.

Board of Directors Meeting AGENDA

1. New data 2. New story behind the curves

3. New partners 4. New information on what works.

5. New information on financing

6. Changes to action plan and budget

7. Adjourn

1. New data 2. New story behind the curves

3. New partners 4. New information on what works.

5. New information on financing

6. Changes to action plan and budget

7. Adjourn

SUMMARY

3 - kinds of performance measures. How much did we do? How well did we do it? Is anyone better off?

RBA in a Nutshell 2 – 3 - 7

2 - kinds of accountability Population accountability Performance accountability

7 - questions from ends to means in less than an hour.

plus language discipline Results & Indicators Performance measures

Baselines & Turning the Curve

“If you do what you always did,

you will get what you always got.”

Kenneth W. Jenkins President, Yonkers NY NAACP

“We will succeed together… or fail separately.”

- Winston Churchill or - Sonny and Cher

|

THANK YOU !

Book - DVD Orders amazon.com

resultsleadership.org

SCORECARD

“Data is the unblinking eye

of reform.”

Barbary Curley, Area Director, Hyde Park Office, Boston

EXERCISES

Kids Count Data Center

2016 2018


Creating a Working Baseline from Group Knowledge

Now

Indicator or Performance Measure

65% Not OK?

Backcasting Forecasting

CARDIFF, WALES Community Outcomes

1. People in Cardiff are healthy

2. Cardiff has a clean attractive and sustainable environment

3. People in Cardiff are safe and feel safe

4. Cardiff has a thriving and prosperous economy

5. People in Cardiff achieve their full potential.

6. Cardiff is a great place to live, work and play.

7. Cardiff has a fair, just and inclusive society.

Turn the Curve Exercise: Population Well-being 5 min: Starting Points

- timekeeper and reporter - geographic area - two hats (yours plus partner’s)

10 min: Baseline - pick a result and a curve to turn - forecast (to 2017) – OK or not OK?

15 min: Story behind the baseline - causes/forces at work - information & research agenda part 1 - causes

15 min: What works? (What would it take?) - what could work to do better? - each partners contribution - no-cost / low-cost ideas - information & research agenda part 2 – what works

10 min: Report convert notes to one page

Two pointers to action

ONE PAGE Turn the Curve Report: Population

Result: _______________ Indicator

(Lay Definition) Indicator Baseline

Story behind the baseline --------------------------- --------------------------- (List as many as needed) Partners --------------------------- --------------------------- (List as many as needed) Three Best Ideas – What Works 1. --------------------------- 2. --------------------------- 3. ---------No-cost / low-cost

Sharp Edges

4. --------- Off the Wall


Turn the Curve Exercise: Program Performance 5 min: Starting Points

- timekeeper and reporter - identify a program to work on - two hats (yours plus partner’s)

10 min: Performance measure baseline - choose 1 measure to work on – from the lower right quadrant - forecast (to 2018) – OK or not OK?

15 min: Story behind the baseline - causes/forces at work - information & research agenda part 1 - causes

15 min: What works? (What would it take?) - what could work to do better? - each partners contribution - no-cost / low-cost ideas - information & research agenda part 2 – what works

10 min: Report convert notes to one page

Two pointers to action

Program: _______________ Performance Measure

(Lay definition) Performance Measure Baseline

Story behind the baseline --------------------------- --------------------------- (List as many as needed) Partners --------------------------- --------------------------- (List as many as needed) Three Best Ideas – What Works 1. --------------------------- 2. --------------------------- 3. ---------No-cost / low-cost

ONE PAGE Turn the Curve Report: Performance

Sharp Edges



Turn the Curve Exercise – Lessons Talk to Action in an hour

1. How was this different from other processes? What worked and what didn’t work?

2. Why did we ask for: a. Results before indicators? b. Forecast? c. Story? d. No cost / low cost? e. Two hats? f. Crazy idea? g. Only 3 best ideas?

3. Do you think a lay audience could understand the reports?

4. How many think you could lead this exercise with a small group? (2+ curves at the same time)


Provision 1. Specify the 3 to 5 most important performance measures (from the How well did we do it? and Is anyone better off? categories).

Provision 2. Specify that the contractor will use a continuous improvement process (the RBA 7 Questions).

Provision 3. Specify how the funder and contractor will work in partnership to maximize LR customer results (quarterly meetings using the 7 questions as the agenda).

Provision 4. Specify that the funder will work with the funding community to simplify and standardize contracting and performance reporting.

Provision 5: : Clear articulation of role in population/community well-being using the language of contribution not attribution.

Provision 6: 10% for quality management and administration. Provision 7: Multi-year funding using 3 year rolling contracts

Provision 8: Use of targets that are fair and useful.

Provision 9: Fund flexibility and virtual funding pool: transfer of up to 10% across line items and program lines.

Provision 10: Request for Results: Getting past the sometimes negative effects of competitive RFP contracting or tendering.


If you could

make one number

better in the next 3 years,

what would it be?

NEXT STEPS

What’s one thing I can personally do with what I learned today?

Resources



www.raguide.org www.resultsaccountability.com

RBA Facebook Group

THANK YOU !



SCORECARD

The Simplest Way TO IMPLEMENT RBA

POPULATION ACCOUNTABILITY 1. Pick an important indicator curve 2. Run the Population Turn the Curve Exercise 3. Take action Repeat

PERFORMANCE ACCOUNTABILITY 1. Pick an important performance measure curve 2. Run the Performance Turn the Curve Exercise 3. Take action Repeat

results-based accountability · part 2: secondary indicators part 3: data development agenda 3 to 5...

Documents