performance measurement tools for justice information technology projects

1

Performance Measurement Tools Performance Measurement Tools for Justice Information for Justice Information

Technology ProjectsTechnology Projects

Facilitated by:Facilitated by:

Center for Society & Law & JusticeCenter for Society & Law & Justice

FacultyFaculty

2

Performance Measurement Tools Performance Measurement Tools for Justice IT Projectsfor Justice IT Projects

Grant # 2002LD-BX K002Grant # 2002LD-BX K002

A CSLJ ongoing projectA CSLJ ongoing projectFunded by the Funded by the

Bureau of Justice AssistanceBureau of Justice Assistance

3

Project Background and PurposeProject Background and Purpose

This project is designed to identify and validate This project is designed to identify and validate an inventory of performance measures an inventory of performance measures

appropriate for justice information technology appropriate for justice information technology projects and develop field-friendly performance projects and develop field-friendly performance

measurement tools.measurement tools.

4

Fact:Fact: There is an increasing focus on There is an increasing focus on measuring performance in both measuring performance in both

government and industry.government and industry.

5

Fact:Fact: The demand for performance The demand for performance

measures in law enforcement andmeasures in law enforcement andcriminal justice settings is high.criminal justice settings is high.

6

QuestionQuestion

Why do Why do youyou think there is currently think there is currently a high demand for performance a high demand for performance

measurement in law enforcement measurement in law enforcement and criminal justice settings?and criminal justice settings?

7

Some Reasons Why There is a High Some Reasons Why There is a High Demand for Performance MeasuresDemand for Performance Measures

Performance measurement is Performance measurement is increasingly being mandatedincreasingly being mandated

Tight budgetsTight budgets There is a need to justify expendituresThere is a need to justify expenditures

8

ProblemProblem

Even though the demand for Even though the demand for performance measures in law performance measures in law

enforcement and criminal justice enforcement and criminal justice settings is high, technology settings is high, technology

integration projects are integration projects are notnot in in advanced stages of performance advanced stages of performance

measurement.measurement.

9

QuestionQuestion

What are some of the unique What are some of the unique challenges or “road blocks” to challenges or “road blocks” to performance measurement?performance measurement?

10

Some Unique Challenges or Some Unique Challenges or “Road Blocks” to Performance “Road Blocks” to Performance

MeasurementMeasurement Many people are unfamiliar with Many people are unfamiliar with

performance measurement or reluctant to performance measurement or reluctant to use it.use it.

Measurements are imposed from the Measurements are imposed from the outside.outside.

Most measures are not outcome oriented.Most measures are not outcome oriented. Existing measures are not sufficient.Existing measures are not sufficient.

11

A Definition of Performance A Definition of Performance MeasuresMeasures

Performance Measures:Performance Measures: a particular value or a particular value or characteristic used to objectively measure characteristic used to objectively measure results. This means using results. This means using validvalid and and reliablereliable indicators. indicators.

Valid:Valid: Do they truly measure what they are Do they truly measure what they are intended to measure?intended to measure?

Reliable:Reliable: Do they consistently do so ? Do they consistently do so ?(Time A, Time B, Time C, etc.) (Time A, Time B, Time C, etc.)

12

A Useful AcronymA Useful Acronym

Valid performance measures are Valid performance measures are

SS pecificpecific

MM easurableeasurable

AA ccountableccountable

RR esults-Orientedesults-Oriented

TT imeboundimebound

13

The Two “O’s” The Two “O’s” of of

Performance MeasurementPerformance Measurement

Performance measurement Performance measurement can be defined as the can be defined as the process of routinely process of routinely measuring the measuring the outputsoutputs and and outcomesoutcomes produced by a produced by a project, thereby allowing project, thereby allowing one to assess the one to assess the effectiveness of project effectiveness of project investments and activities.investments and activities.

14

Outputs and Outcomes:Outputs and Outcomes:The Logic Model Approach to The Logic Model Approach to

Performance MeasurementPerformance Measurement

““A program logic model … provides a A program logic model … provides a roadmap of your program, highlighting roadmap of your program, highlighting

how it is expected to work, what activities how it is expected to work, what activities need to come before others, and how need to come before others, and how

desired desired outcomesoutcomes are achieved.” are achieved.”

Source: WK Kellogg Foundation Evaluation Handbook (1998)Source: WK Kellogg Foundation Evaluation Handbook (1998)

15

The Logic Model Approach

ACTIVITIES OUTCOMESINPUTS OUTPUTS

INPUTS are your resources

OUTPUTS are the product of an activity

OUTCOMES are the consequences of the program/initiative efforts -- changes in conditions, attitudes or behavior of individuals or outcomes for agencies and communities, not what the program or initiative itself does.

16

ACTIVITIESExample:

Mobile Data

Terminals inPatrol Cars

OUTCOMES

INPUTSExample:Money,

Personnel,Hardware,Software

OUTPUTSMore Rapid

IDof Suspects

INPUTS are your resources

OUTPUTS are the product of an activity

OUTCOMES are the consequences of the program/initiative efforts -- changes in conditions, attitudes or behavior of individuals or outcomes for agencies and communities, not what the program or initiative itself does.

Increased public safety

The Logic Model Approach

17

ACTIVITIES OUTCOMESINPUTS OUTPUTS

When creating performance measures, you can use the Strategic Plan and the Theory of Change to link together the:

•Inputs•Outputs•Outcomes

The Logic Model Approach

18

The Logic Model Approach Helps You The Logic Model Approach Helps You to Define Yourto Define Your

A theory of change is a statement of how A theory of change is a statement of how your program intends to proceed from your program intends to proceed from initial outputs to produce long-term initial outputs to produce long-term outcomes.outcomes.

Once you articulate and reach consensus on Once you articulate and reach consensus on your theory of change, you can then move your theory of change, you can then move on to identify those vital, few performance on to identify those vital, few performance measures that are aligned with your measures that are aligned with your organization’s mission, goals and objectives.organization’s mission, goals and objectives.

19

Goals

In order to solve these

Problems

Activities

We will do the followingactivities and

Events

Targets

For thesepeople and

for this amount of

time

Initial/Short &

IntermediateTerm

Outcomes

We will know these changeshave occurred

if:___________

Theoryof

Change

This activitywill lead to changes in

these factors__________,which in turnwill lead to

solving theseproblems

LongTerm

Outcomes

We will knowwe are

reachingour goals

if:___________

Thinking Through a Logic Model and Articulating a Theory of Change

20

A Logic Model and Theory of ChangeExample

Goals

In order to solve these

Problems

Activities

We will do the followingactivities and

Events

Targets

For thesepeople and

for this amount of

time

Initial/Short &

IntermediateTerm

Outcomes

We will know these changeshave occurred

if:___________

Theoryof

Change

This activitywill lead to changes in

these factors__________,which in turnwill lead to

solving theseproblems

LongTerm

Outcomes

We will knowwe are

reachingour goals

if:___________

21

Developing a Theory of ChangeDeveloping a Theory of Change

Use a Focus Group FormatUse a Focus Group Format Start with Long-term OutcomesStart with Long-term Outcomes Work Backwards Toward Initial ActivitiesWork Backwards Toward Initial Activities Map Required Existing ResourcesMap Required Existing Resources Reconcile Multiple Theories of ChangeReconcile Multiple Theories of Change

Adapted from J.P. Connell et al., 1995, “New Approaches to Adapted from J.P. Connell et al., 1995, “New Approaches to Evaluating Community Initiatives.” Aspen Institute.Evaluating Community Initiatives.” Aspen Institute.

Recommended Process for Surfacing & Recommended Process for Surfacing & Articulating a Theory of ChangeArticulating a Theory of Change

22

The Logic Model/Theory of Change Method The Logic Model/Theory of Change Method for Developing Performance Measuresfor Developing Performance Measures

Program Logic Model and Chain of Events

Category Program Feature and

Activity

Initial Outcomes

Intermediate Outcomes

Intermediate Outcomes

II

Final Outcomes/Goals

Accomplished

Measures: 1._______2._______3._______4._______

1._______2._______3._______4._______

1._______2._______3._______4._______

1._______2._______3._______4._______

1._______2._______3._______4._______

23

Summary: Summary: Why Measure Performance?Why Measure Performance?

To monitor project implementationTo monitor project implementation Because IT projects are high-riskBecause IT projects are high-risk To demonstrate improvementsTo demonstrate improvements To correct problems and make adjustmentsTo correct problems and make adjustments To ensure accountabilityTo ensure accountability To illustrate progress and justify additional To illustrate progress and justify additional

funding for your integration effortfunding for your integration effort

24

Some Suggested Steps…Some Suggested Steps…1.1. Invest in Performance MeasurementInvest in Performance Measurement2.2. Carefully select and form a teamCarefully select and form a team3.3. Identify the Business Process to be mappedIdentify the Business Process to be mapped4.4. Specify the logic model using the Theory of Change Specify the logic model using the Theory of Change

methodmethod5.5. Decide the Audience/Level of the MeasuresDecide the Audience/Level of the Measures6.6. Keep in mind that the Model should be Plausible, Keep in mind that the Model should be Plausible,

Doable and TestableDoable and Testable7.7. Reconcile conflicting assumptions among Reconcile conflicting assumptions among

stakeholdersstakeholders8.8. Revise and Refine model and corresponding Revise and Refine model and corresponding

measures as neededmeasures as needed