performance measurement tools for justice information technology projects
DESCRIPTION
Performance Measurement Tools for Justice Information Technology Projects. Facilitated by: Center for Society & Law & Justice Faculty. Performance Measurement Tools for Justice IT Projects. A CSLJ ongoing project Funded by the Bureau of Justice Assistance. Grant # 2002LD-BX K002. - PowerPoint PPT PresentationTRANSCRIPT
1
Performance Measurement Tools Performance Measurement Tools for Justice Information for Justice Information
Technology ProjectsTechnology Projects
Facilitated by:Facilitated by:
Center for Society & Law & JusticeCenter for Society & Law & Justice
FacultyFaculty
2
Performance Measurement Tools Performance Measurement Tools for Justice IT Projectsfor Justice IT Projects
Grant # 2002LD-BX K002Grant # 2002LD-BX K002
A CSLJ ongoing projectA CSLJ ongoing projectFunded by the Funded by the
Bureau of Justice AssistanceBureau of Justice Assistance
3
Project Background and PurposeProject Background and Purpose
This project is designed to identify and validate This project is designed to identify and validate an inventory of performance measures an inventory of performance measures
appropriate for justice information technology appropriate for justice information technology projects and develop field-friendly performance projects and develop field-friendly performance
measurement tools.measurement tools.
4
Fact:Fact: There is an increasing focus on There is an increasing focus on measuring performance in both measuring performance in both
government and industry.government and industry.
5
Fact:Fact: The demand for performance The demand for performance
measures in law enforcement andmeasures in law enforcement andcriminal justice settings is high.criminal justice settings is high.
6
QuestionQuestion
Why do Why do youyou think there is currently think there is currently a high demand for performance a high demand for performance
measurement in law enforcement measurement in law enforcement and criminal justice settings?and criminal justice settings?
7
Some Reasons Why There is a High Some Reasons Why There is a High Demand for Performance MeasuresDemand for Performance Measures
Performance measurement is Performance measurement is increasingly being mandatedincreasingly being mandated
Tight budgetsTight budgets There is a need to justify expendituresThere is a need to justify expenditures
8
ProblemProblem
Even though the demand for Even though the demand for performance measures in law performance measures in law
enforcement and criminal justice enforcement and criminal justice settings is high, technology settings is high, technology
integration projects are integration projects are notnot in in advanced stages of performance advanced stages of performance
measurement.measurement.
9
QuestionQuestion
What are some of the unique What are some of the unique challenges or “road blocks” to challenges or “road blocks” to performance measurement?performance measurement?
10
Some Unique Challenges or Some Unique Challenges or “Road Blocks” to Performance “Road Blocks” to Performance
MeasurementMeasurement Many people are unfamiliar with Many people are unfamiliar with
performance measurement or reluctant to performance measurement or reluctant to use it.use it.
Measurements are imposed from the Measurements are imposed from the outside.outside.
Most measures are not outcome oriented.Most measures are not outcome oriented. Existing measures are not sufficient.Existing measures are not sufficient.
11
A Definition of Performance A Definition of Performance MeasuresMeasures
Performance Measures:Performance Measures: a particular value or a particular value or characteristic used to objectively measure characteristic used to objectively measure results. This means using results. This means using validvalid and and reliablereliable indicators. indicators.
Valid:Valid: Do they truly measure what they are Do they truly measure what they are intended to measure?intended to measure?
Reliable:Reliable: Do they consistently do so ? Do they consistently do so ?(Time A, Time B, Time C, etc.) (Time A, Time B, Time C, etc.)
12
A Useful AcronymA Useful Acronym
Valid performance measures are Valid performance measures are
SS pecificpecific
MM easurableeasurable
AA ccountableccountable
RR esults-Orientedesults-Oriented
TT imeboundimebound
13
The Two “O’s” The Two “O’s” of of
Performance MeasurementPerformance Measurement
Performance measurement Performance measurement can be defined as the can be defined as the process of routinely process of routinely measuring the measuring the outputsoutputs and and outcomesoutcomes produced by a produced by a project, thereby allowing project, thereby allowing one to assess the one to assess the effectiveness of project effectiveness of project investments and activities.investments and activities.
14
Outputs and Outcomes:Outputs and Outcomes:The Logic Model Approach to The Logic Model Approach to
Performance MeasurementPerformance Measurement
““A program logic model … provides a A program logic model … provides a roadmap of your program, highlighting roadmap of your program, highlighting
how it is expected to work, what activities how it is expected to work, what activities need to come before others, and how need to come before others, and how
desired desired outcomesoutcomes are achieved.” are achieved.”
Source: WK Kellogg Foundation Evaluation Handbook (1998)Source: WK Kellogg Foundation Evaluation Handbook (1998)
15
The Logic Model Approach
ACTIVITIES OUTCOMESINPUTS OUTPUTS
INPUTS are your resources
OUTPUTS are the product of an activity
OUTCOMES are the consequences of the program/initiative efforts -- changes in conditions, attitudes or behavior of individuals or outcomes for agencies and communities, not what the program or initiative itself does.
16
ACTIVITIESExample:
Mobile Data
Terminals inPatrol Cars
OUTCOMES
INPUTSExample:Money,
Personnel,Hardware,Software
OUTPUTSMore Rapid
IDof Suspects
INPUTS are your resources
OUTPUTS are the product of an activity
OUTCOMES are the consequences of the program/initiative efforts -- changes in conditions, attitudes or behavior of individuals or outcomes for agencies and communities, not what the program or initiative itself does.
Increased public safety
The Logic Model Approach
17
ACTIVITIES OUTCOMESINPUTS OUTPUTS
When creating performance measures, you can use the Strategic Plan and the Theory of Change to link together the:
•Inputs•Outputs•Outcomes
The Logic Model Approach
18
The Logic Model Approach Helps You The Logic Model Approach Helps You to Define Yourto Define Your
A theory of change is a statement of how A theory of change is a statement of how your program intends to proceed from your program intends to proceed from initial outputs to produce long-term initial outputs to produce long-term outcomes.outcomes.
Once you articulate and reach consensus on Once you articulate and reach consensus on your theory of change, you can then move your theory of change, you can then move on to identify those vital, few performance on to identify those vital, few performance measures that are aligned with your measures that are aligned with your organization’s mission, goals and objectives.organization’s mission, goals and objectives.
19
Goals
In order to solve these
Problems
Activities
We will do the followingactivities and
Events
Targets
For thesepeople and
for this amount of
time
Initial/Short &
IntermediateTerm
Outcomes
We will know these changeshave occurred
if:___________
Theoryof
Change
This activitywill lead to changes in
these factors__________,which in turnwill lead to
solving theseproblems
LongTerm
Outcomes
We will knowwe are
reachingour goals
if:___________
Thinking Through a Logic Model and Articulating a Theory of Change
20
A Logic Model and Theory of ChangeExample
Goals
In order to solve these
Problems
Activities
We will do the followingactivities and
Events
Targets
For thesepeople and
for this amount of
time
Initial/Short &
IntermediateTerm
Outcomes
We will know these changeshave occurred
if:___________
Theoryof
Change
This activitywill lead to changes in
these factors__________,which in turnwill lead to
solving theseproblems
LongTerm
Outcomes
We will knowwe are
reachingour goals
if:___________
21
Developing a Theory of ChangeDeveloping a Theory of Change
Use a Focus Group FormatUse a Focus Group Format Start with Long-term OutcomesStart with Long-term Outcomes Work Backwards Toward Initial ActivitiesWork Backwards Toward Initial Activities Map Required Existing ResourcesMap Required Existing Resources Reconcile Multiple Theories of ChangeReconcile Multiple Theories of Change
Adapted from J.P. Connell et al., 1995, “New Approaches to Adapted from J.P. Connell et al., 1995, “New Approaches to Evaluating Community Initiatives.” Aspen Institute.Evaluating Community Initiatives.” Aspen Institute.
Recommended Process for Surfacing & Recommended Process for Surfacing & Articulating a Theory of ChangeArticulating a Theory of Change
22
The Logic Model/Theory of Change Method The Logic Model/Theory of Change Method for Developing Performance Measuresfor Developing Performance Measures
Program Logic Model and Chain of Events
Category Program Feature and
Activity
Initial Outcomes
Intermediate Outcomes
Intermediate Outcomes
II
Final Outcomes/Goals
Accomplished
Measures: 1._______2._______3._______4._______
1._______2._______3._______4._______
1._______2._______3._______4._______
1._______2._______3._______4._______
1._______2._______3._______4._______
23
Summary: Summary: Why Measure Performance?Why Measure Performance?
To monitor project implementationTo monitor project implementation Because IT projects are high-riskBecause IT projects are high-risk To demonstrate improvementsTo demonstrate improvements To correct problems and make adjustmentsTo correct problems and make adjustments To ensure accountabilityTo ensure accountability To illustrate progress and justify additional To illustrate progress and justify additional
funding for your integration effortfunding for your integration effort
24
Some Suggested Steps…Some Suggested Steps…1.1. Invest in Performance MeasurementInvest in Performance Measurement2.2. Carefully select and form a teamCarefully select and form a team3.3. Identify the Business Process to be mappedIdentify the Business Process to be mapped4.4. Specify the logic model using the Theory of Change Specify the logic model using the Theory of Change
methodmethod5.5. Decide the Audience/Level of the MeasuresDecide the Audience/Level of the Measures6.6. Keep in mind that the Model should be Plausible, Keep in mind that the Model should be Plausible,
Doable and TestableDoable and Testable7.7. Reconcile conflicting assumptions among Reconcile conflicting assumptions among
stakeholdersstakeholders8.8. Revise and Refine model and corresponding Revise and Refine model and corresponding
measures as neededmeasures as needed