1

Performance Management for Justice Information Sharing

David J. RobertsGlobal Justice Consulting

Steve PrisocChief Information Officer

New Mexico State Courts

2006 BJA/SEARCH Regional Information Sharing Conference

June 4, 2007 Jacksonville, Florida

2

“Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.”

—H. James Harrington

3

WHY evaluate performance?

• Information is control• Provides feedback to improve program

performance• Provides information for resource

allocation• Enables effective planning• Tests generalizations based on

experiences and assumptions• Market and develop support among

funding bodies, constituents, and staff.

4

Landscape of Performance Management

• Investment appraisal and benefits realization– What is the actual investment we’re making?– How are benefits going to be collected and

tracked?

• Solid program management and tracking– Is the project on track?– How do we ensure it remains on track?

• Achievement of the strategic objectives– Fundamentally, what is it that we’re trying to

achieve in our information sharing initiative?

5

Process vs. Impact Evaluations

• Process evaluations focus on how the initiative was executed; the activities, efforts, and workflow associated with the response. Process evaluations ask whether the response occurred as planned, and whether all components worked as intended. Fundamentally, a process evaluation posits the question, “Are we doing the thing right?”

• Impact evaluations focus on the outcome (the what) of the initiative; the output (products and services) and outcome (results, accomplishment, impact). Did the problem decline or cease? And if so, was the response the proximate cause of the decline? Fundamentally, the impact evaluation posits the question, “Are we doing the right thing(s)?”

6

Balanced Scorecard

Originally developed in business by Kaplan & Norton

1. Financial – How do we look to stakeholders?

2. Customer – How well do we satisfy our internal and external customers’ needs?

3. Internal Business Process – How well do we perform at key internal business processes?

4. Learning and Growth – Are we able to sustain innovation, change, and continuous improvement?

7

Balanced Scorecard for Law Enforcement (Mark Moore, et al)

1. Reduce criminal victimization2. Call offenders to account3. Reduce fear and enhance personal

security4. Guarantee safety in public spaces5. Use financial resources fairly, efficiently,

and effectively6. Use force and authority fairly, efficiently,

and effectively7. Satisfy customer demands/achieve

legitimacy with those policed

8

Trial Court Performance Standards

1. Access to Justice

2. Expedition and Timeliness

3. Equality, Fairness and Integrity

4. Independence and Accountability

5. Public Trust and Confidence.

9

Corrections Performance

• Security– Drug Use– Significant Incidents– Community Exposure…

• Safety– …of inmates– …of staff– …of environment…

• Order– Inmate misconduct– Use of force– Perceived control…

• Care– Stress & illness– Health care– Dental care…

• Activity– Work & industry– Education & training– Religion…

• Justice– Staff fairness– Use of force– Grievances (# & type)...

• Conditions– Space– Pop density– Freedom of movement…

• Management– Satisfaction– Stress & burnout– Turnover…

10

Universal IJIS Elements

• Definition: The ability to access and share critical information at key decision points throughout the whole of the justice enterprise.

• Scope: Recognition that the boundaries of the enterprise are increasingly elastic—engaging not only justice, but also emergency & disaster management, intelligence, homeland security, first responders, health & social services, private industry, the public, etc.

• Goal: Get the right information, to the right people, all of the time—underscores the need for dynamic information exchange.

11

Information Sharing Objectives

• What is the problem we’re addressing?• What information do we have regarding

current levels of performance?• What is it that we’re trying to do?• 3 Universal Objectives:

– Improve Public Safety and Homeland Security;– Enhance the Quality and Equality of Justice;– Gain Operational Efficiencies, Effectiveness, and

demonstrate Return on Investment (ROI).

12

Sample Public Safety Measures

• Decrease the amount of time it takes to serve a warrant

• Decrease the amount of time for law enforcement to have details on protection orders.

• Reduce the amount of time it takes users of the integrated justice system to respond to a request from the public

• Reduce the time it takes to complete a criminal history background check

• Reduce the number of agencies that can’t communicate with each other.

• Increase the percentage of court dispositions that can be matched to an arrest—this will improve the quality of the computerized criminal history records

• Decrease the average response time to establish a positive identification following an arrest

• Reduce the number of incidents of criminal records being associated with the wrong person

• Reduce recidivism• Reduce the fear of crime

in target neighborhoods

13

JNET: Improved Public Safety & Homeland Security

• Notifications– Timely notification of critical events

• Arrest, disposition, warrant, violation, death, etc

– Offender accountability and increased public safety.

• Confirmed Notifications– FY01/02         3,645  – FY02/03         18,349– FY03/04         29,980– FY04/05         33,264– FY05/06         46,424– Total = 178,339 confirmed notifications

14

Sample Quality of Justice Measures

• Reduce the average time a defendant is held while waiting for a bond decision

• Reduce the time it takes for correctional facility intake

• Reduce the number of days it takes to process cases from arrest to disposition

• Reduce the number of false arrests.

• Reduce the amount of missing data.

•Reduce the number of civilian complaints against local law enforcement•Reduce the number of continuances per case that result from scheduling conflicts between the courts, law enforcement, and prosecution•Reduce the number of cases without a next scheduled event•Reduce the average number of days or hours from arrest to arraignment

15

JNET: Improvement in the Quality of Justice

• Improved decision making– At key decision points, providing the required information in

a timely, usable method• Traffic Stop

– Who is this person? Positive identification (photo, SID, etc)– Is this person wanted? Outstanding warrants/wants.– Is this person a threat? Previous history of violent behavior, firearms,

etc.

• Enhanced Overall Data Quality • Reduction of errors• Accurate and timely statistical reporting

• Improve Business Process• Minimize offender processing time• Reduction in “holding” time.

16

Sample Efficiency/Effectiveness Measures

• Reduce the number of hours that staff spend entering data manually or electronically

• Reduce the costs of copying documents for justice organizations

• Reduce the number of hours spent filing documents manually

• Reduce the number of hours spent searching other governmental databases

• Increase the number of law enforcement personnel performing community policing tasks, instead of administrative tasks

• Reduce the amount of missing information in criminal justice databases

• Reduce the number of corrections needed in databases maintained by CJIS agencies

• Decrease the number of warrants that never get entered into the state registry

• Increase the number of query hits on each agency database

• Reduce the number of hours it takes to enter a court disposition into the state criminal history repository

17

JNET: Efficient and Effective ROI

• PennDOT (DMV) Certified Drivers History via JNET

– In 2003, the PennDOT processed 157,840 certified driving history requests for local police, district attorneys, and the Courts

– One clear performance measure is highlighted by the dramatic reduction in processing costs for PennDOT. The personnel cost metric is based on the time required to process a paper copy of the driver history request, including the manual application of an embossed certification seal. PennDOT calculates their personnel cost at $1.50 per certified history processed, and when incorporating a combined printing and mailing cost of $.50 per copy, the total cost to manually generate a certified driver history equates to $2.00 per request.

– During August 2006, the 56,126 certified driving history requests process by JNET saved PennDOT $112,252 in monthly operating expenses. Only 4767 were processed in the traditional fashion.

– PennDOT has reallocated personnel to support and process other areas of business such as ‘paid’ requests from individual citizens and pre-employment screeners.

18

Critical Assumptions

• Baseline data exist regarding current or historical performance of the system

• Access, ability and willingness to capture data regarding on-going performance

• Timely, accurate and complete data collection• Appropriate and sufficiently detailed analysis

techniques• Staff to conduct the analysis and reports• Effective communication mechanisms to:

– Monitor on-going baseline performance– Constantly assess the impact and operations

• Political will and operational capacity to do something as a result of what the measures show!

19

Performance Dashboards

What we’re NOT talking about:

The threat level in the airline sector is HIGH or Orange 3/1/07

20

What we ARE talking about…

21

Sample Performance Dashboard

Draft dashboard assessing performance on a series of dimensions that have been agreed by key decisionmakers.

This requires effective data collection and routine reporting from operational systems in place throughout the County and agreement that we’re going to do something with the data in order to respond to critical performance elements.

22

23

Establishing a Performance Management Program

The Six Steps to Establishing a Performance-Based Management Program

Source: Will Artley, DJ Ellison and Bill Kennedy, The Performance-Based Management

Handbook, Volume 1: Establishing and Maintaining a Performance-Based Management

Program (Washington, DC: U.S. Department of Energy, 2001)

24

Outcomes and performance measures

• Outcomes are the benefits or results gained by reaching goals, achieving objectives and resolving strategic issues

• Performance Measures are specific, measurable, time-bound expressions of future accomplishment that relate to goals, objectives and strategic initiatives

• Goals, objectives and strategic initiatives should ideally lead to outcomes

• Pragmatic performance measurement planners recognize that not all things that need to be measured can always be empirically linked to outcomes.

25

It is important that performance measures be based on criteria that correspond to desired outcomes; however, it is often difficult or even impossible to obtain objective measures of certain key outcomes.

Not all outcomes easily lend themselves to measurement

Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.—Albert Einstein

26

Populated logic modelProgram Logic Model and Chain of Events

Category Program Feature and Activity

Initial Outcomes

Intermediate Outcomes

Intermediate Outcomes II

Final Outcomes Reached

Measures Rap sheet information of appropriate scope, timeliness, accuracy and ease of use available to magistrate judge at first court appearance/bond hearing

1. Greater use of Rap sheet information when setting bail/bond and conditions of release

1.More appropriate conditions of release and establishment of bail/bond appropriate to both the arrest charges and the criminal history and past warrant information

1. Fewer crimes committed by those awaiting trial2. Fewer failures to appear3. More timely disposition of criminal cases

1.Enhanced justice process

2. Positive influence on lessening total number of crimes committed

27

Use scenario approach to reach agreement and define performance

• Bring stakeholders together to reach consensus on the desired state of integration

• Define the current state of integration (baseline)

• Quantify gap between current state and desired state

• Define desired outcomes

• Develop objectives and performance measures that can be linked to desired outcomes

28

Stakeholders must agree on performance measures in advance

• Perceived performance is an agreed-upon construct

• Criteria for defining performance should be negotiated by stakeholders (and governing body) prior to developing measures

• Stakeholders will value outcomes differently depending on their role within (or relative to) the justice enterprise

29

Characteristics of good measures

• Measures link back to goals, objectives and mission statements

• Measures drive the right behavior from employees, partners and consultants

• Collecting data on measures is feasible and cost effective• Measures are understood by most employees• Measures lend themselves to being tracked on an

ongoing basis so that drops in performance can be detected when there is time to do something about it.

• Measures represent aspects of performance that we can actually change

30

Performance measurement caveats

• Most people (including your employees and consultants) can learn to make measures come out the way they think you want them to, without actually improving a process

• Always question the measures you’ve defined, keeping in mind that the people applying them could find ways of boosting the measures without really improving anything

• Test each measure to determine if it operates as expected. Does it always go one way when things get better and the other when things get worse?

31

The Russian Nail Manipulating a single metric allowed Soviet managers to appear successful even though their efforts did not always lead to expected outcomes.

Success was typically measured by singular metrics of gross output, such as weight, quantity, square feet, or surface area. Gross output indicators played havoc with assortments, sizes, quality, etc., and frequently

resulted in products like Khrushchev’s chandeliers – so heavy “that they pull the ceilings down on our heads.”

A famous Soviet cartoon depicted the manager of a nail factory being given the Order of Lenin for exceeding his tonnage. Two giant

cranes were pictured holding up one giant nail.

My Time with Soviet Economics by Paul Craig Roberts(Published in The Independent Review, v.VII, n.2, Fall 2002,pp. 259– 264.)

32

Behavior driven the wrong way

• The Soviet Union wasted billions searching for oil because it rewarded drilling crews on the basis of the number of feet drilled. Because it is easier to drill many shallow wells than a few deep wells, drillers drilled lots of shallow wells, regardless of what was advisable geologically.

• The 1983 Chicago Sun Times article reported a Soviet hospital that had turned away a seriously ill patient because "they were nearing their yearly quota for patient deaths—and would be criticized by authorities if they exceeded it."

33

Family of related measures

• Produce x widgets per hour• Produce x widgets per hour without exceeding y dollars• Produce x widgets per hour without exceeding y dollars with

only one full-time employee• Produce x widgets per hour without exceeding y dollars and

with only one full-time employee and generating z units of waste

• Produce x widgets per hour without exceeding y dollars with only one full-time employee and generating z units of waste and at a zero defect rate

• Produce x widgets per hour without exceeding y dollars with only one full-time employee and generating z units of waste and at a zero defect rate and without contributing to global warming

34

The “widget” family

• Number produced within specified time period

• Cost of producing widgets

• People required

• Waste generated

• Defect rate

• CO2 produced

35

Specific Justice Example

36

Legislatively imposed measure

Number of calls from users requesting assistance (lower

number indicates superior performance)

37

Measure: Length of time to resolve a call for service and the quality of service call resolution as measured by the following two dimensions:

1. Average time from the opening of a service ticket to the closing of a service ticket. JID will also report the median and standard deviation with the average.

2. The quality of service as measured by a regular user surveys designed to measure the quality of the service provided to the caller. Survey respondents are selected randomly

Replacement multi-dimensional measure

38

Strategic Goal 3: Identify and recommend cost

effective biometric identification applicationsObjective 3.1:By September 2004, research, identify, and recommend technological applications that support biometrics for rapid identification.Objective 3.2:By September 2004, research, identify, and evaluate the costs and benefits ofbiometric identification applications.Outcomes:• Increased knowledge of biometric technologies• Improved cost-effective biometric identification solutionsPerformance Measures:• Number of research projects on biometric technological solutions completed by September 2004• Number of research projects on costs and benefits of biometrics completed by September 2004• Number of research reports presented to the Governing Body

39

Justice performance measures

• Average law enforcement response time to calls for service for incidents involving a threat to citizen safety

• Percent of arrest records in state repository with final dispositions

• Number of automated information exchanges within and between criminal justice agencies

• Number of crimes cleared using AFIS system(s)• Number of arrests made of wanted individuals resulting from

the use of electronically available warrant and detainer information

• Number of electronic checks of justice databases performed to identify high risk individuals

• Average time from arrest to final case disposition for felony arrests

40

What makes a performance measure effective?

• First and foremost, to be effective a measure must be an incentive to a person or group of persons to change behavior in such a way that things really improve.

• A performance measure should provide feedback to a person or group of persons. Without feedback no information is available on whether the target implied by the measure is being met.

• A performance measure (or family of measures) should be precise and comprehensive so as to prevent the possibility of the measure being met without actually leading to expected outcomes.

41Project Management

Strategic Planning

Performance Management

Three-legged Stool

42

The role of project plans

• Project plans can augment a performance plan by ensuring that outputs are completed on time and on budget

• Rigorous project management can ensure that tasks are actually performed before they are measured.

• Project planning, along with strategic planning, is an essential adjunct to any performance management program.

43

There’s more to management than measurement

If you can’t measure it, you can’t manage it.—Peter Drucker

Drucker’s saying has convinced some managers that measurement is management, which is a bit of an overstatement; however, measurement is one of the most powerful tools in management toolbox

44

Final points

1. If you don’t monitor your performance it will probably get worse.

2. You can’t devise performance measures in a vacuum, you must involve stakeholders and measure what’s valued.

3. Don’t devise measures for which you lack data.

4. Performance measurement can be expensive and time consuming so why bother unless you intend to use the results to provide ongoing process feedback.

5. Errors in devising measures will lead to unexpected consequences

45

So inscrutable is the arrangement of causes and consequences in this world that a two-penny duty on tea, unjustly imposed in a sequestered part of it, changes the condition of its inhabitants.

Thomas Jefferson

46

Performance MeasurementBJA’s Perspective

Michael DeverPolicy Advisor

47

Purposes: Performance Measures

• Linking people and dollars to performance

• Linking programs and resources to results

• Justification of continued funding

• Learning and management tools for us, for you

48

What Does BJA Do With the Data?

• GPRA: Government Performance and Results Act

• PART: Program Assessment Rating Tool (www.expectmore.gov)

• Budget formulation

• MD&A: Management Discussion and Analysis

49

How You’ll Report Performance Measures

• Via the semi-annual progress report submitted electronically via GMS (Grant Management System) Due Jan 30st and July 30st

• Report only on grant-funded activities during the specified reporting period

• Progress reports will not be accepted without complete data

50

Resources• Will Artley, DJ Ellison and Bill Kennedy, The Performance-Based

Management Handbook, Volume 1: Establishing and Maintaining a Performance-Based Management Program (Washington, DC: U.S. Department of Energy, 2001) at http://www.orau.gov/pbm/pbmhandbook/pbmhandbook.html

• John E. Eck, Assessing Responses to Problems: An Introductory Guide for Police Problem-Solvers (Washington, DC: Center for Problem-Oriented Policing, no date), at http://www.popcenter.org/Tools/tool-assessing.htm

• Michael Geerken, The Art of Performance Measurement for Criminal Justice Information System Projects, (Washington, DC: U.S. Department of Justice, Bureau of Justice Assistance, 2006 [forthcoming])

• Robert H. Langworthy (ed.), Measuring What Matters: Proceedings from the Policing Research Institute Meetings, (Washington, DC: NIJ/COPS, July 1999, NCJ 170610), pp. 37-53.

• David J. Roberts, Law Enforcement Tech Guide: Creating Performance Measures that Work! A Guide for Law Enforcement Executives and Managers, (Washington, DC: U.S. Department of Justice, Office of Community Oriented Policing Services, 2006) at http://www.cops.usdoj.gov/mime/open.pdf?Item=1968