using data to inform good agency practice nebraska justice alliance september 15, 2015 ryan spohn,...

Using Data to Inform Good Agency Practice

Nebraska Justice Alliance September 15, 2015

Ryan Spohn, DirectorNebraska Center for Justice Research

Anne Hobbs, DirectorJuvenile Justice Institute

Interactive Session - Please go to ResponseWare.com

What Did You Eat for Breakfast?

A. Coffee OnlyB. YogurtC. CerealD. EggsE. Some combo of the

aboveF. Nothing

Coffee O

nly

Yogurt

Cereal

Eggs

Some co

mbo of the a

bove

Nothing

17% 17% 17%17%17%17%

NOT “data driven”• Sutcliffe & Weber (2003), in a Harvard Business

Review article, concluded that the way that senior executives interpret their business environment is more important for performance than the accuracy of data they have about their environment

• Their capacity to interpret what data they had predicted high performance more than data quality or accuracy

NOT “data driven”• This suggests that experienced professionals,

administrators, and practitioners will never be replaced by number crunchers

• Some individuals use statistics as a drunk man uses lamp-posts — for support rather than for illumination

NOT “data driven”• “An expert is someone who has succeeded in

making decisions and judgment simpler through knowing what to pay attention to and what to ignore.” ~Edward deBono

• That being said…useful data collection and analysis on important performance measures and outcomes improves agency performance, all else equal

Objectives• Outline the importance of data-based decision making• Describe ways that data can be strategically integrated

into the decision-making process• Describe how using data can result in successfully

meeting agency goals• Describe basic research concepts useful to agency

leaders• Provide examples of how research data can inform

agency-related decisions

Data Rich and Information Poor• Many agency professionals report that “too

much data can’t be trusted and too much data lies fallow in vast, unexamined warehouses”

• Potential solutions: 1. Improve the data

2. Invest in people

3. Strive to empower all with dataRedman and Walker: “Make Data Work Throughout Your Organization”

Have you ever been required to religiously collect data, but no one ever looked at it?

A. YesB. No

50%50%

Have you ever mandated to report data that was seriously flawed?

A. NeverB. Occasionally C. Happens all the

time!

Never

Occas

ionally

Happens all t

he time!

33%33%33%

Data-Informed Decision-Making• Organizations that use data to inform decision

making perform measurably better than those who do not.*

1. Improve the data• As accurate as possible

• Systematic

• Tie it to the most central goals and mission of your agency

*Erik Brynjofsson, Professor at the MIT Sloan School of Management

Data-Informed Decision-Making2. Invest in people• Do not concentrate data in the hands of an

individual, but integrate it into day-to-day operations

• Train employees to be more data literate

• Teach “evaluative thinking”

Jim Giles, Economist Intelligence Unit

Data Rich and Information Poor3. Strive to empower all with data• Democratize the data

Jim Giles, Economist Intelligence Unit

I have expertise in my field, but I would not be helpful if we did an evaluation of my program

A. TrueB. False

50%50%

A program should not identify it’s own outcomes

50%50%

A. TrueB. False

Evaluation Myths

Goals of Data Informed Practice• Validate your assumptions• Your experience is invaluable, but test your

assumptions with data to see if they hold up

• Your agency environment is constantly changing and what worked yesterday may fail today

Slater and Moss, 2013

Goals of Data Informed Practice• Gain insight into behavior• New practices, rules, regulations, interventions,

etc. will produce expected outcomes and unintended consequences

• Data provide the best evidence for agency improvement

Slater and Moss, 2013

Goals of Data Informed Practice• Create a productive framework for decision

making• For better or worse, data has a certain

dispassionate element to it

• Using data can help to temper emotion-laden decisions

Slater and Moss, 2013

Example: Justice Reinvestment

Source: Urban Institute

Source: Urban Institute

Source: Urban Institute

Jail example• 35% of inmates are found to be booked into the county jail

multiple times within one year

• Looking at the data, stakeholders discovered that this 35% of inmates was consuming nearly 70% of resources

• Data collection focused on that 35% to determine their characteristics & reasons for jail return

• Goal: develop strategies to maximize cost savings and enhance public safety

• This “frequent user” population was determined to be largely homeless; therefore, stakeholders developed interventions to provide supportive housing with the help of local community agencies

Two Basic Assertions1. Making a commitment to support the

development of a knowledge base in your field is best demonstrated when one possesses a basic understanding of the underlying principles of research

2. Having this basic level of appreciation and understanding of data analysis and research is a responsibility of agency leaders

Nine Stages of Research1. Development of the research objectives and

understanding the decisions that will be made based on the research

2. Defining the sample and how it is selected

3. Determining the research approach (e.g. qualitative, quantitative, or mixed)

4. Creating data collection instruments (surveys, administrative data, etc.)

5. Collecting the dataSource: Slater & Moss, 2013

Nine Stages of Research6. Cleaning and analyzing the data

7. Writing the report or other deliverables

8. Presenting the findings

9. Working with staff to understand the results of the research and implementing changes based on these results

Source: Slater & Moss, 2013

Evidence Based Nebraska• Research objectives and understanding the

decisions that will be made based on the research– What works in Nebraska?– What if my program isn’t shown to be effective?

• Defining the sample and how it is selected - What works in Nebraska?

• Determining the research approach (e.g. qualitative, quantitative, or mixed)– Some measures will be similar across the state– Some operate differently and the measures cannot

be the same

• Creating data collection instruments (surveys, administrative data, etc.)– One common data set– Work with programs to create pre and post surveys

• Cleaning and analyzing the data-- Using Common Definitions

• Writing the report or other deliverable and presenting the findings

• Working with staff to understand the results of the research and implementing changes based on these results

-- Writing and presenting the report to the program (i.e. Crisis response )

Things You Always Wanted to Know About Data Analysis (But Were Afraid to Ask)

• What is an IRB?• Advantages of qualitative versus quantitative

research• What does multivariate mean?• What is sample selection bias?• Other?

Conclusions• Our work is too important not to use data to inform the

decisions of agency professionals

• Data should not “drive” decision-making, but should inform progress and innovation

• Data-informed decision making is one way of improving agency practice—add to your toolset

• We cannot predict the future impact of any innovation, intervention, or environmental influence, even those labeled “evidence-based”; we must collect the data and examine impacts in our jurisdictions

Questions or comments?Ryan Spohn, NCJR

rspohn@unomaha.edu

Anne Hobbs, JJIahobbs@unomaha.edu