using data to inform good agency practice nebraska justice alliance september 15, 2015 ryan spohn,...
TRANSCRIPT
Using Data to Inform Good Agency Practice
Nebraska Justice Alliance September 15, 2015
Ryan Spohn, DirectorNebraska Center for Justice Research
Anne Hobbs, DirectorJuvenile Justice Institute
Interactive Session - Please go to ResponseWare.com
What Did You Eat for Breakfast?
A. Coffee OnlyB. YogurtC. CerealD. EggsE. Some combo of the
aboveF. Nothing
Coffee O
nly
Yogurt
Cereal
Eggs
Some co
mbo of the a
bove
Nothing
17% 17% 17%17%17%17%
NOT “data driven”• Sutcliffe & Weber (2003), in a Harvard Business
Review article, concluded that the way that senior executives interpret their business environment is more important for performance than the accuracy of data they have about their environment
• Their capacity to interpret what data they had predicted high performance more than data quality or accuracy
NOT “data driven”• This suggests that experienced professionals,
administrators, and practitioners will never be replaced by number crunchers
• Some individuals use statistics as a drunk man uses lamp-posts — for support rather than for illumination
NOT “data driven”• “An expert is someone who has succeeded in
making decisions and judgment simpler through knowing what to pay attention to and what to ignore.” ~Edward deBono
• That being said…useful data collection and analysis on important performance measures and outcomes improves agency performance, all else equal
Objectives• Outline the importance of data-based decision making• Describe ways that data can be strategically integrated
into the decision-making process• Describe how using data can result in successfully
meeting agency goals• Describe basic research concepts useful to agency
leaders• Provide examples of how research data can inform
agency-related decisions
Data Rich and Information Poor• Many agency professionals report that “too
much data can’t be trusted and too much data lies fallow in vast, unexamined warehouses”
• Potential solutions: 1. Improve the data
2. Invest in people
3. Strive to empower all with dataRedman and Walker: “Make Data Work Throughout Your Organization”
Have you ever been required to religiously collect data, but no one ever looked at it?
A. YesB. No
50%50%
Have you ever mandated to report data that was seriously flawed?
A. NeverB. Occasionally C. Happens all the
time!
Never
Occas
ionally
Happens all t
he time!
33%33%33%
Data-Informed Decision-Making• Organizations that use data to inform decision
making perform measurably better than those who do not.*
1. Improve the data• As accurate as possible
• Systematic
• Tie it to the most central goals and mission of your agency
*Erik Brynjofsson, Professor at the MIT Sloan School of Management
Data-Informed Decision-Making2. Invest in people• Do not concentrate data in the hands of an
individual, but integrate it into day-to-day operations
• Train employees to be more data literate
• Teach “evaluative thinking”
Jim Giles, Economist Intelligence Unit
Data Rich and Information Poor3. Strive to empower all with data• Democratize the data
Jim Giles, Economist Intelligence Unit
I have expertise in my field, but I would not be helpful if we did an evaluation of my program
A. TrueB. False
50%50%
A program should not identify it’s own outcomes
50%50%
A. TrueB. False
Evaluation Myths
Goals of Data Informed Practice• Validate your assumptions• Your experience is invaluable, but test your
assumptions with data to see if they hold up
• Your agency environment is constantly changing and what worked yesterday may fail today
Slater and Moss, 2013
Goals of Data Informed Practice• Gain insight into behavior• New practices, rules, regulations, interventions,
etc. will produce expected outcomes and unintended consequences
• Data provide the best evidence for agency improvement
Slater and Moss, 2013
Goals of Data Informed Practice• Create a productive framework for decision
making• For better or worse, data has a certain
dispassionate element to it
• Using data can help to temper emotion-laden decisions
Slater and Moss, 2013
Example: Justice Reinvestment
Source: Urban Institute
Source: Urban Institute
Source: Urban Institute
Jail example• 35% of inmates are found to be booked into the county jail
multiple times within one year
• Looking at the data, stakeholders discovered that this 35% of inmates was consuming nearly 70% of resources
• Data collection focused on that 35% to determine their characteristics & reasons for jail return
• Goal: develop strategies to maximize cost savings and enhance public safety
• This “frequent user” population was determined to be largely homeless; therefore, stakeholders developed interventions to provide supportive housing with the help of local community agencies
Two Basic Assertions1. Making a commitment to support the
development of a knowledge base in your field is best demonstrated when one possesses a basic understanding of the underlying principles of research
2. Having this basic level of appreciation and understanding of data analysis and research is a responsibility of agency leaders
Nine Stages of Research1. Development of the research objectives and
understanding the decisions that will be made based on the research
2. Defining the sample and how it is selected
3. Determining the research approach (e.g. qualitative, quantitative, or mixed)
4. Creating data collection instruments (surveys, administrative data, etc.)
5. Collecting the dataSource: Slater & Moss, 2013
Nine Stages of Research6. Cleaning and analyzing the data
7. Writing the report or other deliverables
8. Presenting the findings
9. Working with staff to understand the results of the research and implementing changes based on these results
Source: Slater & Moss, 2013
Evidence Based Nebraska• Research objectives and understanding the
decisions that will be made based on the research– What works in Nebraska?– What if my program isn’t shown to be effective?
• Defining the sample and how it is selected - What works in Nebraska?
• Determining the research approach (e.g. qualitative, quantitative, or mixed)– Some measures will be similar across the state– Some operate differently and the measures cannot
be the same
• Creating data collection instruments (surveys, administrative data, etc.)– One common data set– Work with programs to create pre and post surveys
• Cleaning and analyzing the data-- Using Common Definitions
• Writing the report or other deliverable and presenting the findings
• Working with staff to understand the results of the research and implementing changes based on these results
-- Writing and presenting the report to the program (i.e. Crisis response )
Things You Always Wanted to Know About Data Analysis (But Were Afraid to Ask)
• What is an IRB?• Advantages of qualitative versus quantitative
research• What does multivariate mean?• What is sample selection bias?• Other?
Conclusions• Our work is too important not to use data to inform the
decisions of agency professionals
• Data should not “drive” decision-making, but should inform progress and innovation
• Data-informed decision making is one way of improving agency practice—add to your toolset
• We cannot predict the future impact of any innovation, intervention, or environmental influence, even those labeled “evidence-based”; we must collect the data and examine impacts in our jurisdictions
