the role of policy evaluation in violence preventionc.ymcdn.com/sites/ role of policy evaluation in...
Post on 02-Apr-2018
218 Views
Preview:
TRANSCRIPT
1
The Role of Policy Evaluation in Violence Prevention
Jennifer L. Matjasko, PhD Behavioral Scientist
Division of Violence Prevention National Center for Injury Prevention and Control
Centers for Disease Control and Prevention
National Center for Injury Prevention and Control Division of Violence Prevention
2
Overview
� Overview of the policy process
� The role of policy evaluation in the policy process
� Types of policy evaluation
� Policy evaluation and violence prevention (examples and illustrations weaved throughout)
� Conclusions
2
3
Overview
� Overview of the policy process
� The role of policy evaluation in the policy process
� Types of policy evaluation
� Policy evaluation and violence prevention
� Conclusions
3
5
Overview
� Overview of the policy process
� The role of policy evaluation in the policy process
� Types of policy evaluation
� Policy evaluation and violence prevention
� Conclusions
5
6
The Role of Policy Evaluation in the Policy Process
“Application of evaluation principles and methods to examine content, implementation, or impact of policy.”
Purpose: • Choosing among policy options • Informing implementation • Demonstrating impacts and value of
policy • Informing evidence base and future
policies • Accountability for resources invested
7
The Role of Policy Evaluation in the Policy Process
Problem Identification
Describe Policy Options
Assess and Select Policy
Options
Develop Strategy to Implement
Policy Implementation & Compliance
Evaluate Effects and
Impact
Implementation Evaluation
Impact Evaluation
Problem Identification Policy Analysis
Strategy and Policy
Development Policy
Enactment Policy
Implementation
8
Overview
� Overview of the policy process
� The role of policy evaluation in the policy process
� Types of policy evaluation
� Policy evaluation and violence prevention
� Conclusions
8
9
The Role of Policy Evaluation in the Policy Process
Problem Identification
Describe Policy Options
Assess and Select Policy
Options
Develop Strategy to Implement
Policy Implementation & Compliance
Evaluate Effects and
Impact
Implementation Evaluation
Impact Evaluation
10
Content Evaluation
� Does policy content clearly articulate goals, implementation, and intended change?
� What was the context of policy development? ��Essential in interpreting implementation and impact evaluation results
� Tools/Resources
� Legislative Records/Databases (Lexis Nexus, THOMAS, govtrack, state and local government websites)
� Logic Models
11
Content Evaluation Example: State-Level Anti-Bullying Laws
� The US Department of Education’s Analysis of State Anti-Bullying Laws � http://www2.ed.gov/rschstat/eval/bullying/state-bullying-
laws/state-bullying-laws.pdf
� To what extent do states’ bullying laws cover key legislative and policy components?
�Content analysis of anti-bullying laws according to Dept of Ed key components �Phase of the policy process?
US Dept of Ed (2011)
12
Content Evaluation Example: State-Level Anti-Bullying Laws
� Some findings
� Forty-six states have bullying laws and 45 of those laws direct school districts to adopt bullying policies.
� Thirty-six states include provisions in their education codes prohibiting cyberbullying or bullying using electronic media.
US Dept of Ed (2011)
13
Implementation Evaluation
� Was policy implemented as intended? � Understanding how a policy was implemented including critical differences between
planned and actual implementation ��Can be critical in interpreting impact evaluation results � Tools/Resources
� Constituent interviews � Case studies � Observations
14
Implementation Evaluation Example: State-Level Anti-Bullying Laws
� Another component of US Department of Education’s Analysis of State Anti-Bullying Laws � How are state laws translated into practice at the district and
school levels?
�Key constituent interviews in select states and districts
�Phase of the policy process?
US Dept of Ed (2011)
15
Implementation Evaluation Example: State-Level Anti-Bullying Laws
� Some findings
� Districts located in states with more expansive legislation
produced the most expansive school district policies. � However, there were some exceptions. Several school districts
in states with less expansive laws also substantially expanded the scope and content of their policies beyond the minimum legal expectations.
US Dept of ed (2011)
16
Impact Evaluation
� Did policy produce intended outcomes/impact? Can those changes be attributed to the policy?
� Short-term or intermediate outcomes as well as long-
term changes or impacts � Compare costs of a policy to the resulting benefits
� Tools/Resources
� Administrative data � Cost data from key constituents � Methods for constructing comparison groups � Policy evaluation experts
17
Impact Evaluation Example: Business Improvement Districts (BIDS) and Violent Crime
� Business Improvement Districts � Community economic development model � Special assessments on commercial properties to augment
public services
�Pre/post among 30 BID neighborhoods in Los Angeles
�Phase of the policy process?
MacDonald et al. (2010)
Problem Identification Policy Analysis
Strategy and Policy
Development Policy
Enactment Policy
Implementation Evaluate Effects
and Impact
18
Impact Evaluation Example: Business Improvement Districts (BIDS) and Violent Crime
� Some findings
� BID implementation was associated with a 12% reduction in the incidence of robbery and an 8% reduction in the total incidence of violent crimes.
� Implementation data informed these results: private security example.
MacDonald et al. (2010).
19
Overview
� Overview of the policy process
� The role of policy evaluation in the policy process
� Types of policy evaluation
� Policy evaluation and violence prevention
� Conclusions
19
20
Policy Evaluation and Violence Prevention
� A broad policy landscape
� Few policies have been rigorously evaluated
� Evaluations are needed at all stages of the policy process � To identify promising policy options � To specify the characteristics and active ingredients of a policy
intervention � To ascertain fidelity of implementation � To identify impact and potential unintended consequences of
policies
21
Violence Prevention Policy Evaluation Things to consider
� What is the policy? � Who is impacted? � What violence-related knowledge or behaviors do you
want to change? Or, what are the violence-related knowledge or behaviors that might be changed by a particular policy intervention?
� What is a realistic time frame to see change? � What are the potential unintended consequences? � What data sources are available to evaluate policy? � What other policies might confound findings?
22
Data Sources to Inform Evaluation
� Crime data � Public surveys � Medicaid and other claims data � Behavioral health data � ED and hospitalization data � Mortality data (e.g., NVDRS) � Census data
� Others? � New data sources to conduct evaluation?
24
Policy Evaluation Challenges/Solutions
Challenges � Evaluation is often an
afterthought � Data sources may not
be available � Bringing the right
players together may be difficult
� Statistical challenges with multiple interventions at the same time
Solutions � Include evaluation
planning early on � Identify data needs
prior to determine feasibility of the eval
� Identify key stakeholders as early as possible in the eval planning process
� Partner with academic institutions, other experts to conduct evaluations
25
Conclusions
� Policy evaluation is critical
� Plan evaluation before the implementation of policy (when possible)
� Demonstrating impact can be challenging when multiple interventions have been implemented
� CDC and others are available to provide technical assistance
top related