key issues of measuring the effectiveness of compliance assurance eugene mazur oecd environment...
TRANSCRIPT
Key issues of measuring the effectiveness of compliance
assurance
Eugene MazurOECD Environment Directorate
Why measure outcomes?
To know that policies and laws are actually implemented!
• Meeting legal and policy obligations• Internal management needs• Budget justification• Demand for enhanced external
accountability
The traditional focus on input and output measures is not enough to assess effectiveness!
2
What to measure?
Intermediate outcomes – changes in compliance knowledge & behaviour of the regulated community
• Compliance/noncompliance rates• Measures of recidivism/duration of
noncompliance• Pollution release indicators• Indicators of improved environmental
management practices and reduced risk• Measures of effectiveness of compliance
assistanceFinal outcomes – improvements in
environmental conditions as a result of compliance assurance
• Environmental quality indicators
3
How good are the indicators?
• Measurability – available at a reasonable cost, regularly updated via a reliable procedure
• Analytical soundness – well founded in technical and scientific terms– Scope definition, statistical validity, observation
periods, link between activities and outcomes• Policy relevance – user-friendly, easy to
interpret, provides a basis for international comparisons
• Phase 1 OECD project (2009-2010): analysis of practices in 10 countries
4
Assessment matrix
5
Measurability Analytical soundness
Policy relevance
Compliance rates M L M
Recidivism rates M L M
Pollution releases H M H
Improved management L L M
Effectiveness of assistance L L M
Environmental quality H M M
A toolbox – impossible to identify a “best practice” approach or a set of “flawless” indicators.
Key implementation principles• Targeting outcome indicators on concrete
regulatory priorities (pollutants, sectors) improves analytical soundness but reduces comparability– Develop compliance rates for specific sectors and
regulations– Focus pollution release indicators on priority
pollutants, use number of pollution incidents– Use only environmental quality indicators that
correspond to targeted, problem-oriented efforts• Link outcome indicators with time-specific
targets• Trends analysis in conjunction with input,
output indicators greatly increases policy relevance
6
Main issues going forward
• How to make compliance rates more meaningful?
• Using composite indices to characterise compliance outcomes?
• How many compliance indicators to measure (costs and benefits)?
• Is it feasible to develop a core set of indicators comparable nationally (decentralised systems) and internationally?
Phase 2 OECD project (2013-2014)
Compliance rates
Two main problems with compliance rates:
• Statistical bias as a result of ever-increasing emphasis on risk-based targeting
• Binary nature (in or out of compliance)– Differentiate non-compliance based on the
potential seriousness of environmental impact
8
Composite indicators• Despite the cost and misleading nature of
composite indicators (indices), they continue to be developed:– Perceived usefulness as a communication tool– Measure effectiveness on a scale
• Essentially, three categories of compliance-related indices:– Pollution release indices, for priority pollutants,
weighted for their relative harm (NSW, Australia)– Weighted compliance rates to distinguish
between procedural and substantive violations (Ontario, Canada)
– Risk scores: about more than compliance, but allows a comparable indicator of number of high-risk facilities
9
Number of indicators: costs and benefits
• Key challenge: how to measure time/cost requirements for data collection of certain types of outcome indicators
• Prefer intermediate outcome measures with the highest “measurability” (e.g. pollution releases, incidents)
• Pick those most closely linked to input and output indicators
• Promising “value for money” projects in the UK and Ireland
10
Comparability
• Designing comparable indicators not analytically impossible but politically difficult
• Primary need – in decentralised systems of environmental governance (level playing field across sub-national jurisdictions)
• Some “candidates”:– Rate of serious non-compliance for 3-4
“marker” requirements (e.g. linked to EU Directives)
– Number of high-risk installations (top score band)
– Number of serious pollution incidents per number of high-risk installations
– Monetary value of complying actions
11