institute for policy research evidence for agile policy makers the contribution of transformative...
TRANSCRIPT
Institute for Policy Research
Evidence for Agile Policy Makers
The Contribution of Transformative Realism
Graham Room
14 February 2014
Victoria University Wellington
Institute for Policy Research
Evidence for Policy
Assess impacts of different interventions by systematic review of evidence
‘What Works Centres’:•National Institute for Health and Clinical Excellence
.
•Education Endowment Foundation
•Centre for Local Economic Growth
Institute for Policy Research
X
Y
Institute for Policy Research
Institute for Policy Research
Institute for Policy Research
The Hierarchy of Evidence and Rigour
1a Evidence from a systematic review of randomized controlled trials
1b Evidence from at least one randomized controlled trial
II a Evidence from at least one study without randomization
II b Evidence from at least one other type of quasi-experimental study
III Evidence from non-experimental descriptive studies (e.g., case control studies, correlation studies)
IV Expert consensus
Institute for Policy Research
Pawson, R. (2006), Evidence-Based Policy: A Realist Perspective
‘Impact’ (medical) versus ‘engagement’ (social policy)
Interventions unfold in institutionally complex environments
Rarely is it the ‘same’ intervention that is delivered across all contexts
Pawson’s Realist Critique
Institute for Policy Research
Pawson’s contingencies of intervention
Institute for Policy Research
Darwin’s Tree of Life
Institute for Policy Research
‘Transformative realism’
A
A1 A2
A3
B
B1 B2
B3
Institute for Policy Research
EBPM: Evidence of impact of a given intervention: universal and unconditional
Pawson: Impact of a given intervention under a range of contingencies
Transformative realism: Dynamic synergies among interventions
Three Paradigms of Evidence-Based Policy-Making
Institute for Policy Research
The Choice of ParadigmsAssess the scope
and strength of any dynamic synergies between the policy intervention and the policy ecosystem -
using evidence from other settings?
Strong
Apply paradigm 3
Weak
Apply paradigm 2
They are active and
agile and this is important
for the effectiveness
of the intervention
How active and agile are the subjects of the intervention?
How important is it for the effectiveness of the
intervention that they should apply and adapt it to their
circumstances?
Apply paradigm 1
Inactive
Institute for Policy Research
Paradigm 1: Well-evidenced generalisations about impact (universal truths?)
Paradigm 2: Well-based theory as to out-turn under specific local contingencies
Paradigm 3: Need real-time watching and learning-by-doing
Types of Knowledge
Institute for Policy Research
• How to tune social developments in real time?• Where to probe and where to avoid?• What signs of tipping points to avoid or approach? • What data in real time? Big Data?• Ian Stewart (1997), Does God Play Dice?)
• Tuning a complex system (engineering and medicine)• Can this approach be applied to social systems?
Real-Time Watching and Learning-by-Doing
Institute for Policy Research
• How to ‘read’ the ‘sea conditions’?• Networks of ‘navigators’ – tracking how policy ecosystem is developing and
deciding what ‘tuning’ is needed • Elizabeth Eppel, Amanda Wolf, David Taylor: Complex Policy Implementation: The
Role of Experimentation and Learning• ‘Ordinary’ situations - rule of thumb • Anomalous (extraordinary) situations – examine carefully and devise a specific
strategy • What thresholds to take as triggers for concern?
Real-Time Watching and Learning-by-Doing
Institute for Policy Research
• Danger of de-politicising policy• What works for whom?• Corporate and other interests try to shape our
policy futures: is there any shared national interest?
• Good research on impact is necessary but not sufficient for good policy
Concluding Reflections
Institute for Policy Research
Behavioural Insights Team (2012), Test, Adapt, Learn, UK Cabinet Office
Eppel, E., Turner, D., & Wolf A. (2011). ‘Complex policy implementation: The role of experimentation and learning’. In B. Ryan & D. Gill (Eds.), Future State: Directions for public management in New Zealand (pp. 182-212).
Pawson, R. (2006), Evidence-Based Policy: A Realist Perspective
Room, G. (2011), Complexity, Institutions and Public Policy: Agile Decision-Making in a Turbulent World
Room, G. (2013), 'Evidence for Agile Policy Makers :The Contribution of Transformative Realism', Evidence and Policy, 9(2): 225-244.
Stewart, I (1997), Does God Play Dice?
References