evidence and the policymaking process cfhss congress 2006 york university louise shaxson...

Evidence and the Policymaking ProcessCFHSS Congress 2006York University

Louise Shaxsonlouise@shaxson.com

The Series• Effective Research for Development

Policy: How researchers can maximize their influence on policy.

• Evidence and the Policy Making Process: How do policy makers access research, what constitutes evidence.

• Action Research for Maximum Impact: Some "good news" case studies, and practical research tools for practitioners.

This Workshop• Exercise: Food in primary schools

• Drivers of change

• Policy development trajectory

• Evidence and analysis

• What policymakers want

• The role of analysts

• EBPM in practice

• Further information

Exercise: Food in Primary Schools

Drivers of change• Increasing emphasis on the quality of

evidence and its use (Modernising Government);

• To underpin & inform strategy, policy, regulatory work, foresight; and to mitigate risk;

• Importance of challenge to evidence (BSE inquiry, Science Advisory Committees)

• Depth and breadth of future evidence needs will increase given complex and overlapping strategic priorities

• Smaller policy core (efficiency drive in the public sector)

• Getting rid of the ‘generalist/specialist’ label

• Improving intelligent customer capability

• Future evidence needs, when set against strategic priorities, are complex & overlapping. How do we deal with ‘sustainability’?

Policy development trajectory

…any robust information that helps to turn a Department’s strategic priorities into something concrete, manageable and achievable.

The nature of the evidence you need is proportional to the nature of the risk associated with the decision that is being made.

Evidence for policy is…

Evidence is:

Facts (data, known trends), judgements, opinions, analyses, syntheses, arguments, costings, reviews, qualitative & quantitative survey data

Analysis is:

Lines of argument (strategy-policy), research, interpretation

What is evidence and analysis?

…but the evidence base is built upon

• Data

• Lines of argument (analysis)

• Stakeholder opinions

Decision makers like numbers…

Evidence-based policy making is not a sacred cow:

There are policies that:

Use good information… …and use it well…

…and use it poorly…Use poor information…

Analysis & evidence for policy

Procuring, managing and carrying out research to provide new evidence

Scoping the issue, asking the question, deciding what sort of evidence is needed

Interpreting & applying new or existing evidence, monitoring & evaluating the policy once implemented

Evidence and analysis needed rapidly to answer pressing policy questions

Longer-term policy and strategy development

Components of robust evidence & analysis (supply side)

• Is the evidence credible?

• Can we make generalisations from it?

• Is it reliable enough for M&E or impact assessments?

• Is it objective? How do we account for bias?

• Is it rooted in an understanding of the framing assumptions?

Components of robust evidence & analysis (demand side)• Is the evidence policy relevant?

• Is it timely? Has it been delivered fast enough to inform policy decisions?

• Is it accessible to all key stakeholders, not just researchers?

• Is the evidence cost-effective?

• Is it interdisciplinary enough to address cross-cutting issues?

Scoping the issue, asking the question, deciding what sort of evidence is needed

Procuring, managing and carrying out new

research

Interpreting & applying new or existing evidence, monitoring & evaluating the policy once implemented

Quality assessment, peer review, advisory councils, G2000

Negotiate how to apply the evidence; consultancy role

Negotiate the question, advise on alternative sources of evidence

The role of analysts… …is to provide appropriate evidence & analysis throughout the policy making process

• A smaller policy core won’t have time to do the integration

• Intelligent customers need intelligent suppliers

• Multiple tools: social, economic, modelling, scientific, technical, institutional analysis…

• Help policy makers lift their eyes from their desks to see what is possible

but

• Help them maintain their focus on the deliverable.

The role of analysts

• Defra’s Evidence & Innovation Strategy:

• Matching supply-side and demand-side criteria

• Based on the question ‘does it make good policy?’ rather than ‘is it good science?’

• Develop a clear line of sight between policy priorities and evidence provision – for efficient delivery

• Analyse in relation to the policy cycle – evidence needs change from strategy through to delivery

• Need a better understanding of innovation

EBPM in practice

• the need to answer policy’s immediate questions…(with robust & cost-efficient evidence)

• while developing their role…(as people who understand policy processes)

• to underpin broader & more strategic approaches to policy (involving other evidence providers, particularly other disciplines)

How can analysts balance…

Selected Bibliography• Shaxson, L.J. (2005) “Is your evidence robust

enough? Questions for policy makers and practitioners”. Journal of Evidence & Policy 1(1): 101-111. http://www.ingentaconnect.com/content/tpp/ep/2005/00000001/00000001/art00006

• Evidence-based policy making: guidance for policy makers. Available at http://www.defra.gov.uk/science/how/evidence.htm

• Jones, K.E. (2005) Understanding risk in everyday policy making. Defra: UK. Available at http://www.defra.gov.uk/environment/risk/policymaking0509.pdf

• Defra (2005) Evidence & Innovation Strategy 2005–2008 (Part 1: Aims of the consultation / introduction). Available at http://www.defra.gov.uk/science/how/documents/PDFs%20in%20Parts/Part%20I.pdf

RAPID Stuff• ODI Working Papers

• Bridging Research and Policy Book

• Meeting series Monograph

• RAPID Briefing Paper

• Tools for Policy Impact

• Communication Tools

• Policymaker Tools

• RAPID CDROM

• www.odi.org.uk/rapid