Independent evaluation: the experience of undertaking policy evaluations commissioned by the Department of Health in England

Nicholas MaysDirector, Policy Innovation Research UnitDepartment of Health Services Research & PolicyWorkshop on ‘New approaches to health policy evaluation: what role for independence?’, Congress of the Swiss Evaluation Society (SEVAL), 4 September 2015

Improving health worldwide

www.lshtm.ac.uk

What does ‘independence’ mean in research and evaluation? • Means different things to different participants (academics, think

tanks, commercial agencies, not-for-profits)• Can be defined in terms of structure, source of finance (e.g.

endowment, government, charitable, commercial), ideology, ethos, ability to choose what to study and how to approach it (intellectual freedom)

• How independence is ‘framed’ can affect degree of influence• I see it as always a relative concept, along a series of continua

– always a (different) set of constraints (e.g. performance metrics in universities including to raise research funding) or incentives/disincentives & rewards

DH Policy Research Programme

• A national programme of independent research since 1970s to provide evidence that can be used by DH officials to advise their Ministers

• Based on simple ‘customer-contractor’ model, mostly uses university researchers

• Three main modes of (competitive) funding:– Policy Research Units, usually 5 years, potentially renewable– initiatives, consisting of groups of linked, commissioned projects– single commissioned projects

How DH typically commissions evaluations

1. Detailed invitation to tender inviting proposals from external research teams2. One or two-stage proposal process3. External peer reviews and comments from DH policy ‘customers’4. Response to reviewers’ reports5. Assessment of proposals by commissioning panel of officials and academics,

usually independently chaired6. Recommendation to minister of successful team (and/or negotiation of

refinements of original proposal)7. Contract, including rights to publish after 28-day notice/comment period, with

maximum 3-month delay– ‘Publication of scientifically robust research results is encouraged’

8. 10-12 month process if two stages

Assessment criteria

• RELEVANCE of the proposed research to the research brief

• QUALITY of the research design• QUALITY of the work plan and proposed management

arrangements• STRENGTH of the research team• IMPACT of the proposed work• VALUE for money (justification of the proposed costs)• INVOLVEMENT of patients and the public

Assumptions underpinning the DH approach• External, independent evaluation is superior to the alternative• Objectivity is a realistic and important goal• Conflicts of interest should be minimised by ensuring that evaluators

are not involved in what they are studying– No action-research or ‘engaged’ form of evaluation

• Emphasis on (mainly positivist) ‘science’• Centrality of peer review at all stages• Evaluation can remain relevant, timely and applicable while showing

the above features– no tensions between ‘independence’ and applicability

• DH and Ministers are the dominant ‘customers’

Advantages• Evaluation not subject to day to

day interference from government– formal rules of engagement

• Evaluators can report what they find

• Evaluators can focus on research skills rather than being expected to be implementers, management consultants, etc.

Disadvantages• DH can comment in ways that

may put researchers under pressure to self-censor– researchers have to determine

their own balance between being ‘helpful’ and ‘unresponsive’

• Requires expert intermediaries• Works best with a stable

‘customer’ group

Experience of the DH approach from the point of view of different participants

Advantages• Value for public money in that

findings are published• Degree of ‘independence’

maintains the value of using external evaluators– possible to provide some

challenge function

Disadvantages• Can appear slow &

cumbersome to busy policy officials– Researchers can be seen to be

‘remote’

• Still tensions when findings do not correspond to policy makers’ expectations

• Accusations of ‘policy-based evidence’ since researchers do not choose topics

Experience of the DH approach from the point of view of different participants

Advantages• Government can easily distance

itself from unpalatable findings

Disadvantages• Can be used to maintain a

façade of EBP– ‘independence’ no guarantee of

greater use of findings– can undermine engagement with

policy-relevant issues

• Ignores 60 years’ experience of more ‘engaged’ forms of evaluation

• Local implementers may not see value of ‘national’ evaluations– may not cooperate

Experience of the DH approach from the point of view of different participants

Different stakeholder perspectives on pilots and evaluations

DH

Pilot sitesEvaluators

Other Government departmentsHM Treasury

Local stakeholders, e.g. politicians, frontline staff;

providers; users/patients

Academic community, e.g. peer reviewers , commentators