phm analyst academy · 2019-10-10 · (ex-ante) design stage evaluations / impact assessments vs...

Population Health Management

NHS England and NHS Improvement

PHM Analyst Academy8th October

08.45 - 09:15 Arrival and Registration Lead

09:15 - 10:00 Welcome & introduction to the day

Mentimeter questions

(Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations

L Hawkins, SU

G Wrench, PHE

S Wyatt, SU

10:00 - 10:45 Introduction to Logic models & Qualitative Methods F Battye, SU

10:45 - 11:05 Coffee/Tea Break

11:05 - 12:30 Modelling approaches (RIGHT Framework) S Wyatt, SU

12.30 - 13.25 Lunch

13:25-13:30 Impromptu networking

13:30 - 14:00 Business case development – multi organisation – STP / ICS P Taylor, Provex

14:00 - 15:30 Experimental study designs

- Randomised controlled trials

- Cluster randomised trials

- Stepped wedge designs

Observational study designs

- Interrupted time series analysis

(exercise)

- Matched cohort design

- Synthetic controls

P Seamer, SU

14:45 Coffee/Tea Break

15:50 - 16:15 Tabletop reflections & Mentimeter questions

Review and look ahead to Session 3

M A

Mohammed, SU

G Wrench, PHE

16:30 – 18:00 Optional informal networking to include a 40 min Seminar on ‘Understanding the Arrival and Waiting

Patterns of Ambulances at Emergency Departments using Discrete Event Simulation Modelling’

S Croft, SU



Welcome

Lucy Hawkins and Fraser Battye

midlandsphmacademy.nhs.uk



Mentimeter

Gareth Wrench



Evaluation and Impact Assessments

Steven Wyatt

•Before implement - design stage evaluation / ex-ante modelling

•After implement - quantitative (ex-post) evaluation

When should we estimate the effect of a planned change?

14

time

Ex-ante

Ex-post

known

unknown

Intervention

or RiskOutcome

or Impact Applications

• commissioning plans

• service plans

• opportunity assessment

• business case

• roll-out

• decommission

• adjust plans / contracts

• remedial action



What’s your theory of change?

A brief introduction to logic models

Fraser Battye

Every action has a causal theory

16

“If we do x, then we’ll get y”

Sometimes explicit; usually implicit

“If we deliver our training package, then we will

improve the skills of care homes staff...

If staff are more skilled, then they will be more able to

cope in the event of a crisis...

If staff are more able to cope in a crisis, then there will

be fewer unplanned admissions to hospital....

If there are fewer unplanned admissions, then more

people will die in a setting of their choice. They will have

a better death; we will make better use of resources.”

‘An apple a day keeps the doctor

away…’

Giving free fruit to primary schools in order

to improve health

What’s the causal story? How do you

(logically) get from action to outcome?

Pulling out this ‘theory of change’ is essential for evaluation – and for designing initiatives

We learn via advances in theory

So the unit of analysis for evaluation

should be ‘the theory’

Results then refine, reject, (provisionally)

support the theory

Apples

Delivered

Apples

Eaten

Vitamin

Levels

Raised

Health

Outcomes

Improved

Interpretation

❌ ❌ ❌ ❌ Implementation Failure

☑ ❌ ❌ ❌Engagement Failure

(first causal link)

☑ ☑ ❌ ❌Theory Failure

(early causal link)

☑ ☑ ☑ ☑ Consistent with theory

☑ ☑ ☑/❌ ☑Theory Failure

(later causal link)

☑ ☑ ☑/❌ ☑/❌Partial Theory Failure

Works in some contexts

☑ ☑ ❌ ☑Theory Failure

(different causal path)

“An apple a day...” (Ref: Funnell & Rogers, 2013)

Logic models are one tool for representing these theories. There aredifferent approaches; all share common elements*

* Terminology varies but basic concepts remain constant

Inputs

Resources

used

Activities

Things done

(measured

by outputs)

Outcomes

Effects of

activities

Impacts

Broader

societal

‘goods’

Simple logic model for refurbishing a house (the lazy way)…

Inputs

£

Time

Activities

Source and

manage

experts

Outcomes

Improved

domestic

environment

Impacts

Increased

house value

Improved area

Inputs

£

Time

Activities

Book place to

stay, find

things to do

together

Outcomes

Reduced

stress (?)

Increased

happiness

Impacts

Improved

family

functioning

…or going on a family holiday

“If we deliver our training package, then we

will improve the skills of care homes staff...

If staff are more skilled, then they will be more

able to cope in the event of a crisis...

If staff are more able to cope in a crisis, then

there will be fewer unplanned admissions to

hospital....

If there are fewer unplanned admissions, then

more people will die in a setting of their choice.

They will have a better death; we will make

better use of resources.”

This shows the

theory that

connects activity…

…to outcomes…

…to impacts

•Impacts are the final effects that you are working towards – e.g. increased life expectancy, reduced health inequality, more sustainable services, etc

•Normally expressed at a high level. Triple / quadruple aim a useful framework

•Changes at this level only indirectly attributable to your intervention –you ‘contribute to’, rather than ‘cause’. Contextual factors a significant influence

Work back: what change do you ultimately want to see?

•These are the changes that you are trying to make / that would (logically / evidence suggests…) result from your activities

•Can be broken down into:

• Intermediate outcomes – changes in knowledge / awareness / skills / access

• Outcomes – changes in behaviour / condition / status

•Language suggesting change is therefore important: ‘reduced, increased, improved, better, worse’

Being more detailed: what outcomes do you need to achieve your impact?

•The things you do (e.g. establish apple scheme, etc)

•Measured by outputs (e.g. # people eating apples; # apples eaten)

•You don’t need to be detailed – just the main activities - the logic model is not a plan

•What ‘mechanism’ links activities to outcome? A causal ingredient?

What will you do to achieve these outcomes?

Often at this point, you’ll find

yourself working backwards

and forwards

•Should be fairly straightforward: these are the resources you have to do the things you do

•Usually measured in £

•For most programmes, cash funding is the largest element – but maybe there are in-kind inputs too, e.g. if partners have assigned staff to your programme, if you have lots of volunteers, if you are given ‘free’ facilities, etc..

And what resources will you use?

“In an Integrated Care System, NHS organisations,

in partnership with local councils and others, take

collective responsibility for managing resources,

delivering NHS standards, and improving the health

of the population they serve.”

NHS England

Inputs

Resources

used

Activities

Things done

(measured

by outputs)

Outcomes

Effects of

activities

Impacts

Broader

societal

‘goods’

Take a step back and reflect on:

1: Assumptions in the model:o Practical (e.g. shows significant reliance on recruitment of…)

o Evidential (e.g. implied connection between activity x and effect y)

o Contextual (e.g. that there is no significant change in regulation of x)

Can this be used in programme planning? Is this showing risks to be managed? Would more evidence help design? What does it mean for evaluation?

2: Your overall theory of change. Policies generally use:o Sticks (beat / regulate things into place)

o Carrots (incentivise / ease the change you want)

o Sermons (eulogise and persuade)

What is the mix in your theory? Does this seem optimal given the task? If not, what is missing and can this be managed?

Parting note 1: Logic models and economic evaluation

Cost

(Economy)

Efficiency (£ per output)

Effectiveness / Benefit (£ per outcome)

Parting note 2: your model should reflect your theory, which might not be linear

Further resources / guidance

1. www.strategyunit.co.uk – search ‘logic model’ for fuller guide to using logic models

2. www.betterevaluation.org

3. HMT ‘Magenta Book’ – good all round guidance on evaluation

http://www.betterevaluation.org/



An incredibly brief introduction to qualitative

methods for evaluation

Fraser Battye

Qualitative Vs

Quantitative?

There are many different types of qualitative method / approaches for evaluation…

Individual

interviews

Group

interviews

Focus groupObservation

Document

analysis

Participatory

approaches

Diaries

Social media

analysisQualitative

surveysCase studies

Ethnography

…all underpinned by some common purposes

Understanding things as

they are experienced by

the people involved

(context)

Exploring – what matters, to

whom, why

Explaining (e.g.) why did

events unfold in this way?

Creating (e.g.) what

should we do now?

(Not always, but…) democratising

the evaluation process

Interesting to note what is

not on this list that would

be there for quantitative

method

Choice of method depends on many factors

•The question! First and most important consideration…

•Time and resource. Qualitative methods can be expensive / time consuming

•Ethics. Accessing participants, what you might find, power dynamics (etc)

•Skills. Done well, it looks easy; but so easy to do badly. Analysis of non-quantitative data is a specialist skill too

•Stage of the evaluation process. Drawing out theory? Designing approaches (defining outcomes that matter)? Tracing process / implementation? Explaining effects? Interpreting findings / working up recommendations?

•Related point: where does this fit with quantitative methods?



Break



Design stage evaluationSteve Wyatt

What it is that makes each of these a model?

What are common characteristics of models?

y ~ x

•Models help us understand things that would otherwise be obscured by the complexity of the real world.•Associations, casual relationships and core dynamics – descriptive / explanatory

•Consequences, forecasts - predictive

•Help us understand what we need to do - prescriptive

•Models make us document our assumptions.

•Models help us test things that would be too costly / risky / impractical to try in real life.

•Models can act as guides or templates for complicated actions / developments.

Why do we need models in healthcare?

Modelling and simulation techniques for supporting healthcare decision making: a selection framework

A collaboration of 6 universities

•Cambridge Engineering Design Centre, University of Cambridge

•The School of Information Systems, Computing and Mathematics, Brunel University

•Brunel Business School, Brunel University

•The School of Management, University of Southampton

•The Information Engineering Research Group, University of Ulster

•The School of Mathematics, Cardiff University

The RIGHT Frameworkhttps://www-edc.eng.cam.ac.uk/downloads/right.pdf

https://www-edc.eng.cam.ac.uk/downloads/right.pdf

“This workbook is intended to provide guidance for people who are making decisions in healthcare. It is aimed at anyone who wants to find out more about different modelling and simulation techniques –what they are, when to apply them, and what resources are required to use them. It will not only help decision makers commission more appropriate modelling work, but also assist professional modellers and business consultants to expand their modelling repertoire in order to meet the diverse needs of their clients.

The workbook is not a “how-to-do” guide to modelling and simulation, rather a “what-is-it” introductory guide. That said, the further reading section at the end of the workbook will help locate further details for each technique. The RIGHT research team would also welcome any contact regarding the applications of these techniques.”

RIGHT Modelling

Framework

Design Stage

Evaluation

•A visual aid to explore how different variables in a system are related.

•A modelling method in its own right – but can also underpin quantitative modelling methods

•Comprised of

•a set of nodes, each representing a quantity or variable

•a set of arrows indicating the influence of one quantity on another

Causal loop diagrams (directed acyclic graphs)

X Y

an increase (decrease) in X tends to cause

an increase (decrease) in Y…

… all other things being equal

Population size

Number of

people with a

mental health

problem

Smoking

prevalance

Population life

expectancy

X Y

an decrease (increase) in X tends to cause

an increase (decrease) in Y…

… all other things being equal

X Y

X influences Z but through Y (i.e. not directly)

Z

Prevalence of

cardio-

vascular

disease

Premature

deaths from

cardio-vascular

disease

Smoking

prevalnce

Population life

expectancy

Prevalence of

cardio-

vascular

disease

Premature

deaths from

cardio-vascular

disease

Smoking

prevalence

Population life

expectancy

Incidence of

cancer

Premature

deaths from

cancer

hungerfood

consumptionb

a balancing loop

bank balance interestr

a reinforcing loop

•Balancing and reinforcing loops can be made up of more the 2 nodes

•A loop is

Balancing – if there are an odd number of negative arrows (- - - →)

Reinforcing – if there are no or an even number of negative arrows (- - - →)

Balancing and reinforcing loops

b

r

•Work in groups of 2 or 3

•Draw out these nodes on a asheet of paper

•Join the nodes to indicate influence

•When discussions concluded, you may want to produce final / tidy version

Exercise – Join the dots patients who

need to be

admitted

admissionsaverage

length of stay

admission

threshold

discharge

threshold

unoccupied

beds

all beds


need to be

admitted

admissionsaverage

length of stay

admission

threshold

discharge

threshold

unoccupied

beds

all beds


need to be

admitted

admissionsaverage

length of stay

admission

threshold

discharge

threshold

unoccupied

beds

all beds

b

bb


need to be

admitted

admissionsaverage

length of stay

admission

threshold

discharge

threshold

unoccupied

beds

all beds

b

bb

r

Roemer’s Law

"in an insured population, a hospital bed built is a filled bed“

Shain, M; Roemer, MI (April 1959). "Hospital costs relate to the supply of beds". Modern Hospital. 92 (4): 71–3

patients who

need to be

admitted

admissionsaverage

length of stay

admission

threshold

discharge

threshold

unoccupied

beds

all beds

b

bb

r

www.kumu.io

Last year…

•Hip replacements completed – 1253

•Mean length of stay – 7 days

•Overnight bed occupancy – 80%

•Beds – 30

•Mean waiting time – 91 days

You speak with the service and clinical lead and they say that in 5 years time they expect…

•Demand will have increased by 12%

•Mean length of stay will fall by 14%

• It would like overnight bed occupancy to reduce to 70%

•How many beds will we need?

•What assumptions are needed to reach this view?

Quantitative modellingan elective inpatient hip replacement service

Last year…

•Hip replacements completed – 1253

•Mean length of stay – 7 days


•Beds – 30



•Demand will have increased by 12% for all levels of need

•24% of cases have only a moderate level of need. They would like to divert these cases to receive a non-surgical service in another unit

•Mean length of stay for the non-moderate cases will fall by 14%





Last year…

•Hip replacements completed – 1253•Moderate need - 301•High need - 752•Severe need - 200

•Mean length of stay – 7 days•Moderate need – 3.0 days•High need – 7.0 days•Severe need – 13.0 days


•Beds – 30










Last year…




•Beds – 30







•How many beds will we need?•What if we only manage to divert 12% of moderate cases?•What if we live with 80% occupancy?



Last year…




•Beds – 30







•How many beds will we need?•What if we only manage to divert 12% of moderate cases?•What if we live with 80% occupancy?

•What would the average waiting time fall to?



•What if the questions was;

•If we divert moderate need patients from the beginning of year 4, how long will it take before the average waiting time falls below 84 days?

•What proportion of patients will spend more than 15 days in hospital?

•How frequently will occupancy exceed 85%?

What if the length of stay reduction is heavily dependant on the occupancy rate?

In such circumstances, basic rules-based models are not sufficient.

Two options worth considering;

•Systems dynamics modelling

•Discrete event simulation

Time dependant

I need to know about the model variables at a number of time points –

not just the start and end state of the model and I expect the changes

to evolve non-linearly over time

⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫

Individual

I need to know about variation between patients (its not sufficient to

know the average effect for groups of patients) and I have patient level

data to populate the model.

⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫

Interactions / dynamic

What happens to one patients strongly and importantly effects what

happens to others.

⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫

Stochastic

I need to know about the degree of uncertainty in the model results

(either due to randomness or uncertainty of inputs / parameters) – a

point estimate with / without sensitivity analysis is not sufficient.

⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫

Basic rule based model ⚫

Systems dynamics model ⚫ ⚫ ⚫ ⚫

Discrete event simulation model ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫

Discrete event simulation (DES)

Models of sequences of discrete events within a system where future events are a condition of the current system state.

•e.g. patients flowing through a hospital.

Probability distributions used to describe the arrival and treatment of patients.

https://www.youtube.com/watch?v=4BHBJlzv4RA

Systems dynamics modelling (SD)

Used to model nonlinear behaviour in systems over time.

Causal loop models used to describe stocks and flows within a systems.

Calculus used to define the system.

https://www.youtube.com/watch?v=nTD1SL2qp3o

https://www.youtube.com/watch?v=4BHBJlzv4RA

https://www.youtube.com/watch?v=nTD1SL2qp3o

Evidence is best – but not always available. It’s often necessary to rely on expert opinion.

Establish a reference group

•One opinion is better than none, but multiple opinions are better still (https://en.wikipedia.org/wiki/Wisdom_of_the_crowd)

•Be clear what you need to know

•Provide the reference group with useful context

Ask your reference group members to provide •a central estimate (best guess) is good, but ranges are better still•Ask for best guess, low & high estimate – and use triangular distributions or ask for 90% confidence intervals •You can calibrate you reference group member’s views (https://hubbardresearch.com/publications/how-to-measure-anything-book/)

•Delphi methods can be used to aggregate multiple views•Ask reference group members for views –share these and ask or second set of estimates (https://en.wikipedia.org/wiki/Delphi_method)

Tips on parameritising a model?

https://en.wikipedia.org/wiki/Wisdom_of_the_crowd

https://hubbardresearch.com/publications/how-to-measure-anything-book/

https://en.wikipedia.org/wiki/Delphi_method



Lunch



Business CasesWhat’s a Business Case and Why Do We

Need One?

Paul Taylor

•A written justification to take a recommended course of action

•The size of the business case is proportionate the scale of the investment

•HMT prescribes the format for large public sector spending

•Trusts and CCGs often have their own rules

What’s a Business Case & Why Do we Need One?

Let’s Buy a Car

Three Main Elements of a Business Case

The Benefits

Case

Affordability

Economic Case

•What’s the cost over the lifetime of the project

•Discounted Cash Flows

•Transfer Costs

•Risk Costs Valued

Economic Case

•Sometimes wrapped into the economic case – particularly in large cases

•Quantifiable Benefits

•Non-quantifiable Benefits

•You may end up ranking the options from both perspectives

•Often use surveys or focus groups for non-quantifiable asects

Benefits Case

•How much will it cost?

•Who’s paying?

•Are they willing to pay?

Affordability

TREASURY VIEWQuick Overview of the Treasury Guide

https://www.gov.uk/government/publications/

the-green-book-appraisal-and-evaluation-in-

central-governent or Google “Treasury Green

Book”

https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-governent

TREASURY 5 CASE MODEL

The

Case

Strategic

Case

Economic

Case

Commercial

Case

Financial

Case

Management

Case

The

question

Is the proposal

needed?

Is it

value for money?

Is it viable?

Is it affordable?

Is it achievable?

What the Business Case

must demonstrate

Will it further the

aims & objectives?

Have a range

of options

been considered?

Is there a supplier

who can

meet our needs?

Are the costs

realistic and

affordable?

Are we capable

of delivering

the project?

Is there a

clear case

for change?

Is it the best

balance of cost,

benefits and risk?

Can we secure a

value for money

deal?

Is the required

funding available

and supported?

Do we have robust

systems and

processes in place?

Adapted from:

www.gov.uk/government/uploads/system/uploads/attachment_data/file/190603/Green_Book_guidance_checklist_for_assessi

ng_business_cases.pdf ?

Guide to proportionate and well structured business cases asking:

•Were are we now?

•Where do we want to be?

•How we propose to get there?

•Strategic Case

•Economic Case

•Commercial Case

•Financial Case

•Management Case

TREASURY GREEN BOOK:5 CASE MODEL

Rationale for proposal, background & objectives to be achieved, strategic policy context & fit with wider policy objectives and corporate plans:

STRATEGIC CASE

• What is the case for change?

– What problem are we trying to solve and/or what is the opportunity for improvement?

• What are the investment objectives?

– What do we want to achieve e.g. increased quality, cost improvements?

• Strategic alignment

– How does the investment support national, regional & local objectives?

•Assess the economic costs and benefits to society as a

whole for the whole period of the proposal

•Costs and Benefits for each year of the proposal

•Must have a sufficiently wide consideration of options

including a “do nothing” option

ECONOMIC CASE

The economic case blends financial and non-financial value. Demonstrates that the spending proposal optimises public value. The case should be judged on this and will consider “what is the best answer to the question” including:

The Commercial Case demonstrates that the “preferred option” will result in a viable procurement and well structured Deal

•Procurement approach

•Service Requirements

•Charging Mechanism

•Risk Transfer (if any)

•Key Contractual Arrangements

•Personnel (TUPE) Implications if any

COMMERCIAL CASE

The Financial Case demonstrates that the “preferred option” will result in an affordable deal and how it will affect the organisations finances

•What is the impact on net expenditure

•Where will the money come from to pay for this? Which budgets?

FINANCIAL CASE

The Management Case asks how will we deliver the change into the Trust

•Who is doing what and when?

•What are the project benefits?

•What are the project risks?

MANAGEMENT CASE

This includes:

• Project governance e.g. SRO and project manager

• Delivery plan and milestones

• Project benefits and how will we know we have achieved them?

•the Strategic Outline Case (SOC) – which makes the case for change and refines the long list of options into a shortlist

•Outline Business Case (OBC) – building on the SOC to confirm the solution which offers optimal value for money

•Full Business Case (FBC) – building on the OBC, taking the chosen option, putting in place delivery plans and providing the final detailed costing of the scheme

TREASURY GREEN BOOK

Example of Business Case Benefits you might be dragged into

•Telephone contract for 5 years

•Vodafone

•Contract Extension and Tender

•NHS England & CCGs

•Public Health England and Wales/ Scotland

The context

5 year Benefits and Costs

Net Present Value

Used in the benefits (or economic) part of the equation

Treasury Test Discount Rate currently 3.5% per annum (and has been for a

number of years)

Treasury borrows money almost every day!

Avoided A&E attendances

AvoidedEDattendances27,285,593 CallstoNHS11112months

less 2,754,264 LeadtoanAmbulancedispatchless 1,859,887 LeadtorecommendationtoattendA&E

equals 22,671,442 NonA&Ereferredcalls

19% EstimateofhowmanycallsleadtoapatientNOTtravellingtoA&E(%ofserviceuserswhowouldhaveusedA&EinabsenceofNHS111minus%ofNHS111userswhowererecommendedtoattendA&E)

equals 4,309,680

83.33£ MeanAveragesavingforusingNHS111(costofEDvisitminusNHS111callcost)

equals £359,114,645 Maxsaving

40% Confidencetheestimatesaboveareright

equals £143,645,858

•Call Costs

•Risks Costed

•Optimism Bias

NHS England isn’t directly benefiting from the reduced volume of A&E Attendances – so this doesn’t make it into the affordability analysis

Affordability

Social Cost Effectiveness

Direct Public Costs

Indirect Public Costs

Wider Costs to Society

Risk Costs

Direct Public Sector Benefits

Indirect Public Sector Benefits

Wider Societal Benefits

Benefits can be:

• Cash Releasing

• Monetisable non-cash releasing

• Quantifiable not monetisable

• Qualitative unquantifiable

•Preferred Option already known/assumed

•Funding Body not interested

•No Funds!

•Badly written/ ill conceived documents

•Not sized for purpose

• Needs NHS England or HMT approval!!

So Why Do Business Cases Go Wrong so frequently

•Multiple formal governance arrangements – so who is going to agree it?

•Who gets the benefit?

•Affordable to whom?

•Transfer costs are complicated

•ICS financial framework shouldn’t be important but may determine motivation

It’s always more complicated with ICS/STP business cases

Questions



Break



ReflectionsMohammed Mohammed and Gareth Wrench



Many thanks for your participation!

phm analyst academy · 2019-10-10 · (ex-ante) design stage evaluations / impact assessments vs...

Documents