© Nuffield Trust 30 June 2015

Reflections on the challenges of capturing patient

and staff thoughts in evaluations

Ruth Thorlby

© Nuffield Trust

What is the Nuffield Trust’s approach to research and

evaluation?

“.. There is a real danger of circularity here – opinion sources are quoted as evidence, and subsequent readers may fail to look at the original source, assuming that what is quoted rests on good quality evidence”

“I could write the Nuff's report on the back of a fag packet between the Hobnobs and a cuppa-builders. So could you”

© Nuffield Trust

Evaluations: how ‘realist’ can we be?

‘What works, for whom, and under what circumstances’

Nick Tilley (2000)

http://evidence-basedmanagement.com/wp-content/uploads/2011/11/nick_tilley.pdf

• Theory (why might it work?)

• Embedded (professional, organisational and individuals)

• Active (requires people to do something)

• Open systems (other things happening at the same time) (Pawson and Tilley)

“A central distinguishing tenet of critical realism is that there are both visible and hidden forces at play in the generation of outcomes”

Jagosh et al;; Res Syn Meth 2013

© Nuffield Trust

Capturing user and staff experiences of admission

avoidance schemes

Evaluation of six volunteer-delivered projects across England

Expectations of commissioners: reducing hospital use and improving patient experience

Mixed methods evaluation

User experience?

Staff/volunteer experience?

© Nuffield Trust

Cabinet Office

© Nuffield Trust

Project Sites

Age UK, Cornwall

Royal Volunteer

Society, Leicester

Age UK / British Red

Cross, Leeds

Westbank, Devon

Age UK, South

Lakelands

British Red Cross,

Derbyshire

Age UK, Oxfordshire

© Nuffield Trust

© Nuffield Trust

The visible and the hidden: staff attitudes?

“she’s got some of the relationships in the hospital, so that’s hugely helpful because it’s quite difficult going to work in an acute hospital like that where there’s a bit of a blame culture in terms of delays ... everyone tends to say it’s someone else’s fault and obviously you’ve got mini-fiefdoms within a big hospital in terms of who actually gets to control what”

Voluntary sector project officer

© Nuffield Trust

Attitudes towards the statutory sector

“.. so they [the GPs] saw that as extra work, more people in the community identifying more issues that they ought to be dealing with for which they’re not receiving extra funding, so ‘why would we support you to go out there and find more patients for us to deal with?’”

Voluntary sector project officer

© Nuffield Trust

Local strategies for collecting user experience?

Warwick Edinburgh Mental Wellbeing Scale

Adult Social Care Outcomes Toolkit (ASCOT)

PHQ-9 (depression screening tool)

Generalised Anxiety Disorder (GAD-7)

Feedback forms

Bespoke forms (“Guided Conversation”; Compass)

© Nuffield Trust

Developing a user reported measure of care coordination

Funded by the Aetna Foundation (US)

• King’s Fund

• Picker Europe

• National Voices

• International Foundation for Integrated Care

• Nuffield Trust

• 18 months to develop a user questionnaire to explore whether care (health and social care) coordinated

© Nuffield Trust

© Nuffield Trust

Understanding ‘integration’ from a user perspective

• 2011 ‘Narrative’

published, with ‘I

Statements’

• ‘Narrative’ is based on

the statements that

patients and service

users could make if the

care they experienced

was person centred and

coordinated.

• Based on extensive

engagement with

patients and patient

organisations

© Nuffield Trust

Example of I Statements: communication domain

But: many of these are not straightforward to

measure!

© Nuffield Trust

Methods

• Focus groups with users (over 65), stakeholders (GPs, hospital and

local authority providers)

• Access to National Voices’ latest work with ‘older people’s narrative’

based on in-depth interviews with 75 older people

• Literature review, including policy and organisational documents from

local projects/innovation

• Expert advisory group, with international academic experts

• Cognitive testing n=30

• Piloting: paper questionnaire to approximately 3,000 older people

with one chronic condition, accessed via GP lists in 3-4 areas in

England, including mix of ethnic groups and income distribution

© Nuffield Trust

What matters to older people?

© Nuffield Trust

Current questionnaire status

39 questions, organised under headings:

• Your health and wellbeing

• Managing your health day to day

• Support from social services

• Planned care

• Urgent care

• Hospital care

+ standard demographic questions: age, gender, ethnicity, health status

© Nuffield Trust

Understanding your own culture in order to ‘see’ patients

Inside the system: • Hierarchical

• Scientific/bureaucratic

• Rational/hard

• Measurable

• Boundaries are

guarded

• ‘The patient’ as seen

by medicine

• ‘Patients’ as proxies

Patients’ worlds

• Unstructured/messy

• Emotions/feelings

• Real complicated lives

• Things that are hard to deal

with ‘inside’

• The ‘public’- latent patients

Adapted from: Putting patients first: what do we really mean?

Ereaut G and Whiting E Linguistic Landscapes 2013

© Nuffield Trust

“Focusing on the patient is difficult because it makes you

confront things you don’t necessarily want to face.” Hospital consultant cited in PFCC Final report (2014)

http://www.health.org.uk/media_manager/public/75/programme_docs/PFCC

%20final%20report.pdf

© Nuffield Trust

Concluding thoughts

Realist approach doesn’t require extensive theoretical expertise, but..

• Focus on the theories behind the intervention, the reality of the teams/individuals implementing it, how it might change as it is implemented and the changes happening around it

• Focus on the visible and the hidden

• Any user experience tools need to be relevant and useable!