Findings so far from outcomes data and looking to the future

Miranda Wolpert

CYP IAPT National Informatics LeadChair Outcomes and Evaluation GroupDirector CAMHS Outcomes Research Consortium (CORC)Director Evidence Based Practice Unit (EBPU) UCL & AFC

http://www.iapt.nhs.uk/cyp-iapthttp://www.corc.uk.net

AcknowledgmentsOutcomes and Evaluation Group

• Duncan Law- Clinical and outcome monitoring expertise in CAMHS and lead for COOP

• Margaret Murphy- Clinical and outcome monitoring expertise in CAMHS• David Clark - Adult IAPT measurement and analysis expertise• Kate Martin – Young Minds, service user participation expertise• Jessica Deighton – CAMHS measures and psychometric analysis expertise • Paul Wilkinson – CAMHS Psychiatrist with focus on outcome evaluation• Barbara Rayment – Youth Access, Young people’s counselling services

expertise• Margaret Oates – Adult IAPT data collection and collation expertise • Cathy Troupp – Research in use of ROMS in psychotherapy• David Trickey – Service Lead and expert in PTSD• Paul Stallard - CBT and service development expertise• Stephen Scott – Parenting expertise • Ro Rossiter – Learning Disabilities• Emma Morris - Service lead, Cultural issues• Philippe Mandin - Cultural issues• Rabia Malik - Cultural issues• Anne York – CAMHS S specialist • Anne O Herlihy - Service transformation• David Lowe - expert in use of proms in Family therapy• Peter Stratton – Family work• Cathy Street – CYP participation

CORC Nexus consortium• Dr Isobel Fleming, CORC Programme Lead • Jenna Bradley, CORC Project Manager• Rachel Argent, CORC Research Assistant• Dr Dan Brown, MegaNexus Operations Lead• Lee Murray, MN Project Manager• Phanindra Kaza, MN Senior Software Engineer• Nadia Kuftinoff, MN Project Support• Dan Reader-Powell, MN Systems Administrator

Data managers• Alex Papdakis• Andrea Shand• Barbara Snaith• Bill Clarke• Claire Newall• Craig Colling• David Markwick• Diana Viscusi• Elaine Blagden• Emma Broda• Fatima Blade• Hannah Mendoza-Wolfsaon• Husman Rafiq• Jacqui legge• Jess Parsons• Julia Yu• Laura Cunnen

• Liam Connolly• Maris Vainre• Mark Coughlin• Mark Lowe• Mary Urquhart• Mel Jarvis• Michelle McFarlane• Michelle Adams• Lesley French• Mike Presneill• Muhammad Akram• Paul Fry• Paul Townley• Sara Barnes• Sarah Oliver• Wendy Geraghty

Outcomes for practice Outcomes for evaluation

Outcomes for practice and evaluation

Findings so far: outcomes for practice

Outcomes for practice

Assessment/

Choice

• What’s the problem?” (assessment) This is understanding the issue the young person or family have come for help with

• “What do you want to change?” (goals or aims of therapy) – this is understanding the specific goals the young person or family have - the things they want to work on in coming to a service

Partnership/

ongoing work

• How are we getting on together?” (engagement or alliance) It is important to get things right from the start

• “How are things going?” (Symptom/goal tracking) – this is tracking to see if things are progressing during and intervention

Review & Close

• Have we done as much as we can/need to?” (collaborative decision to close or refer on) – re-review of question 1 e.g. Time2 SDQ (if not used as tracker in long-term case)

• “How has this experience been generally?” (experience of service overall).

Practice rules

The 4M principles

Minimal Burden No unnecessary form-filling.

Multiple Perspectives Consider collecting different people’s views.

Meaningful UseOnly select forms that could provide meaningful

information that will be used by somebody.

Missing Something?Consider whether any of the forms could fill a gap that

you may not have covered in some other way that might be useful to know about.

Practice rules

S om e Do ’s and Don ’ts of us ing c l in ica l ou tcom e too ls

Do

Make su re you have the fo rm s you need, ready before the s e s s io n

Do

A lw a ys e xp la in why yo u a re a sk in g a n yon e to fill out a fo rm

Do

L o o k at the a n swe rs

Do

D iscuss the a n sw e rs with se rv ice u s e rs

Do

S h a re the information in s u p e rv is io n

Do

A lw a ys u se information from th e fo rm s in conjunction with other

c lin ica l information

Don’t

G ive out a q u e s tio n n a ire if you think the p e rso n d o e s n ’t understand

why they a re b e in g a ske d to co m p le te them

Don’t

G ive u se any forms if you don’t u n d e rs ta n d why you are u s in g it

Don’t

In s is t o n someone f illing o ut forms if they are too d is tre s s e d

Don’t

S ee the n u m b e rs g e n e ra te d from o u tco m e to o ls a s a n a b s o lu te fact

Don’t

S ee your c lin ica l ju d g m e n t a s a n a b s o lu te fact

Findings in practice: young peoples’ views

Quotes from young people from YoungMinds consultation in Devon. Reference : Talking About Talking Therapies/Devon CAMHSViews of members of VIK Young Minds

It is important to monitor outcomes to make sure the person feels better not worse

It makes us feel like there is a point to our therapy

It means if we go off track or get a bit lost along the way, we can both figure out how to find the way back

again.

PROMs help make the balance of power

more equal.

Makes us feel like it’s a shared experience between us and the clinician... like we’re in

this together.

Enables us to get an in-depth understanding of what we’re feeling, why we’re feeling it

and what we can do about it.

Gives us a shared understanding of…where we’re starting from.

…where we’re heading to.…how we’re going to get there.

Findings in practice

“Using outcome measures in a therapy session needs to be done in a collaborative way with the young person and they must feel that it is important. Looking at the data and assessing it both with the young person and away from the session is key. Don’t just gather the information; use it to make the therapy better!”Young person with experience of service use To join the conversation and see user perspective go to..http://www.myapt.org.uk

Findings so far from outcomes for evaluation data

Outcomes for evaluation

Evaluation rules

The dreaded 90%!Time 1 Patient Reported Outcome Measure (PROM)

Time 2 PROM Education, employment and training (EET)

PROM with clinical norms. PROM from same reporter using same questions as Time 1

EET information recorded by practitioner

[1] The last recorded measure at point of closure will be used in calculations of outcome[2] If EET is recorded only once this will be taken as both T1 and T2 measure

Evaluation rules: PROMs with clinical norms

•SDQ (and/or impact scale separately)

•RCADS (and/or subscales for depression and different forms of anxiety)

•C/ORS

•Impact of Event Scale

•Behavioural difficulties child and parent measures.

Evaluation data : who seen Y1Jan-Dec 2012

• 1366 cases opened, 255 cases closed• Mean age 11.9 (SD=4.2; peaks at 8 and 15 yrs)• 41.9% male, 57.7% female (0.4% missing)• 59.2% White British, 11.6% other ethnicity

(29.2% missing)• 18.4% of closed cases were only seen once

– of these, 42.6% closed on professional advice

Annual Report

Evaluation data : What sort of problems (NB mainly from CYP IAPT trainees)

PsychosisSelective mutism

Gender discomfortToilet difficulties

Avoids going outSelf harming

Eating difficultiesAttachment difficulties

Habit problemCarer management difficulty

Anxious generallyAvoids specific things

Traumatic eventAnxious away from home

Severe inter-personal difficultiesADHD

Compelled to do or thinkAnxious socially

Panic disorderDepression, low mood

Substance abusingFamily relationship difficulties

Behaviour difficulties

0.0 5.0 10.0 15.0 20.0 25.0

MildModerateSevere

Percentage

N= 565, Missing N= 678

Note high proportion of missing data. Not sure this is representative of all data submitted.

Annual Report

Evaluation data : Interventions Offered(NB mainly from CYP IAPT trainees)

N= 688, Missing N= 678

Note: Categories are not mutually exclusive.

Note high proportion of missing data

Annual Report

Evaluation data : Information about outcomes

  FrequencyPercentage % (of all closed

cases)

Minimum cases sent by

individual site (percentage of closed cases in

submission)

Maximum cases sent by

individual site (percentage of closed cases in

submission)

Closed cases with a symptom specific outcome

149 85.1 0.0 100.0

Closed cases with symptom or general functioning outcome

154 88.0 0.0 100.0

Closed cases with symptom and education outcome

42 24.0 0.0 84.6

Closed cases either symptom or general outcomes and educational outcome

44 25.1 0.0 84.6

Closed cases seen at least three times with a measure completed at assessment

Annual Report

Evaluation data : emerging outcomes

• Significant improvement in average scores between first and last time points for:– CYP rated panic– CYP rated OCD– CYP rated separation anxiety– CYP rated generalized anxiety– CYP rated general wellbeing (CORS)– Parent rated depression– Goals

• No significant improvement in average scores between first and last time points for:– CYP rated depression

Annual Report

Looking to the future: Evaluation reports

Looking to the future: Practice GuidanceThe guide to collecting and using service user feedback and outcomes

informationEditors:

Duncan Law and Miranda Wolpert

Contributors include: • David Trickey• Cathy Street• Peter Stratton• Cathy Troup• Gill Walker• Barry Nixon• Andy Fugard• David Low• Emma Kawartzki• Melanie Jones• Jenna Bradley• Celia Beckett• Mark Dadds• Shona Falconer• Peter Fonagy• Evette Girgis

• Sajid Humayun• Karl Huntbach• Steve Kingsbury• Duncan Law• Claire Maguire• Anita Marsden• Susannah Marks• Nick Midgley• Scott D. Miller• Emma Morris• Kate O’Hara• Kathryn Pugh• Rebecca Putz• Barbara Rayment• Stephen Scott• Brigitte Squire

• Sarah Stewart-Brown• Cathy Street• Frances Taggart• Nick Waggett• Sally Westwood• Paul Wilkinson• Miranda Wolpert• Matt Woolgar• Ann York• Young Sessional Workers from the GIFT Team

Looking to the future: practice and evaluation