Hands’ SMH Evaluation Summary Feedback and Recommendations
2014
Evaluators: Tracey Curwen, PhD & Glen Sharpe, EdD
Acknowledgements: Erin Brock, Nathalie Ouelette, Michelle Dermenjian
Goals
Provide research evidence re SMH programs
Identify most effective Hands’ SMH service
Provide recommendations based on evidence
Literature research and Hands’ SMH outcomes
Curwen & Sharpe, 2014
Summary
Overall, Hands’ SMH is beneficial
On average, clients improve, similar to literature
No differences among service models: mobile,
group, or classroom effectiveness
SMH services overall appear to be directed at few
children with moderate to severe concerns (*cf SNAP)
Most effective with younger population and those
who have greater difficulties
Focus on this population
with clear protocols of identifying mental health need
using most cost effective method
Curwen & Sharpe, 2014
Process
1) Literature review
2) Program information
3) Program outcome data
Curwen & Sharpe, 2014
1) Literature Review
2 models
School-based SMH services (SBMH)
Provided by school personnel
Expanded SMH services (ESMH)
combines traditional SBMH community-based mental health
programs/multidisciplinary care
Focus on ESMH - model used by Hands
Curwen & Sharpe, 2014
1) Literature review continued
Most samples did not meet “clinical cut-off” criteria
Significant decrease/improvement
moved from “no concern” to a lower score in the “no
concern” range
Others in clinical range
Significant decrease
Remain in clinical range
Not meeting criteria for program efficacy
1st Review of Literature
Curwen & Sharpe, 2014
1) Literature review continued
Refocused review
Why programs are effective or ineffective
Program Components & Characteristics
Dosage and duration
Service administrator
Fidelity (implementation consistency)
Theory/framework/focus
Delivery
School characteristics
Intervention characteristics
Curwen & Sharpe, 2014
3) Hands’ SMH Outcomes
Data provided by Hands
Last 3 years
Data included:
Age
Gender
BCFPI at intake to agency
CAFAS pre- and post- SMH
Curwen & Sharpe, 2014
3) Hands’ SMH Outcomes
N = 543 children and youth
70% male
Age: M = 10.53 (SD = 3.27)
Range 4 – 18 years
4-5 = 4% 6-12 = 67% 13-18 = 29%
All children/youth had at least 1 SMH program
16 different SMH services
32% multiple services concurrent
37% services following SMH
35% services prior to SMH
Only 1st SMH considered
Participants
Curwen & Sharpe, 2014
1st SMH Participants
0
10
20
30
40
50
60
70
80
90
4-5 6-12 13-18
Perc
en
tag
e
Age Group
Male Female
Curwen & Sharpe, 2014
First SMH Service
48.9%
25.6%
15.8%
5.9%
1.3
0.9
0.7
0.4
0.2
0.2
Mobile Group Consult
Classroom Student Specific Consultation Summer School Program
Monitoring Transitions Follow-up
SIS Intense Intervention
Curwen & Sharpe, 2014
Outcomes
Of 543 SMH cases
232 CAFAS at pre- & post-1st SMH
All 232 combined:
Significant Improvement No improvement
Total Score Community
School/Work Substance Use
Home Thinking
Behaviour Towards Others Caregiver – material needs
Moods/emotions Caregiver - social support
Self-Harm
p < .01
Curwen & Sharpe, 2014
Recall: Criteria for Evaluation
1) Investigate clinically significant change initially met or exceeded clinically elevated cutoffs
improved to a non-clinical range post-treatment
Investigated actual scores
2) The change must be considered reliable level of change must be sufficient enough to rule out random
fluctuations or measurement error
Used a stringent p value
CAFAS concern levels:
0 = none,
10 = minimal,
20 = moderate,
30 = severe
Curwen & Sharpe, 2014
Pre-post CAFAS Scores
0
2
4
6
8
10
12
14
16
18
20
CA
FA
S S
co
re
Pre-SMH Post- SMH
moderate
minimal
Curwen & Sharpe, 2014
BCFPI Mean Scores
0
10
20
30
40
50
60
70
80
Mobile Group Classroom Consult
T-s
co
res
Internalizing** Externalizing** Total Problems**
Clinically concerning
Sub-clinical
Curwen & Sharpe, 2014
Moderate – Severe Sample
Selected only those in the moderate to severe
range in any of the following:
Home
Behaviour towards others
Moods/emotion
n = 185
mobile (61%), group (24.3%), classroom (10.8%), and
consultation (3.8%)
Consultation dropped: only 8 participants
n = 177
Curwen & Sharpe, 2014
Change scores by SMH service
0
1
2
3
4
5
6
7
8
9
10
Mean
Ch
an
ge S
co
re
Mobile (n = 113) Group (n = 45) Classroom (n = 20)
MANCOVA with age and pre-scores covaried
Curwen & Sharpe, 2014
Single or Concurrent Hands’ Services
CAFAS scale
Only SMH
(n = 110)
Concurrent SMH+
(n = 68)
School/Work
6.3 (8.2)
6.2 (10.8)
Home
4.8 (8.3)
3.5 (11.3)
Behaviour Towards Others**
9.6 (10.9)
7.1 (10.1)
Moods/Emotions**
6.2 (9.1)
3.4 (6.9)
Note: pre-SMH CAFAS scores and age at SMH service were covaried
**p < .01
Curwen & Sharpe, 2014
Summary
Many empirical investigation samples are NOT in
the clinical range pre-SMH
Similar profiles for Hands’ SMH clients
Many clients did not have pre-SMH assessment
measures completed
Unclear WHY referred to SMH
Does not fit clinical level of MH need
Presenting issue is unclear and thus its EBP
How is the service to be received determined?
Curwen & Sharpe, 2014
Summary continued
Age is important to outcomes
Pre-SMH scores are important to outcomes
No difference in outcomes between mobile,
group, or classroom at exit
For those with at least one elevated Mental Health
score
Level of functioning 3-6 months later not assessed
Curwen & Sharpe, 2014
General Recommendations
Identify those most in need of mental health
services
Concerns in clinical or sub-clinical range
Services and intervention based on empirical evidence
Conduct assessments including:
Protocol linking specific need (e.g., depression) and
service most likely to benefit (e.g. mobile)
Parent, teacher, and child
Evaluate changes seen by each
Curwen & Sharpe, 2014
General Recommendations continued
Focus services on Younger age groups
With more concerning functioning scores
Implement the most cost effective service among mobile, group, classroom
Ensure clients meet eligibility criteria Or consider refocusing the purpose of SMH services
Do not provide too many services at once Clear assessment should allow for more focused
service/fewer services at one time
Keep the number of goals manageable *
Curwen & Sharpe, 2014
General Recommendations continued
Consider new working relationship with school
personnel
Many children were not clinically concerning
May be a classroom or teacher-student issue
How SMH providers may work with/assist teachers differently
Curwen & Sharpe, 2014
Limitations and Recommendations
Many children did not have an outcome measure
standard protocol to make SMH decisions
Re-consider data collection system and measures
Unclear how specific service is deemed necessary
Why group vs mobile vs consult?
Why multiple services?
Curwen & Sharpe, 2014
Service System Partnership
School Boards and Hands
You and we are doing what the literature reveals
Our story is not unique-but not as effective as it could be
Students/Children and Youth
All can be referred to Hands for a variety of services
Priority is to link clinical levels of need with appropriate system
response (level of need, intervention type, location, capacity)
SMH Services System Partnership
Provide most cost effective, ethical, evidence based service at
appropriate levels in our collective
Which Student Should Be Priorized?
Severe Problem Identification
CAS
Pediatrician
School
…or probation, or family who’s environment is disrupted
Opportunities
Increased inclusion of teacher in assessment and
treatment planning process for children and youth
Generalization of skills (from therapy) into classroom
With intensive intervention modelling and coaching
Increasing capacity of classroom milieu to support
child and not solely seeking child change
ongoing mental health issues (1:5) can be reflected in
chronic behavioural challenges whose intensity
intervention can moderate, but not necessarily eliminated
Maximizing Service System Capacity
75 schools and limited SMH staffing
We, collectively, need to do something different in
the system with collective mental health resources
Current Capacity
Hands serves over 2000 files across CYMH programs for
students who attend schools which should be priorized for
school based mental health services
Maximizing Service System Capacity
WHAT SYSTEM DO WE WANT FOR OUR
CHILDREN AND YOUTH?
Partnering to provide equitable services
for children and youth