enhancing evidence based services: ontario perception of care and addiction screening &...
TRANSCRIPT
Enhancing Evidence Based Services:
Ontario Perception of Care and Addiction Screening & Assessment
Presentation Outline
Looking Back Results of Drug Treatment Funding Program (DTFP) research,
development and piloting of both the Ontario Perception of Care tool for Mental Health & Addictions and the new Staged Screening and Assessment process for addictions
Current State Status of both projects
Coming Soon! Next steps for Ontario wide implementation
• Enhance knowledge regarding the purpose and impact of the Ontario Perception of Care tool for Mental Health and Addictions
• Increase knowledge and understanding regarding the staged screening and assessment protocol for addictions and concurrent disorders
• Enhance knowledge regarding implementation plans and system readiness for implementation
Presentation Objectives
Drug Treatment Funding Program (DTFP) is funded by Health Canada
Overarching goal of enhancing quality of addiction treatment systems
Resources flowed to provinces and territories as well as selected national projects
Ministry of Health and Long Term Care funds projects to enhance the Ontario addiction system
What is DTFP?
5Looking Back…..
Drug Treatment Funding Program Staged Screening
and Assessment for Addictions
Brief Project HistoryMore detail at: http://eenet.ca/dtfp/best-practice-assessment-procedures-project/
Last Round of DTFP: Informed Process
• Evaluation of ADAT (pre-DTFP)• Background literature review • Environmental scan of Ontario’s addiction agencies• Consultation with large Advisory Committee and close
engagement of Working Group• New process piloted within 5 treatment agencies• Recommendation: to move forward with
implementation of new suite of tools
Conceptual FrameworkDevelopmental Perspective
Assessment
Sta
ge
of
Cli
ent
En
gag
emen
tDIVERSITY LENS
Screening
Stage 1(case finding)
Stage 2(case definition)
Treatment & Support
Stage 1(within-treatment
monitoring)
Transitional Youth
Stage 1(placement matching)
Stage 2(modality matching)
Child Adolescent Older Adult Adult
Recovery Monitoring
Stage 2(post-treatment
monitoring)
Placement Matching Initial client assignment to a treatment setting with a
certain resource intensity (therefore important cost implications)
Modality Matching Creation of a case conceptualization and formulation
leading to an individualized and adaptable treatment plan
Grounded in person’s overall life situation and problem-focused, including trauma informed
Definitions
Selected and Piloted1st
Stage
Screener
• GAIN-SS
2nd Stag
e Screener
• MMS (Modified Mini Screen)• PDSQ (Psychiatric Diagnostic Screening Questionnaire)• POSIT (Problem Oriented Screening Instrument for Teens)
1st Stag
e Assessme
nt
• GAIN Q3 MI ONT
Engaged five treatment agencies to pilot all components of Staged Screening and Assessment Including one youth agency
234 clients recruited
Feedback Gathered On….Feasibility Potential usefulness at clinical and agency levels
Pilot Sites
Perceived STRENGTHS of New Process
• Screeners (stage 1 and 2):• Quick and easy to administer• Facilitates a comprehensive look at relevant psychosocial
domains• Stage 2 screener facilitates referrals for mental health
services
• GAIN-Q3 (stage 1 assessment):• The assessment is comprehensive and standardized • Concrete recommendations for treatment planning and
referral• Reasons and readiness to change questions highly valued• Strong foundation for outcome determination
• Screeners:• Established cut offs for stage 1 screener (GAIN-SS)• Some redundancy between the screeners
• GAIN-Q3:• Insufficient coverage of substance use history• Challenge for administration in a group intake context• Sensitivity around the perceived invasiveness of some
questions• Timing of administration for youth (impact on rapport)• Length of administration• More training needed to edit and interpret summary reports
Perceived CHALLENGES of New Process
Compared to ADAT….. Reported Value-add:• Staged approach seen as efficient and
comprehensive• Better quality data• Less variability in clinical
interpretation• Better treatment planning• Better support for referrals to mental
health services for clients with concurrent disorders
• Summary reports for clients provide better documentation
Possible Value-loss:• Flexibility of group intake• Concern about potential
impact on wait times – quality vs. quantity
• Collection of detailed information regarding substance use history
Bottom Line from Pilot Work• Support for staged model• General support to replace part or all of ADAT with
screening and assessment tools• Some revisions needed for the GAIN-SS and the
GAIN-Q3 (i.e. more on substance use) • Need to build on existing infrastructure and
provincial processes underwayRecommendation: Move forward with Ontario
implementation of new suite of tools
Responding to Pilot Feedback
• Addressing the concerns/challenges identified in advance of broad implementation including:Only one second stage screener for adults (mitigate
redundancy)Pilot agency has been successfully engaging in client self-
administration process (allow group intake, mitigate possible wait times)
Adapting GAIN Q3 MI ONT to include more detailed substance use and treatment history (respond to need for more depth of information in this area)
Pilot Sample Data:Supporting Agency & System
Planning
Analysis of Initial Assessment Data – combined substance useCase Defined As
Also Experienced
Heavy Alcohol Use
(53.5% of sample)N = 76
Heavy Marijuana Use
(30.3% of sample)
N = 43
Heavy Cocaine/ Crack Use(10.3% of sample)N = 14
Heavy Heroin/ Methadone/ Opioid Use (12.5% of sample)N = 17
Heavy Alcohol Use 48.8% 71.4% 70.6%
Heavy Marijuana Use
27.6% 28.6% 35.3%
Heavy Cocaine/Crack Use
13.5% 9.5% 11.8%
Heavy Heroin/ Methadone/Opioid Use
16.2% 14.3% 14.3%
Analysis of Initial Assessment Data – % severe problem areas by gender (n=150)
Male Female Total Problem Domain
SU 62.8 84.3 70.3Int. MH 64.9 88.5 73.3Ext. MH 38.3 55.8 44.5Physical 35.8 54.7 42.6Work 7.4 16.7 10.7Stress 43.2 54.7 47.3Risk Behav. 17.2 48.1 28.3Crime-Viol. 2.2 8.3 4.390-days Trauma 32.3 57.7 41.490-days Victimization 26.9 47.1 34.0
Analysis of Initial Assessment Data – % using community services in past 90 days
Analysis of Initial Assessment Data – % using community services in past 90 days by gender
Value of GAIN-Q3 MI for Outcome Monitoring
• In 2013/14 the project team also completed analysis of pilot outcome data
• 117 clients followed for 3 and then 6 months• Results show the value of the GAIN-Q3 for outcome
monitoring 90 day re-administration of Q3 MI ONT showed:
Decrease in substance use Positive changes in mental health, stress, physical health and risk
behaviours
Fidelity use of GAIN Q3 MI linked to better client outcomes
Ontario Perception of Care Tool for
Mental Health and Addictions (OPOC-MHA)
Brief Project HistoryMore detail at: http://eenet.ca/dtfp/client-satisfaction-project/
• Measures of client experience are widely used by customer-oriented businesses and healthcare services and settings
• Recognized as an important indicator of the quality of care as it is a direct measure of whether a client received services that met expectations and needs
Ontario Perception of Care Project
• Satisfaction a measure of the reaction to the services received • Respondents tend to report high levels of satisfaction even
though dissatisfaction might be voiced in open-ended questions or other feedback formats such as focus groups
• Measures of perception of care ask more directly about the care experience in relation to what is expected as standard practice • Range of responses likely to be wider as respondents may be
more willing to report infrequent exposure or use of a practice than express dissatisfaction about it
Satisfaction versus Perceptions of Care
• Literature review completed• Environmental scan
– All mental health and addiction agencies in Ontario using ConnexOntario database (30% response rate)
– Asked about current tools and processes to assess client perceptions of care
– Most agencies using something– Majority of tools developed in-house
• Extensive stakeholder consultation through Advisory Structure and project Working Group
Previous DTFP Iteration
• Developed new tool with support of Working Group• All tools evaluated in the literature review were assessed
according to validity, usage, length, and relevance• 8 tools were selected from this process• All items from these 8 tools were collated according to
domain for review by the Working Group and served as the foundation for the new tool
• Can be used in addictions, mental health, and concurrent disorder settings
• Translated into French
OPOC- MHA
Domain Sample Question
Access/Entry “The location of services was convenient for me”.
Services Provided “I had a good understanding of my treatment and support plan”.
Participation/Rights “I felt comfortable asking questions about my treatment and support, including medication”.
Therapists/Support Workers/Staff
“I found staff knowledgeable and competent”.
Environment “I felt safe in the facility at all times”.
Discharge/Leaving the Program
“I have a plan that will meet my needs after I leave the program”.
Overall Experience “The services I have received have helped me deal more effectively with my life’s challenges”.
Domains of the OPOC-MHA
CLIENT Version (38 items)• Registered clients of the program
– Those receiving services for their own treatment/support– Family members/significant others/supporters who are receiving
services in their own right)
Note: 6 items specific to inpatient/residential treatment services only
FAMILY/SUPPORTERS (17 items) • Family members/significant others/supporters who are not
registered clients but who are also receiving services from the program (such as parent who has a child in the program)
OPOC-MHA Versions
• Respondent’s age, gender, sexual orientation, ethnic background, and stage in the treatment process are included
• Information can be used for subgroup analyses and from an equity perspective
• Two open-ended questions to allow for comments about what the respondent found most and least helpful in their experience with the program, as well as room for comments throughout the questionnaire
OPOC-MHA Additional Questions
Data collected April 1/12 through June 30/12 1, 753 respondents Administration process varied by site
Least disruptive to program Administered at various points in time
Pilot Sites• 23 pilot sites from both mental health & addiction sectors• Sites represented a diversity of clients and services across
Ontario (i.e. inpatient/community, gender-specific, youth, Indigenous, ethno-racial and immigrant etc.)
OPOC-MHA Pilot Process
We have a tool and it works!• Overall feedback from staff and clients about the
OPOC-MHA was positive• Significant interest from mental health and addiction
agencies in the province (and elsewhere) to implement the tool
• The OPOC-MHA demonstrated strong psychometrics
OPOC-MHA Pilot Results
OPOC-MHA
Pilot Sample Data
34
OPOC-MHA: Referral to ServicesItem 11: I was referred or had access to other services when needed
(including alternative approaches)( Total 1,208 Responses )
45.1%
1.6%6.0%
47.3%
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
100.0
Strongly disagree Diasagree Agree Strongly agree
35
Item 22: Overall, I found the facility welcoming, inclusive, and comfortable ( Total 1,409 Responses )
55.1%
41.4%
2.6%0.9%0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
100.0
Strongly disagree Diasagree Agree Strongly agree
OPOC-MHA: Environment
36
Item 25: I felt safe in the facility at all times( Total 1,389 Responses )
62.7%
33.3%
2.8%1.2%0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
100.0
Strongly disagree Diasagree Agree Strongly agree
OPOC-MHA: Feeling Safe
37
Item 31: The services I have received have helped me deal more effectively with my life’s challenges
( Total 1,288 Responses )
55.9%
39.8%
3.4%0.9%0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
100.0
Strongly disagree Diasagree Agree Strongly agree
OPOC-MHA: Effectiveness
Sub-group Differences in Responses - Some Findings• Patients and supporters did not differ substantially in
their opinion on most of the statements
• Respondents from MH programs answered strongly agree less often on some statements compared to respondents from A and CD programs
• Young respondents (age <=18) appeared to answer strongly agree less often on most of the statements
Differences in Responses - Some Findings
• LGBQT respondents were less likely to endorse strongly agree on all the statements
• Respondents who have been mandated by court, medical authority, etc. appeared to answer strongly agree less frequently on all statements than respondents who voluntary participated in the program/treatment
40
• Staged Screening and Assessment Process being implemented in Ontario MoHLTC funded Addiction agencies
• OPOC-MHA being implemented in Ontario MoHLTC funded addiction, mental health and concurrent disorder programs
• Implementation beginning in about half of the LHINs, with other half beginning early 2016
Provincial Implementation
Research Based Approach to Implementation • Implementation Science (IS) is a proven approach to
bringing evidence-based research into practice to improve client outcomes
• IS means purposeful, planned and active implementation, supporting fidelity and sustainability
• Planned and supported implementation results in higher implementation with fidelity
More info on IS: http://nirn.fpg.unc.edu/learn-implementation
Reference: Greenhalgh et el. 2004.Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations. The Milbank Quarterly, Vol. 82(4), 581 – 629.
Implementation Science Supports • Implementation supports increase direct
practice change from 5% to as much as 95% • Implementation supports (i.e. fidelity monitoring
and supervision/coaching) help staff see the new evidence based practice is not ‘just another change/project/model’
• Increases staff’s ability to provide the new evidence-based practice competently & flexibly
Implementation Planning to Date
• Developed Provincial implementation plan using implementation science framework
• Fine tuning tools and processes to support implementation
• Developing infrastructure (i.e. catalyst)• Capacity building for DTFP implementation team
– Certification to trainer level of GAIN assessment tool
DTFP Implementation Team
• DTFP Implementation Team includes:– DTFP Implementation Supervisor– Implementation Coordinator (assigned to LHIN)– Implementation Coach (assigned to LHIN)– Evaluation Coordinator– Knowledge Broker– Research Analyst (OPOC-MHA)
• Dr. Brian Rush providing ongoing consultation and guidance to the team
46
The Details:What is Being Implemented
Best Practices for Screening and Assessment – Key Principles
• Structured information gathering with validated tools is vital: unstructured interviews miss co-morbidity – you have to ask!
• Staged approach saves time and resources for the longer screening and assessment
• Tools that cover both mental health and addictions enhance communication & relationship building across sectors
Stage 1 Screener: GAIN SS
If score is 2 or greater on
internalizing scale, proceed to
2nd stage screener.
If score is less than 2 on internalizing
scale, proceed to 1st stage assessment.
Stage 2 Screener:
MMS (18+)POSIT (12-17)
Stage 1 Assessment: GAIN Q3 MI ONT
(with detailed SU & tx. history)
Staged Screening & Assessment Process
• Available in English and French• Will be accessible via Catalyst/DATIS
– Ease of administration– Supports development of clinical profile/system level data
All Tools
• Studied and used in a number of different settings including Canada
• Valid and reliable down to 10 years of age – Canadian validation with adults – recommend for age 12 and up
• Cost: $100 agency licensing fee for 5 years unlimited use • Self- or clinician-administered (via GAIN ABS or paper
and pencil) • Reported to take 5 - 10 minutes to complete • Pilot used the CAMH-modified version – 7 additional
questions
Stage 1 Screening: GAIN-SS
• Validated in public sector settings in the U.S.• No cost• 22 items divided into 3 sections to capture the
three major categories of mental illness (mood, anxiety and psychotic disorders)
• Paper and pen: self-/clinician- administered• Estimated 15 minutes to complete
Stage 2 Screening (Adults): Modified Mini Screen (MMS)
• Valid and reliable• Designed to identify problems and potential
treatment/service needs in 10 areas including substance abuse, mental and physical health and social relations
• Estimated 20-30 minutes to administer, 2-5 minutes to score
• Administered by self/clinician and with paper/computer
• For use with clients aged 12 - 17
Stage 2 Screening (Youth): Problem Oriented Screening Instrument for Teenagers (POSIT)
• Developed by Chestnut Health Systems in Illinois• Good psychometric properties• One of main instruments in the GAIN family of
assessments• Ontario version was developed to increase the tool’s
relevance to the provincial context – Incorporated items around trauma and barriers– Cross-walk with ADAT; mapped to strengths and needs
criteria
Stage 1 Assessment: GAIN-Q3 MI(Ontario Version)
• Multi-purpose tool that identifies a wide range of life problems
• For use among adolescents and adults in both clinical and general populations
• For use in diverse settings• Established with strong focus on subsequent outcome
monitoring
Stage 1 Assessment: GAIN-Q3 MI(Ontario Version)
• School Problems• Work Problems• Physical Health • Sources of Stress• Mental Health
Content of GAIN-Q3 MI(Ontario Version)
• Risk Behaviours for Infectious Diseases
• Substance Use• Crime and Violence• Life Satisfaction
Basic Domains Covered:
• Each domain includes motivational-based questions specific to readiness for change and reasons for changing behaviour
• Estimated 45 minutes to complete face-to-face administration Computer/ABS (accessed via Catalyst) paper-and-pencil
• Self administration in a group format being used regularly at one of the pilot sites (quite successfully)
On the GAIN-Q3 MI (Ontario Version)
• Supports system-level client profile mentioned previously• A number of reports can be generated to support clinical
decision making and referral process– Individual Clinical Profile (ICP)– Personalized Feedback Report (PFR)– Recommendation Referral Summary (RRS)– Validity Report
On the GAIN-Q3 MI (Ontario Version)
Examined Complementarities of GAIN-Q3 MI ONT with other Tools
• Crosswalk between GAIN-Q3 and OCAN, RAI and LOCUS was conducted to explore relevant contributions of the GAIN-Q3
• Results: • Tools can complement each other• Only the GAIN Q3 provides the level of information
needed for substance use assessment and treatment planning
Embedded within the GAIN Q3 MI ONT:• Provide greater depth of information regarding
substance use and treatment history• Supports referral and/or treatment planning for clients
with more complex issues• Clinician may administer in same session as Q3 MI ONT
or subsequently if more information deemed necessary
Substance Use Grids
In Summary, New Staged Process……• Includes screeners:
That are quick and easy to administer Provide comprehensive look at relevant psychosocial
domains Facilitate referrals for mental health services
• Includes assessment that: Is comprehensive and standardized Provides concrete recommendations for treatment planning
and referral Includes reasons & readiness to change questions that are
highly valued Provides a strong foundation for outcome determination
Value Add at Multiple LevelsClinical:
• Assessment a process that should occur over time• Assessment needs to be motivationally oriented and
comprehensive across bio-psycho-social and spiritual domains of health
• Assessment alone can contribute to outcomes• GAIN assessment fidelity is linked to clinical outcomes
• Assessment plus evidence-informed interventions improves outcomes
Value Add at Multiple Levels
Organizational:• Analysis/summary of initial assessment data• Organizational clinical profile informs program
planning and quality improvement• Electronic administration and printable validity reports
(that notes administration inconsistencies) can support performance monitoring and coaching
Value Add at Multiple LevelsSystem:• Structured data:
• Basic information on demographic characteristics of clients that can be compared to the community profile to identify under-served populations
• Development of clinical/severity-based profiles of people in treatment (i.e. to confirm that those with the most severe profiles are being served in the most cost-intensive services)
• Provide a baseline for follow-up outcome assessment, which also has value at the individual level
• Tools with sound psychometrics and track record• Comprehensive assessment and resulting reports
facilitate better treatment planning• Higher concurrent disorder capacity• High level of detail on substance use and overall
severity: holistic view of client need• Fully integrated process incl. report generation• Agency clinical and psychosocial profile = informed
planning
Clinical & Agency Value-Add Summary
• Placement matching: Initial client assignment to a treatment setting with appropriate resource intensity (important cost implications and better use of existing resources)
• Detailed client profiles for community gap analysis, project planning (i.e. reducing ER and hospital use) and performance monitoring
• High potential for outcome monitoring • Comparable data with other LHIN’s for potential
benchmarking
LHIN/System Value-Add Summary
OPOC-MHAImplementation
Objective:• Systematically implement OPOC-MHA across all
MoHLTC funded substance use, concurrent disorder and mental health services
Projected Outcomes:• Standardized information regarding client
satisfaction/perception of care• Enhanced quality improvement and accountability
processes at both service and system levels across Ontario
OPOC-MHA Implementation
• Details of tool administration can and will vary by agency• Each agency and/or program can determine how and
when the tool should be administered• Four key requirements:
– Provide entire questionnaire – Ensure anonymity– Ensure completion is voluntary– Provide facilitation as needed
OPOC-MHA Administration Essentials
• Variety of ways the OPOC-MHA can be administered. No prescribed way, as this depends on the practices of each agency and/or program
• Distributed to clients in many ways, i.e. group setting, individually, in person at the agency or through email/mail
• Current administration via paper and pencil or electronically through Fluid Survey database– Data will be collected and analyzed centrally
• Future administration through DATIS/Catalyst (for both MH and A)
OPOC-MHA Data Collection
Timing• At any point in the person’s treatment process
One question asks which part of the treatment process a participant is currently engaged
Scoring• Likert scale
strongly agree, agree, disagree strongly disagree, N/A
OPOC-MHA Administration
Scales # of Items Items Scoring
Overall Perception of Care 23 1-8, 12-15, 17-18, 20-25, 30-32
Average score of the 23 items
Accessing Services 6 1-6Average score of the 6 items
Within Services 17 7-8, 12-15, 17-18, 20-25, 30-32Average score of the 17 items
Analyzing OPOC-MHA Data• Analysis & interpretation of OPOC-MHA data may involve individual
or grouped items• Responses to each item may be reported as % OR averages • “Overall Perception of Care” score
– Plus subscale scores for “Accessing Services” & “Within Services”
• OPOC-MHA designed to capture information on quality improvement indicators such as:– safety, accessibility, client-centredness, equity,
integration, effectiveness, and appropriate use of resources
• OPOC-MHA identified by Accreditation Canada as an instrument approved for use for assessing client satisfaction/perception of care for accreditation purposes
OPOC-MHA and Quality Improvement
Implementation Next Steps
• LHIN Engagement - Scope of the work– # of agencies– key contacts (A and MH networks?)– current process for system access etc.
• Development of LHIN-specific implementation plans– Using Implementation Science– Connected to overall Provincial plan– Contextualized to the LHIN
• Development and implementation of LHIN and agency supports
Note on Coordinated/Central Access
• Staged process lends itself more to coordinated access models than previous ADAT assessment process
– who does what component can vary but the package is well-developed and supported
• LHIN-level implementation plans will be developed for the specific community context
– Supported and tailored approach– Implementation will work within access model to
determine how the tools best fit
Key Milestones and Timelines
Milestone Anticipated TimelineEngage with early adopter LHINs May/June 2015Assemble DTFP Advisory Committee and SS&A/OPOC Working Group
June 2015
Develop LHIN Specific Implementation Plans
July 2015
LHIN Implementation Team Development
August 2015
Develop Agency Implementation Plans (incl. coaching support)
August/September 2015
Key Milestones and TimelinesMilestone Anticipated TimelineTraining and Capacity Building Begins September/October
2015
Initial Implementation begins in Early Adopter LHINs
October 2015
Track implementation, monitor progress, engage in developmental evaluation
October 2015 – April 2016
Engage with remaining LHINs February/March 2016Full Implementation in Early Adopter LHINs
May – December 2016
78
Training: Staged Screening & Assessment
• Mixed modalities• Web based training on some elements
• Introduction to the process• GAIN – SS• MMS• POSIT
• Face to face training• GAIN Q3 MI ONT• Motivational Interviewing• Implementation
79
SS & A Training Plan Overview
Chestnut Health Systems
Trains DTFP Team to Trainer Certification Level on GAIN Q3 MI ONT
Trains Team of LHIN Based Trainers to
Trainer Certification Level of GAIN Q3 MI
ONT
DTFP Team
Trains early adopter/champion
agencies in each LHIN on entire SS&A process,
including administration certification on GAIN Q3
MI ONT
Trains Team of LHIN Based Trainers on all other
elements of the SS&A process
LHIN Based Trainers
Trains all other implementing
agencies on the entire process
(with DTFP Team support) and
provides ongoing training for
sustainability
80
OPOC-MHA Capacity Building
• Web based orientation and training• Understand the tool• Administration details• Data gathering• Quality Improvement implications
Kim Baker Brian Rush DTFP Implementation Supervisor Project Consultant [email protected] [email protected]
Linda Sibley Donna RogersAddiction Services of Thames Valley [email protected] [email protected]