enhancing evidence based services: ontario perception of care and addiction screening &...

Enhancing Evidence Based Services:

Ontario Perception of Care and Addiction Screening & Assessment

Presentation Outline

Looking Back Results of Drug Treatment Funding Program (DTFP) research,

development and piloting of both the Ontario Perception of Care tool for Mental Health & Addictions and the new Staged Screening and Assessment process for addictions

Current State Status of both projects

Coming Soon! Next steps for Ontario wide implementation

• Enhance knowledge regarding the purpose and impact of the Ontario Perception of Care tool for Mental Health and Addictions

• Increase knowledge and understanding regarding the staged screening and assessment protocol for addictions and concurrent disorders

• Enhance knowledge regarding implementation plans and system readiness for implementation

Presentation Objectives

Drug Treatment Funding Program (DTFP) is funded by Health Canada

Overarching goal of enhancing quality of addiction treatment systems

Resources flowed to provinces and territories as well as selected national projects

Ministry of Health and Long Term Care funds projects to enhance the Ontario addiction system

What is DTFP?

5Looking Back…..

Drug Treatment Funding Program Staged Screening

and Assessment for Addictions

Brief Project HistoryMore detail at: http://eenet.ca/dtfp/best-practice-assessment-procedures-project/

Last Round of DTFP: Informed Process

• Evaluation of ADAT (pre-DTFP)• Background literature review • Environmental scan of Ontario’s addiction agencies• Consultation with large Advisory Committee and close

engagement of Working Group• New process piloted within 5 treatment agencies• Recommendation: to move forward with

implementation of new suite of tools

Conceptual FrameworkDevelopmental Perspective

Assessment

Sta

ge

of

Cli

ent

En

gag

emen

tDIVERSITY LENS

Screening

Stage 1(case finding)

Stage 2(case definition)

Treatment & Support

Stage 1(within-treatment

monitoring)

Transitional Youth

Stage 1(placement matching)

Stage 2(modality matching)

Child Adolescent Older Adult Adult

Recovery Monitoring

Stage 2(post-treatment

monitoring)

Placement Matching Initial client assignment to a treatment setting with a

certain resource intensity (therefore important cost implications)

Modality Matching Creation of a case conceptualization and formulation

leading to an individualized and adaptable treatment plan

Grounded in person’s overall life situation and problem-focused, including trauma informed

Definitions

Selected and Piloted1st

Stage

Screener

• GAIN-SS

2nd Stag

e Screener

• MMS (Modified Mini Screen)• PDSQ (Psychiatric Diagnostic Screening Questionnaire)• POSIT (Problem Oriented Screening Instrument for Teens)

1st Stag

e Assessme

nt

• GAIN Q3 MI ONT

Engaged five treatment agencies to pilot all components of Staged Screening and Assessment Including one youth agency

234 clients recruited

Feedback Gathered On….Feasibility Potential usefulness at clinical and agency levels

Pilot Sites

Perceived STRENGTHS of New Process

• Screeners (stage 1 and 2):• Quick and easy to administer• Facilitates a comprehensive look at relevant psychosocial

domains• Stage 2 screener facilitates referrals for mental health

services

• GAIN-Q3 (stage 1 assessment):• The assessment is comprehensive and standardized • Concrete recommendations for treatment planning and

referral• Reasons and readiness to change questions highly valued• Strong foundation for outcome determination

• Screeners:• Established cut offs for stage 1 screener (GAIN-SS)• Some redundancy between the screeners

• GAIN-Q3:• Insufficient coverage of substance use history• Challenge for administration in a group intake context• Sensitivity around the perceived invasiveness of some

questions• Timing of administration for youth (impact on rapport)• Length of administration• More training needed to edit and interpret summary reports

Perceived CHALLENGES of New Process

Compared to ADAT….. Reported Value-add:• Staged approach seen as efficient and

comprehensive• Better quality data• Less variability in clinical

interpretation• Better treatment planning• Better support for referrals to mental

health services for clients with concurrent disorders

• Summary reports for clients provide better documentation

Possible Value-loss:• Flexibility of group intake• Concern about potential

impact on wait times – quality vs. quantity

• Collection of detailed information regarding substance use history

Bottom Line from Pilot Work• Support for staged model• General support to replace part or all of ADAT with

screening and assessment tools• Some revisions needed for the GAIN-SS and the

GAIN-Q3 (i.e. more on substance use) • Need to build on existing infrastructure and

provincial processes underwayRecommendation: Move forward with Ontario

implementation of new suite of tools

Responding to Pilot Feedback

• Addressing the concerns/challenges identified in advance of broad implementation including:Only one second stage screener for adults (mitigate

redundancy)Pilot agency has been successfully engaging in client self-

administration process (allow group intake, mitigate possible wait times)

Adapting GAIN Q3 MI ONT to include more detailed substance use and treatment history (respond to need for more depth of information in this area)

Pilot Sample Data:Supporting Agency & System

Planning

Analysis of Initial Assessment Data – combined substance useCase Defined As

Also Experienced

Heavy Alcohol Use

(53.5% of sample)N = 76

Heavy Marijuana Use

(30.3% of sample)

N = 43

Heavy Cocaine/ Crack Use(10.3% of sample)N = 14

Heavy Heroin/ Methadone/ Opioid Use (12.5% of sample)N = 17

Heavy Alcohol Use 48.8% 71.4% 70.6%

Heavy Marijuana Use

27.6% 28.6% 35.3%

Heavy Cocaine/Crack Use

13.5% 9.5% 11.8%

Heavy Heroin/ Methadone/Opioid Use

16.2% 14.3% 14.3%

Analysis of Initial Assessment Data – % severe problem areas by gender (n=150)

Male Female Total Problem Domain

SU 62.8 84.3 70.3Int. MH 64.9 88.5 73.3Ext. MH 38.3 55.8 44.5Physical 35.8 54.7 42.6Work 7.4 16.7 10.7Stress 43.2 54.7 47.3Risk Behav. 17.2 48.1 28.3Crime-Viol. 2.2 8.3 4.390-days Trauma 32.3 57.7 41.490-days Victimization 26.9 47.1 34.0

Analysis of Initial Assessment Data – % using community services in past 90 days

Analysis of Initial Assessment Data – % using community services in past 90 days by gender

Value of GAIN-Q3 MI for Outcome Monitoring

• In 2013/14 the project team also completed analysis of pilot outcome data

• 117 clients followed for 3 and then 6 months• Results show the value of the GAIN-Q3 for outcome

monitoring 90 day re-administration of Q3 MI ONT showed:

Decrease in substance use Positive changes in mental health, stress, physical health and risk

behaviours

Fidelity use of GAIN Q3 MI linked to better client outcomes

Ontario Perception of Care Tool for

Mental Health and Addictions (OPOC-MHA)

Brief Project HistoryMore detail at: http://eenet.ca/dtfp/client-satisfaction-project/

• Measures of client experience are widely used by customer-oriented businesses and healthcare services and settings

• Recognized as an important indicator of the quality of care as it is a direct measure of whether a client received services that met expectations and needs

Ontario Perception of Care Project

• Satisfaction a measure of the reaction to the services received • Respondents tend to report high levels of satisfaction even

though dissatisfaction might be voiced in open-ended questions or other feedback formats such as focus groups

• Measures of perception of care ask more directly about the care experience in relation to what is expected as standard practice • Range of responses likely to be wider as respondents may be

more willing to report infrequent exposure or use of a practice than express dissatisfaction about it

Satisfaction versus Perceptions of Care

• Literature review completed• Environmental scan

– All mental health and addiction agencies in Ontario using ConnexOntario database (30% response rate)

– Asked about current tools and processes to assess client perceptions of care

– Most agencies using something– Majority of tools developed in-house

• Extensive stakeholder consultation through Advisory Structure and project Working Group

Previous DTFP Iteration

• Developed new tool with support of Working Group• All tools evaluated in the literature review were assessed

according to validity, usage, length, and relevance• 8 tools were selected from this process• All items from these 8 tools were collated according to

domain for review by the Working Group and served as the foundation for the new tool

• Can be used in addictions, mental health, and concurrent disorder settings

• Translated into French

OPOC- MHA

Domain Sample Question

Access/Entry “The location of services was convenient for me”.

Services Provided “I had a good understanding of my treatment and support plan”.

Participation/Rights “I felt comfortable asking questions about my treatment and support, including medication”.

Therapists/Support Workers/Staff

“I found staff knowledgeable and competent”.

Environment “I felt safe in the facility at all times”.

Discharge/Leaving the Program

“I have a plan that will meet my needs after I leave the program”.

Overall Experience “The services I have received have helped me deal more effectively with my life’s challenges”.

Domains of the OPOC-MHA

CLIENT Version (38 items)• Registered clients of the program

– Those receiving services for their own treatment/support– Family members/significant others/supporters who are receiving

services in their own right)

Note: 6 items specific to inpatient/residential treatment services only

FAMILY/SUPPORTERS (17 items) • Family members/significant others/supporters who are not

registered clients but who are also receiving services from the program (such as parent who has a child in the program)

OPOC-MHA Versions

• Respondent’s age, gender, sexual orientation, ethnic background, and stage in the treatment process are included

• Information can be used for subgroup analyses and from an equity perspective

• Two open-ended questions to allow for comments about what the respondent found most and least helpful in their experience with the program, as well as room for comments throughout the questionnaire

OPOC-MHA Additional Questions

Data collected April 1/12 through June 30/12 1, 753 respondents Administration process varied by site

Least disruptive to program Administered at various points in time

Pilot Sites• 23 pilot sites from both mental health & addiction sectors• Sites represented a diversity of clients and services across

Ontario (i.e. inpatient/community, gender-specific, youth, Indigenous, ethno-racial and immigrant etc.)

OPOC-MHA Pilot Process

We have a tool and it works!• Overall feedback from staff and clients about the

OPOC-MHA was positive• Significant interest from mental health and addiction

agencies in the province (and elsewhere) to implement the tool

• The OPOC-MHA demonstrated strong psychometrics

OPOC-MHA Pilot Results

OPOC-MHA

Pilot Sample Data

34

OPOC-MHA: Referral to ServicesItem 11: I was referred or had access to other services when needed

(including alternative approaches)( Total 1,208 Responses )

45.1%

1.6%6.0%

47.3%

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Strongly disagree Diasagree Agree Strongly agree

35

Item 22: Overall, I found the facility welcoming, inclusive, and comfortable ( Total 1,409 Responses )

55.1%

41.4%

2.6%0.9%0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Strongly disagree Diasagree Agree Strongly agree

OPOC-MHA: Environment

36

Item 25: I felt safe in the facility at all times( Total 1,389 Responses )

62.7%

33.3%

2.8%1.2%0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Strongly disagree Diasagree Agree Strongly agree

OPOC-MHA: Feeling Safe

37

Item 31: The services I have received have helped me deal more effectively with my life’s challenges

( Total 1,288 Responses )

55.9%

39.8%

3.4%0.9%0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Strongly disagree Diasagree Agree Strongly agree

OPOC-MHA: Effectiveness

Sub-group Differences in Responses - Some Findings• Patients and supporters did not differ substantially in

their opinion on most of the statements

• Respondents from MH programs answered strongly agree less often on some statements compared to respondents from A and CD programs

• Young respondents (age <=18) appeared to answer strongly agree less often on most of the statements

Differences in Responses - Some Findings

• LGBQT respondents were less likely to endorse strongly agree on all the statements

• Respondents who have been mandated by court, medical authority, etc. appeared to answer strongly agree less frequently on all statements than respondents who voluntary participated in the program/treatment

40

• Staged Screening and Assessment Process being implemented in Ontario MoHLTC funded Addiction agencies

• OPOC-MHA being implemented in Ontario MoHLTC funded addiction, mental health and concurrent disorder programs

• Implementation beginning in about half of the LHINs, with other half beginning early 2016

Provincial Implementation

Research Based Approach to Implementation • Implementation Science (IS) is a proven approach to

bringing evidence-based research into practice to improve client outcomes

• IS means purposeful, planned and active implementation, supporting fidelity and sustainability

• Planned and supported implementation results in higher implementation with fidelity

More info on IS: http://nirn.fpg.unc.edu/learn-implementation

Reference: Greenhalgh et el. 2004.Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations. The Milbank Quarterly, Vol. 82(4), 581 – 629.

Implementation Science Supports • Implementation supports increase direct

practice change from 5% to as much as 95% • Implementation supports (i.e. fidelity monitoring

and supervision/coaching) help staff see the new evidence based practice is not ‘just another change/project/model’

• Increases staff’s ability to provide the new evidence-based practice competently & flexibly

Implementation Planning to Date

• Developed Provincial implementation plan using implementation science framework

• Fine tuning tools and processes to support implementation

• Developing infrastructure (i.e. catalyst)• Capacity building for DTFP implementation team

– Certification to trainer level of GAIN assessment tool

DTFP Implementation Team

• DTFP Implementation Team includes:– DTFP Implementation Supervisor– Implementation Coordinator (assigned to LHIN)– Implementation Coach (assigned to LHIN)– Evaluation Coordinator– Knowledge Broker– Research Analyst (OPOC-MHA)

• Dr. Brian Rush providing ongoing consultation and guidance to the team

46

The Details:What is Being Implemented

Best Practices for Screening and Assessment – Key Principles

• Structured information gathering with validated tools is vital: unstructured interviews miss co-morbidity – you have to ask!

• Staged approach saves time and resources for the longer screening and assessment

• Tools that cover both mental health and addictions enhance communication & relationship building across sectors

Stage 1 Screener: GAIN SS

If score is 2 or greater on

internalizing scale, proceed to

2nd stage screener.

If score is less than 2 on internalizing

scale, proceed to 1st stage assessment.

Stage 2 Screener:

MMS (18+)POSIT (12-17)

Stage 1 Assessment: GAIN Q3 MI ONT

(with detailed SU & tx. history)

Staged Screening & Assessment Process

• Available in English and French• Will be accessible via Catalyst/DATIS

– Ease of administration– Supports development of clinical profile/system level data

All Tools

• Studied and used in a number of different settings including Canada

• Valid and reliable down to 10 years of age – Canadian validation with adults – recommend for age 12 and up

• Cost: $100 agency licensing fee for 5 years unlimited use • Self- or clinician-administered (via GAIN ABS or paper

and pencil) • Reported to take 5 - 10 minutes to complete • Pilot used the CAMH-modified version – 7 additional

questions

Stage 1 Screening: GAIN-SS

• Validated in public sector settings in the U.S.• No cost• 22 items divided into 3 sections to capture the

three major categories of mental illness (mood, anxiety and psychotic disorders)

• Paper and pen: self-/clinician- administered• Estimated 15 minutes to complete

Stage 2 Screening (Adults): Modified Mini Screen (MMS)

• Valid and reliable• Designed to identify problems and potential

treatment/service needs in 10 areas including substance abuse, mental and physical health and social relations

• Estimated 20-30 minutes to administer, 2-5 minutes to score

• Administered by self/clinician and with paper/computer

• For use with clients aged 12 - 17

Stage 2 Screening (Youth): Problem Oriented Screening Instrument for Teenagers (POSIT)

• Developed by Chestnut Health Systems in Illinois• Good psychometric properties• One of main instruments in the GAIN family of

assessments• Ontario version was developed to increase the tool’s

relevance to the provincial context – Incorporated items around trauma and barriers– Cross-walk with ADAT; mapped to strengths and needs

criteria

Stage 1 Assessment: GAIN-Q3 MI(Ontario Version)

• Multi-purpose tool that identifies a wide range of life problems

• For use among adolescents and adults in both clinical and general populations

• For use in diverse settings• Established with strong focus on subsequent outcome

monitoring

Stage 1 Assessment: GAIN-Q3 MI(Ontario Version)

• School Problems• Work Problems• Physical Health • Sources of Stress• Mental Health

Content of GAIN-Q3 MI(Ontario Version)

• Risk Behaviours for Infectious Diseases

• Substance Use• Crime and Violence• Life Satisfaction

Basic Domains Covered:

• Each domain includes motivational-based questions specific to readiness for change and reasons for changing behaviour

• Estimated 45 minutes to complete face-to-face administration Computer/ABS (accessed via Catalyst) paper-and-pencil

• Self administration in a group format being used regularly at one of the pilot sites (quite successfully)

On the GAIN-Q3 MI (Ontario Version)

• Supports system-level client profile mentioned previously• A number of reports can be generated to support clinical

decision making and referral process– Individual Clinical Profile (ICP)– Personalized Feedback Report (PFR)– Recommendation Referral Summary (RRS)– Validity Report

On the GAIN-Q3 MI (Ontario Version)

Examined Complementarities of GAIN-Q3 MI ONT with other Tools

• Crosswalk between GAIN-Q3 and OCAN, RAI and LOCUS was conducted to explore relevant contributions of the GAIN-Q3

• Results: • Tools can complement each other• Only the GAIN Q3 provides the level of information

needed for substance use assessment and treatment planning

Embedded within the GAIN Q3 MI ONT:• Provide greater depth of information regarding

substance use and treatment history• Supports referral and/or treatment planning for clients

with more complex issues• Clinician may administer in same session as Q3 MI ONT

or subsequently if more information deemed necessary

Substance Use Grids

In Summary, New Staged Process……• Includes screeners:

That are quick and easy to administer Provide comprehensive look at relevant psychosocial

domains Facilitate referrals for mental health services

• Includes assessment that: Is comprehensive and standardized Provides concrete recommendations for treatment planning

and referral Includes reasons & readiness to change questions that are

highly valued Provides a strong foundation for outcome determination

Value Add at Multiple LevelsClinical:

• Assessment a process that should occur over time• Assessment needs to be motivationally oriented and

comprehensive across bio-psycho-social and spiritual domains of health

• Assessment alone can contribute to outcomes• GAIN assessment fidelity is linked to clinical outcomes

• Assessment plus evidence-informed interventions improves outcomes

Value Add at Multiple Levels

Organizational:• Analysis/summary of initial assessment data• Organizational clinical profile informs program

planning and quality improvement• Electronic administration and printable validity reports

(that notes administration inconsistencies) can support performance monitoring and coaching

Value Add at Multiple LevelsSystem:• Structured data:

• Basic information on demographic characteristics of clients that can be compared to the community profile to identify under-served populations

• Development of clinical/severity-based profiles of people in treatment (i.e. to confirm that those with the most severe profiles are being served in the most cost-intensive services)

• Provide a baseline for follow-up outcome assessment, which also has value at the individual level

• Tools with sound psychometrics and track record• Comprehensive assessment and resulting reports

facilitate better treatment planning• Higher concurrent disorder capacity• High level of detail on substance use and overall

severity: holistic view of client need• Fully integrated process incl. report generation• Agency clinical and psychosocial profile = informed

planning

Clinical & Agency Value-Add Summary

• Placement matching: Initial client assignment to a treatment setting with appropriate resource intensity (important cost implications and better use of existing resources)

• Detailed client profiles for community gap analysis, project planning (i.e. reducing ER and hospital use) and performance monitoring

• High potential for outcome monitoring • Comparable data with other LHIN’s for potential

benchmarking

LHIN/System Value-Add Summary

OPOC-MHAImplementation

Objective:• Systematically implement OPOC-MHA across all

MoHLTC funded substance use, concurrent disorder and mental health services

Projected Outcomes:• Standardized information regarding client

satisfaction/perception of care• Enhanced quality improvement and accountability

processes at both service and system levels across Ontario

OPOC-MHA Implementation

• Details of tool administration can and will vary by agency• Each agency and/or program can determine how and

when the tool should be administered• Four key requirements:

– Provide entire questionnaire – Ensure anonymity– Ensure completion is voluntary– Provide facilitation as needed

OPOC-MHA Administration Essentials

• Variety of ways the OPOC-MHA can be administered. No prescribed way, as this depends on the practices of each agency and/or program

• Distributed to clients in many ways, i.e. group setting, individually, in person at the agency or through email/mail

• Current administration via paper and pencil or electronically through Fluid Survey database– Data will be collected and analyzed centrally

• Future administration through DATIS/Catalyst (for both MH and A)

OPOC-MHA Data Collection

Timing• At any point in the person’s treatment process

One question asks which part of the treatment process a participant is currently engaged

Scoring• Likert scale

strongly agree, agree, disagree strongly disagree, N/A

OPOC-MHA Administration

Scales # of Items Items Scoring

Overall Perception of Care 23 1-8, 12-15, 17-18, 20-25, 30-32

Average score of the 23 items

Accessing Services 6 1-6Average score of the 6 items

Within Services 17 7-8, 12-15, 17-18, 20-25, 30-32Average score of the 17 items

Analyzing OPOC-MHA Data• Analysis & interpretation of OPOC-MHA data may involve individual

or grouped items• Responses to each item may be reported as % OR averages • “Overall Perception of Care” score

– Plus subscale scores for “Accessing Services” & “Within Services”

• OPOC-MHA designed to capture information on quality improvement indicators such as:– safety, accessibility, client-centredness, equity,

integration, effectiveness, and appropriate use of resources

• OPOC-MHA identified by Accreditation Canada as an instrument approved for use for assessing client satisfaction/perception of care for accreditation purposes

OPOC-MHA and Quality Improvement

Implementation Next Steps

• LHIN Engagement - Scope of the work– # of agencies– key contacts (A and MH networks?)– current process for system access etc.

• Development of LHIN-specific implementation plans– Using Implementation Science– Connected to overall Provincial plan– Contextualized to the LHIN

• Development and implementation of LHIN and agency supports

Note on Coordinated/Central Access

• Staged process lends itself more to coordinated access models than previous ADAT assessment process

– who does what component can vary but the package is well-developed and supported

• LHIN-level implementation plans will be developed for the specific community context

– Supported and tailored approach– Implementation will work within access model to

determine how the tools best fit

Key Milestones and Timelines

Milestone Anticipated TimelineEngage with early adopter LHINs May/June 2015Assemble DTFP Advisory Committee and SS&A/OPOC Working Group

June 2015

Develop LHIN Specific Implementation Plans

July 2015

LHIN Implementation Team Development

August 2015

Develop Agency Implementation Plans (incl. coaching support)

August/September 2015

Key Milestones and TimelinesMilestone Anticipated TimelineTraining and Capacity Building Begins September/October

2015

Initial Implementation begins in Early Adopter LHINs

October 2015

Track implementation, monitor progress, engage in developmental evaluation

October 2015 – April 2016

Engage with remaining LHINs February/March 2016Full Implementation in Early Adopter LHINs

May – December 2016

78

Training: Staged Screening & Assessment

• Mixed modalities• Web based training on some elements

• Introduction to the process• GAIN – SS• MMS• POSIT

• Face to face training• GAIN Q3 MI ONT• Motivational Interviewing• Implementation

79

SS & A Training Plan Overview

Chestnut Health Systems

Trains DTFP Team to Trainer Certification Level on GAIN Q3 MI ONT

Trains Team of LHIN Based Trainers to

Trainer Certification Level of GAIN Q3 MI

ONT

DTFP Team

Trains early adopter/champion

agencies in each LHIN on entire SS&A process,

including administration certification on GAIN Q3

MI ONT

Trains Team of LHIN Based Trainers on all other

elements of the SS&A process

LHIN Based Trainers

Trains all other implementing

agencies on the entire process

(with DTFP Team support) and

provides ongoing training for

sustainability

80

OPOC-MHA Capacity Building

• Web based orientation and training• Understand the tool• Administration details• Data gathering• Quality Improvement implications

Kim Baker Brian Rush DTFP Implementation Supervisor Project Consultant Kim.Baker@camh.ca Brian.Rush@camh.ca

Linda Sibley Donna RogersAddiction Services of Thames Valley FourCastlsibley@adstv.on.ca drogers@fourcast.ca