keeping it fresh: 9 years of ansa implementation
TRANSCRIPT
Presentation Overview1. How to adapt the ANSA over time to
reflect the needs and strengths of the clients served.
2. How to conduct a multi-year analysis to inform an internal CQI process.
3. How to keep the ANSA fresh and relevant in the minds of clinicians after they have been using it for longer periods of time.
2
Introduction: Jewish Child and Family Services (JCFS)
» Mission: To provide help, healing, and caring services infused with Jewish values to strengthen lives in our community
» 6 Primary Program Areas1. Counseling and Support
2. Support for Communities
3. Education
4. Child Welfare
5. Services for People with Disabilities
6. Employment Services
4
History of JCFS CANS, ANSA, and FAST Implementation
5
DCFS requires use of CANS in System of
Care (SOC) now known as Intensive
Placement Stabilization (IPS)
DCFS expands CANS to Specialized Foster
Care/Residential
JCFS adopts CANS for all Foster Care
contracts
2009 - Outpatient Counseling chooses
ANSA
2010 - Maintaining Adoption
Connections (MAC) chooses FAST
JCFS chooses CANS with FAST for
Outpatient Counseling
MAC shifts to CANS and FAST
ANSA Revised
FAST added to ANSA
JCFS Therapeutic Day School implements TDS CANS
Comprehensive
2003 2005 2009-2010 20132013 2017
*DCFS=Department of Children and Family Services*CANS=Child and Adolescent Needs and Strengths*ANSA=Adult Needs and Strengths Assessment*FAST=Family Advocacy and Support Tool
Current Tools in Use at JCFS
» JCFS CANS Comprehensive• Based on National Child Traumatic Stress Network CANS
• Used in Outpatient Counseling and Maintaining Adoption Connections
• FAST integrated into tool
» ANSA 2.0• Based on the Indiana ANSA
• Used in Outpatient Counseling
• FAST integrated into tool
» DCFS CANS 2.0• Used in Foster Care and Intensive Placement Stabilization
» TDS CANS Comprehensive• Used in Therapeutic Day School 6
JCFS Outpatient Counseling Program
» Services: Individual, Couples, Family, and Group Counseling
» Top 3 Primary Treatment Approaches:
• Psychodynamic Therapy, Family Systems Therapy, Cognitive Behavioral Therapy
» Serve ~1,100+ clients yearly
8
Program Goal: Increase, strengthen, and promote adaptive functioning and resilience within adults, children and families
All Clients
» Median length of service: 6.4 months
» Majority: female, Caucasian, Jewish, English-speaking, low-income
» Median age at admission: 26 ranging from <1 year to 90+ years old
» Majority are adult clients (58%)
» Top 5 presenting problems: Depression, Anxiety, Adjustment, Prevention/Support, Relationship Problems (Family)
9
JCFS Outpatient Counseling Program Client Characteristics
Adult Clients (19 and Older)
» Median length of service: 6.6 months
» Largest percentages (33-34%) identify as single or married; 20% divorced
» Median age at admission: 46. Largest percentage (31%) 40 to 54followed by 25 to 39 (27%)
All Clients
Adult Clients
Selecting the ANSA in 2009
» Agency merged and moved from a child-serving program to a program serving all ages
» Updated the logic model unifying the program as one
» Embarked on a search for an adult assessment tool
• Using CAFAS, CBCL, YSR for children, wanted an assessment tool for adults
• Sub-committee looked at potential adult assessments for outcomes
» ANSA preferred choice
» Implemented using the Indiana model
10
» Employment
» Developmental Needs
» Trauma • Traumatic Experiences
• Trauma Adjustment
» Substance Use
» Violence• Historical Risk Factors
• Emotional/Behavioral Risks
• Resiliency Factors
» Sexual Aggression
» Criminal Behavior
» Caregiver Strengths and Needs
11
ANSA 1.0
» Strengths
» Life Domain Functioning
» Acculturation
» Behavioral Health Needs
» Risk Behaviors
Core Domains Specialty Individual Domains Caregiver Specialty Domain
How was the ANSA revised in 2013?
» Clinicians volunteered to serve on a short-term assessment committee
» Reviewed the internal ANSA Item Level Analysis Report• Which items to eliminate?
• Which areas were not reflected on the tool?
• What items were not clearly defined by the manual?
• Reviewed changes made by Indiana and their work on Recovery
12
What did we decide to change?
» Clear gaps:
• Did not have any items on parenting
• Did not have any items focused on family needs
» 2013 Modification – Key changes
• Improved definitions
• FAST integrated into assessment (Family Together, Adult, and Child Functioning domains)
• Added Caregiver Role Functioning item with Caregiver Role specialty module
• Changed Acculturation domain to optional specialty module
• Added Suicide Risk as a specialty module
• Moved Emotional/Behavioral Risk items out of the optional specialty Violence module into the Core Risk Behaviors module
13
» Caregiver/Parenting Role
» Employment
» Developmental Needs
» Cultural Identity and Acculturation
» Trauma • Traumatic Experiences
• Traumatic Stress Symptoms
» Substance Use
» Suicide Risk
» Dangerousness
» Sexual Aggression
» Criminal Behavior
» Family Assessment• Family Together
• Adult Functioning
• Child Functioning
» Caregiver Strengths and Needs
14
ANSA 2.0 Today
» Strengths
» Life Domain Functioning
» Behavioral Health Needs
» Risk Behaviors
Core Domains Specialty Individual Domains Family and Caregiver Specialty Domains
ANSA Implementation Today
» In-person reliability training
» Completed at Intake, 6 Month, 1 Year, Annually from Admission, Discharge
» Entered into client information system
» Client level reporting on demand
» Program level reporting
16
ANSA Individual Summary Report Example
• Intake Assessment• 5 Identified Strengths to build on
• 3 Actionable Individual Needs
• 2 Background Trauma Experiences Needs
• 2 Actionable Family Needs
• 6 Usable Strengths
17
ANSA Individual Summary Report Example
• Discharge Assessment• 1 Strength to build on
• 2 Background Trauma Experiences Needs
• No Individual Actionable Needs
• No Actionable Family Needs
• 10 Usable Strengths
18
ANSA Individual Summary Report Example
• Change in Domain Scores Over Time• Reflects Domain Scores for all
Assessments
• Reliable Improvement in:• Life Domain Functioning
• Strengths
• Acculturation
• Family Together
• Career/Employment
19
ANSA for Continuous Quality Improvement
» Twice yearly reporting
» Sample based on the fiscal year
» Calculate yearly Reliable Change Index
» Key Outcome: Reliable improvement in any applicable domain
20
9 Year ANSA Analysis –Methodology
» Data cleaning
» Calculate Reliable Change Index
» Determine baseline – highest % actionable
» Refine sample for admission cohort analysis
22
9 Year ANSA Analysis –Data Cleaning
» Started with 9,617 Assessments
» Included only clients admitted 2009 onward (7,797 assessments)
» Included only assessments marked FINAL
» Excluded any data entry errors
• Incorrect program selected
• Assessments dated prior to admission date
» Deleted multiple assessments completed recorded for the same date – kept the most recent based on “Data Entry Date” in the system
23
9 Year ANSA Analysis –Data Cleaning
» Recoded for rating periods
• Intake: Any ANSA completed within the first 90 days
• 6-Month: Any ANSA completed >90 days and less than 9 months
• 1 year: 9 months to 1.49 years
• 2 Years and Onward: [year].50 round up
• Discharge: 31 days from discharge or last date of service
» Deleted multiple assessments based on recoded rating period
» 7,557 ANSAs for 2,837 client episodes
24
9 Year ANSA Analysis –Reliable Change Index (RCI)
» What is it?• The amount of change necessary in the domain score above and beyond
measurement error (reliable change). Jacobson & Truax 1991
» How to calculate:• Calculate a domain score
• Calculate the standard deviation for each domain
• Calculate the average reliability score of all ANSA raters
• Plug your standard deviation and average reliability score into this formula:
25
FormulaRCI=1.28*(standard deviation) x SQRT (1-reliability)
9 Year ANSA Analysis –Domain Score
» How to calculate:
• Add all scores on items in a domain to create a sum. Exclude N/A values.
• Divide by the total items completed for the scale, excluding missing and N/A responses.
• 75% of a domain must be rated to calculate a score.
• Multiply the score by 10 to produce uniform 30 point domain score where:
• 0=‘0’ ratings on all items
• 30=all ‘3’ ratings on every item in the domain.
26
9 Year ANSA Analysis –Reliable Change Index (RCI) –Core Domains
Domain N Mean Minimum MaximumStandard Deviation
RCI
Strengths 2,494 12.27 0.00 30.00 6.05 3.46
Life Domain Functioning 2,483 5.21 0.00 23.33 3.29 1.88
Behavioral Health Needs 2,468 4.75 0.00 21.00 3.05 1.75
Risk Behaviors Pre-2013 2,460 0.54 0.00 13.75 1.29 0.74
Risk Behaviors Post-2013 1,345 1.10 0.00 14.62 1.74 0.99
27
9 Year ANSA Analysis –Reliable Change Index (RCI) –Specialty
Domain N Mean Minimum MaximumStandard Deviation
RCI
Family Together 449 10.89 0.00 30.00 4.87 2.79
Identity and Acculturation 1162 1.46 0.00 26.67 3.32 1.90
Traumatic Stress Symptoms 454 7.61 0.00 27.50 5.07 2.90
Employment 343 9.07 0.00 30.00 7.51 4.30
Caregiver Needs and Strengths 120 3.18 0.00 18.33 4.17 2.39
Caregiving/ Parenting Role 165 9.22 0.00 25.00 5.74 3.28
Substance Use 187 11.49 0.00 30.00 8.39 4.80
Suicide Risk 171 5.00 0.00 14.00 3.20 1.83
28
9 Year ANSA Analysis –Baseline Swap
» Why do we swap baselines?
» Baseline Swap Inclusion Criteria• 3 or more ANSAs completed
• Intake and 6 month ANSAs required
» 38% of episodes fit the criteria (1,092 of 2,837 episodes)
» Total % actionable (# items rated 2 or 3 divided by # rated) increased in 34% of eligible cases (372 of 1,092)
29
9 Year ANSA Analysis –Baseline Swap
After swapping baselines…
1 in 4 clients demonstrated:• 15 increased needs
• 8 decreased strengths
Why the change?• Overestimating client strengths
• Discovery
30
» Final Sample: 1,426 client episodes for clients admitted 2009-2016
» Excluded 2017 and 2018 admission cohorts (<75% have discharged)
» Included only clients with a baseline and discharge ANSA for comparison
31
9 Year ANSA Analysis –Refine Sample for Analysis
AdmissionYear
# Client Episodes
2009 141
2010 185
2011 195
2012 196
2013 215
2014 168
2015 190
2016 136
ANSA Analysis –Reliable Improvement in Any Domain
68%65% 65%
70%
61%60%
52%
57%
2009 2010 2011 2012 2013 2014 2015 2016
33
ANSA Analysis –Reliable Improvement in Any Domain
68%65% 65%
70%
61% 60%
52%
57%
45%
49%53%
49% 49%52%
36%
46%
2009 2010 2011 2012 2013 2014 2015 2016
34
Improve in Any Domain
Improve without Decline
ANSA Analysis –Reliable Improvement vs. Decrease % Actionable
68%65% 65%
70%
61%
60%
52%
57%61% 60%
64%
68%67%
62%
57%
63%
2009 2010 2011 2012 2013 2014 2015 2016
35
Improve in Any Domain
Decrease Total Percentage Actionable
ANSA Analysis –Admission Cohort vs. Fiscal Year
68%65% 65%
70%
61%60%
52%
57%
2009 2010 2011 2012 2013 2014 2015 2016
36
Fiscal Year
ANSA Analysis –Reliable Improvement by Core Domain
Strengths
Life Domain Functioning
Behavioral Health
15%
20%
25%
30%
35%
40%
45%
2009 2010 2011 2012 2013 2014 2015 2016
37
ANSA Analysis –Reliable Improvement –Risk Behaviors
0%
5%
10%
15%
20%
25%
2009 2010 2011 2012 2013 2014 2015 2016
38
Pre-2013 Items
Post-2013 Items
*In 2013 Frustration Management, Hostility, Paranoid Thinking, Secondary Gains from Anger, and Violent Thinking moved from the Violence Emotional/Behavioral Risks Specialty Domain to Core Risk Behaviors Domain
ANSA Analysis –Reliable Improvement –Family and Parenting
0%
5%
10%
15%
20%
25%
30%
35%
2013 2014 2015 2016
39
Family Together
Caregiver/Parenting Role
ANSA Analysis – Significant Domain Score Change
0.9
0.4
4.3
4.8
10.8
1.5
0.5
4.9
5.4
12.7
Acculturation
Risk Behaviors
Mental Health
Life Domain Functioning
Strengths
40
BaselineDischarge
p<.05
Increased Strengths
» Family
» Social Connectedness
» Optimism
» Educational
» Job History
» Talents/Interests
» Spiritual/Religious
» Community
» Natural Supports
» Resiliency
» Resourcefulness
» Volunteering41
ANSA Analysis – Significant Item Level Changes
Decreased Needs
Life Domain Functioning
» Family
» Caregiver Functioning
» Employment
» Social Functioning
» Recreational
» Residential Stability
» Legal
» Sleep
» Decision-making
» Cultural Identity
» Depression
p<.05
Decreased Needs
Behavioral Health
» Anxiety
» Interpersonal
» Adjustment to Trauma
» Anger Control
» Eating Disturbance
Risk Behaviors
» Suicide Risk
Specialty Domains
» Job Time
» Cultural Rituals
» Cultural Stress
» Affect Regulation 42
ANSA Analysis – Significant Item Level Changes
Increased Needs
Life Domain Functioning
» Involvement in Treatment
Specialty Domains
» Ability to Listen
» Suicide Intent
p<.05
ANSA Analysis – Improvement in Top Diagnostic Categories
57%
61%
65%
62%
22%
21%
14%
23%
21%
18%
21%
15%
Other Conditions – Relational
Anxiety Disorders
Depressive Disorders
Trauma- and Stressor-Related
43
Improvement 1+ Domain No Change Decline
(n=393)
(n=344)
(n=207)
(n=157)
ANSA Analysis – Significant Differences –Length of Stay
76%
70%
71%
70%
66%
51%
13%
10%
15%
15%
19%
25%
12%
20%
14%
15%
15%
24%
4+ Years
3 Year
2 Year
1 Year
6 Months to 1 Year
<6 Months
44
Improvement 1+ Domain No Change Decline
p<.05
ANSA Analysis – Significant Differences –Counseling Modalities
67%
69%
69%
60%
38%
10%
14%
21%
44%
33%
20%
16%
19%
19%
4
3
2
1
0
45
Improvement 1+ Domain No Change Decline
p<.05*Modalities include: Individual, Couples, Family, and Group*0 Modalities=Case Management/Assessment Only
ANSA Analysis – Significant Differences –Couples
41%
62%
41%
18%
17%
21%
Couples Only
Couples + Other
46
Improvement 1+ Domain No Change Decline
(n=141)
(n=29)
p<.05
ANSA Analysis – Significant Differences –Family
33%
70%
33%
12%
33%
18%
Family Only
Family + Other
47
Improvement 1+ Domain No Change Decline
(n=264)
(n=24)
p<.05
ANSA Analysis – Improvement based on Failed Session Percentage
54%
64%
68%
64%
20%
19%
17%
20%
26%
17%
15%
16%
>32%
18% to 32%
8% to 18%
<8%
48
Improvement 1+ Domain No Change Decline
(n=358)
(n=356)
(n=348)
(n=356)
p<.05
ANSA Analysis –Other Significant Findings
Clients who improved:
» Higher total % actionable at baseline
• Life Domain Functioning, Behavioral Health, Risk Behaviors (pre-2013 scale), Acculturation, Trauma Experiences
» Lower % usable strengths at baseline
» Discharged due to “Planned Service Completion”
49p<.05
ANSA Analysis –What happened in 2015 and 2016?
68%65% 65%
70%
61%60%
52%
57%
2009 2010 2011 2012 2013 2014 2015 2016
50
Actionable NeedsUsable Strengths Length of Stay Sessions
Potential Action Areas for Counseling CQI
» What clinical groups can we offer to supplement other forms of counseling?
» How do we encourage family/partner involvement in treatment?
» How can we increase client engagement in services?
» What does treatment planning look like for clients with fewer needs and more strengths at intake?
» How can we adapt existing strengths to address needs?
» Is it time to revise the ANSA again?
52
Keeping it Fresh
Final Thoughts/Suggestions
1. Convene/engage a subcommittee of program staff to revise the assessment to reflect the needs and strengths of the client population
2. Provide real-time reports for staff to monitor improvement and share with clients
3. Provide booster trainings and additional opportunities for staff to gather and share ideas of how to use these assessments in practice
4. Conduct regular analysis to share significant findings with staff to inform treatment on a program level
53
Melissa Villegas
www.jcfs.org
Thank you!
Questions?
Emily Shapiro