The Wraparound Team Monitoring System (Wrap-TMS): Feasibility, Utility, and Initial Psychometrics of a Wraparound-specific Health Information Technology System
Eric J. Bruns, Ph.D., April Sather, MPH, and Hattie Quick, MSW University of Washington School of Medicine, Seattle, WA
Kelly L. Hyde, Ph.D. Social TecKnowledgy, Inc. 26th Annual Research and Policy Conference on Children’s Mental Health Tampa, FL March 4, 2013
1
Funding provided by:
National Institute for Mental Health (MH095516)
Maryland Department of Social and Health Services / Institute for Innovation and Implementation at the University of Maryland School of Social Work
Center for Medicaid and Medicare Service
Thanks to Vista del Mar, Three Rivers Wraparound, Sacramento Children’s Home, Kentucky Impact, Families and Children First of Clermont County, and Maryland Choices for participating in Pilot testing
Overview of presentation
• Present a rationale for and background on the Wraparound Team Monitoring System
• Provide a quick overview of the system
• Present results of an initial pilot of feasibility and psychometrics of Wrap-TMS scales
Why did we develop Wrap-TMS?
• Research shows that a large portion of variance in treatment outcomes are accounted for by:
• clarity around goals
• monitoring and feedback of data on progress, satisfaction, and alliance
• Outcomes of collaborative care are more positive when decisions are informed by reliable & relevant information
Why Wrap-TMS?
• States, initiatives, and providers have often struggled to find ways to manage data and information in a way that is in line with wraparound principles
• Fidelity and process data collected externally doesn’t help individual teams keep on track
• Outcomes / costs may not be monitored at all
• We need ways to facilitate achievement of wraparound principles such as outcomes-based and collaborative
Why did we develop Wrap-TMS?
• Prominent wraparound initiatives have developed excellent MIS systems that are critical to their success
• However, these are locally specific and don’t always translate easily to new sites/states
• NWI aims to provide wraparound sites with a relevant, field-tested, model adherent IT system AND needed supports: • Customization, training, help-desk, ongoing research and reliability testing,
interoperability, Electronic Health Record
Wrap-TMS provides a way to manage
and access key information on the
wraparound process • Individuals engaged in the process
– Youth and family members, team members, providers, natural and community supports, coordination of care
• Key documentation
– Plans of care, strengths, needs, family stories, family history timeline, meeting and appt. times, meeting notes, contact histories, critical incidents, services and costs
• Service processes
– Family Satisfaction, Fidelity, Progress toward needs
• Outcomes
– youth and family support, residential status, educational environment and behavior, youth functioning
Wrap-TMS is intended to facilitate more
effective implementation of wraparound…
at a family, team, and initiative level
• At a youth/family and team level:
– Aid communication between team members
– Allow transparency in information sharing
– Assist in documentation compliance
– Help keep teams on track:
• What are the priority needs? Are we making progress?
• What is the family’s vision for the future? Are we getting there?
• Is the process working for the family and team?
Wrap-TMS is intended to facilitate more
effective implementation of wraparound…
at a family, team, and initiative level
• At higher levels:
– Supports data-driven supervision
– Facilitates evaluation
• Of staff performance
• Of initiative process and outcomes
– Allows managers and administrators (at program, initiative, county, state levels) to monitor implementation, outcomes, service use, and costs in “real time”
Wrap-TMS Data elements are based on input
about what will be most critical to decision
making in team-based integrated care
• Youth, family, and team information
• Plans of care and crisis plans
• Progress notes and incident reports
• Core outcomes such as:
– Family Support and Connectedness
– Progress Towards Needs and Family Vision
– Core Community Outcomes (residential placement, school achievement and suspensions, involvement with JJ, emotional and behavioral functioning)
– Family and Team Perspectives (Fidelity/Satisfaction)
Home Page – Dashboard Staff
Care Coordinator/Facilitator
Specific Dashboards
Customizable Dashboard Views
Home Page –Dashboard
Supervisor
Supervisor Specific Dashboards
Customizable User
Workflow
Customizable Dashboard Options
Add New Case Record
Draft: Replication of slides not permitted without consent from Social TecKnowledgy
12
User identified Required Data Elements
Waiver Funding Data
Add New Demographic Data
Fields
Adding a New Demographic Data Element
Draft: Replication of slides not permitted without consent from Social TecKnowledgy
13
Add/Edit Core Data Assessments
Draft: Replication of slides not permitted without consent from Social TecKnowledgy
14
Core Assessment History
Review/Update all Core Assessments
Add/Edit Customized and or
Required Assessments
Draft: Replication of slides not permitted without consent from Social TecKnowledgy
15
Users can build their own assessments
Other Behavioral Health Assessments available for use
Youth Dashboard with Work Flow
User Specific Dashboards
Customizable User Workflow
Youth Space
Replication of slides not permitted without consent from Social TecKnowledgy
17
Plan of Care:
Available to auto-sign and email
Replication of slides not permitted without consent from Social TecKnowledgy
Embedded Progress and Contact
Notes with Exportable History
Replication of slides not permitted without consent from Social TecKnowledgy
19
Incident Report Form
Replication of slides not permitted without consent from Social TecKnowledgy
20
Multi-Filter Option for Export of Data by
Graphical Reports and or Spreadsheets
Draft: Replication of slides not permitted without consent from Social TecKnowledgy
21
Choice to de-identify all data for export
User Roles-Core set with flexibility
to add roles as needed
Draft: Replication of slides not permitted without consent from Social TecKnowledgy
22
Permissions Set by User Role
Draft: Replication of slides not permitted without consent from Social TecKnowledgy
23
Results of Initial Pilot
Aims of the initial pilot test
• Program Wrap-TMS items into a web-based user input prototype;
• Conduct usability testing with a diverse group of users nationwide:
• Assess psychometrics of Wrap-TMS items and scales:
• Goal setting and progress monitoring
• Fidelity and satisfaction
• Outcomes (school, residential, and community functioning)
• Youth and family social support
• Pilot test and evaluate the utility of the “team dashboard” reporting function of Wrap-TMS with real-world programs serving youth and families participating in actual WSM teams;
Assessments Completed
Assessments
N Youth
Entered
Community
Outcomes
Family
Satisfact
ion
Family
Support
School
Outcomes
Youth
Needs
Youth
Support
Custom
or Other
Provider 1 5 0 0 2 0 3 1 0
Provider 2 5 0 0 3 1 3 3 2
Provider 3 5 1 4 3 2 3 0 7
Provider 4 18 6 2 9 7 10 10 3
Provider 5 5 0 0 2 4 3 2 0
Provider 6 10 11 9 1 3 10 2 0
Total 48 18 15 20 17 32 18 12
18 users entered 48 youth and 132 assessments
Pilot testers
• 12 Facilitators and 6 supervisors
• Mean years experience with wraparound = 4.61 (SD = 1.24)
• 88% had over 1 year experience
• 45% over 3 years.
Results of usability testing
0% 20% 40% 60% 80% 100%
Youth/family/team data entry easy
Enter data on progress easy
Enter information about plan of care easy
Enter data about process/satisfaction easy
Overall high usability
Would be a benefit to our wrap program
Percent strongly agree
Community Outcomes Assessment Based on Strengths and Difficulties Questionnaire (SDQ)
• 18 assessments completed
• Cronbach’s Alpha = .771
Two sections: 1) Rate the degree to which
difficulties interfere with life a) Scales range from 0 to 3 b) “Overall Functioning”
score calculated 2) Enter number and type of juvenile justice contacts
Community Outcomes Assessment
0
2
4
6
8
10
12
14
16
18
Classroom Learning Friendships Home Life Leisure/Fun
Do the youth's difficulties interfere with...
0 - Not at all
1 - A Little
2 - A Medium amount
3 - A Great deal
Mean = 2.07 Mean = 2.21 Mean = 2.39 Mean = 2.06
Community Outcomes Assessment
0
2
4
6
8
10
12
14
16
18
0 - No 1 - Yes, minor difficulties 2 - Yes, definite difficulties 3 - Yes, severe difficulties
Overall Functioning
Family Satisfaction Assessment
• Scales range from 0 “Very Dissatisfied” to 3 “Very Satisfied”
Items Mean (SD)
Service Satisfaction 2.00 (.62)
Progress Satisfaction 1.62 (.96)
Family Support Assessment
• Rate the degree to which family receives support. Scale ranges from 0 to 3
• 19 assessments completed
• Cronbach’s Alpha = .529.
Scale: 0 = Significant need for support, 1 = Need for support 2 = Strength, 3 = Significant strength
Family Support Assessment
Items Mean (SD) Alpha if Item Deleted
Community 1.12 (.49) .552
Family 1.26 (1.00) .467
Friends 1.12 (.60) .546
School/Work 1.42 (1.07) .265
Service 1.68 (.82) .425
Scale: 0 = Significant need for support, 1 = Need for support 2 = Strength, 3 = Significant strength
Family Support Assessment
0
2
4
6
8
10
12
14
16
18
Community Family Friends School/Work Service
0 "Significant Need for for Support"
1 "Need for Support"
2 "Strength"
3 "Significant Strength"
Youth Support Assessment
• Rate degree to which youth receives support. Scale ranges from 0 to 3
• 16 assessments completed
• Cronbach’s alpha = .323
Youth Support Assessment
0
2
4
6
8
10
12
14
16
Community Family Friends School/Work Service
0 "Significant Need for for Support"
1 "Need for Support"
2 "Strength"
3 "Significant Strength"
School Outcomes Assessment
• Tracks percent of school days attended
• 14 assessments completed
School Outcomes assessment
% Days Attendance 87.83 (26.72)
Current Success 1.50 (1.16)
Suspensions 1
Expulsions 0
Other 0
Youth Needs Assessment
• The user enters the specific needs of the youth/family, sorts them into 14 categories, and provides ratings of progress towards meeting those need
• 22 assessments completed
Youth Needs Assessment Needs category: %Yes
Behavior 63.3
Comp Dev 4.5
Culture 4.5
Emotional 63.6
Family 63.6
Finances 9.1
Health 13.6
Legal 4.5
Logistical 4.5
Place to Live 13.6
Safety 13.6
School 59.1
Social Relation 45.5
Spiritual 4.5
Work 9.1
Other 0
Youth Needs Assessment
0
10
20
30
40
50
60
70
80
90
100
Pe
rce
nt
Sele
cte
d
Needs Categories
Dashboards - Youth
Youth-Specific Dashboards
Dashboard Details
Draft: Replication of slides not permitted without consent from Accountability
Solutions, Inc. 43
Item-Level Correlations
Wrap-TMS Items/Scales Pearson r
Interference with Friendships – Youth Support, Friends -.837*
Interference with Home life – Family Support, Service .878*
Youth Support, Family – Family Support, Family .947**
Youth Support, Community – Family Support, Community .853**
Community Outcomes, Overall Functioning - % School Attendance -.524
Community Outcomes, Overall Functioning – Family Support, Friends -.664
Community Outcomes, Overall Functioning – Youth Support, Friends -.575
Community Outcomes, Learning – Current School Success -.680
Community Outcomes, Learning – Family Support, Community -.768
Community Outcomes, Learning – Family Support, School/work -.625
Community Outcomes, Learning – Youth Support, Service .636
Family Satisfaction, Services – Current School Success -.612*
Family Satisfaction, Services – Family Support, School/Work -.866**
Support for construct validity
• Ran 42 correlations between Wrap-TMS scales or items
• Of the 42 pairs, there were 24 meaningful correlations, with 22 in the hypothesized direction
• Although this exercise is exploratory and highly influenced by small sample sizes, we interpret it as initial evidence for the construct validity of our assessment items and scales.
Pilot of Wrap-TMS Conclusions
• Users overall found system to have high usability and promise for improving efficiency and outcomes
• Areas for improvement:
• Entry of data on Plan of care elements
• Fewer mouse clicks per operation
• Calendar system
• Youth and family support connections assessment
• Psychometrics of Wrap-TMS assessments • Items showed good variability
• Most scales showed good internal consistency
• Exception: Youth and family support connections
• Scales largely correlated in hypothesized directions, lending initial support to construct validity
Next steps • Revise Wrap-TMS to respond to concerns
about items in assessments and overall usability
• Enhancing features to include: • Custom reports • Drag and drop reporting • Costs and service tracking and reporting • Child and Adolescent Needs and Strengths
(CANS) reports, and clinical flags
• Wrap-TMS will be available to programs and wraparound initiatives by July 2013