combining universal screening and data mining from schools...
TRANSCRIPT
Combining Universal Screening and Data Mining From Schools, DSS, Mental Health, Chemical
Dependency, and Probation in a Realist Evaluation of What Works and For Whom
Discussion Session Presented at University of South Florida’s 28th Annual Research & Policy Conference: Child, Adolescent, and Young
Adult Behavioral Health, Tampa, Fl., March 22-25, 2015.
Mansoor A. F. Kazi, PhD, University at Albany, State University of New York Patricia Brinkman, Director, Community Mental Hygiene Services, Chautauqua County Rachel M. Ludwig, Project Coordinator, Tapestry/Children’s SPOA, Chautauqua County Victoria Patti, Early Identification & Recognition Specialist, Chautauqua County, NY
Tapestry of Chautauqua
County, New York
Mansoor A. F. Kazi, Assistant professor Behavioral Health, School of Social Welfare, University at Albany
[email protected] • American Evaluation Association (eval.org) Co-Chair
Human Services Evaluation Topical Interest Group.
Based on Kazi, M. A. F.
(2003) ‘Realist Evaluation
in Practice’, London: Sage
Realist Evaluation Partnerships
SAMHSA’s
Gold Award for
Outstanding
Local
Evaluation
2010 Training
Institute STAKES/THL
Chautauqua Tapestry
Family driven ~ Youth guided ~ Culturally sensitive
Community based ~ Evidence-based
What is Chautauqua Tapestry?
An opportunity and challenge for
Chautauqua County service and support
providers to partner with youth who have
emotional and behavioral challenges and
with their families to create an
accessible, responsive, appropriate and
effective service delivery system.
“Chautauqua Tapestry brings together providers, families
and the community to share responsibility and resources to enhance the delivery
of services and improve the lives of youth and families.”
Seeking Collaborative Partners
Education
Mental Health
Public Health
Child Welfare
Family Court
Objectives for Discussion Session
• 1) How to Work with school districts to select and regularly use universal screening tools
• 2) How to evaluate the effectiveness of changes in the services as the system of care is implemented utilizing the universal screening tools
• 3) How to utilize data dumps from the management information systems of schools, mental health and other services to continuously evaluate alongside the repeated universal screening tools and to promote sustainability
• 4) How to enhance the utility of evaluation from a family perspective that places real meanings to the quantitative findings across thousands of service users
• 5) How to use this data to promote cultural competence, family and youth partnerships, and interagency collaboration.
Specific Topics To Be Covered
• How to utilize the repeated screenings.
• Using this real data to demonstrate how research methods can be applied to investigate the patterns between demographics, interventions and outcomes, in a continuous evaluation (group discussion and analysis of data undertaken with the group)
• Youth and Family perspective—how to interpret these apparently statistical findings in a meaningful way, what do these findings mean for cultural competence, youth and family partnerships, and interagency.
Local Evaluation Strategy
• Evaluation resource and service for you and for each participating agency
• How to access and to use your own MIS data
• How to analyze this data repeatedly to inform practice
• Carried out with you and for you only—you decide who to share with and how to use the findings
Local Evaluation Strategy
• Human service agencies routinely collect data, but this data is not typically used for evaluation.
• Apply this evaluation strategy to make the best use of the available data in their own agencies
• Utilize data dumps from the management information systems of schools, mental health and other services
• Continuously evaluate alongside the repeated universal screening tools and to promote sustainability
Realist Evaluation: What Interventions work & in what circumstances
• A combination of efficacy research & epidemiology traditions • Management Information System (MIS) Data routinely collected but
typically not used for evaluation in agencies • Investigate interrelationships between outcomes, client demographics,
client circumstances, & services provided (interventions) • Methods such as binary logistic regression can predict the likelihood of
effectiveness of an intervention in given circumstances • Use findings at regular intervals to better target and develop services
School Examples of Realist Evaluation
• 13 School Districts (Chautauqua County) including the largest--Jamestown Public Schools
• 2008/09 baseline and comparisons with 2009/10, 2010/11 and each marking period in the current year
• Outcomes: average school grades, state tests, discipline, attendance, drop out rates
• Demographics: ethnicity, gender, lunch status, special educational needs, etc.
• Interventions: school based interventions, summer program, mental health and other services
• 100% school data plus 100% agency data from participating agencies
• What works and for whom in achieving school, agency and system of care outcomes
Data analysis and utilization
• Single system design with each youth and one group pretest posttest design repeated at every marking period
• Comparison of outcomes between baseline and subsequent periods
• Comparisons between those receiving and not receiving interventions
• Investigation of patterns between outcomes, demographics and interventions
• Binary logistic regression to identify predictors at every marking period
• Data analysis carried out in partnership with schools and agencies
• Utilization of evaluation findings to develop and improve services for children and families at regular intervals
Universal Screening Program
• Chautauqua County Department of Mental Health is the host of the 5 year grant through NYS Office of Mental Health
• 36 other counties in NYS have also received the grant
• to screen 2,000 children/youth ages 3-21 each year for early detection of emotional concerns or difficulties
• Provide community education about the importance of early identification and mental health
• Partnerships with organizations, schools, agencies
Emotional Wellness Screenings in Public School Systems
• Screened by teacher who knows the child well and for at least 6 months to observe the child’s behaviors
• Identify children with emotional concerns before they develop into disorders; helps identify stressors
• Increase likelihood that struggling children get the help they need in school, as well as referrals to outside agencies or supports
• Minimize the impact of possible mental health concern on the child’s life
UNIVERSAL SCREENER IN
SCHOOLS
Response To Intervention
Universal Screening
Academic Screeners--- More familiar with
(i.e.., Literacy, Math)
Social/Emotional Screener---Strengths and
Difficulties Questionnaire
JPS partnering with agencies to provide
interventions
Why Universal Screening For Social
Emotional Needs?
Looking for Universal interventions for
school-wide systems (tier 1 of PBIS)
Process of finding the right
students/matching the right interventions to
them
Screeners are one piece of data.
Data Based Decisions
Goals Of Screening
Fast, efficient, and respectful
Include all children and youth in
elementary
Identify focus for interventions
One piece of data
Strengths and Difficulties Questionnaire
(SDQ)
(Goodman, 2005)
Brief behavioral screening questionnaire
for children 3-17 year old.
asks about 25 attributes, some positive, and
others negative.
http://www.sdqinfo.org/
SDQ Items in 5 Scales
Emotional Symptoms
Conduct Problems
Hyperactivity/inattention
Peer Relationships
Prosocial Scale **
Universal Screening Tool: SDQ
• Strengths and Difficulties Questionnaire (SDQ)
• Brief measure of pro-social behaviour and psychopathology, 3-17 yr olds
• Goodman (2001)—reliability .73
• Five factors: emotional symptoms, conduct problems, hyperactivity, peer relationships and pro-social
• Grade levels K to 4 in 2012, 2013 & 2014
SDQ 2012 and 2013 Scores
SDQ 2012 and 2013 Scores
SDQ 2012 and 2013 Scores
SDQ 2012 Normal Scores/Ethnicity
SDQ 2012 Normal Scores/IEP
SDQ 2012 Predictors for
abnormal/borderline total scores
SDQ Change from 2012 to 2013: Total
Scores
Paired Samples Statistics
Mean N Std. Deviation Std. Error Mean
Pair 1 Total_Difficulties2012 9.0907 1511 7.73341 .19895
TOTAL2013 9.0232 1511 7.95077 .20454
SDQ Change from 2012 to 2013: Total
Scores
Jamestown County Mental Health
Distribution of 2012 Total SDQ
Jamestown County Mental Health
Change from 2012 to 2013 Total SDQ
Jamestown County Mental Health
Change from 2012 to 2013 Total SDQ
Striders Elementary Change from 2012 to
2013 Total SDQ
Predictors for Change from 2012 to 2013
Total SDQ
Conclusions for SDQ Screening
•Main contributors to SDQ total score status were hyperactivity and conduct subscales
•Predictors for abnormal/borderline SDQ total scores were IEP, lunch status, gender, father’s level of education
•Significant improvements with Jamestown County Mental Health intervention
•Predictors for SDQ total score improvement were Striders intervention and baseline abnormal/borderline scores
Evaluation Strategy
• We are here to help you—as an evaluation resource
• Help you to use your own data for evaluation
• Can connect to others in Chautauqua County who have done this (e.g. data-dumps)
• Data findings can inform your decision-making and help make your services more effective
• This evaluation will also help you to be in a better position to apply for grants and other funding
• Enhance your own capacity for evaluation
• Utilizing SAMHSA funding, a free evaluation service for all participating school districts, DSS and other child-serving partners
LIVE INTERACTIVE DATA EXAMPLE
• THE SDQ Universal Screening for 2014
• How the situation has changed: Merging 2014 SDQ data with the SDQ data of 2012 and 2013
• Services Provided by Child-Serving Partners: The real impact on SDQ scores