SBHCs & Academic Outcomes: Are We Asking the Right Questions?

Gorette Amaral

Mona E. Mansour

Sara Ann Peterson

Susan Russell Walters

National Assembly on School Based Health Care

June 2003

Why we decided to explore this research road….

First, What Does the Literature Say?

Literature Review

• To see how academics are impacted by SBHCs, we searched for…• Peer-reviewed journal articles,

evaluation studies and reports • Written by private and public research

and government organizations • In health, education, and psychology

fields

The Impact of SBHCs on Academics

+ = Positive impact of intervention on indicator. o = No statistically significant impact on indicator. (Blank) = The study did not measure this particular indicator.

McCord et al1993

Kisker &Brown 1996

BPHC 1993 Jennings et al2000

Warren &Fanscsali

2000

Armbruster &Lichtman

1999

Gall2000

Webber 2003

Promotion to the nextgrade

+ +

Graduation rate +

GPA/grades o

Credit accumulation +

Receipt of failing grade + o

Attendance/absences o o + o + + +

Withdrawal/drop outrates

+ +

Suspension rates o o

Disciplinary referrals + o +

Educational aspirations +

Tardiness +

Studies found a Positive Impact from SBHCs on…

• Absences 4 (of 7) • Promotion to the next grade 2 (of 2) • Withdrawal/drop out rates 2 (of 2) • Disciplinary problems 2 (of 3) • Failing grade 1 (of 2) • Tardiness 1 (of 1)

Limitations of Studies on SBHC’s Impact on Academics

• 3 had no comparison group• No “cookie cutter” SBHC model• 4 examined interventions beyond the

scope of typical SBHCs

• One used anecdotal evidence

Improved Academic

Performance

SBHC Services

Educational Factors

Environmental Factors

Multiple Influences on Academic Performance

Student Health Factors

Social Factors

Summary of Findings Regarding Relationships Between Health Status/Risk Behaviors and Academic Achievement

Factors that an SBHC might impact

+ = Positive association found between intervention and academic indicatorO = No association found between intervention and academic indicator (Blank) = The study did not measure this particular indicator.

Weitzmanet al

(1986)High

school

Klerman(1996)High

school

Bailey-Britton(1987)Elem.

Symonset al

(1997)High

school

Blum et al(2000)

All ages

WestEd(2001)High

school

Glied(2002)

Ages 10-18

ATOD use + + + + +Mental healthproblems

+ + +

Poor diet + +Intentionalinjuries

+ +

Physicalillness

+ +

Low selfesteem &resiliency

+ +

Risky sexualbehavior

+ o

Health careutilization

o +

Studies that found a Positive Relationship Between Health Status/Risk Behaviors and Academics

• AODT use 5 (of 5) • Mental health problems 3 (of 3) • Poor diet 2 (of 2) • Intentional injuries 2 (of 2) • Physical illness 2 (of 2) • Low self-esteem and resiliency 2 (of 2) • Risky sexual behaviors 1 (of 2)

• Health care utilization 1 (of 2)

These are issues that SBHC can impact!

Summary of Findings Regarding Relationships Between Health Status/Risk Behaviors and Academic Achievement

Factors beyond the direct influence of an SBHC

+ = Positive relationship between indicator and academic performance.0 = No statistically significant relationship between indicator and academic performance. (Blank) = The study did not measure this particular indicator.

Weitzmanet al

(1986)High

school

Klerman(1996)High

school

Bailey-Britton(1987)Elem.

Symonset al

(1997)High

school

Blum et al(2000)

All ages

WestEd(2001)High

school

Glied(2002)

Ages 10-18

Educationalfactors

+ + +

Household orfamilycharacteristics

o + +

Demographics + +School safety + +Eatingbreakfast

+

Studies that found a Positive Relationship Between Other Factors and Academics

• Educational factors (3 of 3) • Household/family characteristics (2 of 3)• Demographics (2 of 2)• School safety (2 of 2) • Eating breakfast (1 of 1)

Conclusions from the Literature

• Insufficient evidence to prove a direct link between SBHCs and academic outcomes.

• Evidence of impact of medical and mental health status on academic outcomes.

• Important role of SBHCs in improving student health can contribute, at least indirectly, to improved academic outcomes.

What are Important Methodological Issues?

Methodological Issues

1. Possible Outcome Measures2. Individual Student Data3. Aggregate Data4. Access to Data5. Student or Parent Survey6. Computerized Records7. Data Quality & Linking8. Variable Definitions

1. Possible Outcome Measures

• Graduation rate• Grade promotion• GPA/grades-failing

grades• Credit accumulation• Standardized test

scores• Academic Performance

Index

• Attendance/absence• Cutting class• Withdrawal/drop-out

rates• Suspension rates• Disciplinary referrals• Educational aspirations

2. Individual Student Data

• Advantage of individual student data: • More power to determine differences between

your control and intervention groups

• If individual student data is used, however, researchers may need…• To obtain active parental and/or student consent

• Could get consent along with consent for SBHC use

• To comply with HIPPA issues if research study affiliated with academic institution

• To establish data agreements with school districts

• To deal with “Privacy tag issues”

3. Aggregate Data

• Aggregate data may include students• At the school or district level• Who did or did not use SBHC services,

and/or• Who were or were not enrolled in the

school for the majority of the school year

• Advantage of aggregate data is avoidance of consent issues

4. Access to Data

• Methods of Obtaining Data • Computerized or paper school records• Parent/child/teacher report or survey

• Relationship with school information services/technology division critical• Determines priority of providing data needed• Level of comfort with sharing data• Whether data is obtainable at all

• This is easier if SBHC run by school district

5. Student or Parent Surveys

• Limitations of surveys • Recall bias• How questions are phrased• Low response rates• Non-representative sample

6. Computerized Records

• May be more accurate than surveys, but still some issues• Where student mobility is high, absences in

particular may be inaccurate- absences may be high due to lack of knowledge that student has moved and has not officially withdrawn• Difficulty with decentralized school districts• May not be designed to collect all variables of

interest

7. Data Quality & Linking

• Data Quality• Limits on how data can be verified

• Limited by cost• Limited by how data provided

• Data Linking• Different sources, different challenges• If multiple sources, how will data be linked

• Consent issues• Use of unique identifiers• Making data non-identifiable before link• Data agreements that allow links by certain personnel

8. Variable Definitions

• Caution to make sure there are uniform definitions for variables being measured• Districts using different criteria for

determining and coding whether a child is exempt from academic testing

• Schools within a district using different criteria to determine whether an absence is excused/unexcused or what constitutes being absent ( full day/half day)

Given what we know, what are the next questions to

ask about the relationship between SBHCs and academic outcomes?

Keep in mind….

• We know:• SBHCs improve access to health care• Medical and mental health status of students

impacts their academic success

• And:• All SBHCs are not exactly the same but are

defined by their common elements• SBHCs are a part of a variety of services offered

in schools

However…

• The relationships between SBHCs and academic outcomes that we postulate as research and evaluation questions MAY be direct or indirect.

And this relationship should be…

• Possible (SBHC actually provides the intervention that is expected to influence academic outcome)

• Realistic (takes into account multiple interventions that may occur simultaneously or that may all impact academic outcome)

• Logical (model linking the intervention with the outcome can be described)

• Demonstrable (data for the intervention and outcome or adequate proxies can be reasonably obtained)

Where should we go next?

• Consider the various levels at which the intervention can be defined. Is it…• The SBHC as a whole or• A specified array of services delivered in

the SBHC context or• An intervention delivered to a specific sub-

population in the SBHC?

And…

• Examine the findings when each of these definitions are used. More focused interventions may yield clearer links to academic outcomes.

• Consider how SBHCs contribute to the Coordinated School Health program model as well as youth development and student support programs.

Finally!!

• Consider that the highest likelihood of improving student achievement may come from the synergy of interventions including SBHCs, other health and social programs, youth development opportunities, and educational program improvements.

• Develop a variety of algorithms or models that describe the relationships that we think occur based on our prior research and empirical observations.

Health & Academic Risk

Health Risk

BehaviorsEducationalOutcomes

EducationalBehaviors

AODT use

Mental health

Poor diet

Intentional injuries

Physical illness

Self-esteem

Sexual behaviors

Attendance

Dropout Rates

Behavioral Problems

Graduation

GPA

Standardized test scores