Individual differences in response to intervention: An application of Integrative Data Analysis in Project KIDS

Sara A. HartFlorida State University

shart@fcrr.org@saraannhart

& Chris Schatschneider, Carol Connor & Stephanie Al Otaiba

Expanding our search for moderators of intervention

• A little about me– Behavioral genetics background– Interested to move these findings to schools

• Even with modest effect sizes, individual differences in intervention response

• Bioecological model (Bronfenbrenner & Ceci, 1994)

– Provides framework for differentiating students based on non-intervention related traits• Lets follow individualized medicine

Integrative Data Analysis (IDA)

• Item-level pooled data (Curran & Hussong, 2009)

• Capitalizes on cumulative knowledge – Longer developmental time span– Increased statistical power– Increased absolute numbers in tails

• Controls for heterogeneity – Sampling, age/grade, cohort, geographical, design,

measurement

Project KIDS

• Expanded definition of moderators of response to intervention– Cognitive, psychosocial, environmental, familial/genetic

risk• IDA across 9 completed intervention projects – Approximately 5600 kids

• Data entry of item level data common across at least 2 projects– ~30 different assessments

• Questionnaire data collection

Proof of Concept

• Behavior problems and achievement are associated

• More behavior problems are typically seen in LD populations

• Is adequate vs inadequate response status differentiated by behavior problems?

Method

• Participants– 2007-2008 ISI intervention through FL LDRC (Al Otaiba et al., 2011)

• RCT: 23 treatment, 21 contrast teachers• 556 kindergarteners • A2i recommendations vs enhanced standard practice

– 2009-2010 RTI Intervention through FL LDRC (Al Otaiba et al., 2014)

• RCT: 34 classrooms, kids randomized • 522 1st graders• regular RTI vs dynamic RTI

– 2005-2006 ISI intervention project (Connor et al., 2007)

• RCTish : 22 treatment, 25 contrast teachers, 3 pilot• 821 first graders• A2i recommendations vs standard practice

Method• Measures– WJ Tests of Achievement Letter-Word

Identification (LWID)• Pre- and post-intervention testing periods

– Social Skills Rating Scale: Behavioral Problems subscale • Teacher completed during intervention year

07/08 K ISI LDRCMean (SD)

09/10 1st RTI LDRCMean (SD)

05/06 1 ISIMean (SD)

WJ LWID Fall 12.03 (5.50) 26.66 (9.03) 24.28 (7.97)

WJ LWID Spring 21.75 (7.09) 38.73 (8.07) 36.54 (7.39)SSRS .50 (.44) .34 (.39) .53 (.44)

r=.89 r=-.23

r=-.26

Results: Calibration LWID IDA

• Randomly selected 1 time point/child/project to form “calibration sample” for LWID

• Decision to include only items > 5% endorsement rate

• Reduced item sample from 75 38 – Items 11 to 49

Results: Calibration LWID IDA

Hahahaahaaa.

Results: Calibration LWID IDA

• Non-linear factor analysis– Multiple group analysis to arrive at single factor

for LWID– Using measurement (in)variance principles across

the 3 projects, using modindices in Mplus to adjust for project variant differences • Key decision: We wanted to constrain based on

meaningful differences – Chi-square difference value set to equal Cohen’s d = .20

Metric (in)variance-localized misfit for items 32, 35, 36, 37, 39, & 46-49

Scaler (in)variance-localized misfit for thresholds of items 46, 47 & 49

Residual variance (in)variance-all could be constrained to equal

Factor variance (in)variance-All could be constrained to be equal

Final Model-using all data, set parameters based on final factor model, exported factor scores

Results: SSRS

Results

r = .97 r = .98

Results

r = .96

Now I have equivalent data!!!

• It’s a lot of steps to get a regular old data set• Now I can answer content questions, but with

more kids in a more generalizable sample

• So, are behavior problems bad for treatment response?

Results: Response

• Proc mixed: covariance adjusted LWID score– 1712 children

Results: Response

• 818 treatment children

Results: Response

• 818 treatment children

UnresponsiveCutoff < 20%

N=164!

Results

• Logistic regression– SSRS behavior problems significant predictor of

response status (OR = 1.58, CI = 1.30-1.90)• average behavior problems = 41% probability of being

“unresponsive”• greater than average behavior problems(+ 1SD) = 50%

probability of being “unresponsive”• Less than average behavior problems (-1SD) = 34%

probability of being “unresponsive”

Conclusions

• Response status is differentiated by behavior problems– Mo’ behavior problems, mo’ (reading) problems!

Overall IDA conclusions

• IDA is a “cheap” way to get more power, more n at tails, and show more generalizable effects

• Given how similar many of our projects are, consider doing item-level data entry – Easy potential to combine data– Can you do factor analysis and IRT? You can do IDA*!

• These data are more useful together than apart– IRT within and between samples?– Treatment effectiveness across samples?– Characteristics of lowest responders?

Project KIDS goals

• Intervention response status is known• Can we predict responders/non-responders with questionnaire

data?– Family history

• 1st degree vs 2nd degree, dosage

– Cognitive correlates• Executive functioning, ADHD

– Behavior• Comorbid behavior issues

– Environments• Home literacy environment vs neighborhood vs school

• Individualize instruction based on child traits– No cheek swab needed

Acknowledgements • Stephanie Al Otaiba • Carol Connor• Chris Schatschneider• Great staff & grad students, and many wonderful data

enterers

NICHD grant HD072286

r = .95

r = .78

r = .88

What about without DIF?r = .97

r = .99