distinguishing between treatment effects and dif in a substance abuse outcome measures using...
TRANSCRIPT
Distinguishing between Treatment Effects and DIF in a Substance Abuse Outcome Measures Using Multiple Indicator Multiple Causes (MIMIC) Models
Barth B. Riley, Michael L. Dennis
Chestnut Health Systems
Study supported by National Institute on Drug Abuse Grant (NIDA) No. R37 DA11323.
Overview
Differential Item Functioning and Its Impact The Multiple Indicator Multiple Cause (MIMIC)
Model Demographic differences in substance use
and substance abuse treatment Present Study
Differential Item Functioning
DIF: Two groups differ in their likelihood of endorsing an item after controlling for differences on the measured construct.
Group differences in the likelihood of endorsing an item may be due to: Group differences on the latent trait Differential item functioning (DIF) Both
DIF can also occur over time
Differential Item and Test Functioning The presence of DIF items can reduce the
validity of a measure in making between group comparisons.
If DIF is of sufficient magnitude to cause measurement bias against one group relative to another, efforts to interpret outcomes measures becomes complex.
Did the persons change or did the items in the instrument change?
Analysis of DIF
Several approaches have been employed for the analysis of DIF: T tests comparing item parameters between two
groups Mantel-Haenszel contingency tables Logistic regression IRT Likelihood ratio tests
Most of these approaches are limited to comparisons of two groups on a single factor.
Do not directly assess impact of DIF on person measures.
Multiple Indicator Multiple Causes (MIMIC) Models Combines aspects of confirmatory factor
analysis and structural equation modeling. The basic MIMIC models consist of the
following components: A latent variable—the construct being
measured. A set of measured indicators—items Grouping variables such as race and gender
Basic IRT Model
LatentConstruct
LatentConstruct
Item 1Item 1
Item 2Item 2
Item 3Item 3
Item nItem n
……
Indicators
MIMIC Model, No DIF Assumed
LatentConstruct
LatentConstruct
Item 1Item 1
Item 2Item 2
Item 3Item 3
Item nItem n
……
Latent Variable
EthnicityEthnicity
GenderGender
Indirect effects
Indicators
MIMIC Model with DIF Effects
LatentConstruct
LatentConstruct
Item 1Item 1
Item 2Item 2
Item 3Item 3
Item nItem n
……
Latent Variable
EthnicityEthnicity
GenderGender
Effect of DIF is partialed out of the indirect effects Indicators
Direct DIF effect
Study
The purpose of this study was to examine the effect of DIF by time, gender race on the Global Appraisal of Individual Needs (GAIN) Substance Problem Scale
Data were collected from 446 participants as part of a three-year substance abuse early re-intervention study.
Participants (N=446)
Recruited from community-based substance abuse treatment in Chicago in 2004.
Participants were randomly assigned to either outcome monitoring or recovery management checkups, designed to help relapsing participants to return to treatment.
Followed quarterly for 3 years. Participants were predominantly
Male (54.5%) African American (80.2%) Average age: 38.4 years (SD=8.3)
Primary Drug
AlcoholAmphetaminesMariuanaCocaineOpiates
Substance Problem Scale
The Substance Problem Scale (SPS) measures problems with alcohol/drug use during the past month, including abuse, dependence and substance-abuse health problems.
Consists of 16 dichotomous items Based on DSM-IV-TR criteria for substance
abuse and substance dependence. Internal consistency: .9 Test-retest reliability: .73
Model
In order to assess treatment effects over time, a multilevel framework was used: Level 1: Time: random effect Level 2: Person: fixed effects
Treatment variables: Random assignment to recovery management Days in outpatient, intensive outpatient and residential
treatment DIF factors: gender and ethnicity One and two parameter IRT models were compared.
MIMIC Model: Within Level
SPSSPS
TimeTime
TxParticipation
TxParticipation
SPS 1SPS 1
SPS 2SPS 2
SPS 3SPS 3
SPS nSPS n
……
Control for DIF
MIMIC Model: Between Level
SPSSPS
RaceRace
GenderGender
RMCRMC
SPS 1SPS 1
SPS 2SPS 2
SPS 3SPS 3
SPS nSPS n
……
Control for DIF
OpiatesOpiates
Goodness of Fit
Model CFI TLI RMSEA
1 Parameter IRT
Basic IRT model .974 .970 .099
MIMIC No DIF .972 .961 .076
MIMIC DIF .974 .970 .075
2 Parameter IRT
Basic IRT model .952 .994 .045
MIMIC No DIF .975 .994 .03
MIMIC DIF .976 .994 .03
N Cases = 400, N Observations = 5393
Time DIF
-0.75
-0.5
-0.25
0
0.25
0.5
0.75
1
1.25
1.5
1.75
0 3 6 9 12 15 18 21 24 27 30 33 36
Time (Months)
Ite
m D
iffi
cu
lty
(b)
Hiding Use
Use Weekly
Using/Responsibilities not met
Need more AOD get same effect
Made you depressed
Used more often
Time DIF
Unable to cut down use
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
0 3 6 9 12 15 18 21 24 27 30 33 36
Time (Months)
Item
Dif
ficu
lty
(b)
Gender DIF
0
0.5
1
1.5
2
2.5
Caused Health Problems Made situation unsafe
MaleFemale
Ethnicity DIF
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Causing fights, trouble Widthdrawal symptoms
HispanicNon-Hispanic
Primary Drug DIF
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Made situation unsafe Withdrawal symptoms
OpiatesOther Drugs
Group Differences on DIF Factors
Factor
No DIF Model DIF Model
Z Sig. Z Sig.
Time -0.271 .001 -0.281 .001
Gender 0.217 .041 0.206 ns
Ethnicity -0.733 ns -0.845 ns
Primary Drug 0.333 .015 0.257 ns
Treatment Effects
Treatment Variables
No DIF DIF
Z Sig. Z Sig.
Any outpatient tx. -0.606 .001 -0.605 .001
Times in outpatient tx. 0.003 ns 0.003 ns
Any intensive outpatient -0.325 ns -0.324 ns
Days, intensive outpatient tx. -0.007 ns -0.007 ns
Any residential treatment 0.435 .001 0.434 .001
Nights in residential tx. -0.306 .001 -0.306 .001
Any methadone tx. 0.636 .001 0.635 .001
Days taking methadone -0.023 ns -0.023 ns
Recovery Monitor. Checkups -0.248 .018 -0.249 .018
Conclusions
The MIMIC model is a promising tool for assessing the presence and impact of DIF on at the scale level (DTF).
Controlling for DIF reduced differences in SPS measures as a function of gender and primary drug.
Treatment effects as measured by the SPS were not affected by gender, ethnicity, primary drug or time DIF.
MIMIC: Strengths
Assess DIF and DTF on multiple factors DIF factors can be discrete or continuous
variables Distinguish between treatment and DIF
effects Can be used in conjunction with longitudinal
analysis methods (e.g., multilevel modeling).
MIMIC: Limitations/Caveats
In order to specify the model, at least one item must be free of DIF (or have minimal DIF).
Can not detect non-uniform DIF—DIF in the discrimination parameter
Obtaining group specific item parameters is not straightforward
Assumes consistent factor structure across groups
Useful References
Muthén, B. (1989). Latent variable modeling in heterogeneous populations. Psychometrika, 54, 557-585.
Fleishman, L.A., & Lawrence, W.F. (2003). Demographic variation in SF-12 scores: True differences or differential item functioning? Medical Care, 41(7 Suppl.) III75-III86.
MacIntosh, R. & Hashim, S. (2003). Converting MIMIC model parameters to IRT parameters in DIF analysis. Applied Psychological Measurement, 27, 372-379.
Finch, H. (2005). The MIMIC Model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST, and the IRT likelihood ratio. Applied Psychological Measurement, 29(4):278-295.
Contact Information
A copy of this presentation will be at: www.chestnut.org/li/posters
For information on this method and a paper on it, please contact Barth Riley at [email protected].