computerized adaptive testing in clinical substance abuse practice: issues and strategies barth...
TRANSCRIPT
![Page 1: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/1.jpg)
Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies
Barth RileyLighthouse Institute, Chestnut Health Systems
![Page 2: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/2.jpg)
Overview
• CAT Basics• CAT in Clinical Assessment
• Triage of individuals to support clinical decision making• Measuring Multiple Dimensions• Identifying Persons with Atypical Presentation of Symptoms
![Page 3: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/3.jpg)
Evidence-Based Practice• Requires accurate diagnosis, treatment placement, and
outcomes monitoring• Assessment over a wide range of domains• The cost of evidence-based assessment is:
• Time• Respondent Burden• Increased staff resources (including training
![Page 4: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/4.jpg)
Improving Efficiency• The use of screeners and short-form
instruments has significantly improved the efficiency of the assessment process• Can help determine whether a full
assessment is warranted• But not a substitute for a full assessment
• Lack of precision• Floor and ceiling effects• Limited content validity
![Page 5: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/5.jpg)
CAT Basics
![Page 6: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/6.jpg)
CAT Process
Decreased Difficulty
Decreased Difficulty
Typical Pattern of Responses
Increased Difficulty
Increased Difficulty
Middle Difficulty
Middle Difficulty
Score is calculated and the next best item is selected based on item difficulty+
/- 1
Std
. E
rror
Correct Incorrect
![Page 7: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/7.jpg)
Logical Components of CAT
• Start Rule
• Item Selection
• Measure Estimation
• Stop Rule(s)
![Page 8: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/8.jpg)
The Start Rule
• Used to select first item
• What measure is assigned to the respondent prior to selecting the first item?
• Can be an arbitrary value (0 on the logit scale) or can be based on previously gathered information.
![Page 9: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/9.jpg)
Item Information
0
0.5
1
-4-3
.6-3
.1-2
.7-2
.3-1
.9-1
.5-1
.1-0
.7-0
.3 0.1
0.5
0.9
1.3
1.7
2.1
2.5
2.9
3.3
3.7
Trait
Pro
ba
bili
ty/In
form
ati
on
Prob. Information
Item Difficulty = 0.5
too difficult too easy
Maximum information, Trait level = 0.5
![Page 10: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/10.jpg)
CAT in Clinical Assessment
![Page 11: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/11.jpg)
Clinical Decision Making
• How severe are the symptoms?
• What type of treatment is most appropriate?
• Can CAT be used to answer these questions more efficiently?
![Page 12: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/12.jpg)
Strategy
• Starting Rules•Using screener measures to set
the initial measure and select the first item
• Variable Stop Rules•Tight precision around cut points•Less precision away from cut
points
![Page 13: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/13.jpg)
Riley, Conrad, Dennis & Bezruczko, 2007
• Used CAT to place persons into low, moderate and high levels of substance abuse and dependency.
• Substance Problem Scale (SPS) is a 16 item instrument measuring recency of substance use. •When was the last time you drank
alcohol?
![Page 14: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/14.jpg)
Defining Cut Points
• Cut points can be established by examining where persons with different levels of severity fall onto the measurement continuum.
![Page 15: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/15.jpg)
The Start Rules
•Random: Randomly select an item between -0.5 and 0.5 logits of severity.
•Screener: Select most informative item relative to measure on a previously administered screener (SDScr).
![Page 16: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/16.jpg)
The Variable Stop Rule
• Stop rules set for low, mid and high range of severity.
• Mid range stop rule was set to SE=0.35 for all simulations.
• Low and High range stop rule: SE=0.5 to 0.75
![Page 17: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/17.jpg)
CAT Standard Error
Middle range where
decisions and made and
precision is controlled
High & Low ranges where there is little impact on clinical
decisions and precision is
allowed to vary more
High & Low ranges where there is little impact on clinical
decisions and precision is
allowed to vary more
![Page 18: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/18.jpg)
Start Rule UsingScreener
Select item
Administeritem
EstimateMeasure, SE
Stop?
End test
Yes
No
High range?
Mid range?
Low rangestop rule
High rangestop rule
Mid rangestop rule
Yes
Yes
No
No
CAT Algorithm
![Page 19: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/19.jpg)
Results
• Screener starting rule improved CAT efficiency by 7 percent
• CAT reduced the number of required items by 13 to 66%
• CAT to full-measure correlations ranged from .87 to .99
• Classification of persons into treatment groups based on CAT and full measure (kappa coefficients) ranged from .66 to .71.
![Page 20: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/20.jpg)
Results• Variable stop rules improved
efficiency by 15-38%•Efficiency depended on definition of
the mid range of severity
• Screener start rule and variable stop rules resulted in accurate and efficient estimation of substance abuse severity.
![Page 21: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/21.jpg)
Measuring Multiple Dimensions
![Page 22: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/22.jpg)
Assessment on Multiple Dimensions
• Instruments often measure multiple constructs
• In CAT, treating a multidimensional item bank as unidimensional is problematic:• Some subdimensions may not be adequately
measured• Particularly if subdimensions are not highly
correlated with each other
![Page 23: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/23.jpg)
Strategy: Content Balancing
• Set an item “quota” for each subscale•Maximum number of subscale items
to administer during the CAT
• An item is selected if:• Its subscale quota has not been met•Provides maximum information
![Page 24: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/24.jpg)
Internal Mental Distress Scale
• The IMDS consists of the following subscales:•Depression Symptom Scale•Anxiety/Fear Symptom Scale•Traumatic Distress Scale•Homicidal/Suicidal Scale
![Page 25: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/25.jpg)
Variations of Content Balancing
• Screener: Administers screener items first; no further content balancing.
• Mixed: Administers screener items, then uses content balancing for remaining items.
• Full: Uses content balancing throughout CAT session.
![Page 26: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/26.jpg)
Variations of Content Balancing
• In mixed and full content balancing, the following target number of items is administered from the IMDS subscales:• Depression: 5• Anxiety: 5• Trauma: 5• Homicidal/Suicidal: 3
![Page 27: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/27.jpg)
Content Balancing ResultsScale N Items None Screener Mixed Full
Depression≥ 1 99.1% 100% 100% 100%≥ 3 79.1% 76.7% 100% 100%
Homicidal/Suicidal
≥ 1 20.5% 100% 100% 100%≥ 3 8.2% 7.8% 100% 100%
Anxiety≥ 1 100% 100% 100% 100%≥ 3 100% 100% 100% 100%
Trauma≥ 1 100% 100% 100% 100%≥ 3 99.7% 100% 100% 100%
![Page 28: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/28.jpg)
CAT to Full-Scale CorrelationsScale None Screener Mixed Full
IMDS 0.982 0.982 0.978 0.971
Depression 0.957 0.937 0.956 0.956
Homicidal/Suicidal
0.599 0.828 0.964 0.945
Anxiety 0.962 0.947 0.956 0.957
Trauma 0.968 0.974 0.972 0.969
Average r 0.894 0.934 0.965 0.960
![Page 29: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/29.jpg)
Placement into Triage GroupsMeasure None Screener Mixed Full
IMDS .867 .871 .863 .841
Depression .909 .911 .753 .749
Homicidal/Suicidal
.312 .067 .917 .902
Anxiety .803 .759 .811 .790
Trauma .836 .850 .847 .837
Average Kappa
.745 .692 .838 .824
![Page 30: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/30.jpg)
Results
• Content balancing had the greatest impact on homicidal/suicidal scale.
• Mixed content balancing provided best overall results
![Page 31: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/31.jpg)
Identifying Persons with Atypical Presentation of Symptoms
![Page 32: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/32.jpg)
Implications
• Implications: Clients sometimes endorse severe clinical symptoms that are not reflected by overall scores on standard assessments.
• Misfit in clinical assessment can reflect:• Difficulty understanding the assessment• Cross-cultural effects• Differential effects of treatment on some symptoms but
not others• Unusual symptom profiles
![Page 33: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/33.jpg)
Clinical Implications• Results reveal subgroups who endorse severe
symptoms without endorsement of milder symptoms.• Atypical Suicide profile• Substance dependence symptoms with abuse
symptoms• Persons who commit serious crimes (murder, rape)
who have not committed less serious criminal offenses.
![Page 34: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/34.jpg)
Person Fit Statistics
• Person fit statistics are the most common means of detecting atypical responders.
• Here is a typical (predicted by IRT) pattern of responding:
11111101000000000• Here is an example of an atypical response
pattern:
110111110100000111
![Page 35: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/35.jpg)
Fit Statistics in CAT• Become less sensitive as the number of
administered items decreases.
• In CAT, items are usually selected in which each possible response to the item is equally likely.
• Items for which unusual responses are given may not be administered by the CAT.
![Page 36: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/36.jpg)
Outfit by Number of Items
Admin. Items
Outfit Categories
< 0.75
Proto
Typical
0.75-1.33
Typical
> 1.33
Atypical
16 30.2% 48.1% 21.7%
12 34.3% 51.1% 14.6%
8 38.4% 53.2% 8.4%
4 58.2% 40.0% 1.8%
![Page 37: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/37.jpg)
Strategies• Item selection strategies
• Unidimensional Approach• Examine response patterns for items representing a second-
order construct, such as internal mental distress• Fit statistics: detects all atypical symptom patterns
• Multidimensional Approach• Compare subdimension measures • Detection of a specific response pattern
• Is the persons level of suicide ideation greater than their level of depression?
• How big a difference in measures?
• Combination of the above
![Page 38: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/38.jpg)
Does Item Selection Matter?Atypicalness
Category None Screener Mixed Full
Full
IMDS
Proto Typical 26.7% 34.6% 48.3% 50.5% 49.2%
Typical 69.0% 58.7% 40.8% 38.9% 38.4%
Atypical 4.3% 6.5% 10.9% 10.6% 12.4%
Kappa .27 .32 .48 .50 --
![Page 39: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/39.jpg)
CAT to Full-Measure Person Fit
CAT* Statistic
Full-Measure
OutfitOutfit r= .73
Ei r=.31
Homicidal/Suicidal – Depression
r=.08
Logistic Regression Correct % 91.6%
* Using full content balancing
![Page 40: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/40.jpg)
Suicide-Depression Profile
CAT* Statistic
Full Measure
H/Sa- Depression
Outfit r = .11
Ei r = -.54
H/S-Depression r = .92
Multiple Regression R2 = .86
* Using full content balancing
a Homicidal-Suicidal Scale measure
![Page 41: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/41.jpg)
Conclusions
• Fit statistics and examination of subscale scores appear to capture different response patterns.
• Using effective item selection methods in conjunction with multiple measures of person fit improves our ability to detect atypical symptom patterns.
![Page 42: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/42.jpg)
Potential of CAT in Clinical Practice
• Reduce respondent burden
• Reduce staff resources
• Reduce data fragmentation
• Streamline complex assessment procedures
• Assist in clinical decision making
• Identify persons with atypical profiles
![Page 43: Computerized Adaptive Testing in Clinical Substance Abuse Practice: Issues and Strategies Barth Riley Lighthouse Institute, Chestnut Health Systems](https://reader033.vdocument.in/reader033/viewer/2022051619/56649d9d5503460f94a86593/html5/thumbnails/43.jpg)
Contact Information
• A copy of this presentation will be at: www.chestnut.org/li/posters
• For information on this method and a paper on it, please contact Barth Riley at [email protected]