assessment of reliable change: methods and assumptions michael basso, ph.d. associate professor and...
TRANSCRIPT
Assessment of Reliable Change: Methods and Assumptions
Michael Basso, Ph.D.Associate Professor and Director of Clinical Training
Department of Psychology—University of Tulsa
Clinical Associate Professor
Department of Psychiatry—University of Oklahoma
Objectives• Provide background concerning methods
of assessing reliable change• Describe assumptions and applications of
reliable change scores• Illustrate use of reliable change scores
Assessment of Clinical Change
• Two Basic Approaches– Assessment of Group Differences Across Time– Assessment of Individual Differences Across Time
Assessment of Group Differences Across Time
• Assessment of statistically reliable change– “Does the treatment yield significant benefits for groups of
patients?”
» i.e., do average scores at T1 and T2 come from different distributions
– This approach describes the average rate of change over groups primarily
– It is accomplished with repeated measure ANOVA– Problem: You could have a statistically significant difference
with a very small effect size, but it might not be a clinically meaningful change
• Assessment of Clinically Meaningful Change– “Did the patient’s change in performance at T2 exceed base
rates of change?”
» i.e., did the individual show change that exceeded expectations based on measurement error, practice effects, and regression to the mean?
– This method describes the base rate of change
» Change that is exceeds the base rate is not normal, and is therefore clinically meaningful
– Our focus is on the assessment of clinically meaningful change in individuals, but this method can be applied to group data as well
Assessment of Individual Differences Across Time
• How do you establish the base-rate of change?
• Bear into consideration that:– It would be improbable to obtain the exact same score twice
– There is no perfect test-retest correspondence because of
» measurement error
» regression to the mean
» practice effects
Assessment of Clinical Change for Individuals
• Reliable change Index scores– “Does change exceed what would be expected based on measurement
error alone?”– This method is based on Reliability of measurement– It is used for typical performance tests
» i.e., attitude, personality, psychopathology, etc.
• Standardized Regression-Based Change Scores– “Does change in scores exceed expectations based on T1 (baseline)
scores?”– This method is based on a validity coefficient (i.e., what T2 score is
predicted by the T1 score)– It is used for maximal performance tests
» i.e., IQ, neuropsychological, etc.
Two Methods of Assessing Base Rates of Change
• Elaborated by Jacobson and Truax (1991)– Based on the standard error of the difference
» Which in turn is based on the reliability coefficient– This reflects the sampling distribution of difference scores– it implies the magnitude of differences between two test scores that
vary by chance alone
• Assumptions– Error components are mutually independent and independent of true
pretest and posttest scores– Error is normally distributed with a mean of 0– SE of error is equal for all subjects– These assumptions are questionable in clinical settings (cf. Maassen,
2004)
Reliable Change Index Scores
Standard Normal Curve—Distribution of Difference Scores
-4 -3 -2 -1 0 1 2 3 4
34% 34%
14% 14%2% 2%
50th 84th 98th16th2nd
• To use the RCI, you must compute the SE of difference between two scores
• SEdiff=(2(SD(1-rxx)1/2)2)1/2
• Then, compute a confidence interval for change scores• for 95% confidence, you multiply 1.96 * SEdiff• for 90% confidence, you multiply 1.60 * Sediff• Does the raw score change between T2 and T1 exceed the
confidence interval?– If so, it represents change that exceeds the base rate expected based on
measurement error» Thus, clinically meaningful change has occurred
– If not, then the change is consistent with the base rate expected based on measurement error» Thus, no clinically meaningful change has occurred
Reliable Change Index Scores--Method
• Ferguson, Robinson, & Splaine (2002)– SF-36 in 200 patients who had undergone a Coronary Artery
Bypass Grafting (CABG) surgery– SF-36 contains 8 scales
» Physical Functioning» Role Functioning Physical» Bodily Pain» General Health» Vitality» Social Functioning» Role Functioning-Emotional» Mental Health
Reliable Change Index Scores—An Example
• Ferguson, Robinson, & Splaine (2002)– Physical Functioning
» Reliability=.93 (from normative sample of 2474)» Mean of normative sample=84.15» SD of normative sample=23.28
» SEdiff=(2(SD(1-rxx)1/2)2)1/2
• SEdiff=(2(23.28(1-.93)1/2 ) 2) )1/2=9.85
• 95% CI: (SEdiff)*1.96=19.32
• T1 Mean=40.97
• T2 Mean=63.39
• Mean Diff=22.42
• The mean difference exceeds 19.32
• Thus, clinically meaningful change has occurred as a result of surgery
Reliable Change Index Scores—An Example
• Ferguson, Robinson, & Splaine (2002)– Mental Health
» Reliability=..84 (from normative sample of 2474)» Mean of normative sample=75.01» SD of normative sample=21.40
» SEdiff=(2(SD(1-rxx)1/2)2)1/2
• SEdiff=(2(21.40(1-..84)1/2 ) 2) )1/2=10.92
• 95% CI: (SEdiff)*1.96=21.40
• T1 Mean=72.08
• T2 Mean=71.84
• Mean Diff=-0.24
• The mean difference fails to exceed 21.40
• Thus, no clinically meaningful change has occurred as a result of surgery
Reliable Change Index Scores—An Example
• Elaborated by Charter (1996)• Used primarily for maximal performance tests• The RCI of Jacobsen and Truax is used for
typical performance tests– It assumes that errors between test scores at baseline and time
2 are uncorrelated– This assumption is untenable in maximal performance tests
because of practice effects
Standardized Regression Based Change Scores
• Based on the standard error of prediction– SEpred=SDY2((1-rY1Y2
2)1/2)
• The SE reflects the sampling distribution of predicted scores• It implies the range of scores that might be expected at time two that may
be expected from the baseline score and prediction error• This method requires you to compute the estimated true score
– Y2True=M+((rY1Y2)(Y1-M))
• The T2 score is prone to error, and this formula permits an unbiased estimate of the true score
• The SEpred is used to compute a confidence interval around the estimated true score
Standardized Regression Based Change Scores
Standard Normal Curve—Distribution of Standard Error of Prediction Around
Estimated True Score
-4 -3 -2 -1 0 1 2 3 4
34% 34%
14% 14%2% 2%
50th 84th 98th16th2nd
• To use the SRB, you must compute the estimated true T2 score
• Compute the confidence interval around this estimated true T2 score
• For 95% confidence, you multiply 1.96 * SEpred• For 90% confidence, you multiply 1.60 * SEpred• Does the obtained T2 score fall outside the confidence
interval around the estimated true score for T2?– If so, it represents change that exceeds the base rate expected based
on measurement error, regression to the mean, and practice» Thus, clinically meaningful change has occurred
– If not, then the change is consistent with the base rate expected based on measurement error, practice, and regression to the mean» Thus, no clinically meaningful change has occurred
Standardized Regression Based Change Scores--Method
• Basso, Carona, Lowery, & Axelrod (2002)– WAIS-III re-tested in a group of control subjects over a 3-6 month interval– FSIQ
» Test-Retest Reliability=.90 » T1 Mean T1=109.4 (11.6)» T2 Mean T2=115.0 (12.1)
» SEpred=SDY2((1-rY1Y22)1/2)
• SEpred=(12.1((1-.902) 1/2))=5.29
• 95% CI: (SEdiff)*1.96=10.36• Mean Diff=5.60
– The mean difference fails to exceed the 95% CI– No individual had a score exceeding the 95% CI
• To apply the SRB, the T2 True Score is estimated– If the obtained score falls within the CI around the T2 True score, then no
clinically meaningful change has occurred
Standardized Regression Based Change Scores--An Example
• Basso, Carona, Lowery, & Axelrod (2002)– An example application:– T1 obtained score=104– T2 obtained score=116– Estimated True T2 Score
» YTrue=M+((rY1Y2)(Y1-M))
» YTrue=100+(.90)(104-100)=103.6
» 116 exceeds 10.36 points from 103.6
» Thus, meaningful change has occurred
Standardized Regression Based Change Scores--An Example
• Basso, Carona, Lowery, & Axelrod (2002)– An example application:– T1 obtained score=103– T2 obtained score=106– Estimated True T2 Score
» YTrue=M+((rY1Y2)(Y1-M))
» YTrue=100+(.90)(106-100)=105.4
» 105 falls within 10.36 points of 106
» Thus, no meaningful change has occurred
Standardized Regression Based Change Scores--An Example
Questions?