quality measures for rehabilitation: policy, provider and patient perspectives measuring clinical...
TRANSCRIPT
Quality Measures for Rehabilitation: Policy, Provider and Patient Perspectives
Measuring Clinical Change: Quality Indicators
ACRM-ASNR Pre-Conference Institute
Kenneth J. Ottenbacher
University of Texas Medical Branch
Disclosures
National Institutes of Health Grants: R24 HD065702; R01 AG017638; K12 HD055929; T32 HD007539PI: Kenneth J. Ottenbacher
National Center for Medical Rehabilitation Research (NICHD), National Institute on Neurological Disorders and Stroke, and National Institute on Aging
Department of Education Grants: H133G080163; H133P110012 PI: Kenneth J. Ottenbacher
National Institute on Disability and Rehabilitation Research
Learning Objectives
• Describe three approaches to measuring clinical change
• Discuss the difference between clinical change at the individual level and facility level
• Identify challenges in measuring clinical change
How NOT to measure clinically important change:
Measuring clinically important change is complex. Multiple methods are available with various advantages and disadvantages.
p<.05
Statistical Significance versus Clinical Importance
Design sensitivity: statistical power for
experimental research Mark W. Lipsey
The Cult of Statistical Significance
Stephen T. Ziliak & Deirdre N. McCloskey
Investigators and clinicians have a long history of interpreting statistically significant findings as also being clinically significant.
Example: Smith reports a rehabilitation outcome study with statistically significant results. Jones conducts a replication study with (null) non-significant results.
Assume that both trials are well-designed and their internal validity is good.
Results:
Smith, t = 2.21 (df = 78, p < 0.05)
Jones, t = 1.06 (df = 18, p = ns)
The interpretation is that Smith found a statistically significant difference (change) in the outcome measure and Jones did not find a statistically significant difference (change).
And… that the treatment produced a significant change in the Smith study, but not the Jones study.
An alternative way to look at the results of the Smith and Jones studies is to examine the degree to which the null hypothesis is false in both investigations.
Cohen (1988) refers to this as the “effect size.”
If we compute an effect size for each study, we will find the following:
Formula: d = 2(t)/ df
Smith d = 0.50
Jones d = 0.50
The effect size is the same for the two trials - the degree to which the null hypothesis is false is identical for both the Smith and Jones study.
The amount of change associated with the outcome is the same in both studies.
Why are the findings in one study statistically significant and the other study non-significant?
Things are not always what they appear to be.
Power for the Jones trial = 0.18
The chance of a Type 2 error is 0.82 (82%)
From: Day, R.H. (1972). Science, Vol. 175, p.1335.
Reading made Don Quixote a gentleman, but believing what he read made him insane.
G. B. Shaw
Who is the Target:
1. Assessment of individual change
2. Assessment of facility change
3. Assessment of population change
Measuring Clinical Change
Approaches to Assessing (individual) Change
• Comparison of change in raw scores or ratings
• Effect size (standardized mean difference)
• Reliable change index (Minimal detectable change)
• Minimal clinically important difference• Standardized regression-based change
scores
There are two approaches to measuring (individual) change.
• Distribution-based approaches that rely on expressing change scores in terms of an underlying sampling distribution.
• Anchor-based approaches that require an external, independent standard to ‘anchor’ the meaning of clinical
importance.
Examples:
Reliability Change Index (RCI)
Need the following information: a) baseline score, b) follow-up score, and c) the standard error of measurement (SEM) for the test. (SEM = S1 1-r) where S1 = standard deviation, r = reliability coefficient.
RCI = (X2 – X1) / SEM
The RCI provides an estimate of whether a change is greater than would be expected by chance (measurement error).
Examples:
Minimal Clinically Important Difference (MCID)
Requires identifying a clinically important external standard. For example, 5 motor FIM points. 5 FIM points represents ~ 5% of total scale range on the motor domain. This standard can be based on clinical experience, research literature, patient goals, or some combination.
Use of UDS burden of care values to determine external anchor.
Mean Decrease in Minutes of Help Per Day per FIMTM Point Change
Spinal Cord Injury Study, 1990-91 Hamilton et. al.
13 - 26 26 - 39 39 - 52 52 - 65 65 - 78 78 - 91
0
1
2
3
4
55
6
Min
ute
s o
f H
elp
Minutes per FIM Point
Motor FIM Score Ranges
UDSMR
Converting individual measures of clinical change to facility or population measures of clinical change (quality indicators).
Examples:
Percent of persons in a Case Mix Group achieving Minimal Detectable Change or Reliability Change Index criteria.
Proportion of persons in a Case Mix Group achieving a Minimal Clinically Important Difference.
Examples (continued):
Percent of persons in a Case Mix Groupachieving above benchmark expectations.
Polar GraphAdmission
Discharge
Benchmark
Ea GrBa
DU
DL
To
Bl
BoCh TToT
TuTWC
St
Co
Ex
Si
PsM e
UDSMR
Challenges:
1. How to risk Adjust?
2. How to make the process patient centered?
3. What are the unintended consequences?
The End
Doing outcomes research is a lot like raising children… you always think you are going to do a better job next time.
http://rehabsciences.utmb.edu/R24/ACRM.asp