measuring the quality of intensive care - critical care · pdf file ·...
TRANSCRIPT
![Page 1: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/1.jpg)
Measuring the quality of intensive care
Gordon D. Rubenfeld, MD MScProfessor of Medicine, University of TorontoChief, Program in Trauma, Emergency, and Critical CareSunnybrook Health Sciences Centre
![Page 2: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/2.jpg)
Outline
• What are the challenges of measuring the quality of intensive care?– Are quality measures valid? – Are quality measures and reporting effective?– Are quality measures and reporting safe?
• Should we be reporting more publically?
![Page 3: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/3.jpg)
Identifying the best doctors and hospitals looks easy
![Page 4: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/4.jpg)
Risk Adjusted OutcomeThe Standardized Mortality Ratio
it’s just long division< 1 = good, > 1 = bad
![Page 5: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/5.jpg)
Risk Adjusted OutcomeThe Standardized Mortality Ratio
it’s just long division< 1 = good, > 1 = bad
![Page 6: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/6.jpg)
Risk Adjusted OutcomeThe Standardized Mortality Ratio
it’s just long division< 1 = good, > 1 = bad
95 – 162 0.88 (0.5 – 1.8)
![Page 7: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/7.jpg)
Every ICU cohort looks like this
![Page 8: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/8.jpg)
Problems with risk adjusted mortalityIf you can’t reduce your numerator, increase your denominator
• Upcoding for quality– In response to outcome assessment of CABG, coding of
comorbidities jumped abruptly at some hospitals, including a 1 year jump in “COPD” from 1.8% to 52.9%
– “Good” hospitals more likely to overcode severity of illness.
Chassin MR, Hannan EL, DeBuono BA. Benefits and hazards of reporting medical outcomes publicly. N Engl J Med. Feb 8 1996;334(6):394-398.
![Page 9: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/9.jpg)
Problems with risk adjusted mortality If you dont like your SMR, use a different risk model
• Comparing hospital rankings for 5 severity models with similar statistical performance in 3 diseases in 108 hospitals – Good agreement (κ > 0.7) in only 11 pairs of assessments
• 40 Dutch ICUs compared using 3 risk adjustment models (APACHE 2, SAPS 2, MPM 2)– 21 ICUs were identified as performance outliers by one of the
3 models– Only 1 ICU was identified by all 3 models as an outlier
Iezzoni LI. JAMA. 1997;278:1600-1607.Bakshi-Raiez, et al, CCM 2007; 35:
![Page 10: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/10.jpg)
Problems with risk adjusted mortality If you don’t like your SMR, transfer out not in
• Transfer in– Patients transferred to academic medical centers’ ICUs have
double the mortality predicted by APACHE score (even though this model contains a variable for transfer)
– Importing patients increases SMR• Transfer out
– Death is attributed to the receiving hospital– Exporting patients reduces SMR
Ann Intern Med. 2003 Jun 3;138(11):882-90Chest. 2007 Jan;131(1):68-75
![Page 11: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/11.jpg)
Problems with risk adjusted mortality If you don’t like your SMR, spin the wheel again
• Several studies use simulated populations of hospitals with “perfect” risk adjustment– 56-82% of the excess death rates were explained by random
variation– 65% of bad outlier hospitals in any given year were due to
chance• Data on > 1000 cases improve accuracy• Outliers in > 1 year improve accuracy
JAMA. 1990;264:484-90.Hosp Health Serv Adm. 1997;42:3-15.Med Care. 1996;34:737-53.
![Page 12: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/12.jpg)
Case volumes for most procedures are too low to use mortality as outcome …
the same is probably true for many ICUs
Dimick, J. B. et al. JAMA 2004;292:847-851.
![Page 13: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/13.jpg)
Does SMR correlate with other measures of quality?
![Page 14: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/14.jpg)
Even US News and World Reports ranking does not correlate with process of care
0
25
50
75
100
US News Top Hospital
Worst QuartileMiddle QuartilesBest Quartile
Williams, et al, Circulation. 2006 Aug 8;114(6):558-64
Compared 41 US News “Best Cardiac Hospitals”to 733 other hospitals by process of care
![Page 15: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/15.jpg)
Level 1 Trauma CentersVariation in outcome for severe injury
Arch Surg. 2008;143(2):115-119
![Page 16: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/16.jpg)
In most cases, discrepancies between observed and expected outcomes should serve to trigger a review of the process of care to determine if quality problems in fact exist.
Hadorn D, Keeler E, Rogers W, Brook R. Santa Monica, CA: Rand, 1993.
![Page 17: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/17.jpg)
[…] when used as a measure of quality for individual hospitals, risk-adjusted mortality rates are seriously inaccurate. Publication of hospital mortality rates misinforms the public about hospital quality.
Medical Care Research and Review. 1998;55:371-404.
![Page 18: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/18.jpg)
Interim conclusions
• Risk adjusted outcome (SMR) is unreliable, gameable, and does not correlate with other measures of quality
• Chance plays a large part in assigning rank particularly in smaller centers
• Risk adjusted outcome (SMR) does not always correlate with other measures of quality
![Page 19: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/19.jpg)
Does measuring and reporting quality have an effect on patients or physicians?
![Page 20: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/20.jpg)
Effect of publication of quality measures
• On patients– CABG patients unaware of outcome data– More likely to base decisions on anecdotal reports than data
• On physicians– Minimal effect on referral– Perception that it affected surgeon willingness to do high-risk
cases• On hospitals
– Depends on results – no one complains (or does anything about) low SMR
– High SMR – complaints about data qualityJAMA. 2000 Apr 12;283(14):1866-74.
![Page 21: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/21.jpg)
No effect of quality on market share
Baker DW, et al. The effect of publicly reporting hospital performance on market share and risk-adjusted mortality at high-mortality hospitals. Med Care. Jun 2003;41(6):729-740.
![Page 22: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/22.jpg)
Quasi-randomized trial of performance rating data in Wisconsin
24 hospitals Alliance Group 91 hospitals in Wisconsin
Confidential report
No report
Public report
Health Affairs. 2003 Mar-Apr;22(2):84-94
Caveats apply: public report hospitals were not a randomized group, administrative data reports, could not distinguish activities in response to reports from pre-existing activities
![Page 23: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/23.jpg)
Validity of performance ratings are in the eye of the beholder
• Perception of validity of performance rating and hospital performance correlated (r=0.13) but only in public reporting hospitals
2.00
2.75
3.50
4.25
5.00
1
PublicNo reportConfidential
P<0.05
Repo
rt m
ore v
alid
→
In general we don’t believe performance ratings, but we really don’t believe them if we look bad and other people will see the results
Health Affairs. 2003 Mar-Apr;22(2):84-94
![Page 24: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/24.jpg)
Public report hospitals more likely to have quality improvement activities
“Low” performing public report hospitals more likely to engage in QI
![Page 25: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/25.jpg)
• Cluster RCT in 86 hospitals in Canada• Early or delayed public release of cardiac report cards• No effect on 12 AMI or 6 CHF processes of care• But
– Increased quality improvement activities at early intervention– Improved (small absolute) mortality in AMI and CHF with
poor LV function
JAMA. 2009;302(21):2330-2337
![Page 26: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/26.jpg)
If public reporting of quality measures is questionnably effective – is it at least safe?• Improvements in CABG outcomes with public reporting
associated with decreased access– By high risk patients– By racial minorities perceived to be at higher risk
• Focus on process measures without selective review may lead to indiscriminate overuse– 90% of patients who did not receive colorectal screening were excluded
appropriately– Time to antibiotics in ED – PIP/TAZO for CHF?
• Perception that using complications as quality measure will lead to decreased screening for complication
JAMA. 2005 Mar 9;293(10):1239-44.
![Page 27: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/27.jpg)
What about critical care?
• Evidence based process measures are increasingly problematic
• Mortality is distal and much of what we do is palliative• VAP and CR-BSI are … VAP and CR-BSI
![Page 28: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/28.jpg)
Some issues may be hospital specific
+Night time discharges
![Page 29: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/29.jpg)
A roadmap for the future?
![Page 30: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/30.jpg)
Conclusions
• Domains of quality (access, structure, process, outcome, patient experience) do not converge and may be orthogonal
• The SMR (or ranking based on it) is misleading and is neither a reliable nor valid measure of quality
• Public reporting of outcome data can have harmful effects that must be considered but may be necessary to be effective
• What you measure is not nearly as important as what you do with what you measure
![Page 31: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/31.jpg)
Conclusions
• Only intensivists may care about the “quality” of intensive care – patients and payers probably care more about the quality of care overall
• The best (and safest) way to use quality measures is not known– Public or private reporting– Administrative data or locally collected– Explicit tools, coaching, data feedback in what form?– Pay for performance?
![Page 32: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/32.jpg)
Recommendations for measuring and improving ICU performance
• Quality should be assessed along multiple domains• Benchmarking absolute performance between ICUs
should be discouraged in favor of comparing improvement in performance
• If benchmarking, assess as outlier only after considering multiple risk adjustment tools and serial status
• Given relative weakness of evidence in critical care, ICUs should be allowed to select their own performance measures from a menu
![Page 33: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/33.jpg)
We cannot ignore quality data even if we disagree on its use and validity
![Page 34: Measuring the quality of intensive care - Critical Care · PDF file · 2017-11-10Measuring the quality of intensive care Gordon D. Rubenfeld, MD MSc ... If you can’t reduce your](https://reader034.vdocument.in/reader034/viewer/2022051722/5a9ec3177f8b9a8e178bd045/html5/thumbnails/34.jpg)
Slides and Conflict Vitae available on [email protected]