developing quality indicators when there is limited evidence: the example of injury care
DESCRIPTION
Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care. H. Thomas Stelfox, MD, PhD University of Calgary February 9, 2012. Objectives. Describe a quality indicator development process Review strategies for determining the need for quality indicators - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/1.jpg)
Developing Quality Indicators When There is Limited Evidence:
The Example of Injury Care
H. Thomas Stelfox, MD, PhDUniversity of Calgary
February 9, 2012
![Page 2: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/2.jpg)
Objectives
1. Describe a quality indicator development process– Review strategies for determining the need for
quality indicators– Review the value of environmental scans– Discuss the role of consensus methodologies
2. Review lessons learned from the process
![Page 3: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/3.jpg)
Objectives
1. Describe a quality indicator development process– Review strategies for determining the need for quality
indicators• Are there opportunities to improve care?
![Page 4: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/4.jpg)
16 yo Post Motor Vehicle Collision
![Page 5: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/5.jpg)
Decompressive Craniectomy
• Patient scheduled for OR
• Pre-op blood work ordered
• aPTT > 5 X normal
![Page 6: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/6.jpg)
Ordered Administered
![Page 7: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/7.jpg)
Problem – Quality of Care
• 98,000 die in US hospitals from error each year
• Canadian study of adverse events:– 7.5% of hospitalized patients– 1/3 of adverse events judged preventable
![Page 8: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/8.jpg)
Patient Safety Publications Before & After IOM Report
Stelfox et al. IQSH 2006
![Page 9: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/9.jpg)
Personal Experience with Error
Response Physicians(N=831)
Public(N=1207)
P-value
Error made in own or family member’s care
35% 42% <0.001
Health consequences
Serious 18% 24% <0.001 Minor 10% 13% 0.03 None 7% 5% 0.06Blendon et al. NEJM 2002
![Page 10: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/10.jpg)
Quality of Trauma Care• Medicine’s quality problem includes
trauma:– Half of all patients do not receive
recommended care
– Medical errors common in critically ill trauma patients
– 2.5% - 14% trauma deaths in hospital are preventable
![Page 11: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/11.jpg)
What is the Quality of Trauma Care?
![Page 12: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/12.jpg)
The Challenge – Quality Measurement & Improvement
“If you can’t measure it you can’t manage it”
Peter Drucker
![Page 13: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/13.jpg)
To Address This Challenge
Research program to develop population-based & evidence-
based indicators of quality of care in critically injured adult patients
![Page 14: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/14.jpg)
Research ProgramResearch Synthesis International Audit of
QI Practices
Potential Quality Indicators
Quality Indicators for Evaluation of Implementation
Evaluation of Implementation
Final Quality Indicators
Multi-Step QI Development Process
Years 1-2
Year 3
Years 4-5
![Page 15: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/15.jpg)
Objectives
1. Describe a quality indicator development process– Review strategies for determining the need for quality
indicators• Are there opportunities to improve care?• What quality indicators currently exist?
![Page 16: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/16.jpg)
Research Synthesis
To systematically review the literature about quality indicators (QI) for
evaluating trauma care
![Page 17: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/17.jpg)
Scoping Review Quality Indicators
Stelfox et al. Arch Surg 2010
![Page 18: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/18.jpg)
Pediatric Patients
• Deficiencies in care 8%-45% pts.
• 6%-32% deaths judged preventable
• Need for pediatric specific measures
• No evaluations of validity or reliability
Stelfox et al. Crit Care Med 2010
![Page 19: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/19.jpg)
Adult PatientsBest Indicators
• Preventable death
Indicators To Avoid• Time to craniotomy in TBI
Potential Indicators• Complications• Non-fixation of femur fracture• Scene time• ↓ LOC & airway management• Unplanned return to OR• Time to emergency laparotomy• ↓ LOC & time to CT head• Readmission to hospital• Time to basic diagnostics• Reintubation <48 hrs• Preventable morbidity• Missed injury• Statistically unexpected death
Stelfox et al. Crit Care Med 2011
![Page 20: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/20.jpg)
Impact on Quality of CareSource Outcome Result
Before After
Chadbunchachai et al. 200155
Hospital mortalityPreventable deathTreatment pitfalls
Pitfalls → mortality
2.6%3.0%
43 per 1000 patients29 per 1000 patients
2.4%2.0%
23 per 1000 patients12 per 1000 patients
Chadbunchachai et al. 200354
Hospital mortalityPreventable deathTreatment PitfallsPitfalls → mortality
2.4%2.0%
23 per 1000 patients12 per 1000 patients
1.4%1.3%
17 per 1000 patients10 per 1000 patients
Ruchholtz et al. 200256
Assessment criteria
Hospital mortality 17%
Improved both centers 25%*
Improved one centre 45%*
Improved neither centre 30%11%
*p<0.05
Stelfox et al. Crit Care Med 2011
![Page 21: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/21.jpg)
What does the Evidence Tell Us?
• QI literature for evaluating trauma care:
– Adults more than children
– Acute care more than post-acute care
– Supported by limited scientific evidence• A few promising indicators• May be associated with improved care
![Page 22: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/22.jpg)
Objectives
1. Describe a quality indicator development process– Review strategies for determining the need for quality
indicators– Review the value of environmental scans
• How are quality indicators used?
![Page 23: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/23.jpg)
Environmental Scan of Trauma Centres
• Goal – describe real world trauma centre performance improvement activities
• U.S., Canada, Australia & New Zealand
• Electronic surveys to 328 trauma centres– 249 centres responded (76%)
• Follow up interviews of 76 centres
![Page 24: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/24.jpg)
Quality Indicators
Quality Indicators USA(N=198)
CND(N=35)
AUS(N=16)
Structure indicators 88% 66% 75%
Process indicators 98% 91% 75%
Outcome indicators 99% 91% 88%
*
*
** p<0.05
Stelfox et al. Ann of Surg 2012 in press
![Page 25: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/25.jpg)
Performance Improvement Practices
PI Practices USA(N=198)
CND(N=35)
AUS(N=16)
M & M Conference 97% 94% 88%
Quality Audits 93% 89% 88%
Report Cards 53% 26% 31%Internal Benchmark 81% 63% 69%Extern Benchmark 81% 37% 63%
* p<0.05
***
![Page 26: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/26.jpg)
Phase of Care Evaluated by QIs
Prehosp
ital
Hospita
l
Post-Hosp
ital
2° Prev
entio
n
Prehosp
ital
Hospita
l
Post-Hosp
ital
2° Prev
entio
n
Prehosp
ital
Hospita
l
Post-Hosp
ital
2° Prev
entio
n0
5
10
15
20
25
30
USACNDAUS
No. I
ndic
ator
s pe
r Cen
tre
*
* *
Structure Process Outcome
*
![Page 27: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/27.jpg)
IOM Aims Evaluated by QIs
Safe
Effecti
ve
Patien
t-Cen
tered
Timely
Efficien
t
Equitable
0
10
20
30
40
USACNDAUS
No.
Qua
lity
Indi
cato
rs p
er C
entre
* p<0.05
*
* * *
![Page 28: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/28.jpg)
Do Trauma Centres use the Same Indicators?
Santana et al J Trauma 2012 in press
![Page 29: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/29.jpg)
10 Most Common Quality IndicatorsQuality Indicator Percentage of Centres (n=247)Appropriate admission service/MD 53
Hospital mortality 43
Secure airway in comatose patient 40
Time to laparotomy 39
Scene time 38
Time to craniotomy 36
Length of stay 35
Reintubation 34
Non-surgical gunshot wound management 32
Unplanned return to operating room 30
Santana et al J Trauma 2012 in press
![Page 30: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/30.jpg)
How did you decide on your QIs?• Accreditation
“A lot of it is driven by formal regulations as what is expected of us as a trauma center either by
our State authority or the ACS.”
• Local Issues“We had cases with certain issues that were recurrently popping up, and these issues were
chosen for monitoring.”
![Page 31: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/31.jpg)
How have PI activities impacted your trauma program?
• Culture of quality“The whole program is driven by safety and quality.”
• Standardized care“Set standards so we have clear expectations on how
trauma patients should be care for.”
• Improved processes & outcomes of care“It has improved it tremendously, we keep getting better
at what we are doing, it helps us drive down our morbidity and mortalities.”
![Page 32: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/32.jpg)
How has your trauma program evaluated its PI activities?
• No Evaluation“No it’s hard – you can take forever – you
can evaluate the evaluation of the evaluation”
• Informal Evaluation“Nothing formally, but we meet once a week
and go through what’s working, what’s not working, what we want to change.”
![Page 33: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/33.jpg)
What can a program do to improve its quality of trauma care?
• Better measures: “It’s a vicious circle. If we had the stats to prove we make a difference it would be easier to get funding. But how do you get there?”
• Better benchmarking: “In an ideal world I would love to benchmark outcomes, but people don’t use the same data dictionary and it is hard to benchmark when not comparing apples to apples”
• More concurrent: “My dream would be that we would track our indicators in real time, review charts within a day or two of them being flagged and fix problems while still active.”
![Page 34: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/34.jpg)
Summary of Environmental Scan Observations
• Trauma centres spend a lot of energy and time on quality measurement & improvement
• Significant variation exists in how trauma centres measure & manage the quality of care they deliver
• Significant gaps exist within the observed quality improvement processes
![Page 35: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/35.jpg)
What Next?
![Page 36: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/36.jpg)
Objectives
1. Describe a quality indicator development process– Review strategies for determining the need for quality
indicators– Review the value of environmental scans– Discuss the role of consensus methodologies
• How to bridge the gap between the evidence base and the need for developing applied measures?
![Page 37: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/37.jpg)
Quality Indicator Development
Expert Panel ReviewRound 1
Potential Quality Indicators
Quality Indicators for Evaluation of Implementation
Expert Panel ReviewRound 2
Expert Panel Workshop
![Page 38: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/38.jpg)
Expert Panel
![Page 39: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/39.jpg)
Sample Quality Indicator
![Page 40: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/40.jpg)
Rating Scale
![Page 41: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/41.jpg)
Quality Indicator Development Process74 •Round 1: 22 accepted, 52 neutral, 0 rejected, 5 proposed
57 •Round 2: 10 accepted, 49 neutral, 1 rejected, 2 proposed
84 •Workshop: 43 accepted, 14 neutral, 27 rejected, 24 merged
43 •Working Groups: 43 merged into 33, mini SRs for 33 QIs
33 •Final Review: 32 accepted, 1 rejected
32
![Page 42: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/42.jpg)
Research ProgramResearch Synthesis International Audit of
QI Practices
Potential Quality Indicators
Quality Indicators for Evaluation of Implementation
Evaluation of Implementation
Final Quality Indicators
Multi-Step QI Development Process
Years 1-2
Year 3
Years 4-5
![Page 43: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/43.jpg)
Stepping BackAre we on target?
Gruen et al. BJS 2012
![Page 44: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/44.jpg)
Indicators Developed
Phase of care Structure Process Outcome
Prehospital 3 5 1
Hospital 2 14 5
Posthospital X 1 2
2˚ Prevention X 1 1
* 1 to 2 indicators per cell of our conceptual model
![Page 45: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/45.jpg)
Objectives
1. Describe a quality indicator development process– Review strategies for determining the need for quality
indicators– Review the value of environmental scans– Discuss the role of consensus methodologies
2. Review lessons learned from the process
![Page 46: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/46.jpg)
Lesson #1Clear Purpose & Goals for the Quality
Indicator
“...I would submit that the end that we seek here is to try and draw the line between the indicator in question and its measurability and whether or
not the patients are going to have better outcomes as a consequence”
![Page 47: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/47.jpg)
Lesson #2Incorporating Evidence, Expertise & Patient
Perspectives
“The timing I’m not sure about, there’s no class one evidence to support it … At the same time I
think there is some value added to early management … prevention of certain secondary
complications … that’s why I advocate for it.”
![Page 48: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/48.jpg)
Lesson #3Contextual Considerations & Variation
“…again this is going to be a local guidelines issue because different hospitals and systems will triage to their resuscitation room differently
depending on their volumes, …[this difference is] just going to influence the wording [of the
indicator ]…”
![Page 49: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/49.jpg)
Lesson #4Data Collection & Management
“… anything that encourages the whole trauma system to … improve the data that is submitted … if we’re going to have a measure that forces
the issue that we need better data collection, I’m all for it .”
![Page 50: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/50.jpg)
One Additional Lesson
Take Advantage of Potential Gaps / Opportunities Identified During
the Process
![Page 51: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/51.jpg)
Missed Opportunity
“I discussed the paper with one other editor and we felt that it would be substantially strengthened if it had evidence of program outcomes of
these included”
Dr. Ginny BarbourChief Editor, PLoS Medicine
![Page 52: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/52.jpg)
Captured Opportunities• No patient-centered quality indicators
Research program to develop patient-centered indicators
• Exclusive focus on high income countries Initiated an evaluation of trauma quality
improvement in low & middle income countries
![Page 53: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/53.jpg)
![Page 54: Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care](https://reader036.vdocument.in/reader036/viewer/2022062520/56815ab5550346895dc868f3/html5/thumbnails/54.jpg)
AcknowledgementsMentors• Sharon Straus• David Bates• Bill Ghali• John Kortbeek• Tom Noseworthy• Don Redelmeier
Funding Agencies• CIHR• Alberta Innovates, Health
Solutions
Partners• Trauma Association of Canada• National Trauma Registry
Collaborators• Avery Nathens • Russell Gruen• Andrew Kirkpatrick• Morad Hameed• Anna Gagliardi• Sean Bagshaw• John Tallon
Research Team• Barbara Artiuch• Nancy Clayden• Jamie Boyd• Farah Khandwala• Maria Santana