day 2: session ii presenter: jeffrey geppert, battelle ahrq qi user meeting september 26-27, 2005
TRANSCRIPT
Day 2: Session IIDay 2: Session II
Presenter: Jeffrey Geppert, BattellePresenter: Jeffrey Geppert, BattelleAHRQ QI User MeetingAHRQ QI User MeetingSeptember 26-27, 2005September 26-27, 2005
Methods for Creating Methods for Creating Aggregate Performance Aggregate Performance
Indices Indices
AHRQ QI User MeetingAHRQ QI User Meeting
September 27, 2005September 27, 2005
Jeffrey GeppertJeffrey Geppert
Battelle Health and Life SciencesBattelle Health and Life Sciences
OverviewOverview
Project objectivesProject objectives Why composite measures?Why composite measures? Who might use composite measures?Who might use composite measures? Alternative approachesAlternative approaches Desirable features of a compositeDesirable features of a composite Proposed approach for the AHRQ QIProposed approach for the AHRQ QI Questions & AnswersQuestions & Answers
Project ObjectivesProject Objectives
Composite measures for the AHRQ QI Composite measures for the AHRQ QI included in the National Healthcare included in the National Healthcare Quality Report and Disparities ReportQuality Report and Disparities Report
Separate composites for overall quality Separate composites for overall quality and/or quality within certain domains and/or quality within certain domains (e.g., cardiac care, surgery, avoidable (e.g., cardiac care, surgery, avoidable hospitalizations, diabetes, adverse hospitalizations, diabetes, adverse events)events)
A methodology that can be used at the A methodology that can be used at the national, state and provider/area levelnational, state and provider/area level
Project ObjectivesProject Objectives
FeedbackFeedback Does the proposed approach meet Does the proposed approach meet
user needs for a composite?user needs for a composite? What analytic uses should the What analytic uses should the
composite address?composite address? What are the important policy What are the important policy
issues?issues? How should the composite be How should the composite be
incorporated into the AHRQ QI incorporated into the AHRQ QI software?software?
Goals of Goals of National Healthcare ReportsNational Healthcare Reports
National LevelNational Level– Provide assessment of quality and Provide assessment of quality and
disparitiesdisparities– Provide baselines to track progressProvide baselines to track progress– Identify information gapsIdentify information gaps– Emphasize interdependence of quality and Emphasize interdependence of quality and
disparitiesdisparities– Promote awareness and changePromote awareness and change
State / Local / Provider LevelState / Local / Provider Level– Provide tools for self-assessmentProvide tools for self-assessment– Provide national benchmarksProvide national benchmarks– Promote awareness and changePromote awareness and change
Unique challenges to quality Unique challenges to quality reporting by statesreporting by states
States release comparative quality information in a political States release comparative quality information in a political environmentenvironment– Either must adopt defensible scientific methodology or make conservative Either must adopt defensible scientific methodology or make conservative
assumptionsassumptions
Examples of reporting decisions:Examples of reporting decisions:– Small numbers issuesSmall numbers issues– Interpretive issues (better/worse, higher/lower)Interpretive issues (better/worse, higher/lower)
Purchasers demanding outcomes and cost information from Purchasers demanding outcomes and cost information from statesstates
Why Composites?Why Composites?
Summarize quality across multiple Summarize quality across multiple measuresmeasures
Improve ability to detect quality Improve ability to detect quality differencesdifferences
Identify important domains and drivers of Identify important domains and drivers of qualityquality
Prioritize actionPrioritize action Make current decisions about future Make current decisions about future
(unknown) healthcare needs(unknown) healthcare needs Avoids cognitive “short-cuts”Avoids cognitive “short-cuts”
Why Not Composites?Why Not Composites?
Mask important differences and Mask important differences and relationships among components (e.g. relationships among components (e.g. mortality and re-admissions)mortality and re-admissions)
Not “actionable”Not “actionable” Difficult to identify which parts of the Difficult to identify which parts of the
healthcare system contribute most to healthcare system contribute most to qualityquality
Detract from the impact and credibility of Detract from the impact and credibility of reportsreports
Independence of componentsIndependence of components Interpretation of componentsInterpretation of components
Who Might Use Them?Who Might Use Them?
Consumers – To select a hospital either Consumers – To select a hospital either before or after a health eventbefore or after a health event
Providers – To identify the domains and Providers – To identify the domains and drivers of qualitydrivers of quality
Purchasers – To select hospitals in Purchasers – To select hospitals in order to improve the health of order to improve the health of employeesemployees
Policymakers – To set policy Policymakers – To set policy in order to in order to improveimprove the health of a populationthe health of a population
ExamplesExamples
““America’s Best Hospitals” (U.S. News & World America’s Best Hospitals” (U.S. News & World Report)Report)
Leapfrog Safe Practices Score (Leapfrog Safe Practices Score (27 procedures to reduce 27 procedures to reduce
preventable medical mistakespreventable medical mistakes)) NCQA, “America’s Best Health Plans”NCQA, “America’s Best Health Plans” QA Tools (RAND)QA Tools (RAND) Veteran Health Administration Veteran Health Administration (Chronic Disease Care (Chronic Disease Care
Index, Prevention Index, Palliative Care Index)Index, Prevention Index, Palliative Care Index) Joint Commission Joint Commission (heart attack, heart failure, pneumonia, (heart attack, heart failure, pneumonia,
pregnancy)pregnancy)
National Health Service (UK) Performance RatingsNational Health Service (UK) Performance Ratings CMS Hospital Quality Incentive Demonstration ProjectCMS Hospital Quality Incentive Demonstration Project
ExamplesExamples
Measures
Com
ponents
Com
posite
Goals Utility
Alternative ApproachesAlternative Approaches
ApproachApproach GoalGoal UtilityUtility
OpportunityOpportunity Appropriate careAppropriate care Volume of Volume of opportunitiesopportunities
BurdenBurden Minimize excess Minimize excess death/costsdeath/costs
Measures with most Measures with most excessexcess
Expected qualityExpected quality Better than referenceBetter than reference Lowest ratioLowest ratio
VariationVariation Better than referenceBetter than reference OutliersOutliers
Latent qualityLatent quality Reduce variationReduce variation Measures with Measures with greatest variationgreatest variation
Desirable FeaturesDesirable Features
Valid - Based on valid measuresValid - Based on valid measures Reliable – Improve ability to detect differencesReliable – Improve ability to detect differences Minimum Bias – Based on unbiased measuresMinimum Bias – Based on unbiased measures Actionable – Interpretable metricActionable – Interpretable metric Benchmarks or standardsBenchmarks or standards TransparentTransparent Predictive – Should guide the decision-maker Predictive – Should guide the decision-maker
on likely future quality based on current on likely future quality based on current information.information.
Representative – Should reflect expected Representative – Should reflect expected outcomes for populationoutcomes for population
Proposed ApproachProposed Approach
A modeling-based approachA modeling-based approach Latent quality – observed correlation in Latent quality – observed correlation in
individual measures is induced by variability individual measures is induced by variability in latent qualityin latent quality
Individual measures with highest degree of Individual measures with highest degree of variation have larger contribution to variation have larger contribution to compositecomposite
Theoretical interpretationTheoretical interpretation Consistent with goal of reducing overall Consistent with goal of reducing overall
variation in qualityvariation in quality
Proposed ApproachProposed Approach
Latent Quality
Com
ponent
Com
ponent
Com
ponent
Com
ponent
AdvantagesAdvantages
Avoids contradictory results with Avoids contradictory results with individual measures or the creation of individual measures or the creation of composites that may misleadcomposites that may mislead
Construction of the composite increases Construction of the composite increases the power of quality appraisalsthe power of quality appraisals
Allows for both measure-specific Allows for both measure-specific estimates and compositesestimates and composites
Allows for validation with out-of-sample Allows for validation with out-of-sample predictionprediction
Advantages (Continued)Advantages (Continued)
Hierarchical – for small numbers, the Hierarchical – for small numbers, the best estimate is the pooled average rate best estimate is the pooled average rate at similar hospitalsat similar hospitals
Allows for incorporation of provider Allows for incorporation of provider characteristics to explain between-characteristics to explain between-provider variability (e.g., volume, provider variability (e.g., volume, technology, teaching status)technology, teaching status)
Gives policymakers information on the Gives policymakers information on the important drivers of qualityimportant drivers of quality
Overview of AHRQ QIsOverview of AHRQ QIs
Prevention Prevention Quality Quality IndicatorsIndicators
Inpatient Quality Inpatient Quality Indicators Indicators
Patient Safety Patient Safety IndicatorsIndicators
Ambulatory care sensitive Ambulatory care sensitive conditionsconditions
Mortality following procedures Mortality following procedures Mortality for medical conditionsMortality for medical conditions Utilization of proceduresUtilization of procedures Volume of proceduresVolume of procedures
Post-operative complicationsPost-operative complications Iatrogenic conditionsIatrogenic conditions
ExamplesExamples
IQI Surgical Mortality
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
High Medium Low
Latent Quality
PANCREATIC RESECTION
AAA REPAIR
CABG
CRANIOTOMY
ExamplesExamples
IQI Medical Mortality
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
High Medium Low
Latent Quality
HIP REPLACEMENT
AMI
CHF
STROKE
GI HEMORRHAGE
HIP FRACTURE
PNEUMONIA
ExamplesExamples
Prevention Quality Indicators
0.000
0.002
0.004
0.006
0.008
0.010
0.012
0.014
0.016
High Medium Low
Latent Quality
DIABETES SHORT TRM COMPLICATN
DIABETES LONG TERM COMPLICATN
PEDIATRIC ASTHMA
COPD
PEDIATRIC GASTROENTERITIS
HYPERTENSION
CONGESTIVE HEART FAILURE
DEHYDRATION
BACTERIAL PNEUMONIA
URINARY INFECTION
ANGINA
DIABETES UNCONTROLLED
ADULT ASTHMA
LOWER EXTREMITY AMPUTATION
ExamplesExamples
PSI Postoperative Complications
0.000
0.002
0.004
0.006
0.008
0.010
0.012
0.014
0.016
0.018
0.020
High Medium Low
Latent Quality
POSTOP PHYSIO METABOLDERANGEMENT
POSTOP RESPIRATORY FAILURE
POSTOPERATIVE PE OR DVT
POSTOPERATIVE SEPSIS
ExamplesExamples
PSI Technical Adverse Events
0.000
0.020
0.040
0.060
0.080
0.100
0.120
0.140
0.160
High Medium Low
Latent Quality
DECUBITUS ULCER
FAILURE TO RESCUE
INFECTION DUE TO MEDICAL CARE
ExamplesExamples
PSI Technical Difficulty
0.000
0.001
0.001
0.002
0.002
0.003
0.003
0.004
0.004
0.005
High Medium Low
Latent Quality
IATROGENIC PNEUMOTHORAX
POSTOP HEMORRHAGE OR HEMATOMA
ACCIDENTAL PUNCTURE/LACERATION
Hierarchical ModelsHierarchical Models
Also referred to as smoothed rates or reliability-adjusted ratesAlso referred to as smoothed rates or reliability-adjusted rates Endorsed by NQF for outcome measuresEndorsed by NQF for outcome measures Methods to separate the within and between provider level Methods to separate the within and between provider level
variation (random vs. systematic)variation (random vs. systematic) Total variation = Within provider + Between provider Total variation = Within provider + Between provider (Between = (Between =
Total – Within)Total – Within)
Reliability (w) = Between / TotalReliability (w) = Between / Total– Signal ratio = signal / (signal+noise)Signal ratio = signal / (signal+noise)
Hierarchical ModelsHierarchical Models
Smoothed rate is the (theoretical) best predictor of Smoothed rate is the (theoretical) best predictor of future qualityfuture quality
Provides a framework for validation and forecastingProvides a framework for validation and forecasting Smoothed rate (single provider, single indicator) = Smoothed rate (single provider, single indicator) =
Hospital-type rate * (1 – w) +Hospital-type rate * (1 – w) +Hospital-specific rate * wHospital-specific rate * w
Multivariate versionsMultivariate versions– Other Years (auto-regression, forecasting)Other Years (auto-regression, forecasting)– Other Measures (composites)Other Measures (composites)– Non-persistent innovations (contemporaneous, Non-persistent innovations (contemporaneous,
nonsystematic shocks)nonsystematic shocks)
Outcomes and ProcessOutcomes and Process
Source: Landrum et. al. Analytic Methods for Constructing Cross-Sectional Profiles of Health Care Providers (2000)
Hierarchical ModelsHierarchical Models
Provider T
ype
Provider
PatientPolicy Quality
Policy and PredictionPolicy and Prediction
The best predictor of future performance is often The best predictor of future performance is often historical performance + structurehistorical performance + structure
The greater the reliability of the measure for a The greater the reliability of the measure for a particular provider, the more weight on historical particular provider, the more weight on historical performanceperformance
The less the reliability of the measure for a particular The less the reliability of the measure for a particular provider, the more weight on structureprovider, the more weight on structure
Volume often improves the ability to predict Volume often improves the ability to predict performance for low-volume providersperformance for low-volume providers
Other provider characteristics (e.g. availability of Other provider characteristics (e.g. availability of technology) do as welltechnology) do as well
Area characteristics (e.g., SES) do as wellArea characteristics (e.g., SES) do as well
Socio-Economic StatusSocio-Economic Status
The Public Health Disparities Geo-coding Project - The Public Health Disparities Geo-coding Project - Harvard School of Public Health (PI: Nancy Krieger)Harvard School of Public Health (PI: Nancy Krieger)
Evaluated alternative indices of SES (e.g. Townsend Evaluated alternative indices of SES (e.g. Townsend and Carstairs)and Carstairs)
Occupational class, income, poverty, wealth, education Occupational class, income, poverty, wealth, education level, crowdinglevel, crowding
Gradations in mortality, disease incidence, LBW, Gradations in mortality, disease incidence, LBW, injuries, TB, STDinjuries, TB, STD
Percent of persons living below the U.S. poverty linePercent of persons living below the U.S. poverty line– Most attuned to capturing economic depravationMost attuned to capturing economic depravation– Meaningful across regions and over timeMeaningful across regions and over time– Easily understood and readily interpretableEasily understood and readily interpretable
Socio-Economic StatusSocio-Economic Status
PQI #1 Diabetes Short-term Complication
0.0000
0.0002
0.0004
0.0006
0.0008
0.0010
0.0012
0.0014
0.0016
1 2 3 4 5 6 7 8 9 10
% Below U.S. Poverty Decile
LimitationsLimitations
Measures and methods difficultMeasures and methods difficult Restrictive assumptions on correlationRestrictive assumptions on correlation Correlations may vary by provider typeCorrelations may vary by provider type Requires a large, centralized data Requires a large, centralized data
sourcesource
ExpansionsExpansions
Flexibility in weighting the componentsFlexibility in weighting the components Empirical – domains driven entirely by Empirical – domains driven entirely by
empirical relationships in the dataempirical relationships in the data A priori – domains determined by clinical A priori – domains determined by clinical
or other considerationsor other considerations Combination – empirical when the Combination – empirical when the
relationships are strong and the relationships are strong and the measures precise, otherwise a priorimeasures precise, otherwise a priori
Welfare-driven CompositesWelfare-driven Composites
Measures
Com
posites
Meta-com
posites
Welfare-Driven CompositesWelfare-Driven Composites
Making current decisions about future Making current decisions about future needs – maximize expected outcomes, needs – maximize expected outcomes, minimize expected costsminimize expected costs
Policymaker focus – for a populationPolicymaker focus – for a population A provider focus A provider focus –– for their patientsfor their patients A employer focus – for their employeesA employer focus – for their employees A consumer focus – based on individual A consumer focus – based on individual
characteristicscharacteristics
AcknowledgmentsAcknowledgments
Funded by AHRQFunded by AHRQ Support for Quality Indicators II (Contract No. 290-04-0020) Mamatha Pancholi, AHRQ Project Officer Marybeth Farquhar, AHRQ QI Senior Advisor Mark Gritz and Jeffrey Geppert, Project Directors, Battelle Health
and Life Sciences
Data used for analyses:Data used for analyses:Nationwide Inpatient Sample (NIS), 1995-2000. Healthcare Cost and Nationwide Inpatient Sample (NIS), 1995-2000. Healthcare Cost and
Utilization Project (HCUP), Agency for Healthcare Research and Utilization Project (HCUP), Agency for Healthcare Research and QualityQuality
State Inpatient Databases (SID), 1997-2002 (36 states). Healthcare State Inpatient Databases (SID), 1997-2002 (36 states). Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and QualityResearch and Quality
AcknowledgementsAcknowledgements
We gratefully acknowledge the data organizations in participating states that contributed data to We gratefully acknowledge the data organizations in participating states that contributed data to HCUP and that we used in this study: the Arizona Department of Health Services; California Office HCUP and that we used in this study: the Arizona Department of Health Services; California Office of Statewide Health Planning & Development; Colorado Health & Hospital Association; Connecticut of Statewide Health Planning & Development; Colorado Health & Hospital Association; Connecticut - Chime, Inc.; Florida Agency for Health Care Administration; Georgia: An Association of Hospitals & - Chime, Inc.; Florida Agency for Health Care Administration; Georgia: An Association of Hospitals & Health Systems; Hawaii Health Information Corporation; Illinois Health Care Cost Containment Health Systems; Hawaii Health Information Corporation; Illinois Health Care Cost Containment Council; Iowa Hospital Association; Kansas Hospital Association; Kentucky Department for Public Council; Iowa Hospital Association; Kansas Hospital Association; Kentucky Department for Public Health; Maine Health Data Organization; Maryland Health Services Cost Review; Massachusetts Health; Maine Health Data Organization; Maryland Health Services Cost Review; Massachusetts Division of Health Care Finance and Policy; Michigan Health & Hospital Association; Minnesota Division of Health Care Finance and Policy; Michigan Health & Hospital Association; Minnesota Hospital Association; Missouri Hospital Industry Data Institute; Nebraska Hospital Association; Hospital Association; Missouri Hospital Industry Data Institute; Nebraska Hospital Association; Nevada Department of Human Resources; New Jersey Department of Health & Senior Services; Nevada Department of Human Resources; New Jersey Department of Health & Senior Services; New York State Department of Health; North Carolina Department of Health and Human Services; New York State Department of Health; North Carolina Department of Health and Human Services; Ohio Hospital Association; Oregon Association of Hospitals & Health Systems; Pennsylvania Health Ohio Hospital Association; Oregon Association of Hospitals & Health Systems; Pennsylvania Health Care Cost Containment Council; Rhode Island Department of Health; South Carolina State Budget Care Cost Containment Council; Rhode Island Department of Health; South Carolina State Budget & Control Board; South Dakota Association of Healthcare Organizations; Tennessee Hospital & Control Board; South Dakota Association of Healthcare Organizations; Tennessee Hospital Association; Texas Health Care Information Council; Utah Department of Health; Vermont Association; Texas Health Care Information Council; Utah Department of Health; Vermont Association of Hospitals and Health Systems; Virginia Health Information; Washington State Association of Hospitals and Health Systems; Virginia Health Information; Washington State Department of Health; West Virginia Health Care Authority; Wisconsin Department of Health & Department of Health; West Virginia Health Care Authority; Wisconsin Department of Health & Family Services. Family Services.
Questions & AnswersQuestions & Answers
Questions And AnswersQuestions And Answers