positive predictive value of ahrq patient safety indicators in a national sample of hospitals...
TRANSCRIPT
POSITIVE PREDICTIVE VALUE OF AHRQ PATIENT POSITIVE PREDICTIVE VALUE OF AHRQ PATIENT SAFETY INDICATORS IN A NATIONAL SAMPLE OF SAFETY INDICATORS IN A NATIONAL SAMPLE OF
HOSPITALSHOSPITALSAcademyHealth Annual Research Meeting
June 9, 2008
Team presenter: Pat Zrelak PhD, CNRN, CNAA-BCTeam presenter: Pat Zrelak PhD, CNRN, CNAA-BC11
Team: Patrick RomanoTeam: Patrick Romano11; Garth Utter; Garth Utter11; Richard White; Richard White11; Dan Tancredi; Dan Tancredi11; ;
Ruth BaronRuth Baron11; Banafsheh Sadeghi; Sheryl Davies; Jeff Geppert; Banafsheh Sadeghi; Sheryl Davies; Jeff Geppert22. .
1 University of California Davis, Sacramento, CA. 1 University of California Davis, Sacramento, CA. 2 Battelle Memorial Institute, Elk Grove, CA.2 Battelle Memorial Institute, Elk Grove, CA.
Background of PSIs Background of PSIs
Set of quality indicators designed to identify potentially Set of quality indicators designed to identify potentially preventable problems that patients may experience as preventable problems that patients may experience as a result of contact with the health care systema result of contact with the health care system
Based on severity adjusted inpatient hospital Based on severity adjusted inpatient hospital discharge datadischarge data
Initially developed through a contract with UC-Stanford Initially developed through a contract with UC-Stanford Evidence-based Practice CenterEvidence-based Practice Center
Little is known about the criterion validity of the PSIs Little is known about the criterion validity of the PSIs across multiple hospitalsacross multiple hospitals
AHRQ PSI Validation Pilot Goals AHRQ PSI Validation Pilot Goals
Gather evidence on the criterion validity of the PSIs Gather evidence on the criterion validity of the PSIs based on medical record reviewbased on medical record review
Improve guidance about how to interpret & use the Improve guidance about how to interpret & use the datadata
Evaluate potential refinements to the specificationsEvaluate potential refinements to the specifications
Develop medical record abstraction toolsDevelop medical record abstraction tools
Develop mechanisms for conducting validation Develop mechanisms for conducting validation studies on a routine basisstudies on a routine basis
Positive Predictive Value Positive Predictive Value
The The positive predictive value positive predictive value or or post-test probability post-test probability is the is the proportion of flagged cases who actually had the event. proportion of flagged cases who actually had the event.
The Positive Predictive Value (PPV) can be further defined The Positive Predictive Value (PPV) can be further defined as: as:
Methods Methods
Retrospective cross-sectional study designRetrospective cross-sectional study design Volunteer sample of collaborative partners Volunteer sample of collaborative partners
– Facilitating organizations (e.g., Arizona)Facilitating organizations (e.g., Arizona)– Hospital systemsHospital systems– Individual hospitalsIndividual hospitals
Sampling based on administrative data Sampling based on administrative data Sampling probabilities assigned using AHRQ QI Sampling probabilities assigned using AHRQ QI
software to generate desired sample size software to generate desired sample size nationallynationally
Data collection methodsData collection methods
Each hospital identified chart abstractors Each hospital identified chart abstractors
Training occurred via webinarsTraining occurred via webinars
Medical record abstraction tools & guidelinesMedical record abstraction tools & guidelines– Pretested in the Sacramento areaPretested in the Sacramento area– Targeted the ascertainment of the event, risk Targeted the ascertainment of the event, risk
factors, evaluation & treatment, and related factors, evaluation & treatment, and related outcomesoutcomes
TimelineTimeline
■ 10 indicators- divided into 2 phases of 5 each
■ Phase I review- Training early 2007 Chart review 4 month process 4th Qtr 2005, 2006, & 1st Qtr 2007
■ Phase II review – Waiting OMB approval
Phase III –sensitivity determination
Patient Safety IndicatorsPatient Safety Indicators
Phase I Phase II
Accidental puncture and laceration
Foreign body left in during procedure
Selected infection due to medical care
Postoperative hemorrhage or hematoma
Postoperative pulmonary embolism or deep vein thrombosis
Postoperative physiologic and metabolic derangement
Postoperative sepsis Postoperative respiratory failure
Iatrogenic pneumothorax Postoperative wound dehiscence
Medical record sampleMedical record sample
Phase I Hospitals Sample
Accidental puncture and laceration 43 249
Iatrogenic pneumothorax 38 205
Postoperative PE/DVT 37 155
Selected Infection due to Medical Care 37 194
Postoperative Sepsis 33 164
Overall 47 967
N=249N=249
90% of events occurred during the hospitalization90% of events occurred during the hospitalization
10% were false positives 10% were false positives – 8% identified by abstractor as miscoded8% identified by abstractor as miscoded– 2% present on admission 2% present on admission
Accidental Puncture or LacerationAccidental Puncture or Laceration
Iatrogenic PneumothoraxIatrogenic Pneumothorax
N=205N=205
89% of events occurred during the hospitalization89% of events occurred during the hospitalization
11% were false positives 11% were false positives – 7% present or suspected at admission7% present or suspected at admission– 4% no documentation of event (miscoded) 4% no documentation of event (miscoded)
Postoperative DVT or PEPostoperative DVT or PE
N=155N=155
68% occurred during the index hospitalization68% occurred during the index hospitalization
32% were false positives32% were false positives– 16% had no surgical procedure performed in the 16% had no surgical procedure performed in the
OROR– 16% did not have a new post-op PE or DVT 16% did not have a new post-op PE or DVT
Selected Infection due to Selected Infection due to Medical CareMedical Care
N=194N=194
61% occurred during the index hospitalization61% occurred during the index hospitalization
39% were false positives 39% were false positives – 17% were present on admission17% were present on admission– 22% had no documentation of an infection22% had no documentation of an infection
Postoperative SepsisPostoperative Sepsis
N=164N=164
40% of the events occurred during the hospitalization40% of the events occurred during the hospitalization
60% were false positive 60% were false positive – 17% had no documentation of bacteremia, septicemia, 17% had no documentation of bacteremia, septicemia,
sepsis or SIRSsepsis or SIRS– 17% had infection (=14%) or sepsis (=3%) POA 17% had infection (=14%) or sepsis (=3%) POA – 25% did not have elective surgery25% did not have elective surgery
Summary of PPVsSummary of PPVsPreliminary estimatesPreliminary estimates
PSI PPV%
Accidental puncture or lacerationAccidental puncture or laceration 90%90%
Iatrogenic pneumothoraxIatrogenic pneumothorax 89%89%
Postoperative DVT/PEPostoperative DVT/PE 68%68%
Selected infections due to medical careSelected infections due to medical care 63%63%
Postoperative sepsisPostoperative sepsis 40%40%
Recognizing limitationsRecognizing limitations
Data elements available via chart reviewData elements available via chart review
Time constraints (burden on collaborators)Time constraints (burden on collaborators)
Inter-hospital variation Inter-hospital variation
Volunteer sample Volunteer sample
Further analysis of:Further analysis of:– Potential preventabilityPotential preventability– Management/treatment & patient outcomesManagement/treatment & patient outcomes– Inter-hospital variationInter-hospital variation
Evaluation of alternative ICD-9-CM specificationsEvaluation of alternative ICD-9-CM specifications– Can we improve PPV through numerator or denominator Can we improve PPV through numerator or denominator
changes?changes?
AHRQ QI Validation PilotAHRQ QI Validation PilotNext steps for analysisNext steps for analysis
Policy implications Policy implications
““Present at admission” data would substantially improve the PPVPresent at admission” data would substantially improve the PPV– Current PSI software has POA optionCurrent PSI software has POA option– October 2008 release will require POAOctober 2008 release will require POA
Coding changes are needed to enhance specificity and PPV in Coding changes are needed to enhance specificity and PPV in some areas some areas – AHRQ proposed codes for upper extremity and thoracic venous AHRQ proposed codes for upper extremity and thoracic venous
thromboses, and to distinguish acute from sub-acute thrombosesthromboses, and to distinguish acute from sub-acute thromboses– New codes for catheter-associated bloodstream infectionNew codes for catheter-associated bloodstream infection
Several PSIs have been endorsed by NQF Several PSIs have been endorsed by NQF – Accidental Puncture & Laceration (phase I)Accidental Puncture & Laceration (phase I)– Iatrogenic pneumothorax (phase I)Iatrogenic pneumothorax (phase I)– Foreign body (phase II) Foreign body (phase II) – Wound dehiscence (phase II) Wound dehiscence (phase II)
AHRQ project teamAHRQ project team– Mamatha Pancholi &Marybeth FarquharMamatha Pancholi &Marybeth Farquhar
Battelle training and support teamBattelle training and support team– Laura Puzniak & Lynne JonesLaura Puzniak & Lynne Jones
All of the validation pilot partners!All of the validation pilot partners!
AcknowledgmentsAcknowledgments