risk informed evaluation of patient safety training anthony d. slonim, md, drph vice president...
TRANSCRIPT
Risk Informed Evaluation of Patient Safety Training
Anthony D. Slonim, MD, DrPH
Vice President Medical AffairsCarilion Medical Center
Senior Staff, Departments of Medicine and PediatricsCarilion Clinic
Professor, Medicine and PediatricsVirginia Tech-Carilion School of Medicine
Objectives
• Decision-making science How do we normally make decisions?
• Picking up on Level 4…Can we train to improve our decision making results?
Decision Making Science
Bayes Theorem
P (A/B)=P(B/A) * P(A)
P(B)
Pattern Recognition
How many squares do you see?
Jumping to
conclusions too quickly.
Is there flashing in the
squares?Your
mind will play
tricks on you!
Decision-Making
Medical Decision-Making Process1. Perception/Data gathering (training-H and P, Labs, Rads)
• “Amber light” is showing
2. Interpretation (training-pattern recognition and probability)• “Amber light” means prepare to stop, maybe
3. Decision making (based on probability + experience)• Stop or go
4. Action taking (reflex/“gut level response”/programming)• Hit the brake or accelerator
Marx, D
Marx D and Slonim AD: Assessing patient safety risk before the injury occurs: An Introduction to Socio-Technical Probabilistic Risk Assessment. Quality and Safety in Healthcare 2003; 12 Suppl 2: 33-38.
Medical Decision-Making
Perception/Data GatheringInterpretatio
n
Decision-Making
Act
ion
Triage Nurse
ED Nurse
Physician
XGet Help:
Cardiology Consultation
Make a Dx and Treat
Do more testing-which
test?
(Pre-test probabilities)
Medical Decision-Making
Perception/Data GatheringInterpretatio
n
Decision-Making
Act
ion
Triage Nurse
ED Nurse
Physician
XGet Help:
Cardiology Consultation
Make a Dx and Treat
Do more testing-which
test?
(Pre-test probabilities)
Expert Decision Making:Practice, Practice, Practice
* Expert – pattern matching against large mental library, quick, accurate if confirm correct answer
* Novice – library is empty – slow, error prone process
* Certain Diagnoses are Favored- Frequent, Recent, Serious
* Heuristics – fixating on the wrong pattern
Pattern Recognition
Picking up on Level 4:Can we train for results?
Kirkpatrick’s Levels• Level I Reactions
• How well trainees liked training
• Level II Learning• The extent to which trainees understand and retain
principles, facts, and techniques
• Level III Behavior• The extent to which behavior changes as a result
of training
• Level IV Results• Impact of training on organizational criteria
Data AnalyticsData AnalyticsProcess Process
ImprovementsImprovementsChange Management
Improved Outcomes
Program Identification &
Prioritization
Elements of Quality Programs
Quality Functions
Research Education / Training
Why is there a safety problem ?
• Considerable variation in practice • Based on opinion or consensus • Evidence-based guidelines-unsupported• Failure to create fail-safe processes• Our providers may not know their work
• Policies and procedures
• We’re learning to work together• We’re not sure of the results we’re looking for
Process Analysis• Processes:
• A series of sequential steps governing interactions
• Between patients and providers• Between providers and providers
• Examples of process analysis techniques:• Root cause analysis-retrospective• HAACP (hazard analysis and crit control points)• FMEA (failure mode effects)• PI methodology
Low-frequency, High Impact Events
• Low frequency, high-impact events • Variable processes and practices
• Wrong site surgery • The abduction of children from hospitals• Deaths or major harm
• Process analysis helps to identify risk and prioritize interventions
• Decision support helps to guide decision making
Probabilistic Risk Assessment
• A hybrid between process analysis and decision support
• Identifies risk points and directs to interventions• Is hierarchical and probabilistic
• Allows disentanglement of patient level risks, provider level risks, and system level risks
• Assigns probabilities for prioritization of risk reduction strategies
• Includes sociotechnical components into the models
Conceptual Framework
Probabilistic Risk Assessment
The Institution
The Providers
The Patient
Quantitative Methods:
Qualitative Methods:
The Prospective Risk Model
Gate1
Patient Harm
Q:0.000146343
Gate2
Patient Harm - Type 1 - Failure
to Detect Initial Severity
Q:8.1143e-005
Gate20
Patient injury prior to
assessment - 1 event
Q:1.94e-008
Gate70
Type 1 harm for those at
risk of harm
Q:8.11236e-005
Gate23
Simulatanous patient, nurse, and
physician failures lead to harm
Q:4.68636e-005
Gate26
Patient escalation, but
patient harmed
Q:2e-006
Gate53
Nurse escalation, but
patient harmed
Q:3.016e-005
Gate57
Physician request for help
fails - 3 events
Q:2.1e-006
Gate71
Patient Harm - Type 2 - Failure to
Detect Critical Change in Condition
Q:6.52e-005
Gate12
Simulatanous patient, nurse, and
physician failures lead to harm
Q:3.4e-005
Gate14
Patient escalation, but
patient harmed
Q:1.48e-005
Gate46
Nurse escalation, but
patient harmed
Q:1.54e-005
Gate50
Physician request for help
fails - 2 events
Q:1e-006
The Top Three Risks
Training Evaluation
• Definition• The systematic collection of descriptive and
judgmental information necessary to make decisions related to instructional activities
• Ensures training• Meets its stated objectives• Changes trainee attitudes• Increases trainee knowledge• Develops trainee skills• Transfers results to the job
Training Evaluation
• Important variables to consider:• Organizational Factors• Individual Factors• Trainee Knowledge, Skills, and Attitudes• Training Transfer• Organizational Outcomes
Merging Kirkpatrick and ST PRA
• Socio-Technical Probabilistic Risk Assessment• Good for examining low base rate events (Six Sigma)• Models contributing causes
• Procedural tasks• Team tasks
• Identifies the impact of an intervention• Evidence base• Empirically based
• Adjust and test the model• Monte Carlo• Changes in the likelihood of outcomes
Traditional ApproachesTraditional Approaches
•Quick Wins•“Fire-fighting”•Burn-out / Fatigue•Difficult-to-Sustain, Short-Term Results
Typical Results
Imp
act
Time
Quality Fusion ApproachQuality Fusion Approach
Time
Imp
act
Quality Fusion Results
Typical Results
Example
What is Escalation?
• Failure to rescue associated with• Interpretation problems• Throughput problems
Put another way…
• When you do not realize the patient is in trouble OR you know the patient is in trouble, but you don’t respond as needed.
Common Course
Ideally, we track the illness. As the patient gets worse (line goes up), we respond. As the patient improves we adjust.
The patient conditionThe provider team response
==
Going Off Course
The defect rate in our modelis caused by failures to properly trackthe course of the illness.
The patient conditionThe provider team response
==
Never On Course
The provider team responseThe patient condition=
=
Sometimes, we’re off course right from the beginning and it’s difficult to get back on course.
The Prospective Risk Model
Gate1
Patient Harm
Q:0.000146343
Gate2
Patient Harm - Type 1 - Failure
to Detect Initial Severity
Q:8.1143e-005
Gate20
Patient injury prior to
assessment - 1 event
Q:1.94e-008
Gate70
Type 1 harm for those at
risk of harm
Q:8.11236e-005
Gate23
Simulatanous patient, nurse, and
physician failures lead to harm
Q:4.68636e-005
Gate26
Patient escalation, but
patient harmed
Q:2e-006
Gate53
Nurse escalation, but
patient harmed
Q:3.016e-005
Gate57
Physician request for help
fails - 3 events
Q:2.1e-006
Gate71
Patient Harm - Type 2 - Failure to
Detect Critical Change in Condition
Q:6.52e-005
Gate12
Simulatanous patient, nurse, and
physician failures lead to harm
Q:3.4e-005
Gate14
Patient escalation, but
patient harmed
Q:1.48e-005
Gate46
Nurse escalation, but
patient harmed
Q:1.54e-005
Gate50
Physician request for help
fails - 2 events
Q:1e-006
The Top Three Risks
Conclusions
• A focus on results helps providers and patients• Training on risk points can improve performance
• Leads to better results• Requires alterations in decision making• Enhances empiric data for better understanding training