aapm pqi intro - american association of physicists in...

7/28/2012

1

Methods in CompletingMethods in Completing Performance Quality Improvement (PQI)

Jennifer L Johnson, Karen BrownG ff Ibb tt T dd P li kiGeoffrey Ibbott, Todd Pawlicki

AAPM 54th Annual MeetingCharlotte, NC

Professional Symposium4:30PM - 6:00PM

Disclosures

• None of the presenters have conflicts of i t t t di linterest to disclose.

PQI

• After this course attendees will be better bl table to– Identify and define a PQI project– Identify and select measurement

methods/techniques for use in the PQI project– Describe example(s) of completed projectsp p p j

Outline

• Introduction and the ABRPDSA• PDSA

• Resources and Tools– Incident learning systems, RCA– FMEA– Control Chart– Fishbone– Process Map– Pareto

ABR$PQI$Update$July$30,$2012

Geoffrey$S.$Ibbo>,$Ph.D.

Slides$from$David$Laszakovits,$M.B.A.,Milton$Giberteau,$M.D.,$and

James$Borgstede,$M.D.

Topics

1.$$MOC$AtPAPGlance2.$$PracQce$Quality$Improvement3.$$Public$ReporQng$and$ConQnuous$CerQficaQon$$

3

WHO IS ABMS?

• ABMS sets the standards for the certification process to enable the delivery of safe, quality patient care

• ABMS is the authoritative resource and voice for issues surrounding physician certification

• The public can visit certificationmatters.org to determine if their doctor is board certified by an ABMS Member Board

4

WHAT IS ABMS MOC™?

• A process designed to document that physician specialists, certified by one of the Member Boards of ABMS, maintain the necessary competencies to provide quality patient care

• ABMS MOC promotes continuous lifelong learning for better patient care

ABMS of the Future

More robust

More legislatively active

Continuous MOC rather than 10 year cycles

Involvement and promotion of institutional MOC

Significant presence of primary care boards in ABMS governance

Competition from rogue organizations for stature

MOC$Components

Part$II:$$Lifelong$Learning$and$SelfPAssessment! Category$1$CME$and$Self$Assessment$Modules$

(SAMs)

Part$I:$$Professional$Standing!$$State$Medical$Licensure

Part$III:$$CogniQve$ExperQse! Proctored,$secure$exam$

Part$IV:$$PracQce$Performance$$$! PracQce$Quality$Improvement$(PQI)

Topics

1.$$MOC$AtPAPGlance2.##Prac(ce#Quality#Improvement3.$$Public$ReporQng$and$ConQnuous$CerQficaQon$$

PQI$EvoluQon

PQI$EssenQal$Elements

! Select$project,$metric(s),$and$goal! Collect$baseline$data$! Analyze$data! Create$and$implement$improvement$plan! RePmeasure! SelfPreflecQon

The$Quality$Improvement$Process

• Identify area needing improvement• Devise a measure• Set a goal

•Carry out the measurement

plan ! Collect data

• Develop an improvement plan• Implement for cycle #2

• Analyze the data

•Compare to goal

•Root Cause Analysis

ABR$Individual$and$Group$

PQI$Templates*

*Templates$include$all$essenQal$elements$needed$

to$comply$with$ABR$“meaningful$parQcipaQon”$

requirements

Group$PQI$Criteria

! Group$consists$of$2$or$more$ABR$diplomates! Group$Project$Team$Leader$$designated

• Team$organizaQon,$meeQngs$and$record$keeping• Must$document$team$parQcipaQon

! Project$may$be$group$designed,$societyPsponsored,$or$involve$a$registry

! Requires$at$least$3$team$meeQngs:$$• Project$organizaQon$meeQng• Data$and$root$cause$analysis$meeQng• Improvement$plan$development

Individual$ParQcipant:$$“Meaningful$ParQcipaQon”

! Individual$diplomate$MOC$PQI$credit$requires:• Documented$a>endance$at$>$3$team$meeQngs• PreparaQon$of$a$personal$selfPreflecQon$statement$

describing$the$impact$of$the$project$on$the$group$pracQce$and$paQent$care

• A>estaQon$on$ABR$Personal$Database$(PDB)$• Access$to$project$records$in$the$event$of$an$ABR$

MOC$audit

Changes$in$PQI$A>estaQon

DEMO$P

AGE

A>estaQon$conQnued…

DEMO$P

AGE

DEMO$P

AGE

Topics

1.$$MOC$AtPAPGlance2.$$PracQce$Quality$Improvement3.##Public#Repor(ng#and#Con(nuous#Cer(fica(on##

Specialty Board Certification

State Medical Licensure Quality

Organizations

Private Not-for-Profits

CertificationMOC

Maintenance of Licensure (MOL)

Quality Organizations

Private Not-for-Profits

Changing$Landscape

! Relevance$of$ABMS/ABR$cerQficaQon$must$be$demonstrated$to$the$public,$payers$and$the$government

$!$Medicine$is$experiencing$a$fusion$of$economics,$quality,$safety$and$reimbursement,$so$we$must$work$together$to$effecQvely$project$and$promote$our$specialty$for$the$benefit$of$our$paQents

!Accountability$and$transparency$remain$the$watchwords$for$the$new$millennium

Timeline$Leading$to$ABMS$Public$ReporQng

! March&2009:&ABMS&BOD&adopted&a&standards&document&that&included&a&call&for&ABMS&to&make&info&about&cer>ficate&status&dates&and&MOC&par>cipa>on&status&available&to&the&public

! June&of&2010:&ABMS&BOD&approved&a&twoEpart&resolu>on:& &(1)&approved&public&display&of&MOC&par>cipa>on&by&ABMS&&&&&

&&&&&&&star>ng&Aug&2011&& &(2)&MOC&par>cipa>on&status&reported&using&three&primary&

&&&&&&&designa>ons:!&“Mee>ng&the&Requirements”&of&MOC!“Not&Mee>ng&the&Requirements”&of&MOC!“Not&Required&to&Par>cipate”&in&MOC&(Life>me&Cer>ficates)&

ABMS$Public$ReporQng$cont…

! May&2011:&&&ABMS&MOC&Mee>ng:&Na>onal&Creden>alers&appeared&as&guests&and&stated&interest&in&some&way&to&verify&MOC&par>cipa>on&through&ABMS.

! It&was&recognized&that&the&boards&needed&>me&to&create&communica>ons&and&reach&out&to&their&diplomates,&some&of&whom&would&likely&want&to&enroll&in&MOC.&

! June&2011:&&ABMS&offered&extensions&of&one&year&to&boards&who&wanted&more&>me&to&for&communica>on

! ABR’s&request&for&the&maximum&oneEyear&extension&was&granted.

! If&not&us&then&who:

About$Public$ReporQng ABR$Response$to$ABMS$Public$ReporQng$Requirements

! ABR$online$verificaQon$of$board$eligibility$and$MOC$parQcipaQon$statuses$in$coordinaQon$with$ABMS$reporQng

! Link$from$ABMS$site$to$ABR$site$for$further$$$clarificaQon$on$various$statuses

! Diplomate$lookPup$tool! Immediate,$current$verificaQon$status

ConQnuous$CerQficaQon

! CerQficate$will$no$longer$have$“valid$through”$date$–$instead$conQnuing$cerQficaQon$will$be$conQngent$on$meeQng$MOC$requirements

! Annual$lookPback$used$to$determine$MOC$parQcipaQon$status.$

! No$change$in$MOC$requirements$or$fees

How$does$it$work?MOC&Year LookEback&date& Element(s)&Checked1

2013 3/15/2014 Licensure0

2014 3/15/2015 Licensure

2015 3/15/2016 Licensure,0CME,0SAMs,0Exam,0and0PQI0

2016& 3/15/2017 Licensure,0CME,0SAMs,0Exam,0and0PQI

2017& 3/15/2018 Licensure,0CME,0SAMs,0Exam,0and0PQI

2018 3/15/2019 Licensure,0CME,0SAMs,0Exam,0and0PQI

20XX 3/15/20XX Licensure,0CME,0SAMs,0Exam,0and0PQI

Element Compliance&RequirementLicensure At0least0one0valid0state0medical0license

CME At0least0750Category010CME0in0previous030yearsSAMs At0least060SAMs0in0previous030years

ExamPassed0any0ABR0CerLfying0or0MOC0exam0in0previous0100years

PQI Completed0at0least010PQI0project0in0previous030years

1&Status&Check&for&“Mee>ng&Requirements”

Advantages0of0ConLnuous0CerLficaLon! If0you0have0two0or0more0LmeSlimited0

cerLficates,0they0are0synchronized.! The0number0of0CME0and0SAMs0you0can0count0

per0year0is0unlimited! You0may0take0the0MOC0exam0at0any0Lme,0as0long0

as0the0previous0MOC0exam0was0passed0no0more0than0100years0ago

! BuiltSin0“catchSup”0period0of0one0year0–0sLll0cerLfied

! Aligns0reporLng0more0closely0with0CMS,0TJC,0credenLaling0and0state0licensing0boards

QuesQons?

Thank0You!

PDSA Quality Improvement

Methodology

Plan

Do

Study

Act

Simple Versatile

Powerful

Identify a project

Plan

Ask questions

Make predictions

Set goals

Identify data to be collected

Do

Carry out the plan

Study

Analyze

Compare

Summarize

Act

What changes will be made?

Group Projects

• Individual participation • Access to project materials • Group structure • Meeting minutes

…to be successful at improvement, it takes the will to improve, ideas for improvement, and the skills to execute the changes.

The Improvement Guide: A Practical Approach to Enhancing Organizational Performance

Contact Information

Karen Brown, MHP, CHP, DABR Penn State College of Medicine Milton S. Hershey Medical Center Email: [email protected] P: 717-531-5027

mailto:[email protected]

Resources

• Langley, Gerald J.; Moen, Ronald D.; Nolan, Kevin M.; Nolan, Thomas W.; Norman, Clifford L.; Provost, Lloyd P. (2009-06-03). The Improvement Guide: A Practical Approach to Enhancing Organizational Performance (JOSSEY-BASS BUSINESS & MANAGEMENT SERIES). Wiley Publishing. Kindle Edition.

• The American Board of Radiology. Maintenance of Certification Part IV: Practice Quality Improvement (PQI) 2012. http://www.theabr.org/sites/all/themes/abr-media/PQI_2012.pdf

• Heath, Chip; Heath, Dan (2007-01-02). Made to Stick: Why Some Ideas Survive and Others Die. Random House, Inc.. Kindle Edition.

http://www.theabr.org/sites/all/themes/abr-media/PQI_2012.pdf

Image Resources

• iStockphoto www.istockphoto.com • photoXpress www.photoxpress.com • Everystockphoto www.everystockphoto.com

http://www.istockphoto.com/

http://www.photoxpress.com/

http://www.everystockphoto.com/

7/28/2012

1

PQI – Control Charts, Event Reporting and FMEAReporting, and FMEA

Todd Pawlicki

Elements of PQI Projects

• Relevance to patient careR l t di l t ' ti• Relevance to diplomate's practice

• Identifiable metrics and/or measurable endpoints• Practice guidelines and technical standards• An action plan to address areas for improvement

– Subsequent remeasurement to assess progress and/or improvement

http://www.theabr.org/moc-ro-comp4

Error Management

• Three approaches to error management

– Reactive, Proactive, Prospective

– Incident learning systems• Reactive & Proactive

– Failure Modes & Effects Analysis• Prospective

Basis for Understanding Statistical Process Control

Tolerance Limits

Accept

Action Limits

Action Limits

Target

7/28/2012

2

Control Charts: Individual Values

ual

sXmR

Sample number or Time

Indi

vidu

valu

eXmRchart


Mov

ing

Ran

gemRchart

XmR Chart

1 ∑3 mRUNPL x= + ⋅

Indi

vidu

al

valu

es

1x xN

= ∑31.128

UNPL x +


(n = 1, and use d2 for n = 2)

31.128mRLNPL x= − ⋅

Two Example Control Charts

• Clinical specifications– Set process

requirements

• Control chart limits– Quantify process

performance

Pawlicki, Yoo, Court et al. Radiother Oncol 2008

Event Reporting System

http://www.ihe.ca/publications/library/archived/a-reference-guide-for-learning-from-incidents-in-radiation-treatment

7/28/2012

3

Investigation

• All incidents are investigatedD th d i it f i ti ti d d• Depth and priority of investigation depends on – Severity of incident– Frequency of occurrence

• Assessment – Impact and process domain(s)

• Report – Causal analysis, corrective actions, and follow-up

Choosing A Project From Events• By type

– Clinical, occupational, operational, environmental, security/other

• By impact– Critical, major, serious, minor– Near miss

• By domain– Where in the Radiation Treatment process did the

incident occur?

Corrective Action

• Actions to address causesTarget to improve system performance– Target to improve system performance

• Integrate with other business processes– Capital budgeting– Change management– Training

A i t i di id l• Assign to individuals• Follow up reports / data

Learning

• Lessons learned are distilled and communicatedS i ibl f i ti• Supervisor responsible for communication

• Quality Assurance Committee responsible for overall review of incident patterns

• Communication requirements depending on incident severity– Stop the press vs. Dept email

7/28/2012

4

Example: Forgetting bolus Example: Forgetting bolus

Control Chart

FMEA

• Failure Modes and Effects Analysis

– Provides a structured way of prioritizing risk reduction strategies.

– Helps to focus efforts aimed at minimizing adverse outcomes.

FMEA – Background

• HistoryDeveloped by the Aerospace industry ( 1960s)– Developed by the Aerospace industry (~1960s)

• In the electromechanical age

– Widely applied in automotive and airline industries

• Use– Most effective when applied before a design isMost effective when applied before a design is

constructed – Primarily a prospective tool

7/28/2012

5

FMEA – Vocabulary

• Failure Mode: How a part or process can fail to meet specificationsmeet specifications.

• Cause: A deficiency that results in a failure mode; sources of variation.

• Effect: Impact on customer if the failure mode is not prevented or corrected.

Risk Priority Number (RPN)

Risk Priority Number =

Severity

X

Probability of Occurrence

X

Probability of NOT being detected

FMEA – Metrics

• Occurrence (O) Probability that the failure mode occurs– Probability that the failure mode occurs

• Severity (S) – Severity of the effect on the final outcome resulting

from the failure mode if it is not detected

• Lack of Detectability (D) – Probability that the failure will NOT be detected

1) MD consult2) H&P

3) Database entry4) Prescription

dictated

16) Plaque insert17) Patient survey18) Room survey

No input/control Responsible for operation

9) Sources Ordered10) Sources inventoried11) Sources delivered to Radiation Oncology

12) Sources inventoried into Rad Onc

Processes leading to LDR Implant

Successful LDRImplant

Initial Patient Consult Source Acquisition Implant

Plaque PreparationTreatment Plan

13) Calibration check14) Assembly

15) Sterilization

5) Source type selected6) Hand

calculation 7) Treatment plan8) Source activity

selectedSlide courtesy of Dan Scanderbeg

7/28/2012

6

Process StepPotentialFailure Mode

Effect of Failure Mode O rank

S RankD rank RPN

5) Source type selected

Wrong source type selected

Wrong dose delivered 7 9 9 567

6) Hand calc Wrong depth or duration

Wrong dose delivered 8 9 7 504duration delivered

7) Tx Plan Wrong depth or duration


8) Source activity selected

Wrong source activity selected


9) Order placed Wrong activity ordered


14) Assembly Improper equipment used

Wrong dose delivered/geographic miss

7 9 9 567

I14) Assembly Improper construction Seeds migrate 6 9 4 216

15) Sterilization Improper handling Seeds migrate 5 9 4 180

Slide courtesy of Dan Scanderbeg

Process Step PotentialFailure Mode

Effect of Failure Mode O rank S Rank D rank RPN

5) Source type selected

Wrong source type selected


List sorted in order of RPN (high to low)RPN ~ 550 used as cutoff

7) Tx Plan Wrong depth or duration


14) Assembly Improper equipment used

Wrong dose delivered/geographic miss

7 9 9 567

6) Hand calc Wrong depth or duration


8) Source activity selected

Wrong source activity selected


Wrong activity Wrong dose9) Order placed Wrong activity ordered


14) Assembly Improper construction Seeds migrate 6 9 4 216

15) Sterilization Improper handling Seeds migrate 5 9 4 180


Case Identifier

Type of Case

Physician


Example of data trackingPhysician

Scheduled Time of Case

Scheduled Duration of Case

Physics Start Time for Case

Physics Stop Time for Case

g

Use web-based form to gather data into Excel-type form for

Paperwork & Notes

type form for analysis.

Example of Analysis

• Over 3 weeks – physics brachy schedule was logged using Google Documents


logged using Google Documents

• Results– 20 of 26 (77%) of cases finished later than scheduled– Cases finished later than scheduled time

• Max = 78 minA 31 5 i• Ave = 31.5 min

– 8 occurrences of cases booked back-to-back– 4 occurrences of cases doubled booked

7/28/2012

7

Next Steps

• Create intervention to improve processes

• Document results

Summary

• Control charts for analysis and deciding when to act

• Event Recording System to identify issues

• FMEA to prioritize effortFMEA to prioritize effort

7/28/2012

1

PQI – Fishbone, Process Maps and Pareto Charts

Jennifer L Johnson, MS, MBA

Fishbone Diagram

• Cause-and-effect diagram, Ishikawa didiagram

• Identifies many possible causes for an effect or problem– Brainstorming

S t id i t f l t i– Sorts ideas into useful categories

Tague, N R. The Quality Toolbox 2005

Fishbone DiagramENVIRONMENT

EQUIPMENTPEOPLE

Fishbone Diagram

• Cause enumeration diagramB i t th t d t i – Brainstorm causes, then group to determine headings

• Process fishbone– Develop flow diagram of process steps (<10)– Fishbone each process step

• Time-delay fishbone– Allow people to add over time (1-2 weeks)


7/28/2012

2

Process Maps

• Graphical representation of sequence of t k d ti iti f t t t fi i htasks and activities from start to finish– Flow of inputs, resources, steps, and processes

to create an output– May be color-coded by participant(s)– Value-added vs. nonvalue-added stepsp

• Single diagram or hierarchy of diagrams


Process Maps

• “As-is” – depicts actual, current process in lplace

• “To-be” – depicts future after changes and improvements

• Difference: value-added vs. nonvalue-added stepsadded steps


Perks et al. IJORP 83(4) 2012

Level 0 process flow map for opening an oncology clinical trial.

Dilts D M , Sandler A B JCO 2006;24:4545-4552

©2006 by American Society of Clinical Oncology

7/28/2012

3

Pareto Chart

• Bar graph – Length of bars represent frequency or cost

(money or time)– Arranged from longest (left) to shortest (right)

• Analyze frequency of causes or problemsVisually shows which situations are more – Visually shows which situations are more significant


Pareto Chart

• Decide categories to group items• Decide appropriate measurement

– Frequency, quantity, cost, or time

• Decide period of time• Collect data, recording category or

bl i ti d tassemble existing data


Pareto Chart

• Subtotal measurements in each category• Determine appropriate scale (y-axis)• Construct, label bars for each category

Optional• Calculate percentage (%) for each categoryp g ( ) g y

– (right vertical axis)

• Calculate, draw cumulative sums (%)


Pareto Chart

• Pareto Principle: 80% of effect comes from 20% of the causes20% of the causes

• Measurement choice– Reflective of costs preferred (dollars, time,

etc.)– If causes equal weighting in costs – use

ffrequency

• Weighted Pareto chart (to normalize equal opportunities)


7/28/2012

4

AIM Statement

i h f i ifi To increase the rate of patient-specific quality assurance (PSQA) prior to the first treatment to 100% by July 2011

Baseline Metric IMRT

Avg. Compliance Rate = 71.4%

Cause Analysis

• Create & evaluate process flow• Create & evaluate process flow• Identify potential causes of failures• Create & evaluate tracking data (times,

bottlenecks)• ID & examine cases in which QA was not ID & examine cases in which QA was not

completed

Simulation to Start Treatment Patient Flow

Legend:Physician

Therapy

Dosimetry

Simulation completed-Therapist

Was patient given start

date?Why not?No

START

Physics

Why not?

Whenavailable?

Is it 5 business days?

Are alldata available

to do contours?

Yes

No

Yes

No

Patient

Issue

Dosimetry preparefor contouring MD notified

contours?

Contours completed

Dosimetrist notified

Planning objectives provided?

Why not?

Yes

No

7/28/2012

5

Legend:Physician

Therapy

Dosimetry

Proceed to planning

Dosimetrist competes planMD notifiedMD reviews plan

Plan approved?If IMRT, before

4PM Why not?No Plan reworked?

Yes

Physics

Patient

Issue

Dosimetrist processes plan in

MosaiqIs script in Mosaiq

MD notified to sign plan in Mosaiq

Quality checklist item generated by

4pm

QA/Chart Check prior to beginning

XRT

QA/Chart CheckApproved? Why Not?

Patient start

Yes

No

Yes No

YesMD Notified

Patient starts

Patient startdate changed?

Does patient start

without QA?

Yes

No

MD Notified?Dry Run?

END

Pareto Diagram of Physics Review IMRT QA "After Tx" CausesSep 2010 - Feb 2011

60

70

80

80.0%

90.0%

100.0%

t

0

10

20

30

40

50

60

Num

ber o

f Cau

ses

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

Cum

ulat

ive

Perc

ent

LSDA ES PH BST

Physics Review IMRT QA "After Tx" Causes

LSDA (Late / Same Day Approval) PH (Physics Cause)ES (Early Start) BST (Boost Plan)

Total 165 Cases

Interventions

• Division Grand Rounds (Jan 2011)C i t d i t f QA– Communicated importance of QA

– Discussed ACR accreditation– Developed support from faculty and staff

• Division Guidelines (Apr 1, 2011) – Eliminate late approvals for IMRTpp– Eliminate early patient start times for IMRT– IMRT QA and physics chart check prior to

first treatment now required

7/28/2012

6

After Intervention IMRT

Avg. Compliance

Rate = 99.3%

Acknowledgments

• CS&E teamP j D

• MDACC faculty & staffTh B hh l– Prajnan Das

– Lei Dong– James Kanke– Michael Kantor – Beverly Riley– Tatiana Hmar-Lagroun

– Thomas Buchholz– Liao– Geoffrey Ibbott– Michael Gillin– Rajat Kudchadker– John BinghamTatiana Hmar-Lagroun John Bingham

• Q&S Council members

References

• Tague, Nancy R. The Quality Toolbox. 2nd Edition. Milwaukee Wisconsin: ASQ Quality Press 2005Milwaukee, Wisconsin: ASQ Quality Press, 2005.