webinar slides 12-nov-14: developing qc plans

Webinar Series sponsored by

Developing QC PlansThermo Fisher ScientificDeveloping QC Plans

Part 3. Performing a Risk Assessment to Identify Hazards and Target Failure Modes

James O WestgardJames O. WestgardSten A. WestgardWestgard QC IncWestgard QC, Inc.

Madison, WIwwwwestgard comwww.westgard.com

1

Webinar Series sponsored by Thermo Fisher Scientific

September 17. Regulatory requirements and strategic approachOctober 8. Right‐sizing SQC procedures using “Westgard Sigma Rules”November 12. Performing a risk assessment to identify hazards and target failure modesy gDecember 10. Mitigating risk with an Individualized QC PlanIndividualized QC Plan

2

Part 3. Performing a Risk Assessment to d if d d il dIdentify Hazards and Target Failure Modes

• Review industrial process for risk assessmentReview industrial process for risk assessment• Review CMS hazard identification guidanceId if i bl i i k d l• Identify variables in risk models– 2 vs 3 factors, ranking scales, graphic vs calc of risk

• Illustrate application with qualitative scale– Calculate risk to prioritize failure modes– Identify controls to mitigate risks– Calculate residual risks

3

A Plan for Developing QC Plans

Validate Safety Characteristics

Yes No

Audit QC Practices

Determine Sigma Quality

Analyze2 levels of

controls/day?

DevelopTotal QC Plan

DevelopRisk‐BasedIQC Plan

Monitor Quality & Patient Safety

4

Improve Quality & Safety

Option for IQCPOption for IQCP

Organize Risk Project TeamOrganize Risk Project Team

Assess Hazards, ID Failure Modes

Evaluate Risk of Failure Modes

Identify Controls to Reduce Residual‐Risks

Implement IQC Plan

5

How does industry assess risk? (1)How does industry assess risk? (1)

• 1. Review the process or product• 2 Brainstorm potential failure modes2. Brainstorm potential failure modes• 3. List potential effects of failure modes4 A i SEVERITY ki (SEV)• 4. Assign SEVERITY ranking (SEV)

• 5. Assign OCCURRENCE ranking (OCC)• 6. Assign DETECTION ranking (DET)• 7. Calculate Risk Priority Number (RPN)7. Calculate Risk Priority Number (RPN)

6

Industrial Calculation of Risk (2)Industrial Calculation of Risk (2)

• 3‐factor risk modelRisk = SEV*OCC*DET = RPNRisk SEV OCC DET RPN

• Where SEV is severity• OCC is occurrence• OCC is occurrence• DET is detection• RPN is Risk Priority Number• RPN is Risk Priority Number

7

How does industry assess risk? (3)How does industry assess risk? (3)

• 8. Prioritize the failure modes for action• 9. “Mitigate” high‐risk failure modesg g

– Eliminate or reduce occurrence– Improve detectionImprove detection– Minimize severity of harm

• 10 Calculate RPN again to assess residual risks• 10. Calculate RPN again to assess residual risks and determine acceptability

8

How does industry mitigate risk? (4)How does industry mitigate risk? (4)

• Occurrence ‐ “Design for safety” to eliminate failure‐modes – Disclose “safety characteristics”

• Detection ‐ Build in control mechanismsetection uild in control mechanisms– Provide alerts to identify problems

• Severity Provide instructions for “safe use”• Severity ‐ Provide instructions for safe use and precautions for “use errors”

Inform lab of limitations and recommend QC– Inform lab of limitations and recommend QC

9

Review CMS Guidance for IQCPsReview CMS Guidance for IQCPs

• IQCP includes 3 componentsIQCP includes 3 components– Risk assessment to evaluate hazards (steps 1‐5)– QC Plan to identify control mechanisms (steps 6 7)– QC Plan to identify control mechanisms (steps 6‐7)– QA Plan to monitor quality and safety (step 8)

“T d t i k t th l b t• “To conduct a risk assessment, the laboratory must identify the sources of potential failures

d f t ti d l tand errors for a testing process, and evaluate the frequency and impact of those failures and

f ”sources of errors”10

Review CMS Guidance for IQCPReview CMS Guidance for IQCP

• The Clinical and Laboratory StandardsThe Clinical and Laboratory Standards Institute (CLSI) document EP23‐A –“Laboratory Quality Control Based on RiskLaboratory Quality Control Based on Risk Management – provides helpful guidance to laboratories on the development of qualitylaboratories on the development of quality control plans for test systems.

11

CLSI EP23A Methodology (1)CLSI EP23A Methodology (1)

Identify Project, Assemble Team

1. INPUT INFORMATIONMedical Requirements , Regulatory/Accred. Requirements,

y j ,

Manuf. Test System Info, Test‐Site Requirements

2. HAZARD IDENTIFICATIONReview manufacturer’s Risk Report Map/chart process flowReview manufacturer s Risk Report, Map/chart process flow

Identify failure modes (fishbone), Identify prevention/detection

3. RISK ESTIMATION2‐factor risk model,

Probability of occurrence of harm, Severity of harm

4. RISK EVALUATION

12

4. RISK VA UATIONRisk Acceptability Matrix

CLSI EP23A Methodology (2)CLSI EP23A Methodology (2)5. RISK CONTROL

Identify control mechanisms to optimize detection andIdentify control mechanisms to optimize detection andlower risk to acceptable level

6. LABORATORY QC TOOL BOXQPre‐analytic controls, SQC, EQA, Trueness Controls

Integrated Liquid Controls, Function Tests, Procedural ControlsElectronic Checks, Calibrator Checks, Repeat Patient Testing

Delta Checks, Implausible Values, Patient Population Algorithms (AoN)

7. QUALITY CONTROL PLAN

Delta Checks, Implausible Values, Patient Population Algorithms (AoN)Correlation algorithms, Post‐analytic controls

Review conformance to requirements, Implement control mechanisms

8. QUALITY IMPROVEMENT

13

Monitor performance, Reduce errors, Improve quality

Assess Hazards to Identify Failure Modes

CMS ib ifi t i• CMS prescribes specific categories and conditions that must be i l d d i i k tincluded in your risk assessment– Specimen– Environment– Reagent– Test System– Testing personnel

14

Assess Hazards to Identify Failure Modes

Hazards Checklist for Identification of Targeted Failure Modes (1)Test, Test SystemProject Group, DateSigma Quality=(%TEa‐%Bias)/%CVg y ( )Step/Procedure/Activity in the

Total Testing ProcessWhat might cause a

problem?Data? In‐house or manufacturer?

Targeted Failure Mode?

Specimen

Patient preparationCollectionLabelingStorage, preservation, stabilityTransportationAcceptability and rejection

f l ( h l b )

15

Referral (to other labs)

S /P d /A i i i Wh i h Data? In‐house T d

Hazards Checklist for Identification of Targeted Failure Modes (2)

EnvironmentTemperature

Step/Procedure/Activity in the Total Testing Process

What might cause a problem?

Data? In house or

manufacturer?


TemperatureAirflow/ventilationLight intensityNoise and vibrationH idiHumidityAltitudeDustWaterUtilities (elect stability)Adequate space

ReagentgShipping/receivingStorage condition requirementsExpiration data

16

xpiration dataPreparation

S /P d /A i i i Wh i h Data? In‐house T d




Data? In house or

manufacturer?


Test SystemI d t liInadequate samplingClot detectionInterference detection

HemolysisLipemiaLipemiaIcterusTurbidity

CalibrationMechanical/electronic failures

OpticsPipettes, pipettorsBarcode readers

System controls & function checksProcedural, electronic controlsLiquid controlsTemperature controls

Software/hardwareData transmission to LIS

17

Data transmission to LISResult reporting




Data? In‐house or

manufacturer?


Testing PersonnelTesting PersonnelTrainingCompetencyEducation, experience qualificationAdequate staffing

18

Risk AssessmentCLSI EP23A Model

• Risk – combination of the probability of occurrence of harm and the severity of that harm – Risk = OCC*SEV

• Detection – not considered in EP23 risk assessment even though the purpose of a QC Plan is to provide detection of those failure modes that can not be eliminated or prevented

19

Variables in Risk AssessmentVariables in Risk Assessment• Number of risk factors

– 2‐factor models use OCC and SEV– 3‐factor models add DET

• Ranking scales– 1‐10 common in industrial applications1 10 common in industrial applications– 1‐5 common recommended in CLSI EP23A1 3 useful for “qualitative” assessment– 1‐3 useful for qualitative assessment

• Risk evaluationl l “ k b l ”– RPN calculation or “risk acceptability matrix”

20

Generic Industrial RankingsGeneric Industrial Rankings

RANKING SEVERITY OCCURRENCE DETECTION

10 Highly hazardous Very high Non‐detectable

9 Hazardous Very high Very improbable

8 Very high High Improbable8 Very high High Improbable

7 High High Very low

6 Moderate Moderate Low

5 Low Moderate Moderate

4 Very low Moderate Moderately high

3 Minor Low Highg

2 Very minor Very low Very high

1 None Remote Highly detectable

21

Ref: Dailey. The FMEA Pocket Handbook. Pages 29-34

CLSI EP23A & ISO 14971k l d b lRanking Scales and Acceptability Matrix

S it R kNegligible Minor Serious Critical Catastrophic

Severity Rank

ank

Frequent

Probable U U Unacceptable Risk

nce

Ra

Occasional URemote U UAcceptable Riskcu

rren

Improbable

p

Occ

22

Issues with Occurrence RankingIssues with Occurrence Ranking

• CLSI EP23A relates probability of occurrence p yto frequency of failures– Rank of 1: Frequent, once per weekq , p– Rank of 2: Probable, once per month– Rank of 3: Occasional, once per yearRank of 3: Occasional, once per year– Rank of 4: Remote, once every few years– Rank of 5: Improbable once in life of system– Rank of 5: Improbable, once in life of system

• But, frequency doesn’t reflect number of patients affected by failurepatients affected by failure

23

Patients Potentially HarmedPatients Potentially Harmed

• Depends on frequency of failure and numberDepends on frequency of failure and number of patient test results affected by that failure – Occurrence of hemolysisOccurrence of hemolysis

• Frequency may be daily, but only be a single patient

– Occurrence of calibration errorOccurrence of calibration error• Frequency may be weekly, but would affect an entire run of patients, which could include the whole day or even more than one day

– More difficult to rank patients harmed than f f f ilfrequency of failure

24

Risk AssessmentWestgard Six Sigma Model

• Quantitative approach that focuses on defectQuantitative approach that focuses on defect rate and estimation of number of potentially harmful patient test results producedharmful patient test results produced– Westgard JO. Six Sigma Risk Analysis: Designing analytic QC plans for the medical laboratory. y Q p yMadison WI:Westgard QC, 2011.

– Recommends using TJC “Proactive Risk Reduction” gmethodology rather than CLSI EP23A

– Basis for simpler qualitative methodology here

25

Risk Assessment ExampleWestgard “Qualitative Model”

• 3 factors3 factors• Rank on scale 0, 1, 2Rank Severity Occurrence Detection

0 Very low (no harm) None to a few patients Very low or unknown

1 Moderate (delay) Some patients Moderate (50%)

2 Very high (patient harm) Many patients Very high (≥90%)

• Calculate risk as SEV*OCC*(2‐DET)( )– Estimates will range from 0 to 8– Mitigate failure modes when risk is 2 and higher

26

Mitigate failure modes when risk is 2 and higher

Prepare Electronic Spreadsheet ask blRisk Assessment Table

• Note that CMS does not recommend nor require use of any specific tools, such as Failure Modes and Effects Analysis (FMEA), y ( ),which is the standard tool in industry

• From Q&A in CMS Memo of 8/16/2013From Q&A in CMS Memo of 8/16/2013– Will laboratories be required to use a process map, fishbone diagram, formal risk assessment chartsfishbone diagram, formal risk assessment charts and protocols, etc., in their IQCP?

• No, CLIA will not require the use of these tools in the development of an IQCP.

27

Example Risk Assessment TableFailure Mode and Effects Analysis (FMEA)

Step/Activity Targeted Effect SEV Cause OCC Controls DET RiskFailure Mode

0‐2 0‐2 0‐2 SEV*OCC*(2‐DET)

28

Example POC Analyzer

• Cartridge provides multiple testsCartridge provides multiple tests– Sigma > 5 for triglycerides, glucose, creatinine– Sigma > 4 for cholesterol HDLSigma > 4 for cholesterol, HDL

• Specimen is whole bloodS ll i i fi ti k ll t b– Small specimen size, finger sticks or small tubes

– Automated separation in cartridge to provide serum or plasma samples for analysisserum or plasma samples for analysis

• Analysis time is 5 minutes/cartridge– Other cartridges available for other tests

29

FMEA Example: Specimen acquisitionTurbidity, hemolysis, icteris


0‐2 0‐2 0‐2 SEV*OCC*(2‐DET)

Specimenacceptability

Turbidity Hemolysis

Changes results

1 Photo.Interfer.

1 Manuf.Check???

0 1*1*(2‐0) 2acceptability Hemolysis results Interfer. Check??? = 2

30

FMEA Example: Sample processingInstability when sample is on cells


0‐2 0‐2 0‐2 SEV*OCC*(2‐DET)


Turbidity hemolysis

Changes results

1 Photo.Interfer.

1 Manuf. Check???

0 1*1*(2‐0) 2acceptability hemolysis results Interfer. Check??? = 2

Sampleprocessing

Stability on cells

Lowglucose

2 Cellenzymes

1 SOP, Prompt analysis

0 2*1*(2‐0) = 4

31

FMEA Example: Reagent stabilityff i iAffects enzymatic reactions


0‐2 0‐2 0‐2 SEV*OCC*(2‐DET)


Turbidity hemolysis

Changes results

1 Photo.Interfer.

1 Manuf.Check???

0 1*1*(2‐0) 2acceptability hemolysis results Interfer. Check??? = 2

Sampleprocessing

Stabilityon cells

Lowglucose

2 Cellenzymes

1 SOP, Prompt analysis

0 2*1*(2‐0) = 4

Analysis/ Test system

Reagent stability

Low creat

2 Environ cond’s

2 Electroniccheck

1 2*2*(2‐1) = 4

32

FMEA Example: Operator variability

Step/Activity Targeted Effect SEV Cause OCC Controls DET Risk

Increases test variability

Failure Mode

0‐2 0‐2 0‐2 SEV*OCC*(2‐DET)

Specimen Turbidity hemolysis

Changes results

1 Photo.Interfer.

1 Manuf.Check???

0 1*1*(2‐0) 2hemolysis results Interfer. Check??? = 2

Sampleprocessing

Stabilityon cells

Lowglucose

2 Cellenzymes

1 Prompt analysis

0 2*1*(2‐0) = 4

A l i / T t R t L 2 E i 2 El t i 1 2*2*(2 1)Analysis/ Testsystem

Reagent stability

Low creat

2 Environcond’s

2 Electroniccheck

1 2*2*(2‐1) = 4

Analysis/ Operator

Operator technique

Variableresults

2 Operator skills

2 TrainingSOP

0 2*2*(2‐0) = 8Operator technique = 8

33

FMEA Example: Turnaround TimeWorkload may delay results


0‐2 0‐2 0‐2 SEV*OCC*(2‐DET)

Specimenacquisition

Turbidspecimen

Changes results

1 Photo.Interfer.

1 Manuf.Check???

0 1*1*(2‐0) 2acquisition specimen results Interfer. Check??? = 2

Sampleprocessing

Time on cells

Lowglucose

2 Cellenzymes

1 SOP. Prompt analysis

0 2*1*(2‐0) = 4

Analysis/TestSystem

Cartridgestability

Low creat

1 Environcond’s

2 Electroniccheck

1 1*2*(2‐1) = 2

Analysis/O t

Operator t h i

Variableresults

2 Operator skills

2 Training 0 2*2*(2‐0) Operator technique results skills = 8Resultreporting

TAT Delay 1 Workload 2 SOP,prompt analysis

0 1*2*(2‐0)= 4

34

Prioritization for Risk MitigationPrioritization for Risk Mitigation

• Target any failure mode whose risk score ≥ 2Target any failure mode whose risk score ≥ 2• Prioritize failure modes with higher scores

S f 8 i i– Score of 8 is maximum– Many risk factor combinations yield scores of 4 and 2and 2

– Prioritize score of 2 due to SEVERITY over other scores of 2scores of 2

35

Example of Risk PrioritizationExample of Risk Prioritization

Targeted Failure Mode Risk Mitigation SEV OCC DET Residual Risk g g0‐2 0‐2 0‐2 SEV*OCC*(2‐DET)

Operator variability (8)

Sample stability (4)Sample stability (4)

Turnaround time (4)

Cartridge stability (2)

Specimen acceptability (2)

36

Examples of Risk Mitigation (1)Examples of Risk Mitigation (1)

• Priority 1: Operator variability (8)Priority 1: Operator variability (8)– Monitor by “Repeat Patient Test” control dailyMust establish expected limits of variability– Must establish expected limits of variability

• Repeat Patient Test daily for 20 days• Calculate SD based on differencesCalculate SD based on differences• Set “control limits” for expected variation

• Priority 2: Glucose instability (4)Priority 2: Glucose instability (4)– Reduce processing time & improve storage (cold)Monitor pre analytic time to identify delays– Monitor pre‐analytic time to identify delays

37


• Priority 3: Turnaround Time (TAT) (4)Priority 3: Turnaround Time (TAT) (4)– Utilize patient log to track time of draw, time of analysis and time of reportanalysis, and time of report

– Establish “alert” times to identify workload problemsproblems

– Identify conditions that lead to high workload– Improve process to reduce TATImprove process to reduce TAT

38


• Priority 4: Reagent/test system instability (2)Priority 4: Reagent/test system instability (2)– For POC, analyze stable control material every other day to improve detection, or Repeat Patient Test vs day to improve detection, or Repeat Patient Test vsa reference method/lab

– Qualify new reagent lots using “right‐sized SQC”y g g g

• Priority 5: Specimen acceptability (2)– Materials process training for specimen acquisitionMaterials, process, training for specimen acquisition– Test manufacturer’s control to validate capability to detect lipemia, hemolysis, icterusdetect lipemia, hemolysis, icterus

39

Residual RiskWhat is risk after mitigation?

Targeted Failure Mode Risk Mitigation SEV 0 2

OCC 0 2

DET 0 2

Residual Risk SEV*OCC*(2 DET)0‐2 0‐2 0‐2 SEV*OCC*(2‐DET)

Operator variability (8) Repeat Patient Test 2 2 2 2*2*(2‐2) = 0

Sample stability (4) Improved storage & 2 0 0 1*0*(2‐0) = 0processing

Turnaround time (4) Improve process flow, monitor pre‐

l i i

1 1 1 1*1*(2‐1) = 1

analytic time on patient log

Cartridge stability (2) Analyze stable control or RPT vs

1 2 2 1*2*(2‐2) = 0control, or RPT vs Ref Method/Lab

Specimen acceptability (2)

Instruct patients, validate manuf.

1 1 1 1*1*(2‐1) = 1(2) validate manuf.

control

40

REVIEWRisk Assessment Methodology

• Remember, CMS does not define nor ,recommend any standard methodology!

• Variables in methodology to considerVariables in methodology to consider– 2 or 3 factor risk assessment modelRanking scales for risk factors– Ranking scales for risk factors

– Determination of risk E l ti f t bl i k– Evaluation of acceptable risk

41

REVIEWProcess for IQCP

Organize Risk Project TeamOrganize Risk Project Team

Assess Hazards, ID Failure Modes

Evaluate Risk of Failure Modes

Identify Controls to Reduce Residual‐Risks

Implement IQC Plan

42

REVIEWl kQualitative Risk Assessment Process

• Start with CMS hazards list to identify failure modesStart with CMS hazards list to identify failure modes• Decide whether to use 2 or 3‐factor risk model

– Example here uses SEVerity, OCCurrence, and DETectionExample here uses SEVerity, OCCurrence, and DETection

• Define ranking scale (3, 5, 10 categories)– Example here ranks factors as 0, 1, or 2p , ,

• Define risk evaluation (Graphical vs calculation)– Example here calculates risk as SEV*OCC*(2‐DET)p ( )

• Prioritize risks• Mitigate unacceptable risks g p

43

Risks of Risk ManagementRisks of Risk Management

• A structured risk assessment methodology isA structured risk assessment methodology is needed to develop reliable QC Plans – FMEA is a standard tool but requires education– FMEA is a standard tool, but requires education and training for proper applications

• CLSI EP23A guidance neglects DETECTION and• CLSI EP23A guidance neglects DETECTION and uses arbitrary risk acceptability matrixRi k t d l ti i bj ti• Risk assessment and evaluation is subjective

44

Risk‐Based QC has its own i k i l l “ id l i k ”risks, particularly “residual risks”

• Laboratory is depending on manufacturer toLaboratory is depending on manufacturer to own up to limitations of performance

• Manufacturer passes residual risks on to lab• Manufacturer passes residual risks on to lab• Laboratory implements manufacturer’s

l i i QC Plcontrols in its QC Plan• Laboratory identifies additional controls on basis of its own risk assessment

• Still have “residual risks”

45

Who’s responsible for residual risks in a medical laboratory?

• The director must consider the laboratory’s clinical and legal responsibility for providingclinical and legal responsibility for providing accurate and reliable patient test results versus the cost implications of reducing theversus the cost implications of reducing the quality control frequency testing.– CMS SOM item D5445CMS, SOM item D5445

46

What’s next?What s next?

th b b ll f h• 4th Webinar – December 10 ‐ will focus on the implementation of Individualized QC Plans

47

Thank you for your kind attention!Thank you for your kind attention!

48

Thank you for listening (Questions?)

MUCH, MUCH MORE

49

UC , UC OCan be found onlineAt www.westgard.com

webinar slides 12-nov-14: developing qc plans

Documents