introduction to valid statistical techniques for process...

Introduction to Valid Statistical Techniques for

Process Validation Thursday, April 26, 2012

11:00 am – 1:00 pm Eastern

Peter Knepell, PhD President, Peak Quality Services

[email protected]

Introduction to Valid Statistical Techniques for Process Validation

Introduction to Valid Statistical Techniques for

Process ValidationPeter Knepell, [email protected]

4/26/2012 © 2012 Association for the Advancement of Medical Instrumentation www.aami.org 1

Introducing the Presenter

• Peter Knepell, President of Peak Quality Services

• PhD, Cornell University, Operations Research

• Certified Quality Engineer (CQE) and SoftwareQuality Engineer (CSQE) by the American Societyfor Quality

• Started assisting medical device and pharmaceutical manufacturers in 1994

• Since 1998, specialized in Lean Six Sigma implementation for a variety of industries & organizations

• AAMI faculty for: Statistics, Design of Experiments, Risk Management, and Process Validation workshops



Webinar Agenda

• Setting the Stage

• Illustration of Best Practices• Measurement System Analysis (MSA)• Process Capability• Sampling Plans

• Q&A at the Break and End


Webinar Objectives

• Explore some valid statistical techniques for Process Validation

• Provide a mapping of tools and techniques to IQ, OQ and PQ activities

• Provide examples of industry best practices in this area

• Demonstrate how these tools support regulatory compliance, especially if mandatory or voluntary action is required



Webinar Ground Rules• Questions are good• Presenter will keep it simple (KISS)

• Cover why, what, & where• Briefly cover how to• Avoid statistical complexity

• For greater depth, consider the AAMI workshops on: • Statistical Methods & Tools for a Quality System,

Oct. 15-17, 2012• Webinar: Design of Experiments for Process Validation,

Oct. 11, 2012• Design of Experiments for a Quality System, Spring 2013


Introducing the Subject

• FDA Quality System Regulation (21 CFR 820) is specific about “valid statistical techniques” and “sampling plans … based on valid statistical rationale”

• Application of statistical methods and tools outlined in Global Harmonization Task Force (GHTF) Process Validation Guidance



Setting the Stage I

Min Acceptable

= 10 Pounds

SealStrength

Average15 Pounds

A packaging group is trying to set up their automated equipment so that the Seal Strength is greater than 10 pounds. They adjusted the settings for time, temperature and pressure and processed a few packages. They measured the Seal Strength for each and found the average was 15 pounds.

How are they doing?


“The Customer Does Not Feel The Average”

Min Acceptable

= 10 Pounds

SealStrength

Average15 Pounds

Min Acceptable

= 10 Pounds

SealStrength

Average15 Pounds

“The customer feels the variation.”

Case 1: High Variation

6.7% defects

Case 2: Low Variation

8


Recognition of the Value of Variance Reduction

“Many non-conformities are not the result of errors, instead they are the result of excessive variation and off-target processes. Reducing variation and proper targeting of a process requires identifying the key input variables and establishing controls on these inputs to ensure that the outputs conform to requirements.”

Global Harmonization Task Force (GHTF) Process Validation Guidance, Jan. 2004, p. 14. [emphasis added]


“Where appropriate, each manufacturer shall establish and maintain procedures for identifying valid statistical techniques required for establishing, controlling, and verifying the acceptability of process capability and product characteristics.”

21 CFR 820.250 (a) [emphasis added]

Setting the Stage II



Setting the Stage II

“‘Where appropriate’ is deemed to be ‘appropriate’ unless the manufacturer can document justification otherwise.”

21 CFR 820.1 (a) (3)

“Establish means define, document (in writing or electronically), and implement.”

21 CFR 820.3(k)


Sampling Plans

“Sampling plans, when used, shall be written and based on valid statistical rationale. Each manufacturer shall establish and maintain procedures to ensure that sampling methods are adequate for their intended use and to ensure that when changes occur the sampling plans are reviewed. These activities shall be documented.”

21 CFR 820.250(b) [emphasis added]



Acceptance Activities

Establish and maintain procedures for acceptance activities.Include inspections, tests or other verification activities.

• Receiving acceptance

• In-process

• Final acceptance21 CFR 820.80 (a) – (d)


Statistical Methods & Tools for Process Validation

(Mod 2)(Mods 5-7)

(DOE Workshop)

(Mods 5-6)

(Mod 3)(Mod 7)

IQ Gauge R&R (MSA)

OQ

Historical Data AnalysisDrawing Conclusions Using Data

Predictions Using ProbabilitySampling & Sample SizesComparing Data

Design of ExperimentsScreening ExperimentsResponse Surface StudiesRobust Design Methods

PQDrawing Conclusions Using Data

Sampling PlansCapability StudiesStatistical Process Control (SPC) Charts

(Statistics Workshop, Mod 4)

(Mod 8)



Key Takeaways for Industry Needs and Best Practices

• Reducing variation is key to quality

• The FDA recognizes the need for valid statistical techniques in quality systems

• Statistical methods & tools map well to the requirements of Process Validation


Illustration of Best Practices

• Measurement System Analysis (MSA)

• Process Capability

• Sampling Plans



Gauge R & R

A study for evaluating:

the precision and accuracy of a measurement device

the reproducibility of a measurement device with respect to operators

GHTF Process Validation Guidance, p. 21


Modern Terminology

Gauge R&R = Measurement System Analysis (MSA)

• Attribute MSA – measured values are from finite number of categories (e.g., pass/fail, good/bad, accept/reject)

• Variable MSA – measured values are from a continuum of values (a.k.a., Objective MSA)

Which type of MSA would you prefer to run?Why?



Fundamental Principles of MSA

• A Measurement System is a collection of:• Instruments or gauges• Standards• Personnel• Methods

• Variation is the enemy of a measurement system

• Detecting, measuring, and reducing variation are the goal of MSA

• Fixtures• Software• Environment• Assumptions


Fundamental Principles of MSA

Data collection requires:• Random selection of items to measure

• Repeated measurements of each item, if possible

• Participation of people and instruments expected to make measurements in the future

• A process to avoid bias of any sort (e.g., 3rd party, double blind, random presentation of items)

• Adequate sample sizes



MSA for Subjective Systems(Attribute Data)

21

Measuring a Variable Characteristic

MeasurementProcess

ProductionProcess True

ValueObserved

Value

True Value

Measurement Error

Observed Value = +



Variable MSAThe Result of Repeated Measurements

MeasurementProcess

ProductionProcess σ2

Product σ2Observed

σ2Product σ2

Measurementσ2Observed = +

Manufacturer’s Goals1. Minimize σ2

Measurement (MSA)

2. Minimize σ2Product (Process Capability)


Variable MSA Terminology

Accuracy - The difference between the average of a set of results and a true or expected value.Precision - The dispersion or spread of results.

xxxxxxxxxTruth

Accuracy Error(Bias)

Appraiser’s Repeated

Measurements

PrecisionError



MSA Terminology

Repeatability – ability of a person or instrument to repeat measurements of the same item, given that all other sources of error are insignificant or independently observed.Reproducibility – ability of two or more people or instruments to agree with measurements of the same item, given all other sources of error are insignificant or independently observed.

xxxxxxxxx

ReproducibilityError

RepeatabilityError (A)

xxxxxxxxx

RepeatabilityError (B)

Appraiser A’sMeasurements

Appraiser B’sMeasurements


ANOVA Approach for MSA

xxxxxxxxx

σRepeat(A)

xxxxxxxxxAppraiser A Appraiser B

σRepeat(B)

σReprod

ANOVA = Analysis of Variance

σ2Reprod σ2

Repeatσ2Measurement = +



MSA Goals

1. Quantify σ2Measure

2. Minimize σ2Measure

σ2Reprod σ2

Repeat= +

ANOVA Approach for MSA

σ2Reprod σ2

Repeatσ2Measurement = +

σ2Repeat


Acceptability Criteria #1

Precision to Total Ratio: We want σMeasurement to

be less than 10% of the total σObserved

σObserved

σMeasurementP / Total =

< 0.1 => acceptable system

> 0.3 => unacceptable system

0.1 to 0.3 => may be acceptable based upon: cost of measurement device improvement; cost of poor quality; criticality of application, etc.

Ref: MSA-4, Automotive Industry Action Group, p. 774/26/2012 © 2012 Association for the Advancement of Medical Instrumentation www.aami.org 28


Cost of a Poor Measurement System

4 CasesA. Will definitely be accepted

B. Will definitely be rejected

C. May be falsely accepted

D. May be falsely rejected

If σMeasure is too large, the costs for Cases C & D will be intolerable.

“Safety Gate”

LSL USLσMeasurement

σObserved

±3σMeasurement ±3σMeasurement

CD A B

LSL USLσMeasurement

σObserved

±3σMeasurement ±3σMeasurement

CD A B

NOTE: Some organizations use the term “Guard Band” instead of “Safety Gate” 29

Acceptability Criteria #2

P/Tolerance = USL - LSL

6 σMeasure

Precision to Tolerance Ratio: The region where a good part may be falsely rejected should not exceed 10% of the spec width. This means the “Safety Gate” should be at least 90% of the spec width.

< 0.1 => acceptable system

> 0.3 => unacceptable system

0.1 to 0.3 => may be acceptable based upon: cost of measurement device improvement; cost of poor quality; criticality of application, etc.



Variable MSA ExerciseCaliper Evaluation

Scenario: You are looking at two different designs for a caliper. You are trying to decide if any one is superior to the other.Decide how to set up an MSA to determine if one measurement instrument is superior.


MSA Set Up

• Same operator for both calipers• A 3rd party presents parts and records

measurements• Parts are presented in a random order

Caliper 1 Caliper 2Part # Rep 1 Rep 2 Rep 1 Rep 2

1 98.6 98.0 99.0 99.02 97.5 98.0 98.6 98.53 98.0 97.5 98.2 98.14 100.7 99.9 101.1 101.25 98.0 98.4 98.6 98.6



Illustrating R & R


1 98.6 98.0 99.0 99.0

0.6


1 98.6 98.0 99.0 99.0

0.0

98.3 99.0

0.7

Averages

Repeatability Errors

Reproducibility Error 33

User’s Manual for Variable MSA Reproducibility Error

Reproducibility Error = 0.7

96

97

98

99

100

101

102

1 2 3 4 5Part

Mea

sure

men

t

Caliper 1Caliper 2

34


This point is the max - min of Caliper 1’s readings for Part 1

Rep

eata

bilit

y Er

ror

User’s Manual for Variable MSA Reproducibility Error

Range Chart

0

0.2

0.4

0.6

0.8

1

1.2

1 2 3 4 5 1 2 3 4 5Part

Ran

ge

Caliper 1Caliper 2

35

Key Takeaways for MSA• MSA is fundamental to any quality system• Poor measurement capability leads to huge

costs for: Suppliers, Manufacturers & Customers

• Attribute MSAs are rather simple to conduct & analyze

• Variable MSAs require software for analysis & graphics to diagnose problems

How do you know you have quality if you cannot measure?





• Sampling Plans

If you are not keeping score, you are just practicing.

Vince Lombardi



“Where appropriate, each manufacturer shall establish and maintain procedures for identifying VALID STATISTICAL TECHNIQUES required for establishing, controlling, and verifying the acceptability of PROCESS CAPABILITY and product characteristics.” [emphasis added]

21 CFR 820.250 (a)

Process Capability



Capability Studies

Capability studies are performed to evaluate the ability of a process to consistently meet a specification.The most commonly used capability indices are Cp and Cpk.Capability studies are frequently used towards the end of the validation to demonstrate that the outputs consistently meet the specifications.

GHTF Process Validation Guidance, p. 20


Types of Data

• Variable Data• Output of a temperature sensor (thermistor)• Time to “beep” when temperature stable

• Attribute Data• Pass/Fail test of electronic assembly• Plastic casing cracked or not cracked

OFF ON RESET F C69 8

OFF ON RESET F C69 8

Which type of data would you prefer to have?4/26/2012 © 2012 Association for the Advancement of Medical Instrumentation www.aami.org 40


A Fundamental Statistical Approach to Quality

1. Establish customer requirements.2. Establish upper and/or lower

specifications (USL & LSL) for the critical parameters.

3. Take a sample of the product and measure the critical parameters.

4. Compute the mean ( ) and standard deviation (s) for the critical parameters.

5. Compute process capability measures.

Y


Process Capability Scenario for Variable Data

You sampled 100 temperature sensors from a lot of 1,000 and created a histogram. The upper and lower spec limits (USL & LSL) are shown. Is there an simple way to express how well or poorly your supplier of temperature sensors is performing?

0

5

10

15

20

25

30

35

98.2 98.3 98.4 98.5 98.6 98.7 98.8 98.9

LSL USL



Measures of Quality for Variable Data

• Capability IndicesCpk - Process Capability IndexCp - Process Potential Index

Defects per Million (DPM or ppm)

• Six Sigma MeasuresSigma LevelSigma Capability


Common Process Capability Measures If Metric is Variable

AND Normally Distributed

Process Capability (Actual)Cpk = minimum { USL - μ , μ - LSL }

3 σProcess Potential (Center the Process)

Cp = USL - LSL 6 σ

Sigma Level

σlevel = minimum { USL - μ , μ - LSL }σ

LSL USL

σ

μ

LSL USL

σ

μ



Computing Process Capability Measures from a Sample

Cpk = min (USL- , - LSL) / 3s = D / 3sσlevel = 3 x Cpk = D / s

Cp = (USL – LSL) / 6s

Defects per Million (DPM) = Red Area x 1,000,000

Y Y

= sample averages = sample standard deviationD = distance to closest spec

Red Area = probability of a defect

YD


Procedure to Calculate Quality Measures If Metric is Variable

AND Normally Distributed

1. Establish LSL and/or USL.2. Sample and compute mean ( )

and standard deviation (s).3. Measure the distance from the

mean to the closest spec limit (D).4. Compute: Cpk = min (USL- , - LSL) / 3s = D / 3s

Cp = (USL – LSL) / 6sσlevel = 3 x Cpk = D / s

Defects per Million (DPM) = Red Area x 1,000,000

Y Y

Y



Process Capability Exercise

The packaging team established Seal Strength specs of 10 and 30 lbs. They took a random sample and found an average of 25 lbs and a standard deviation of 2.5 lbs.

Cpk = Min {30 - 25 , 25 – 10 } / (3 x 2.5) = 5 / 7.5 = 0.67

Cp = ( 30 – 10 ) / (6 x 2.5) = 20 / 15 = 1.33

σlevel = min {30 – 25 , 25 - 10} / 2.5 = 5 / 2.5 = 2.0

DPM = 0.02275 x 1,000,000 = 22,750

Compute Cpk, Cp, σlevel, and DPM:

Area = 0.02275


Equivalence of Measures

Cpk σlevel DPM*.33 1 158,655

.67 2 22,750

1 3 1,350

1.33 4 32

1.5 4.5 3.4

Cpk σlevel DPM*.33 1 158,655

.67 2 22,750

1 3 1,350

1.33 4 32

1.5 4.5 3.4

* For non-centered processes (ie, Cp ≠ Cpk)4/26/2012 © 2012 Association for the Advancement of Medical Instrumentation www.aami.org 48


Process Capability Measures for the Sample of 100 Sensors

0

5

10

15

20

25

30

35

98.2 98.3 98.4 98.5 98.6 98.7 98.8 98.9

LSL USL N = 100Mean = 98.59Std Dev = 0.056USL = 98.9LSL = 98.3Cpk = 1.73Sigma Lvl = 5.19Cp = 1.79DPM = 0.1

98.3 98.4 98.5 98.5 98.6 98.7 98.7 98.8 98.9

USLLSL

Should we be happy with our supplier of temperature sensors?


Key Takeaways for Process Capability Measures

• Process capability measures are statistics that predict the ability to deliver defect-free work

• Cpk, σlevel, Cp, & dpm are common measures used for variable data

• Just computing a quality measure from a sample is not adequate – you must quantify the statistical risk of making a wrong conclusion





• Sampling Plans



“Sampling plans, when used, shall be written and based on a valid statistical rationale. Each manufacturer shall establish and maintain procedures to ensure that sampling methods are adequate for their intended use and to ensure that when changes occur the sampling plans are reviewed. These activities shall be documented.”

21 CFR 820.250 (b)

Confidence in Statistical Conclusions Depends on Sampling



Considerations for Sampling Plans

• Risk and/or cost of defective item being accepted

• Cost of inspection

• Sources of Variation• Changes in: personnel, process, equipment, supplier,

materials, environment, etc.

• Maturity of production process

• Supplier performance history

• Contractual arrangements

• MSA results


Technical Considerations

• Variable vs. Attribute Data

• Sampling Lots vs. Continuous Sampling of Production

• Single vs. Multiple or Sequential Sampling Plans

• Risk-based Acceptance Criteria

• Adequate Sample Size to Achieve Acceptance Criteria

• Application of Standards

• Underlying Assumptions



A Very Frequently Asked QuestionWhen starting a new production process, how much should I sample?

A Very Common But Wrong AnswerThe requirement for testing from the first three production lots or batches has been deleted. While the FDA believes that three production runs during process validation is the accepted standard, the FDA recognizes that all processes may not be defined in terms of lots or batches. The number three is, however, currently considered to be the acceptable standard.

Preamble to 21 CFR 820, Comment 85.4/26/2012 © 2012 Association for the Advancement of Medical Instrumentation www.aami.org 55

A Sampling Plan AnswerWhen starting a new production process, how much should I sample?

Your Sampling Plan Should Consider All Potential Sources of Variation.

• Machine Issues: Setup, Degree of Automation, Multiple Lines, Wear & Tear, Calibration Cycle, etc.

• Staff Issues: Experience Level, Shifts, Fatigue, etc.

• Material Issues: Supplier History, Incoming Inspection, Lot Changes, Sensitivity to Environment Changes, etc.

• Environment Issues: Temp & Humidity, Day/Night, Season, etc.

• Process Monitoring: Automated Test, SPC Charts, etc.4/26/2012 © 2012 Association for the Advancement of Medical Instrumentation www.aami.org 56


Statistical Tools That Help Identify Potential Sources of Variation

• Process Flow Charts

• Fishbone Diagrams

• Risk Analysis Tools (FMEA, Fault Tree)

• Graphics of Historical Data (Pareto Charts, Run Charts, Scatter Plots, etc.)

• Measurement System Analysis

• Design of Experiments

• SPC Charts


Definition of an Acceptance Sampling Plan

• An Acceptance Sampling Plan is a pathway for deciding on the disposition of a product based on the inspection of a sample.

• The goal is to minimize the cost of inspection while understanding the risks of making a wrong decision.



A High-Level Acceptance Sampling Plan


Acceptance Sampling Plan for Variable Data

• Decide on a desired quality level (e.g., Cpk > 1.5)• Decide on a Confidence Level and Margin of Error (MOE)

(e.g., 95% confidence & MOE = 0.2)• Estimate the population standard deviation• Compute a sample size (n) using the above• Take a sample and compute Cpk for the sample• If Sample Cpk > Desired Cpk + MOE,

then accept the lot, otherwise:

a) Sample more if you exceed Desired Cpk orb) Make a decision on disposition of the lot



Process Capability Measures for the Sample of 100 Sensors

0

5

10

15

20

25

30

35

98.2 98.3 98.4 98.5 98.6 98.7 98.8 98.9

LSL USL N = 100Mean = 98.59Std Dev = 0.056USL = 98.9LSL = 98.3Cpk = 1.73Sigma Lvl = 5.19Cp = 1.79DPM = 0.1

98.3 98.4 98.5 98.5 98.6 98.7 98.7 98.8 98.9

USLLSL

Should we be happy with our supplier of temperature sensors?

Desired Cpk > 1.5 with 95% confidence


Example Result of Sampling Plan

Based on our sample we are at least 95% confident that the lot has a Cpk of at least 1.5.

Thus, we will accept the lot of 1,000 temperature sensors from our supplier.

Sample Size 100Sample Mean 98.59Sample Standard Deviation 0.056Upper Spec Limit (USL) 98.9Lower Spec Limit (LSL) 98.3Confidence Level 95.00%

Cp 1.79Lower Bound for Cp 1.58

Cpk 1.73Lower Bound for Cpk 1.52

Cp and Cpk Confidence Interval (lower bound)

Statistics and Confidence Intervals

User Defined Parameters



Potential Pitfalls• Computing process capability measures without

applying statistical methods to assure the test’s purpose is met

Should be able to make a statement about confidence and performance level. For example: “We are 95% confident that the defect rate is 0.001% or less.”

• “Intuitive” acceptance sampling plansExample: “Sample 10% of all lots.” This is not statistically valid because it does not quantify your risk.

• Inspecting quality into the productThe process should be stable and capable before using any sampling plan. The higher the defect rate, the greater the chance that a defective product will escape detection and end up in the hands of the customer.


Key Takeaways for Sampling Plans

• Sampling Plans depend on many considerations:• Risk

• Type of data

• Cost of inspection

• Result of sampling plans is a quantification of risks

• An Acceptance Sampling Plan should provide a pathway for deciding the disposition of the lot

• Use valid statistical rationale—don’t rely on intuition for sampling plans

• Past performance

• Types of failures

• Sources of variation



Webinar Summary

• Statistics is much more than collecting data• Use methods & tools to gain Information• Ask “profound questions” to gain Knowledge

• Valid statistical methods & tools have broad application across a wide spectrum of businesses and industries

• There is great value added by applying statistical tools and methods during the entire product lifecycle


Statistical Methods & Tools Not Discussed Today

(Mod 2)(Mods 5-7)

(DOE Workshop)

(Mods 5-6)

(Mod 3)(Mod 7)

IQ Gauge R&R (MSA)

OQ

Historical Data AnalysisDrawing Conclusions Using Data

Predictions Using ProbabilitySampling & Sample SizesComparing Data

Design of ExperimentsScreening ExperimentsResponse Surface StudiesRobust Design Methods

PQDrawing Conclusions Using Data

Sampling PlansCapability StudiesStatistical Process Control (SPC) Charts

(Statistics Workshop, Mod 4)

(Mod 8)



Statistical Methods & Tools for a Quality System

• Process Capability• Acceptance Sampling• SPC Charts

• Graphical Tools & Historical Data Analysis• Measurement System Analysis• Drawing Conclusions Using Data

• Normal Probability• Sampling• Hypothesis Testing

A 3-day, computer-based workshop Next public offering: October 15-17, 2012


Upcoming AAMI Workshops & Webinars

• Design of Experiments for a Quality System, a 3-day, computer-based workshop, Spring 2013

• Webinars:• Valid Statistical Techniques for Design Control,

May 9, 2012• Design of Experiments for Process Validation,

Oct. 11, 2012• Design of Experiments for Design Control,

Dec. 6, 2012



Questions?

Order of questions that I will entertain:

1. General questions on implementation of statistical methods & tools

2. Time permitting, more technical questions

3. Can contact Dr. Knepell at: [email protected]


Closing Reminder

• Please be sure to fill out the evaluation form at:http://aami.confedge.com/ap/survey/s.cfm?s=StatsPV


introduction to valid statistical techniques for process...

Documents