determination of sampling plans utilizing risk analysis · · 2017-06-15determination of sampling...
Post on 29-May-2018
223 Views
Preview:
TRANSCRIPT
Determination of Sampling Plans Utilizing Risk Analysis
Jeff Hanson, ASQ CQE
Manager Technical Services
Upsher-Smith Laboratories, Inc.
Presentation Agenda
•Process Validation Requirements
•Risk-Based Sampling • Attributes• Variables• Comparison Testing
Process Validation Requirements
1. “The sampling plan must result in statistical confidence”
2. “…samples must represent the batch under analysis.”
3. “The number of samples should be adequate to provide sufficient statistical confidence of quality both within and between batches. The confidence level selected can be based on risk analysis as it relates to the particular attribute under examination.”
Reference - Guidance For IndustryProcess Validation: General Principles and Practices
Definitions• Sampling Plan – A Tool used to make a decision about a population
based on a representative portion of the population. A sampling plan has the following elements: • Sample Size
• Sampling Procedure
• Analyzing Procedure
• Acceptance Criteria
Definitions
• Process Acceptable Quality Level – (AQL) – The limit that defines what percentage of nonconformities is acceptable in a process. This limit is typically established by the quality unit for each Critical Quality Attribute (CQA).
• Confidence – Statistical Confidence is the certainty associated with a statement that is based on of the probability that a statement is true. (This study shows that with 95% confidence the process yields less than 1% nonconformities.)
Operating Characteristic Curve (OC Curve)
• OC Curve is unique for each Sampling Plan
• OC Curve plots the Probability to pass a sampling plan versus the actual defect level in the population
Ideal Sampling Plan
Goal of a Sampling Plan•Always want to ACCEPT a
process that is capable of an acceptably low level of nonconformities •Always want to REJECT a
process that produces an unacceptably high level of nonconformities
Pass Validation
Fail Validation
Process AQL
How do you Create an OC-Curve?• Excel using Statistical Distributions (Google)
• Binomial, Hypergeometric, Exponential
• JMP, Minitab, Sampling Plan Analyzer
• Calculator Example:
Binomial (for accept=0)
P(Pass) = (100-defect level)^n = 4.9%
P(Pass) = (100-1)^300 = 4.9%
Sampling Plan N=300, A=0, R=1
% Defective P(Pass)
2% 0.2%
1% 4.9%
0.50% 22.2%
0.10% 74.1%
0.05% 86.1%
0.02% 94.2%
0.01% 97.0%
𝑷 𝒑𝒂𝒔𝒔 = (𝟏𝟎𝟎% −%𝑫𝒆𝒇𝒆𝒄𝒕𝒊𝒗𝒆)𝒏
Note: this formula works for a=0 only
Why do you create an OC-Curve?
“The confidence level selected can be based on risk analysis…”
• The OC-Curve shows us the elements of Risk that we need to consider to meet the three goals of a sampling plan• Want to Pass a capable process with high probability• Want to Fail a non-capable process with high probability• Want to state with adequate and predetermined
confidence that the process will yield acceptable quality product if it passes the sampling plan.
OC Curve – Answering the Risk QuestionsRisk Questions
• What is the probability of failing the sampling plan? 5%
• What is the probability of passing a bad process? 10%
• What is the probability of failing a bad process? (Confidence) 90%
Thus, if you pass, you can state you are 90% confident that your process will have less than 1.5% nonconformities.
Process AQL = 1.5%Process Yields 0.4% nonconformities
Sampling Plan AQL = 0.4%, and RQL = 1.5%
How to choose the sampling plan!
1. Determine what in the process needs to be sampled?
2. Determine how the process will be sampled?
3. How are the samples going to be analyzed?
4. Determine the Acceptance Criteria required for the sampling plan Confidence Level
What in the process needs to be sampled?
Process Critical Quality Attributes (CQA’s)
Data Types Description CQA Examples
Attribute Data
Discrete data, events that are countable
Number of good/bad tablets in a sample, Number of acceptable fill weights, Pass/Fail data
Variables Data
Continuous numeric data
Fill Weight, Cap torque, Peel strength, Assay value, Tablet hardness
How will the process be sampled?
Samples must be representative of the process?
•Random• Stratified• Time-based•Worst-Case •Non-biased
Process Validation Guidance
“…samples must represent the batch under analysis.”
How are you going to analyze the samples?
Preapproved method for analyzing the samples:
•Formal Test Method•Preapproved procedure within the protocol•Ensure method is clear and repeatable•Perform measurement system analysis
Determine the Confidence Level
Risk elements for a Nonconformity:
• Severity
• Detectability
• Frequency
Process Validation Guidance“The number of samples should be adequate to provide sufficient statistical confidence of quality both within and between batches. The confidence level selected can be based on risk analysis as it relates to the particular attribute under examination.”
Determine the Severity of each CQA
• The Severity is the Process AQL for each of the nonconformity type
• This Process AQL is typically determined by the Quality Unit
Defect Type Process AQL Range
Critical 0 – 0.1%
Major 0.25 – 1.0%
Minor 1.5 – 4.0%
Determine the Detectability
Categorize detectability of the nonconformities in the process with current control methods:
Levels Detectability Current Control Method Examples
Highly Detectable > 90% Automated Vision System
Moderately Detectable 50 – 90%100% Operator Inspection, In-process Controls
Marginally Detectable 10 - 50% Operator Awareness
Not Detectable < 10% No Controls – Not visible
What is the Frequency of the nonconformity?
Estimate percent nonconformity levels based on Data from:•Trial Batches•Similar Existing Processes•FAT/SAT/Commissioning•SME/Vendor Estimates
How is Confidence level chosen?Process Validation Guidance - “The sampling plan must result in statistical confidence”
Justify your Confidence level for each sampling plan based on the Risk elements associated with the nonconformity type
Risk Factor Risk Factor Increases Risk Factor Decreases
Severity Increase Confidence Decrease Confidence
Detectability Decrease Confidence Increase Confidence
Frequency Increase Confidence Decrease Confidence
Typical Confidence Levels
ConfidenceLevels
Confidence Ranges
Examples
High 90-99.9%Major defect, with high frequency and low detectability
Medium 75-95%Major defect with moderate frequency and moderate detectability
Low 50-80%Minor defect with moderate frequency and moderate detectability
In-Process 5-20% Typical ANSI/ASQ Z1.4 and Z1.9 Standards
What is an Attribute Sample Plan?
• n is the sample size
• a is the accept number
• r is the reject number
Sample n objects from the process if you get a or less nonconformities then you pass the validation, if you get more nonconformities then you fail the validation effort.
Example: n=230, a=0, r=1
Choosing the Attribute Sampling Plan
• Confidence required (90%)
• Process AQL (1%)
• Frequency - Nonconformity Estimate (0.2%)
SampleSize (n)
Accept Number
(a)
Sampling Plan AQL
(%)
Sampling Plan RQL
(%)
230 0 0.02 1
388 1 0.09 1
531 2 0.15 1
667 3 0.21 1
Attribute Sample Plan AnalysisProcedure1. Collect representative samples
per the procedure (n-667)2. Analyze and document each
sample3. Determine the number of
Nonconformities in the sample4. If this number is equal to or less
than the A (a=3) specified in your sampling plan you meet the acceptance Criteria
ConclusionMeet Acceptance Criteria- Validation concludes that there is a 90% confidence that the process yields less than 1% defects (yields greater that 99% conforming products).-or-Fail Acceptance Criteria- Validation concludes that there is not adequate assurance that the process will yield less that 1% defects.
What is a Variable Sampling Plan?
• n = sample size
• Acceptance Criteria• k – k-value that is used to calculate
the tolerance range of the population (the tolerance range must be in the specification range to pass)
• MSD – Maximum standard deviation as a percentage of the specification range (used for two-sided tests if you use “Sampling Plan Analyzer”)
Example:
Sample Size n = 50
Acceptance Criteria
k = 2.91, MSD = 16.7%
(MSD is optional reduces sample size slightly)
Choose a Variables Sampling Plan • Confidence required (90%)
• Process AQL (1%)
• Frequency - Nonconformity Estimate (0.2%)
SampleSize (n)
K / MSDSampling Plan AQL
(%)
Sampling Plan RQL
(%)
33 2.89 / 16.39% 0.02 1
69 2.70 / 17.25% 0.09 1
101 2.63 / 17.62% 0.15 1
139 2.57 / 17.89% 0.21 1
Variables Sampling Analysis
Variables Calculations (2-sided spec)
• Calculate Average ( 𝑥)
• Calculate the sample Standard Deviation (𝑠)
• K comes from the sampling plan
• Process Tolerance Range 𝑥 ± 𝑘𝑠
• MSD = 100 𝑠
(𝑈𝑆𝐿 −𝐿𝑆𝐿)
Procedure
1. Collect representative samples per the procedure
2. Analyze and document each sample measurement
3. Perform the calculations
4. If this range is within the specification limits and MSD is less than MSD specified then you meet the acceptance Criteria
Example N=139, k=2.57, MSD = 17.89%
Results:
𝑥 = 31.2
Std. Dev. = s = 1.3
Specification = (28.0 – 36.0)
Calculations:
MSD = 100 𝑥 1.3
36.0 −28.0= 16.25%
Process Tolerance Range31.2 ± 2.57 𝑥 1.3 → (27.9 − 34.5)
Analysis:MSD 16.25% < 17.89% (Pass)
Process Tolerance Range (27.9 –34.5) is NOT within the specification range (Fail)
Conclusion: This study shows that this process is not capable of producing <1% nonconformities with 90% confidence
Variables Sampling Plan Conclusion
Meet Acceptance Criteria-Validation concludes that there is a 90% confidence that the process yields less than 1% defects (yields greater that 99% conforming products).
Fail Acceptance Criteria-Validation concludes that there is not adequate assurance that the process will yield less that 1% defects.
Other Sampling Plan Considerations
N Example Types Examples
1 No Variation in the system Equipment Safeties
3 No Variation in the system but want some repeatability
Vision Systems, Rejection mechanisms, Audit Trail systems
10 No Variation in the system but want a high level of repeatability
Serialization commissioning data
? Justify your sample size
Variability - Sampling is needed because processes have variability. If a process has no variability sampling can be treated differently…
Confidence Level for multipart validationsSingle Trial Confidence (%) 1 Trial 2 Trials 3 Trials 4 Trials
50 50 75 87.5 93.75
63 63 86.3 95 98.8
80 80 96 99.2 99.84
90 90 99 99.9 99.99
95 95 99.75 99.99 99.99+
Probability of passing the testing based on Trial 1 = 95%
95 90.2 85.7 81.5
Probability of passing each trial to have a 95% probability of passing
95 97.5 98.3 98.7
Determining EquivalenceWant to know if two processes are equivalent
Example: Is Process 1 and 2 the same?
Use a Hypothesis Test – (t-test)
•Ho : μ1− 𝜇2 = 0 (Null Hypothesis)
•H1: 𝜇1− 𝜇2 ≠ 0 (Alternate Hypothesis)
•α = 0.05
Risk elements of Hypothesis TestingRisk ElementsProducer’s Risk – Fail a validation when the process is capable of reproducibly meeting the specifications
(1-α = Confidence)
Consumer’s Risk – Pass a validation when the process is NOT capable of reproducibly meeting the specifications
Process
Sampling
Plan
Process is
capable
Process Not
capable
Pass
Validation
Right
Decision
β –
Consumer’s
Risk
Fail
Validation
α –
Producer’s
Risk
Right
Decision
Example – Are the two processes the same?
Process 1• Existing process
• Disintegration Rate• Average = 100.0%
• Standard Deviation 1.0%
Process 2• New process
• Disintegration Rate• Average = 99.8%
• Standard Deviation 1.0%
Classic Hypothesis Testing –Trial 1 - n=50
Hypothesis:Ho : μ1 − 𝜇2 = 0
H1: 𝜇1− 𝜇2 ≠ 0
α = 0.05• Sample size n=50• Test Results:
• Process 1 𝑥 = 100.0, s = 1.0• Process 2 𝑥 = 99.8, s = 1.0
• Analysis:t = 1.00, p = 0.159
Conclusion: Since p > 0.05, fail to reject Ho.
There is no evidence to show that there is a difference between Process 1 and 2 at the α=0.05 level
This does not mean that they are the same
Classic Hypothesis Testing ––Trial 2 - n=300
Hypothesis:Ho : μ1− 𝜇2 = 0
H1: 𝜇1− 𝜇2 ≠ 0
α = 0.05• Sample size n=300• Results:
• Process 1 𝑥 = 100.0, s = 1.0• Process 2 𝑥 = 99.8, s = 1.0
• Analysis:t = 2.45 , p = 0.007
•Conclusion: Reject Null Hypothesis. (p<0.05)
Shows conclusively that there is a difference between Process 1 and 2 at the α=0.05 level
This says that the processes are different!
What does Classic Hypothesis Testing conclude?1. Depending on the sample size of the analysis you get
different results
2. You can bias (manipulate) your conclusion based on your sample size
Conclusion:
Choose a small sample size and they won’t be different
Choose a large sample size and it will show a difference
What have we learned?
•The Classic Hypothesis test does not show Equivalence
•Sample Size is very important the analysis
How can you make a conclusion about Equivalence?
Need to change the question that you are asking:
Is the difference between the two processes important?
Define the important difference between the two processes (∆)
Delta: ∆ = 0.5%
Means that as long as the process mean is within 0.5% of each other the process is not importantly different.
Use Confidence Intervals of the Mean of Differences?Determent the Confidence Interval for the Mean
( 𝒙𝟏 − 𝒙2) ± 𝒕𝟐(𝐧 − 𝟏), 𝟎. 𝟎𝟐𝟓( 𝟐 𝒔 𝒏 )
• N = 50
• 95% CI – (-0.19 – 0.59)
• N = 300
• 95% CI – (0.04 – 0.36)
Determine if it is within the Important difference then you can say there is no important difference!
CI is not within (-0.5 – 0.5)
Not Equivalent – (Not significantly different)
CI is within (-0.5 - 0.5)
Equivalent – (But not equal)
Confidence Interval of Process Difference
Inconclusive Result (n = 50)
Equivalent (n = 300)
-0.5 0 0.5
How much data should you collect?
𝑛 =(𝜎1
2+𝜎22) Zα+ Zβ
2
∆ − 𝛿 2
For our example:
• 𝜎1 = 𝜎2 = 1.0% 𝐿𝐴(from initial trials)
• ∆ = 0.5% 𝐿𝐴 (𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑡 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒)
• 𝛿 = 0.2% 𝐿𝐴 (𝑂𝑏𝑠𝑒𝑟𝑣𝑒𝑑 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒)
• 𝛼 = 0.05
• 𝛽 = 0.20
𝑛 =12 + 12 (1.65 + 0.84)2
0.5 − 0.2 2
𝑛 = 138
α or β Z
1% 2.33
5% 1.65
10% 1.28
20% 0.84
Example Revisited
• Sample size: n=138
• Test Results:• Process 1 𝑥 = 100.0, s = 1.0
• Process 2 𝑥 = 99.8, s = 1.0
• Analysis:
( 𝑥1− 𝑥2) ± 𝑡2(n − 1), 0.025 ( 2𝑠 𝑛 ) =
0.2± 1.97( 2(1) 138 )
=0.2 ± 0.24
95% C.I. for the difference of the means is between (-0.06 – 0.44)
• Conclusion: With 95% confidence, the difference between Process 1 and 2 is less than 0.5.
Thus the processes have no important difference or they are equivalent.
Interpretation of Confidence Intervals of Differences
Confidence Interval of Difference
Inconclusive Result
Important Difference
Equivalent
- 0
Summary
“sampling plan must result in statistical confidence”
“…samples must represent the batch under analysis.”
“…confidence level selected can be based on risk analysis ...”
top related