setting quality standards - aacb

John CallejaMelbourne Pathology ServicesAACB Quality SES, Adelaide: 30th Oct.2014

Setting QualityStandards

Discussion Content

John Calleja (M.P.S) Oct 2014 2

Laboratory Error Monitoring vs. Diagnosis Bias, Imprecision & Total Error 6 Sigma Considerations

Revisit Quality Goals – Setting Quality Standards What hierarchy we should use them in How much imprecision or Bias we can allow

Setting QC Targets & Limits• Referencing Quality Goals

Referencing Quality Goals when Assessing An assay shift Deciding which samples to re-run and which results to amend

following run failure

2


1. Laboratory Error


When patient samples arrive atour laboratories – they may havecome to us for one of two mainreasons ..

• For Diagnosis or Screening• Or for Monitoring..

John Calleja (M.P.S) Oct 2014c 5

Quality requirements Differ Slightlyfor each of these sample types..• For Diagnosis / Screening.

We consider Pat. Results vs. Reference Intervals &often against accepted C/O’s – to Rule In / Rule Out

disease

Bias is Important !

+/-2sd = 95.5%Reference Interval !

If we had aPositive Bias!

Patient classificationscould change!

C/O

John Calleja (M.P.S) Oct 2014c 6

• For Monitoring.

We consider Pat. Results relative to theirprevious results .. for: stability, responseto treatment or signs of deterioration .

GoodPrecision isImportant !

0

200

400

600

800

1000

1200

1400

1600

1800

Trop

onin

-T

Time / Date

Troponin-T Monitoring

Troponin-T

Patient hadInfarct ! Want to have confidence

that the decreases inTnT, signify a +ve

response to treatment

– NOT Imprecision !


Whilst Bias & Imprecision areimportant categories forClassifying Error into ..

in the Laboratory

– We also need to considerthe Clinician !


Total Error … Total Error.. considers error from the view-point of.

Clinicians .. who look at error in “Absolute Terms

It recognises that Analytical Error is made up of the ..two components..

* Systematic* Random

.. but views them as 1 parameter.

General Formula .. TEa = | x – u | + 2sd.

1.65sdFor 90% c.i. or 5% error at 1tail


Total Error

_

True Value

u

Total Error: TEa = 1.65s + |x - u |

x - u

Systematic Error

Observed Value

x_

1.65sRandom Error This principle is further

developed by the Six-SigmaConcept

.. which views Bias &Imprecision as part of a Total

Error Budget


Total Error

_

True Value

u

Total Error: TEa = 2s + |x - u |

x - u

Systematic Error

Observed Value

x_

2sRandom Error TEa = 2s + | x - u |

= (2x1) + | 137-140 |

= 5

Example:u = 140x = 137S = 1


• We don’t really know which of these are for diagnosis& which are for monitoring – so minimising thesources of error for both of these sample categoriesis important to us.

• But Monitoring has the tighter requirements so ourQuality Standards should be set for monitoring.

Coming backto our PatientSamples ..


2. Sources of QualityGoals


Consider ..…

What guidelines or Quality Goals do we have .. to indicate

... the level of quality required

.. or the allowable error, we can permit

.. to ensure that our results are medically useful ?

(eg) If we have a CV of 3% for Na ...is this Clinically Acceptable ?


2.1 Quality Goals- Addressed by ....

1. TONKS - 1958 - 25% of Normal Range

2. BARNETT - 1968 - limits based on the opinionsof Clinicians & Lab personnel .

3. COTLOVE/ - 1970 - Based on Intra-IndividualC.G.Fraser - 1990 Biological Variation.

CVa = 0.5 x CVi

4. Govt Bodies. - 1990 - C.L.I.A. -Clinical LaboratoriesInformation Act - 1988

5. QAP Providers - 1982 - RCPA-AACB QAP

6. Evidence Based - 1992 - HBA1c - DCCT Study


2.2 Quality Goals– Hierarchy Provided by the Profession

◦ Evidence Based Studies DCCT Trial - HBA1c ( CV<2.5%)

◦ Biological Goals - Based on CVI CVa < 0.5 CVi, CVa < 0.25 CVi, CVa < 0.75 CVi

◦ Clinician Survey Barnett .. et al

◦ Profession Defined By group of experts

eg. RCPA-QAP Allowable Limits◦ Proficiency Testing Schemes State of the Art Method .or. +/- 2sd of all results submitted

◦ Publication by a Lab or Group

ISO Technical Committee212 Task Force, 1999

IFCCISOAACCRCPAAACB


2.3 Quality Goals

Cotlove et al - 1970- Studied Intra & Inter individual variation of certain analytes- recommend that :

Allowable Limits of Performance, should be based on ...the Relevant Biological Variation,

Work later expandedon by Callum. G. Fraser

late 80’s / early 90’sOptimal Goal CVa < 0.25 x CVi

Minimal Goal CVa < 0.75 x CVi

- Apex of Hierarchy – Biological Variability

… for a result to be medically use-full.

CV < 0.5 x CV

analytic intra individual biological variation


2.4 Quality Goals- Intra-Individual Biological Variability

Take several BloodSamples over 24hrs 02:00 04:00 06:00 08:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00 24:00

Analyse Samples

CVT = (CVA2 + CVi

2) 1/2

Calculate CV(total CV) CVT

Use calculated CVT & known CVA.. to derive CVI

CVI Gluc = 6.5%

CVi = ( CVT2 - CVA

2 ) 1/2


We can illustrate theimportance of Analytical CVon result interpretation !

.. Using inferences fromthe 1993 DCCT Trial ..


Remember ...

When we analyse a patient sample, thereare 2 components of variation:◦ Analytical CVA &◦ Biological CVi

This is represented as

CVT = (CVA2 + CVi

2) 1/2

5

5.5

6

6.5

7

7.5

8

8.5

9

9.5

10

10.5

11

0 20 40 60 80 100 120 140

HBA1c - Effect of CV on result interpretation- CVa=0 %

low risk

High Risk


5.6 Effect of CVA on Result Interpretation

In the DCCT Trial .. 2 Patient Cohorts◦ In Intensively Treated Cohort -> Mean HBA1c was 7%◦ In Conventional ly Treated Cohort -> Mean HBA1c was 9% . .

CVi (HBA1c) = 3.6% ...◦ We can use this study to illustrate the variation due to CVi alone, at these levels

Depiction of pureBiological SignalOnly !CVT = (CVA

2 + CVi2) 1/2

CVT = ( 02 + 3.62 ) 1/2

CVT = 3.6 %


.. So what happens whenwe add in Analytical Variation

to these signals ?


5

5.5

6

6.5

7

7.5

8

8.5

9

9.5

10

10.5

11

0 20 40 60 80 100 120 140

HBA1c - Effect of CVA on result interpretation -CVA=5.0%

Low Risk

High Risk

5.6 Effect of CVA on Result Interpretation

At the CVA = 5.0% (Typical of old HBA1c methodology) There is a significant overlap between cohorts & the

distinction between the groups is blurred !

Significant Overlap Blurringthe distinction between highrisk and low risk Grp for 2ryDiabetic complicationsCVT = (CVA

2 + CVi2) 1/2

CVT = (5.02 + 3.62) 1/2

CVT = 6.16%


5

5.5

6

6.5

7

7.5

8

8.5

9

9.5

10

10.5

11

0 20 40 60 80 100 120 140

HBA1c - Effect of CVA on result interpretation -CVA=1.8 %

Series1

Series2

Effect of CVA on Result Interpretation

CVT = (CVA2 + CVi

2) 1/2

CVT = ( 1.82 + 3.62) 1/2

CVT = 4.02 %

Good Distinctionbetween Low & HighRisk Cohorts

At the Desirable HBA1c CVA Goal = 0.5CVi = 1.8% There is good distinction between cohorts !


.. So .. minimising our analyticalvariation .. clearly assists in patientresult interpretation !

.. But how much is enough ?

.. Given to us by Callum Fraser


.. Minimising our analytical variation ..assists in patient result interpretation !

.. But how much is enough ?

.. Given to us by Callum Fraser


Effects of Imprecision on Test Result Variability

Ref. Biological Variation – Principles to Practice – Callum Fraser

0.25 CVi adds 3% variability.



Desirable Goal.

% increasein variability.

BiologicalCV Goal.


What level of Biasis acceptable ?


What level of Bias is Acceptable ?

Callum Fraser .. Tells us that .. the Reference Interval ismade up of;◦ With-in Subject & CVi◦ Between Subject Variation CVG

For all of us to use the same Reference Interval, .. theanalytical Bias should be less than ¼ of the groupedBiological Variation.

This is represented as:

BA < 0.25 (CVG2 + CVi

2) 1/2

Becomes aDefault Bias

Goal !


Impact of a Shift or Bias

+/-2sd = 95.5%Reference Interval !

Results in False Positives,> 2.5% out of U.R.L.

Shift in Assaycausing +ve Bias

Results in<2.5% out of

LRL

J.Calleja - Melb. Path. - Oct 2014 30

2.1 Impact on Outcomes (eg) Chol

LDLC = Chol – HDLC - (Trig / 2.2 )

Ref: < 3.0

Chol / HDLC Ref: < 4.5

LDL / HDLC

Ref: < 3.5

CHOL Ref: 3.5 – 5.5

Chol.+ ve Bias

5.3

1.7

1.2

3.3

4.4

2.8

TRIG Ref: 0.5 – 2.0

HDLC Ref: > 1.0

5.8 *

1.7

1.2

3.8 *

4.8 *

3.2 *

5.6 *

1.7

1.2

3.6 *

4.7 *

3.0 *

+ 5% + 10%Baseline

Positive Bias


How much of the Population is Displaced by Bias ?

Ref. Biological Variation – Principles to Practice – Callum Fraser

BA < 0.125 (CVI2 + CVG

2)1/2

adds 2%outside of Ref Interval.

BA < 0.250 (CVI2 + CVG

2)1/2

adds 16%

BA < 0.375 (CVI2 + CVG

2)1/2

adds 34%

% of ResultsOutside of URL

% of ResultsOutside of LRL

When Bias = 02.5% on either

side of Ref Limits

Allowable Bias

% Out ofeach

Ref Limit

Bias Goal.


Combining Bias &Imprecision Goals !

Total Error Goal

<0.125(CVi2+CVg2)1/2 + 1.65(0.25CVi)

<0.25(CVi2+CVg2)1/2 + 1.65(0.5CVi)

<0.375(CVi2+CVg2)1/2 + 1.65(0.75CVi)

Bias Goal

<0.125(CVi2+CVg2)1/2

<0.25(CVi2+CVg2)1/2

<0.375(CVi2+CVg2)1/2


CV Goal

Optimal CVa=0.25CVi

Desirable CVa=0.5CVi

Minimum CVa=0.75CVi

2.5 Combining Bias & Precision Goals for Total Error

• Generally we should aim for Desirable Goals.• If we are easily able to achieve them – then go for Optimal.• If the Biol Variabiity goals are very tight & we can’t achieve

them (eg. Na, Ca) – then go for Minimal.• If we can’t achieve Minimal – then should aim for State-of-

the-Art.


Six-Sigma – is a means ofunderstanding & managingOur error .. in terms of an

‘Error Budget’

It’s one thing tohave a goal - but

you need a meansto achieve it

What is Six Sigma !


Six sigma provides a means to monitor the ‘PerformanceCapability’ of a testing system

It was developed by Motorola in the 1980s, so that they couldvirtually eliminate defective products.

Motorola defined this as having .. ‘Six Sigmas (SDs) ofProcess Variation … fitting within the product tolerances.

The effect of ‘how many SDs you have spanning theproduct specifications’ on the defect rate and defects permillion is:

SD range Defect rate (%) Defects/Million± 2 SD 4.5 45,400± 3 SD < 0.27 ¬ 2,700± 4 SD 0.0063 63± 5 SD 0.0057 0.57± 6 SD 0.000002 0.002

Having a 6-SigmaProcesses -

virtuallyeliminates

defects

Six Sigma


%TEA (%ATE) = Total Error Allowable %◦ Source = Biol .Var. Goals (Opt, Des, Min), RCPA,

CLIA, % Bias = Lab’s Bias vs. True Value (Target Value) % CV = Lab’s B/R precision Implication - Not all

6-Sigma estimatesare equivalent !

%TEA - % BiasSigma Metric = --------------------

% CV

Six Sigma- example.


+ ATE = Allowable TotalError

- ATE

= (6-2)/1Sigma = 3

Ref: Total Analytical Error from Concept to Application – Westgard – Sept 13

Bias = 2SDSD = 1


Various Sigma Metrics vs. Specification Limits

Ref. http://sixsigmatutorial.com/defect-based-six-sigma-metrics-dpo-dpmo-ppm-dpu-yield/276/

Consider theimpact of a

+/- 3sigma shift

LSL USL

Compare :a) 6∂ ..

LSL USL

b) 3∂ ..

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6

Most Desirable

LessDesirable

4-Sigmaupwardsdesirable


Method Decision Chart - Westgard

X-Axis max= (1/2 TEA)

Y-Axis max= TEA

Sigma forLab = 4

Draw Lines for each Sigma, From TEA on Y-Axis to:◦ TEA/2 on X-Axis for Sigma = 2 Line◦ TEA/3 on X-Axis for Sigma = 3 Line … etc

Plot Lab “Operating Point,” Bias & CV .. to give Sigma(=4)

Adv: Allowsyou to

Visualise toeffect of Bias &CV on Sigma


Sigma SQC Selection Graph - Westgard

◦ ∆SEcrit = [(ATE – Bias)/SD] -1.65◦ [(ATE – Bias)/SD] = Sigma Metric◦ Sigma = ∆SEcrit + 1.65

Allows Graph tobe rescaled interms of Sigma

Draw vertical linecorresponding to

test Sigma

Read off which QC RulesIntersect w Sigma Line

@ 90 % Pr. of Rejection.

Power FunctionPlot

The size of themedicallyimportantsystematic

error

Various QC Rules


Should we worry about Bias- If so when ?

So when we are considering6 Sigma -


Should we Worry About Bias ? In the US – Yes

◦ Labs are assessed in the context of Regulatory Goals (CLIA) which are aboutTotal Error – of which Bias is a component

In Australia◦ We need to distinguish between Lab Bias & Method Bias

If Lab Bias◦ Definitely -Yes◦ Because – this generally indicates some sort of error in our

implementation of our method. If Method Bias

◦ The general position has been that … so long as we have areference interval that is appropriate to our method - then - No.

However – The pressure is increasingly - “Yes”◦ The existence of accepted Diagnostic, Action & Treatment cut-offs HBA1c, Glucose, Cholesterol, TSH

◦ Emergence of Common Health Records◦ Development of Common Reference Intervals


Effects of usingdifferent TE goals on

Six-Sigma

“Six Sigma” is notan “absolute”Specification !


Comparison of Sigma’s by different TEgoals

Various Goals.Labs Means &

SDs & CVs

AchievedSigmas

Analytes


Comparison of Sigma’s by different TEgoals

Various Goals.Labs Means &

SDs & CVs

AchievedSigmas

For both qclevels


Comparison of Sigma’s by different TE goals –Glucose

Example for Gluc. @ 2 QC concentrations CVs achieved are b/w optimal & desirable. Sigma’ s calculated for different TE Goals. Sigma achieved is different at different concentrations – for above, better at High

Conc. Sigma is better when you have no Bias

◦ Data Shown with & without a slight +ve Bias vs Targets of < 2%

Sigma is “Not an Absolute Parameter”-> If you are quoted a Sigma for a method – Keep all the above factors in

mind.

No Bias

With Bias

CV’s Des. -Opt

Ask: - Which TE goal used?

– Has Bias been included?

Sigma Varies, dependingon which TE Goal

selected

TE Goals -BV, RCPA,

CLIA


Can View Visually – IgA - QC Unity Real Time

Nominate whichTE Goals to use

for givenAnalyte

SuperimposesCalculated Total

Error Range

TE

Plots LJ Plotagainst assigned

Target & SD limits.

SIGMA =(13.5-0.85) / 2.57

= 4.9


3.0 What steps should wefollow when we set our QCTargets & Limits.


3.1 Target Values.


3.1 Various Sources ofTarget Values

QC Package Inserts Consensus Values from a centralised QC DataBase eg. BioRad’s QC-Net, Unity Real Time Set by Your Own Laboratory

.. Will not Discuss Target Values further- Focus will be on Allowable Limits

But – ( some Slides on Target Setting included )


3.1.1 Manufacturer QC Kit Insert Values

The insert statesinstrument / method

specific values !

FT4: +/- 3sd QC Limits7.36 – 11.3 pmol/L22.3 – 35.9 pmol/L52.7 – 80.8 pmol/L


3.1.2 Consensus Values from QC Database

World-Wide ReportMethod Specific Consensus

Values for QC Lot !

DisAdv: Lot must alreadybe in use by other

participants to be of use.


3.1.3 Establish Your Own Target Values** Establish by

- Running the QC Material Several times

.. For all relevant tests

.. On all relevantInstruments / Modules / Channels.. Over a number of

days / runs / calibrations /operators .... Thereby exposing the process to as many possible

sources of variation, as is practical.

- Run the material passively, in parallel to your Existing QCs.

- When sufficient Data has been accumulated;

- Calculate Mean & SD- Remove any Outliers (Exclude values > 3sd)

- Re-Determine the Mean- Set this as the .... Target Value


3.1.3.1 How Many Data Points ?

** Statistic: Standard Error of the Mean (SEM).

This statistic gives us the Amount of Error, associatedwith a given Target Value .. Given the

- SD of the data set &- the Number of QC observations.

The Accuracy of a Target Value .. Increases ...with an Increasing Number of Observations.

S where S = S.D.SEM = --- n = no. of control

n observations.


0

10

20

30

40

50

60

0 1 2 3 4 5 6

Error of the Mean ( Arbitrary Units )

No. of Results

n = 30

n = 5

...No AppreciableDecrease in SEM

for n > 20-30

n = 20

3.1.3.1 How Many Data Points ?- Standard Error of the Mean (SEM)


3.1.4 What if you haveMultiple Instrumentsmeasuring the same test ?


Instrumentsof the same type ?

or ..On more than 1 measuringchannel, of an instrumentof the same type ?

Measure Cell 1 & 2

Instrument 1, 2 & 3

Between.. … DifferentInstrument types ?

Instrument 1, 2 & 3

YesNo


3.1.4 Eg. MPS Glucose – Cobas c701- Are the recoveries across the 3 different c701s .. < 0.33CVi ?

Line 1

Line 2

Line 3

CV Goals, Opt, Des, Min = 1.63, 3.25, 4.9% / Bias Goal 0.33CVi = 2.15%

Max Bias L1: (4.72 – 4.75) = 0.03 | 0.03/4.735 = 0.63%

Max Bias L2: (15.48-15.4) =0.08 | 0.08/15.44 = 0.52%

Yes !Inter-Instrument

Bias is < 0.33CVi


1.4.2 Recoveries differ for different instrumentswith different methodologies & calibrationtechniques

You generally, Can’t use the same targets,across different instrument types !

Roche e602

Siemens Vista

Vitros ECi

Abbott Architect

PSA


3.2 What Steps should weFollow when we set our QCLimits !


Step 2 - Put the labs achieved CV performance into perspective ..Compare to the Instrument / Reagent Manufacturer’s .. statedspecifications for Total CV - Refer to the manufacturers kit insert

Step 3 - Consider the Relevant Quality Goals ( CV Desired) , from the ISO TC212Hierarchy ; Evidence Based / Biol. Var’n / QAP Allowable Limits

Step 4 - Consider the assays Capability – or Six Sigma .

3.2 Setting Appropriate QC Ranges- Steps Involved

Step 1 - Review the SDs & CVs achieved from The Target Setting Studies- Compare this to the Labs Ongoing CVs at similar concentrations

Step 5 - Put all of this information together to determine what the allowable CV( or Quality Goal ) .. for your assay, should be .

Following Running-in Studies of the New QC.. Consider Quality Goals & Method Capability

– Don’t just settle on Mean +/- 2sd !


Tgt = 1.00

SD = 0.051

CV = 5.1%

Range:

Min = 0.9

Max = 1.1

Tgt = 2.2

SD = 0.1

CV = 4.5%

Range:

Min = 2.0

Max = 2.4

Lab about to set targets & sd’s for 2qc’s for Trigs on a Beckman CX7 ..Based on achieved performance …

QC Level-1 QC Level-2

Setting QC Ranges- Acceptable Limits

Are theseranges

appropriate?


? Appropriate QC Limits

3.2.1 Compare Evaluation QC CV toCV of Current QC Lot


Trig - Expected Performance

CVs of current QCbetter (ie)

2.52% vs. 3.07%

Current QC

Evaluation QC



3.2.2 Manufacturer Specifications


Trig - Expected Performance

The Lab’s QCCVs are not as good

@2.0mmol/L

Note: ExpectedCVs

Tgt = 1.00

SD = 0.051

CV = 5.1%

Range:

Min = 0.9

Max = 1.1

Tgt = 2.2

SD = 0.1

CV = 4.5%

Range:

Min = 2.0

Max = 2.4

@ level

3.2



3.2.3 Biologically Based CVa Quality Goals


? Appropriate QC LimitsRefer to “Westgard Web-site”

www. westgard.com

CVi = 20.9%


Lab’s CV Goals areWithin OptimalBiological Goals

Biological Variability

CVi (Triglyceride) = 20.9 %

1. Calculate theAnalytical CV Goals

CVa Minimal = 0.75 CVi= 15.75%

CVa Desirable = 0.5 CVi= 10.5%

CVa Optimal = 0.25 CVi= 5.25%

Tgt = 1.00

SD = 0.051

CV = 5.1%

Range:

Min = 0.9

Max = 1.1

Tgt = 2.2

SD = 0.1

CV = 4.5%

Range:

Min = 2.0

Max = 2.4

2. Compare Lab’s CVsw Biological Goals

Labs Trig QC Goals vs. Biological Limits



3.3.4 QAP Allowable Limits


? How do the Lab’sCV goals

Compare with QAPAllowable Limits

Labs Trig QC Goals .vs. QAP Limits

Tgt = 1.00

SD = 0.051

CV = 5.1%

Range:

Min = 0.9

Max = 1.1

Tgt = 2.2

SD = 0.1

CV = 4.5%

Range:

Min = 2.0

Max = 2.4

Labs CVs


Labs Trig QC Goals .vs. QAP Limits -

What CV do we needto achieve 2,3,4,5,6

Sigma within the QAPAllowable Limits .. ?

Sigma 1sd CV .

2 Sigma (1.71–1.51)/2 = 0.1, (0.1/1.71)x100 = 5.8%

3 Sigma (1.71-1.51)/3 = 0.067 (0.067/1.71)x100 = 3.9%

4 Sigma (1.71-1.51)/4 = 0.05 (0.05/1.71)x100 = 2.9%

5 Sigma (1.71-1.51)/5 = 0.04 (0.04/1.71)x100 = 2.3%


Labs Trig QC Goals .vs. QAP Limits - 2 Sigma

What CV do we needto achieve 95.5% ofall results .. withinthe QAP Allowable

Limits .. ?

• One way we can consider the QAP Target & Allowable Limitsis as .. mean +/- 2 sd (95% C.I.)

• So .. 1sd = (1.71 – 1.51) / 2 = 0.1

• So .. CV required is .. ( 0.1 / 1.71 ) x 100 = 5.8 %


Labs Trig QC Goals .vs. QAP Limits – 3 Sigma

What CV do we needto achieve 99.7% ofall results .. withinthe QAP Allowable

Limits .. ?

• Can consider Allowable range as mean +/- 3 sd (99.7% C.I.)

• So .. 1sd = (1.71 – 1.51) / 3 = 0.067



Labs Trig QC Goals .vs. QAP Limits – 4 Sigma

What CV do we needto achieve results

within +/-4 sigma ..of the QAP Allowable

Limits .. ?

• Can consider Allowable range as mean +/- 4 sd (99.994% C.I.)

• So .. 1sd = (1.71 – 1.51) / 4 = 0.05




3.3.5 State of the Art Performances


? Appropriate QC Limits – State-of-the-Art

Method Median CV = 3.5%

50th percentile CV = 3.2%20th percentile CV = 2.4%

Ranked Within Lab CVs


3.4.6 Put it all Together

Transform Inputs .. into .. Appropriate QC Ranges

Give the Lab QC Limits w aSigma that as best as possible

insulate it against theundesirable impact of shifts &

increases in imprecision


Put it All Together !

Tgt = 1.00

SD = 0.051

CV = 5.1%

Tgt = 2.2

SD = 0.1

CV = 4.5%

Lev 1: Tgt =1.01SD=0.033, CV= 3.3%

Lev 2: Tgt=2.2,1SD=0.065 CV=3.0% 79

Lab’s Evaluation CVs

CVa Minimal = 0.75 CVi= 15.75%

CVa Desirable = 0.5 CVi= 10.5%

CVa Optimal = 0.25 CVi= 5.25%

Manufacturers CV Specs.

RCPA QAP ALEs

QAP State-of-the-Art

Biological Goals

2 sigma QAP ALECV = 5.8%3 sigma QAP ALECV = 3.9%

Lab’s Ongoing CVs

Tgt = 0.95

SD = 0.031

CV = 3.3%

Tgt = 2.0

SD = 0.06

CV = 3.0%

50th %CV = 3.2%20th %CV = 2.4%

Method CV = 3.5%

Set Targets to:

4 sigma QAP ALECV = 2.9%

3.2



3.4.7 Consider Six Sigma Capability


Method Decision Chart – CalculateSigma

X-Axis max= (1/2 TEA)

Y-Axis max= TEA

Sigma forLab = 4.24

Draw Lines for each Sigma, From TEA on Y-Axis to:◦ TEA/2 on X-Axis for Sigma = 2 Line◦ TEA/3 on X-Axis for Sigma = 3 Line … etc

Plot Lab “Operating Point,” Bias & CV .. to give Sigma(=4)

T.E. Goal = Biol.Var. TE Goal

Desirable = 14%


Consider Lab’s Sigma Capability

+ ATE =14%Allowable Total Error

- ATE = -14%

Sigma = 4CV=3.3 % x 3

=9.9%

Can tolerate a+/- shift of:14 - 9.9 = 4.1%

LSL USL

GoodCapability !


3.4.8 A laboratory Toolto Assist the Process


Laboratory Tool - Word Template / 2 Pgs./ Aanalyte

1. LJ Plots – All ActiveChannels

3 Levs - Current & Evaln. QC

7.Manufacturer

PrecisionSpecs

2. Summary Stats –Current & Evaln.

QC

8. Comments & FinalQC Target & 1sd

Settings

3. QAPPerformance Latest

Report

4. BiologicalGoals

Opt, Des, Min

6. State-of-the-Art20th,50th & Mthd

CV

5. QAP ALEs &2, 3 & 4 sigma

CVs


Laboratory Tool - Word Template / 2 Pgs./ Aanalyte




QC3. QAP

Performance LatestReport

4. BiologicalGoals

Opt, Des, Min


CVs


CV

7.Manufacturer

PrecisionSpecs8. Comments & Final

QC Target & 1sdSettings



7.Manufacturer

PrecisionSpecs


QC

8. Comments & FinalQC Target & 1sd

Settings

3. QAPPerformance Latest

Report

4. BiologicalGoals

Opt, Des, Min


CV


CVs


4.0 Using Quality Goalswhen Problems arise !


4.1 How should we Assess /Action a Systematic shift ?


Assessing / Actioning a ShiftSteps to Take:

Attribute a Cause Eg. Reagent or Cal Lot Change Reagent Reformulation

Assess Magnitude of Shift Percent deviation from QC Target or %Bias Pre & Post shift “patient comparison studies” Patient Data Extract – movement in averages & percentiles

Assess Clinical Relevance Clinical Consultation, 0.25 x (CVI

2 +CVG2)1/2, 0.33CVi

Consultation with Manufacturer about a corrective action. New Lot Number of Calibrator / Rgt Calibrator Set Point Reassignment

Change Pt. Ref. Intervals, to compensate for the shift

Apply a corrective Slope &/or Offset to results◦ Derived from Pre & Post shift .. patient comparison studies Examination of influence of shift on patient averages

Change QC Targets

TakeCorrective

Action


4.1.1 Attribute the Causeof the Shift.


Attribute the Cause ?

Start w LJ Plots Draw in dates of calibrator & reagent Lot changes etc .. Rule out Faulty Reagent/ Poor Calibration Attribute the Cause, of the shift -> Shift due to Reagent Lot

Change (Lot 602091)

eg. GGT

R: 23/8 602091

C: 27/8

R: 6/6 698158 R: 18/7 600696

C: 2013 169513


Attribute the Cause ?

Start w LJ Plots Draw in dates of calibrator & reagent Lot changes etc .. Attribute the Cause, of the shift -> Shift due to Reagent Lot

Change (Lot 602091) Estimate Bias Magnitude Assess Significance % Chng < Optimal => Not Sig.

eg. GGT

Target Shift to Diff % Diff159 162 3 1.9 %

R: 23/8 602091

C: 27/8

R: 6/6 698158 R: 18/7 600696

C: 2013 169513


4.1.2 Assess Magnitudeof Shift.


eg. Perform Pt. Comparisons(eg) Assess Ca++ Bias .. Post vs. Pre New Lot of Calibrator

Y = 1.0965x – 0.1723

At URL: 2.6 mmol/LNew Result = 2.679

Bias = +0.0786%Bias = 3.02%

4.1.2.1 Estimate Magnitude of Shift

Results w Fomer Lot

ResultswNew Lot


4.1.2.2 What if we initiallymissed the shift ..

&.. can’t perform

patient samplecomparisons ?


4.1.2.2 Perform Patient Data Extracts

Extract Data from LIS for a period including before &after the shift

Calculate “moving medians”..per 30 .. or .. per 1000 sample results (if lots of data)

Plot .. Moving Median vs. Date/Time Examine for a significant shifts / Estimate Magnitude

Most useful.. when the detection of a shift hasinitially been missed.

How …

- Plot Patient Medians/Percentiles


4.1.2.2 Patient Data Extracts

Write Formulas tocalculate the median forthe preceding 30 samples

=Median(F2:F31)

=Median (F3:F32)

=Median(F4:F33)

- example: raw data

Copy Formulas down spreadsheet .. toget rest of .. rolling 30 sample medians.


4.1.2.2 Example: Plot the MovingMedian versus Date/Time …

Moving MedianSensitive to the ShiftBias ~ 0.15 mmol/L


4.1.3 Assess Significanceof Shift.

Bias Goal

<0.125(CVi2+CVg2)1/2

<0.25(CVi2+CVg2)1/2

<0.375(CVi2+CVg2)1/2


Optimal

Desirable

Minimum

4.1.3 Assessing Significance of Shift

• Assess Shift againstBiological VariabilityBias Goals !


Callum Fraser ‘s .. Quality specifications ..for the allowable differences between twomethods .. used to analyse the same analytein the same laboratory;

4.1.3 To assess differences b/wtwo instruments.

Allowable difference < 0.33 CVi


4.1.3 Estimate Significance of Shifteg. Pt. Comparisons for Ca++ Bias.. Post vs. Pre New Lot of Calibrator

Y = 1.0965x – 0.1723


Bias = +0.0786%Bias = 3.02%

1. Assess against, Inter-instrument Bias Goal:

0.33CVi (1.9) = 0.63%2. Assess against, Opt, Des & Min Biol. Bias Goals:

Desirable Bias: 0.250 (CVI2 + CVG

2)1/2 = 0.85%Minimal Bias Goal: 0.375 (CVI

2 + CVG2)1/2 = 1.27%

Our %Bias =3.02% -> Worse .=> Significant !


4.1.3 Estimate Significance of Shifteg. Pt. Comparisons for Ca++ Bias.. Post vs. Pre New Lot of Calibrator

Y = 1.097x – 0.172


Bias = +0.0786%Bias = 3.02%

Assess against, Opt, Des & Min Biol. Bias Goals:Optimum Bias: 0.125 (CVI

2 + CVG2)1/2 = 0.43%

Desirable Bias: 0.250 (CVI2 + CVG

2)1/2 = 0.85%Minimal Bias Goal: 0.375 (CVI

2 + CVG2)1/2 = 1.27%

Our %Bias =3.02% -> Worse .=> Significant !


4.1.4 What if we decide wehave a ClinicallySignificant Shift ..


4.1.4 Could Derive a Corrective Slope &Offset to realign the performances

Y = 1.097x – 0.172

Derive Corrective Slope & Offset(from Regression Eq):

Method Alignment Improved

Re-Perform Comparisons withSlope & Offset installed !

Corrective Slope = (1/1.097) = 0.91

Corrective Offset = (0.172/1.097) = +0.16


4.1.5 What do we do withQC Targets ? ...


4.1.5.1 What do we do about QC Targets ?

If Shift assessed as Clinically Tolerable

◦ Calculate the mean values for all relevant QC levels.. from the data after the Shift !

◦ Re-Assign the QC Target Values to these values

If Shift assessed as Clinically Significant◦ If Corrective Factors Implemented Should be no need to amend QC Targets

◦ If Other Corrective Action Taken(eg) Reference Intervals modified ..

due to assay Re-Standardisation Calculate the mean values for all relevant QC levels

.. from the data after the Shift ! Re-Assign the QC Target Values to these values


4.1.5.2 What do we do about QC Limits ?

If we have already carefully calculated CV Goals ..-

Re-calculate our SDs .. to achieve equivalentCV goals, at the new target concentration.

◦ CV % = (SD/Mean ) x 100◦ If Former (Pre-Shift) Target = 100, SD = 5, CV Goal = 5%

◦ If Assay Shifted to Mean= 80 To maintain a CV Goal of 5% Re-arrange CV equation to solve for SD SD = (CV/100) x Mean SD = (5/100) x 80 SD = 4


4.2 How should we Assess /Action .. a major QC Failure(eg) due to an Inst. Failure


Considering the Scenario !

◦ Successful QC Event .. @ 10.00am Patient n= 1 Patient n= 2 Patient n= .. Patient n= .. Patient n= 50 Patient n= 51 Patient n= .. Patient n= . . Patient n=100 Patient n=101 Patient n= … Patient n= … Patient n= 150 Patient n= 151 Patient n= .. . Patient n= … Patient n=200

◦ Failed QC Event, > 3SD .. @ 2.00pm

200 Samplesrun in-betweenlast successful

QC & Failed QCEvent !


4.2. What do we do with PatientResults that were Reported inthe Affected Run ?

◦ Consider that the failure .. may have occurred at any-time after the last successful QC event and the knownfailure.

◦ Therefore need to identify an accurate failure time-point!

◦ Procedure: Select representative samples from in-between the last

successful QC event right upto the failed QC Event Pick two to three samples in every 20 - 30 samples (depending

on size of failed batch) – but at shorter intervals towards failedQC Select the samples chronologically Re-run the selected samples Assess all samples for significant differences between the

repeats


Pin-Pointing where run failed !

◦ Successful QCEvent Patient n= 1 Patient n= 2 Patient n= .. Patient n= .. Patient n= 50 Patient n= 51 Patient n= .. Patient n= . . Patient n=100 Patient n=101 Patient n= … Patient n= … Patient n= 150 Patient n= 151 Patient n= .. . Patient n= … Patient n=200

◦ Failed QC Event

◦ Selected Patients Patient n= 1 Patient n= 2 Patient n= 30 Patient n= 31. Patient n= 60 Patient n= 61 Patient n= 90 Patient n= 91 Patient n=120 Patient n=121 Patient n= 150 Patient n= 151 Patient n= 180 Patient n= 181 Patient n= 190 Patient n= 195 Patient n= 200

◦ Failed QC Event

Failure mayhave

occurredanywherewithin the

Batch !

SelectRepresentative

Patientsamples at

regular Time-Points fromBatch -> for

Re-run Checks

Check atsmallerintervalstowards

failed QC

Run Samples &Check where

SignificantDifferences

appear.

If FailurePin-pointed tobetween 150

& 180onwards ->

Rerun allpatients afterpatient 150


4.2.1 Assessment of Differencesin Repeats –

Criteria Derived from :

For serial analysis of the same sample◦ Intra-individual Variation is N/A◦ Simplifies to:

Where .. Z= 1.96 (95% c.i.)

When the same sample is repeated .. asignificant difference is .. If a repeat resultdifference is

RCV (Reference Change Value) = √ 2 x Z x √ ( CVa2 + CVi

2 )

> 2.77 CVa or if the result classificationchanges

= √ 2 x Z x √ ( CVa2 ) = 2.77CVa

AnalyticalVariation

Intra-Individual BiologicalVariation

Two analysis

CoverageFactor

or z=1.65 (unidirectional)

> 2.33 CVa or if the result classification changes


4.2.2 ∆SE or ∆RE critical

Critical Systematic or Random Error that wouldSHIFT or WIDDEN the result distribution enoughto exceed the allowable T.E. specification.

When the same sample is repeated .. asignificant difference is ..

∆SE critical = [ (TEA – Bias ) / CV ] –1.65

% Difference in Results > ∆SE Critical .. or∆RE Critical

Lab CV%Total Allowable Error % Lab Bias %

∆RE critical = [ (TEA – |Bias| ) / ( 1.65 xCV )


Previously Reported ResultsExample: Trig. Problem - 2.77CVa method

Affected Samples ->Amend Results !


Recovery from a Failed Run !

Final Step:

◦ If Repeated Result > 2.77CVa or considered clinicallysignificantly different (eg. Classification Change or byClinical Consultation – w Chemical Pathologist ) Amend Result Re-report with an Amended Report Comment if Critical Change in Result – Phone Dr.


Patient samples come to us for Diagnosis & Monitoring Monitoring has the tighter quality requirement -> Quality

Standards should be based on Precision The Total Error concept & Six Sigma assists us in

considering both. When setting QC Targets & Limits - Consider Quality

Goals & in what Hierarchy to apply them. Construction of a Template incorporating all the relevant

Goals & Information may help the Process Quality Goals can also be used to assist with Assessing a Systematic Shift. Assessing which patients may need to be repeated

when there is a significant Run Failure.

8. Summary

J.Calleja - Melb. Path. - Oct 2014117

References

Chapter 19


The End ...

Thankyou for your attention !

setting quality standards - aacb

Documents