Setting of quality standards

Graham JonesDepartment of Chemical Pathology

St Vincent’s Hospital, SydneyAACB ASM – Adelaide – October 2014

Setting of Quality Standards - 2013

• The 2013 QC workshop revealed that mostAustralian laboratories are not consciouslyadopting performance goals and many are notensuring that their QC algorithms have highdetection of critical errors.

• This is worrying to say the least ….

• Quality Limits are numerical limits for a testoutside which we do not wish to release results(only one of us had them)

NPAAC

G4 Laboratories should set routine performancegoals … based on the clinical use of the testresults.

Approach to Quality Specifications

IFCC, IUPAC, WHO Consensus Conference, Stockholm 1999www.westgard.com/stockholm.html

Stockholm Consensus Conference on QualitySpecifications in Laboratory Medicine

1. Studies on clinical outcomes2. Clinical decisions in general, data from:

– biological variation– clinicians’ opinions

3. Published professional recommendations4. Performance goals set by regulatory bodies or

organisers of External Quality AssessmentSchemes.

5. Goals based on the current state of the art asdemonstrated by data from EQA or from current

Approaches

• Set / Select Quality Limits - design system tomeet limits– 6 sigma– Capability

• Assess performance against standards (UM)– Optimal, Desirable, Minimal– Improve where needed

• Reverse Engineering– Know how good you are

Quality Limits

• Eg: AST result should be within 20% of correctresult (CLIA Guidelines)

• Questions arising1. What is correct result?2. How do we achieve this?

Capability

ImportantChange

Sigma Metric

σ = Change/SD

Capability

ImportantChange

σ = 6, good

1 SD Shift

2 SD spreadσ = 4, OK

σ = 3, poor

2 SD Shift

4 SD Shift

Required shiftdetection

Sigma Metric

Capability

ImportantChange

Sigma Metric

σ = Change/SD

Capability

ImportantChange

Sigma Metric

σ = (Change-Bias)/SD

Power Function Graph (n=2)

12.5s

13.5s

MR

12s

Quality Limits

• Meets process needs• Set goals based on clinical need in advance• Establish QC protocols based on:

– Defined need– Assay Precision

• Issues:– What are the limits to use?– How do I handle Bias?

Assess performance against standards (MU)

• Run assay over a period of time• Obtain Precision data from QC• Compare performance against Highest level of

Stockholm criteria

• Typically within-subject Biological Variation (Cvi)– Level 2a– Compare assay precision (Cva) against CVi

Precision Goals

0%

5%

10%

15%

20%

25%

30%

35%

40%

0 0.2 0.4 0.6 0.8 1

CVa / CVwi

Incr

ease

in to

tal C

V (%

)

Optimal (0.25, 3%)

Minimum (0.75, 25%)

Desirable (0.5, 12%)

Assess performance against standards (UM)

Stockholm Level 2a

• CVa < 0.25 CVi - Optimal• CVa < 0.5 CVi - Desirable• CVa < 0.75 CVi - Minimal

UM Approach

• UM is required by ISO 15189 and NPAAC• Assess performance using real data

– Typically several month’s QC data– For new assay use limited run-in data

• But– Analysis performed AFTER setting up assay

(can use manufacturers estimate of CVa)– QC data may have outliers excluded– If fails higher standard (eg BV), use lower limit

(eg state of the art)– Assessment not planning

Reverse Engineering

Recommended process:

• Set goals, use processes to meet goals

Reverse

• Assess processes to understand performance

“How bad may a result from my lab be?”

Understanding our assays

• For any assay, with the QC protocol in place, weshould be able to say how much analytical errormay occur.

“These rules have the power to cause a STOP 90% ofthe occasions when there is a shift in the assay of 2.8x LSD and cause a PAUSE 90% of the occasionswhen there is a shift in the assay of 2.6 x LSD.”

SydPath Quality Control SOP

So far…

• Quality Limits

• UM

• Reverse Engineering

Quality Limits

• What Limits?

http://www.datainnovations.com/products/ep-evaluator/allowable-total-error-table

http://www.dgrhoads.com/db2004/ae2004.php?B1=Chemistry+A-C&find=&start=1&NOLINKS=

RCPAQAP(%) 5.0 10.0 8.0 15.0 7.8 15.0 15.0CLIA (%) 10.0 20.0 20.0 17.0 30.0 30.0Range: 3-18 5-14 5-22 3-21 5-18 7-30 10-56

RCPAQAP ALP

Limits/acceptability

Meaning of ALP

Basis“Total Error” – Can share reference interval“Imprecision” – Can Monitor across labs

Level“Optimal” – no need to improve“Desirable” – satisfactory“Minimal” – just satisfactory

Revision of ALP

ALP are applied to Total ErrorUsed in interim reportsSingle results include bias and imprecision

Will use categories of CV:1,2,3,4,5,6,8,10,12,15,20,25,30%

Round to nearest category

Change between absolute and percentagebased on precision profile

Revision of ALPTop category (Imprecision):

Within-Subject Biological Variation (Opt, Des, Min)MonitoringSingle Laboratory reaching standard:

Can monitor a patient at labMany Labs within standard:

Can monitor a patient between labsCan share reference intervals

Revision of ALPNext Category:

Total Error (Opt. Des, Min)Within and between subject BV combinedDiagnosisSingle Laboratory reaching standard:

Satisfactory performanceBias and / or precision target for improvement

Multiple Labs meeting standardLikely to be able to diagnose at different labs

(share reference intervals)

Revision of ALP

Final CategoriesState-of-the-art

If unable to meet a higher category

Expert OpinonIf suitable data not available

Using QAP ALP as Quality Limits

Vanessa Lo

Sigma Metrics as Performance IndicatorContributes to Effective Cost and

Man-hour Saving in Chemical PathologyLaboratory

Department of Clinical PathologyChemical Pathology Laboratory

Hong Kong Sanatorium and HospitalRoche Oral Presentation PrizeAACB ASM 2014

>= 5 Sigma17 analytes

ALTPancreatic Amylase

Total AmylaseAST

Direct BilirubinCK

GlucoseGGTLDH

MagnesiumPhosphateTriglyceride

Uric AcidBicarbonate

HDL-CLDL-CCRPLX

4 Sigma5 analytes

ALPCalcium

CholesterolIron

Total Protein

3 Sigma7 analytes

AlbuminTotal Bilirubin

CreatinineUrea

SodiumPotassiumChloride

Chemistry

16* 7 4

SydPath*

AlbuminSodiumChlorideBicarbonate

>= 5 Sigma12 analytesCKMB-STAT

pro-BNPhs-TNT

beta-HCGE2

FSHLH

ProgesteroneProlactin

ThyroglobulinPTH-STAT

Ferritin

4 Sigma1 analyte

B12

3 Sigma2 analytes

SFolateVit. D Total

Immunoassay

Sigma No. of QCdaily

No. of Analytes

Chemistry Immunoassay

>=5 1 17 12

4 2 5 1

3 3 7 2

Vanessa Lo

• Reduction in QC performance

• Saving of 1.6% of reagent costs

• “Uprising in the emotional status of the staff”

USE OF ALP as Quality Limits

• >5 Sigma: Can have confidence that results willbe within RCPAQAP ALP using simple rules

• 4-5 Sigma: need tighter rules to be sure resultswill meet RCPAQAP ALP

• <4 Sigma: Cannot be confident that Results willmeet RCPAQAP ALP

Will see poor results?

RCPA General Serum Chemistry

CYCLE 90 1,2 3,4 5,6 7,8 9,10 11,12, 13,14 15,16Working mean 0.21 0.25 0.24 0.22 0.26 0.18 0.24 0.29Working 80th 0.33 0.40 0.37 0.40 0.38 0.33 0.40 0.45

Outliers 0 0 0 0 0 0 0 0

No outliers* in entire cycle for any analyte!(35 analytes, 1120 Results)

* Method Specific Targets (includes bias)

RCPA General Serum ChemistryCycle Flagged Test

91 4 PO4(2),FT4,GGT90 089 3 hCG(2), Cl88 11 FT4(5),HDL(2), Bic,Lip,Na,K87 2 Ca, FT485 3 Ca, FT4(2)84 1 Ca79 4 Ca,Ferr,TG,Bil,Ca76 8 Ferr,Cl,Bic(2)gluc,Cbil75 1 Ferr

TOTAL 37 0.66% of total

> 3 years: No albumin failures, 1 Na, 1 CO2, 2 ClAll Incapable (Sigma <4)

Roche Modular BCG Albumin (2014)

Albumin

QC Level1:CVa = 2% @ 31 g/L (ALP 6.4%), Sigma = 3.2

QC Level 2:Cva = 1.9% @ 46 g/L (ALP 6.0%), Sigma = 3.2

Albumin CVi = 3.1%(both QC levels meet minimal Standard)

An incapable assay – how does it succeed?

Assay Characteristics

Stable assays:• Performance defined by mean and SD• QC never fails• Results “always” within +/- 2SD

Stable Assay

14

16

18

20

22

24

26

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49

QC-3SD-2SDMean+2SD+3SD

Mean = 20, SD = 195% of QC results between 18and 22.

Assay Characteristics

Unstable Assays• Mean drifts over time (fluctuating bias)• QC process used to detect drifts• Variation in results due to scatter plus drift

Unstable Assay

14

16

18

20

22

24

26

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37

-3SD-2SDMean+2SD+3SDQC

Mean = 20, SD = 1, Plus fluctuating mean.Interpretation: Result of 20 has 95% confidence limit of18 – 22, PLUS undetected bias at time of measurement.

Put all this together…

Put all this together…

• 1. Setting Quality Limits provides the mostrobust approach

• What Limits to use?• RCPAQAP (where capable)• Others (where not capable against RCPAQAP)

• ALP different for different analytes, What limits,Complexity…

What I do…• Compare CVa with CVi• If CVa small relative to CVi:

few rules, wide limitsn=2, 1 x approx 3SD

• If CVa not small relative to Cvi:few rules, tighter limitsn=2, 1 x approx 2.2 SD

• Understand performance risks (ReverseEngineering)

• Compare with quality standards regularly(Cvi, RCPAQAP ALP, state of the art – QAP, PI)

What can we do…

• Define limits based on CVi

• Turn internal process into Logical steps

• Remember stat of the art

Bias: what I do

• Consider separately to precision• Base on Fraser concepts: percent of patients

wrongly classified• Eg ~one 10th of population reference interval

(if Gaussian)

Results Change ProtocolANALYTE ALLOWABLE LIMITSAlbumin +/- 4 up to 40 g/L, then +/- 10%ALT +/- 8 up to 60 U/L, then 15% (NCIRI)Amylase +/- 15% (NCIRI)AP +/- 15% (NCIRI)AST +/- 8 up to 60 U/L, then 15% (NCRI)Bicarbonate +/- 4 mmol/L (6 mmol/L IRI)Bilirubin +/- 8 up to 60 umol/L, then 10% (NCIRI)Calcium +/- 0.2 mmol/L (NCIRI)Chloride +/- 5 mmol/L (NCRI)Cholesterol +/- 0.5 up to 10 mmol/L, then 5%CK +/- 15 up to 100 U/L, then 15%CK-MB +/- 2 ug/L to 7 ug/L, then 15% (NTIRI)Creatinine +/- 0.02 up to 0.2 mmol/L, then 10%Fructosamine +/- 30 up to 300 umol/L, then 10%GGT +/- 8 up to 60 U/L, then 15% (NCIRI)Glucose +/- 0.4 up to 3 mmol/L, then 15% (1 mmol/L IRI)Glucose-GTT +/- 0.4 if crosses barrier otherwise 15% (1)HDL-C +/- 0.2 up to 2 mmol/L then 10%Iron +/- 5 umol/L (8 umol/L IRI)Lactate +/- 1 up to 5 mmol/L, then 20%LD +/- 40 up to 200 U/L, then 20% (NCIRI)Lipase +/- 20 up to 100 U/L, then 20% (NCIRI)Lithium +/- 0.2 mmol/LMagnesium +/- 0.12 mmol/L (0.2 mmol/L IRI)Omolality +/- 8 mmol/kgPhosphate +/- 0.1 up to 1.0 mmol/L, then 10% (0.3 mmol/L IRI)Potassium +/- 0.3 mmol/L (0.5 mmol/L IRI)Protein +/- 6 g/L (8 g/L IRI)Sodium +/- 4 mmol/L (6 mmol/L IRI)Transferrin +/- 0.5 g/LTriglyceride +/- 0.4 up to 3.0 mmol/L then 15%Urate +/- 0.05 mmol/L (0.07 IRI)Urea +/- 1 up to 10 mmol/L, then 10% (2 mmol/L IRI)

Free T4 +/- 3 pmol/L up to 20 pmol/L then 15%TSH +/- 0.6 mU/L up to 4 mU/L then 15% (2)

In the event of re-running an assay following the suspicion of an analytical error,significant changes must be changed in the computer and notified to therequesting/treating doctor. Changes equal to, or greater than those shown belowmay be considered significant (these values based on the RCPA-AACB Allowablelimits of performance). If in doubt, consult the Pathologist or Senior Scientist.

• Thank you