control charts to evaluate long-term performance in

1

1

CONTROL CHARTS TO EVALUATELONG-TERM PERFORMANCE

IN PROFICIENCY TESTS

Angela Sorbo

European Union Reference Laboratory for Chemical Elements in Food of Animal Origin (EU-RL CEFAO)

EURACHEM Workshop October 10-11, 2012

2

CONTENTS

ROLE OF THE EUROPEAN UNION REFERENCE LABORATORIES

EURL CEFAO AS PT PROVIDER AND RATIONALE FOR ITS SCHEME

CONCLUSIONS

PERFORMANCE EVALUATION: Z-SCORES USING “FITNESS FOR PURPOSE APPROACH”

CONTROL CHARTS OF Z-SCORES


This work has been produced with the financial support of the SANCO 2005/Food SAFETY/003-Residues Program of the European Commission. The contents of this presentation are the sole responsibility of the author and can in no way be taken to

reflect the views of the European Commission

2

3

COMMUNITY REFERENCE LABORATORIES FOR RESIDUESOF VETERINARY MEDICINES AND CONTAMINANTS INFOOD OF ANIMAL ORIGIN WAS DESIGNATED IN COUNCILDIRECTIVE 96/23/EC AND REFIXED IN COMMISSIONREGULATION (EC) No 776/2006 of 23 May 2006 AMENDINGANNEX VII TO REGULATION (EC) No 882/2004

Art. 12 CR (EC) 776/2006 ASSIGNED TO THE ISTITUTOSUPERIORE DI SANITÀ (ROME) THE COMMUNITYREFERENCE LABORATORY FOR THE RESIDUES LISTED INANNEX I, GROUP B 3(C) TO DIR. 96/23/EC

EURL (FORMER CRL) IS RESPONSIBLE FOR CHEMICALELEMENTS IN FOOD OF ANIMAL ORIGIN (EU-RL CEFAO)

ROLE OF THE EUROPEAN REFERENCE LABORATORIES


4

Tasks of the CRLs (Art. 32-Reg. 882/2004)

a) providing national reference laboratories with details ofanalytical methods, including reference methods

b) coordinating the national reference laboratories in theapplication of the methods referred to in (a), in particular byorganising comparative testing and by ensuring anappropriate follow-up of such comparative testing inaccordance with internationally accepted protocols, whenavailable

c) coordinating, within their area of competence, practicalarrangements needed to apply new analytical methods andinforming national reference laboratories of advances in thisfield

d) conducting initial and further training courses for the benefit ofstaff from national reference laboratories and of experts fromdeveloping countries

ROLE OF THE EUROPEAN REFERENCE LABORATORIES


3

5

MATRICES, ANALYTES AND LEVEL OF CONCENTRATIONFOR PTs ARE SELECTED TAKING INTO ACCOUNT BOTHTHE RELEVANT EUROPEAN REGULATION (EXISTINGMAXIMUM LEVELS) AND NATIONAL RESIDUES CONTROLPLANS

THE CONCENTRATION OF THE SAMPLES IS OFTENADJUSTED TO THIS AIM

EU-RL CEFAO IS ACCREDITED AS PT PROVIDERACCORDING TO ISO/IEC 17043:2010 (BEFORE 2010: ISO GUIDE

43-1)

THE LABORATORY IS ALSO ACCREDITED IN CONFORMITYWITH ISO/IEC 17025:2005, ENLARGED WITH FLEXIBLESCOPE



6

2006 2007 2008 2009 2010 2011Meat

freeze-dried

2 samples

Meat freeze-dried

1 sample

Liverfreeze-dried

1 sample

Fishfreeze-dried

1 sample

MeatFrozen1 sample

LiverFrozen

1 sample

Milk freeze-dried

2 samples

MilkLiquid

2 samples

MilkLiquid

2 samples

MilkLiquid

2 samples

Milk Liquid1 sample

Meat freeze-dried

1 sample

Fish freeze-dried

1 sample

Meat freeze-dried

1 sample

FishFrozen1 sample

Analytes: Cadmium (Cd), Lead (Pb), total Arsenic (As) and Mercury (Hg)

PROFICIENCY TESTS ORGANIZED IN THE FIVE-YEAR PERIOD 2006-2011



4

7

EUROPEAN FOOD CONTROL NETWORK

OFFICIAL LABORATORIES

NATIONAL REFERENCE LABORATORIES

EUROPEAN REFERENCE LABORATORIES

PROFICIENCY TESTS ORGANIZED BY EU-RL CEFAO AREADDRESSED TO THE NATIONAL REFERENCELABORATORIES (NRLs) OF THE EU MEMBER STATES

NETWORK IS NOW CONSTITUTED BY 27 PARTICIPANTS



8

TO IMPROVE THE PERFORMANCE OF THE NRLsANALYTICAL METHODS AROUND THE MLs

TO HARMONIZE THE INTERPRETATION OFRESULTS IN TERMS OF THE SAMPLEACCEPTANCE (CR (EC) 333/2007)

TO ASSIST NRLs IN FINDING THE CAUSE OFANALYTICAL PROBLEMS LEADING TO ANUNSATISFACTORY PERFORMANCE

GENERAL OBJECTIVES OF PTs according to EU-RL TASKS



5

9

“FITNESS FOR PURPOSE” APPROACH PROPER FOR THE ROLE OF NRLs

PERFORMANCE REQUIRED FROM NRLS SHOULD BEBETTER THAN THAT OF ROUTINE CONTROLLABORATORIES

STANDARD DEVIATION FOR PROFICIENCYASSESSMENT (pEU-RL CEFAO) LOWER THAN THE ONEDERIVED FROM HORWITZ EQUATION

EU-RL CEFAO HAS DEVELOPED PROPER EQUATIONSGIVING A p CORRESPONDING TO THEPERFORMANCE LEVEL THAT NRLs ARE EXPECTEDTO ACHIEVE FOR THE DIFFERENT COMBINATIONMATRIX/ANALYTE/CONCENTRATION LEVEL

PERFORMANCE EVALUATION


10

MEAT AND FISH

Cd

0

10

20

30

40

10 70 130 190 250 310 370 430 490 550 610

%

120

g/kg

p =10% → C > 601

Pb

0

10

20

30

40

20 120 220 320 420 520 620 720 820

%

g/k

p =10% → C > 801

ii

p



Thompson

Horwitz

EU-RL CEFAO

6

11

As

0

10

20

30

40

120 180 240 300 360 420 480 540

%

g/kg

C < 120 →p =22% p =14.5% → C > 501

MEAT AND FISH



0

10

20

30

40

30 160 290 420 550 680 810

%

Hg

g/kg

C < 30 →p =18%

p =10% → C > 801

p

Thompson

Horwitz

EU-RL CEFAO

12

Cd

0

10

20

30

40

50

1 2 3 4 5 6 7 8 9 10 11

%

g/kg

p =15% → C > 11

Pb

0

10

20

30

40

10 15 20 25 30 35 40 45 50 55

%

g/kg

C < 10 →p =30%

p =18% → C > 51

As

0

10

20

30

40

10 15 20 25 30 35 40 45 50 55

%

g/kg

C < 10 →p =40%

p =15% → C > 51

MILK



p

Thompson

Horwitz

EU-RL CEFAO

7

13

Shewhart Control Chart for z-scores(ISO 13528:2005*, Par. 9.2.1)

To prepare this charts the z-scores for a laboratory areplotted as individual points, with action and warninglimit set at ±2.0 and ±3.0 (…). When severalcharacteristics are measured in each round, the z-scores for different characteristics may be plotted onthe same graph (…)

The Shewart control chart is an effective method ofidentifying problems that cause large erratic values ofz-scores

* ISO 13528:2005: Statistical methods for use in proficiency testingby interlaboratory comparisons



14

Rules(ISO 13528:2005*, Par. 9.2.1)

An out of control signal is given when:

b) two out of three successive points fall outside the same warning limit (±2,0)

a) a single point falls outside the action limits (±3,0)




8

15



16



9

17



18

-4

-3

-2

-1

0

1

2

3

4

2006Meat A

2006Meat B

2007Meat

2007Fish

2008Liver

2008Meat

2009Fish

2010Fr. Meat

2010Fr. Fish

2011Fr. Liver

2011Meat

z-s

core

PT

Cd

Pb

z=±2

z=±3

-6.2

CC Meat, Fish and Offal

2009: Visit of the EU-RL CEFAO representatives

2010: Change of analytical technique



10

-4

-3

-2

-1

0

1

2

3

4

2006Meat A

2006Meat B

2007Meat

2007Fish

2008Liver

2008Meat

2009Fish

2010Fr. Meat

2010Fr. Fish

2011Fr. Liver

2011Meat

z-sc

ore

PT

As

Cd

Pb

Hg

z=±2

z=±3


19

2010: Change of staff and technique (from more to less sensitive!)



-4

-3

-2

-1

0

1

2

3

4

10th-2A 10th-2B 11th-1A 11th-1B 12th-1 A 12th-1 B 13th-1 A 13th-1 B 14th-2 16th

z-s

co

re

PT

Cd

Pb

z=±2

z=±3

CC Milk

20

2010: Change of technique (from less to more sensitive!)



11

21

EU-NATIONAL REFERENCE LABORATORIES CONTROL CHART OF THE z-scores

(2006-2012) MILK

Edited by: A. Colabucci Verified by: L. Ciaralli Issued: 19.07.2012



22

CC Milk

-4

-3

-2

-1

0

1

2

3

4

2006-A 2006-B 2007-A 2007-B 2008-A 2008-B 2009-A 2009-B 2010 2012

z-s

core

PT

As

Cd

Pb

z=±2

z=±3

-4

-3

-2

-1

0

1

2

3

4

2006-A 2006-B 2007-A 2007-B 2008-A 2008-B 2009-A 2009-B 2010 2012

z-sc

ore

PT

As

Cd

Pb

z=±2

z=±3

Change of the staff in a laboratory with high standard of QS: excellent and steady performance

Change of staff: improved performance



12

-4

-3

-2

-1

0

1

2

3

4

2006-A 2006-B 2007-A 2007-B 2008-A 2008-B 2009-A 2009-B 2010 2012

z-sc

ore

PT

As

Cd

Pb

z=±2

z=±3

CC Milk

Steady performance in spite of no participation in PTs for 3 years and the change of the sample physical state

23

Analytical method under control



-4

-3

-2

-1

0

1

2

3

4

2006 Meat-A

2006 Meat-B

2007 Meat

2007 Fish

2008 Meat

2008 Liver

2009 Fish

2010 Fr. Meat

2010 Fr. Fish

2011 Fr. Liver

2011 Meat

z-sc

ore

PT

As

Cd

Pb

Hg

z=±2

z=±3

-4

-3

-2

-1

0

1

2

3

4

2006 Meat-A

2006 Meat-B

2007 Meat

2007 Fish

2008 Meat

2008 Liver

2009 Fish

2010 Fr. Meat

2010 Fr. Fish

2011 Fr. Liver

2011 Meat

z-sc

ore

PT

As

Cd

Pb

Hg

z=±2

z=±3

24




pEU-RL CEFAO

pHORWITZ/THOMPSON

13

25

The NRL was advised to accredit the method based on the most sensitive technique

-4

-3

-2

-1

0

1

2

3

4

2006-A 2006-B 2007-A 2007-B 2008-A 2008-B 2009-A 2009-B 2010 2012

z-sc

ore

PT

-4

-3

-2

-1

0

1

2

3

4

2006-A 2006-B 2007-A 2007-B 2008-A 2008-B 2009-A 2009-B 2010 2012

z-sc

ore

PT

Dry ashing + FAAS

MAD + ICP-MS

Pb

Pb

Cd

AsCd

As



26

CUSUM Control Chart for z-scores(ISO 13528:2005*, Par. 9.3.1)

To prepare this chart, the cumulative sum (cusum) of

the z-scores achieved by a laboratory over severalrounds of a PT is calculated and plotted.

The CUSUM control chart is an effective method ofidentifying problems that cause a bias in thedetermination of a characteristic that persists overseveral rounds.




14

27


-10

-8

-6

-4

-2

0

2

4

6

8

10

2006-A 2006-B 2007-A 2007-B 2008-A 2008-B 2009-A 2009-B 2010 2012

z-sc

ore

PT

As

Cd

Pb

THE ‘PERFECT’ CUSUM CHART


-10

-8

-6

-4

-2

0

2

4

6

8

10

2006MEAT-A

2006MEAT-B

2007MEAT

2007FISH

2008MEAT

2008LIVER

2009FISH

2010MEAT

2010FISH

2011LIVER

2011MEAT

cu

su

m o

f z-

sc

ore

s

PT

As

Cd

Pb Cusum CC

28

-4

-3

-2

-1

0

1

2

3

4

2006Meat A

2006Meat B

2007Meat

2007Fish

2008Liver

2008Meat

2009Fish

2010Fr. Meat

2010Fr. Fish

2011Fr. Liver

2011Meat

z-sc

ore

PT

As

Cd

Pb

z=±2

z=±3Shewart CC

Meat, Fish and Offal



Pb

-10

-8

-6

-4

-2

0

2

4

6

8

10

2006 Meat-A

2006 Meat-B

2007 Meat

2007 Fish

2008 Meat

2008 Liver

2009 Fish

2010 Fr. Meat

2010 Fr. Fish

2011 Fr. Liver

2011 Meat

PT

Cu

sum

of

z-sc

ore

s

15

29


-10

-8

-6

-4

-2

0

2

4

6

8

10

12

14

16

18

20

2006Meat-A

2006Meat-B

2007Meat

2007Fish

2008Meat

2008Liver

2009Fish

2010Fr. Meat

2010Fr. Fish

2011Fr. Liver

2011Meat

Cu

sum

of

z-sc

ore

s

PT

As

Cd

Pb

-4

-3

-2

-1

0

1

2

3

4

2006Meat A

2006Meat B

2007Meat

2007Fish

2008Liver

2008Meat

2009Fish

2010Fr. Meat

2010Fr. Fish

2011Fr. Liver

2011Meat

z-sc

ore

PT

As

Cd

Pb

z=±2

z=±3

Cusum CC

Shewart CC


30

CONCLUSIONS

The organization of recurrent PTs on the same matrix/analytecombination and at a similar level of concentration allowsparticipants to monitor the performance of their analyticalmethods and the PT provider to constantly check the qualitylevel of the network

The use of proper Control Charts is an effective means ofmonitoring the trends in the results produced by thelaboratory

The evaluation of CUSUM Charts together with ShewartControl Charts can be a useful way of highlighting bias in theanalytical methods

The analytical background of the Provider is an importantbasis to join all the information from the PT to assistparticipants in understanding their performance


16

31


Many thanks to dr. Rosa Giordano for the invaluable support

control charts to evaluate long-term performance in

Documents