qcl training seminar, tanzania | 5-7 dec 07 1 |1 | proficiency tests john h mcb miller laboratory...

QCL Training Seminar, Tanzania | 5-7 Dec 071 |

Proficiency Tests

John H McB Miller

Laboratory Department (DLab)

European Department for the Quality of Medicines

Council of Europe

Strasbourg, France


Interlaboratory StudiesInterlaboratory Studies

Interlaboratory Testing

• Proficiency testing

• Collaborative study

• Certification study

• Co-operative study

Description

• Continuing assessment of technical competence

• Validation of a specific method

• Establishing the best estimate of the time value of an analyte in a reference material

• Laboratory assessment of samples and methods (eg educational studies)


Proficiency testing by interlaboratory comparisons

To determine the competence of individual laboratories to perform specific tests or measurements

To monitor the performance of laboratories overtime


ISO Guide 43-1

Interlaboratory comparisons may be used to:

a) determine the performance of individual labs for specific tests or measurements and to monitor labs’ continuing performance;

b) identify problems in labs & initiate remedial actions which may be related to, for example, individual staff performance or calibration of instrumentation;


ISO Guide 43-1

c) establish the effectiveness & compatibility of new test or measurement methods & similarly to monitor established methods

d) provide additional confidence to lab clients;

e) identify interlaboratory differences;


ISO Guide 43-1

Interlaboratory comparisons are conducted for a number of purposes and may be used by participating laboratories and other parties eg establishment of a reference material


ISO Guide 43-2

6. Use of results by laboratory accreditation bodies

6.1 The results from proficiency testing schemes are useful for both participating laboratories & accreditation bodies. There are, however, limitations ............. that proficiency testing alone should not be used by laboratory accreditation bodies in their accreditation processes.


6.2 If a laboratory submits a result(s) which fall outside acceptance criteria for a specific scheme, a laboratory accreditation body should have procedures for acting on such results

6.3 Such procedures should include early reporting to the laboratory of its results with an invitation for the laboratory to investigate and comment on its performance.

ISO Guide 43-2


Quality Manual

Laboratories should have a section in their Quality Manuals:

- covering participation in proficiency testing

- how the results are used to demonstrate the competence of the laboratory

- procedures to be followed when unsatisfactory performance is reported

- records of participation in PT scheme

- corrective action reports


Z-score = c tSD

c = calculated reported value t = true value SD = target

RSZ = Zm

m = number of tests

RSSZ = Z2m

PT Scoring System


Assigned Value,

The assigned value ( ) may be the “true” or the consensus value.

True value may be a theoretical value or known value from “spiking’ of known quantity of a known quantity of analyte to a sample.

Consensus value based on the results of the participants. The consensus value is determined by the application of robust statistics (eg median value, mean interquartile range, Huber’s robust mean), to avoid the influence of “outliers” in the overall mean.

x

x


Target standard deviation (TSD)

This is set based on experience, reported or expected precision of the techniques used and according to fitness-for-purpose.

The TSD must be realistic.

The TSD should be consistently applied from round to round for the same technique/procedure in a PT scheme so that performance can be assessed over time.


Ranking

Z-score ≼ 2 satisfactory

≽ 2≼ 3 questionable> 3 unsatisfactory


Outliers are indicated using three test statistics:

Cochran’s test for outlying variances, Grubbs’ single test for outlying means and Grubbs’ paired test for outlying means, to be applied in this order. If a laboratory is excluded, the cycle is repeated from Cochran’s test until no outliers remain.


CochranÕs test for outlying variances is applied as follows: Calculate for each

laboratory the within laboratory variance 2ö and then 2

2

ö

ömax

. This value is compared

with the tabled critical values for the number of individual determinations r. If it exceeds the critical value, the laboratory with the maximum variance is removed and the cycle is continued with CochranÕs test.


Grubbs’ single test for outlying means is applied as follows: Calculate the between laboratory standard deviation of the mean values. Calculate the standard deviations L and H excluding the lowest and the highest mean value respectively

and let * be the smallest of these two values. Calculate

ˆ

ˆ1

*

and compare it with the tabled critical values. If it exceeds the critical value, the corresponding laboratory is excluded and the cycle continued with Cochran’s test


GrubbsÕ paired test for outlying means is applied as follows: Calculate the between laboratory standard deviation ö of the mean values. Calculate the standard deviations L2ö , H2ö and LHö excluding the two lowest, the two highest, and the

highest and the lowest mean value respectively and let *ö be the smallest of these

three values. Calculate

ö

ö1

*

and compare it with the tabled critical values. If it

exceeds the critical value, the corresponding laboratories are excluded and the cycle continued with CochranÕs test.


Robust Statistics

Huber’s mean for calculation of the « consensus » value is preferred to using elimination of results by tests for outliers


Scoring over time

RSD - Used for detecting consistent bias

RSSZ - Magnitude of deviations.

Cancellation of significant Z-scores if opposite sign is limited


The rescaled sum of squared z-scores RSSZ is calculated as m

z 2

=RSSZ where m

is the number of squared z-scores added (one for each sample). Critical values of RSSZ depend on m according to the table:


m ‘Doubtful’ ‘Unacceptable’ 2 3 4 5 6

> 3.00 > 2.60 > 2.37 > 2.21 > 2.10

> 4.61 > 3.78 > 3.32 > 3.02 > 2.80


Proficiency Testing

Corrective actions:

- PT schemes can be either mandatory or voluntary

- Voluntary schemes: PT records examined by external auditors during ISO 17025 assessment

- Mandatory schemes: Corrective action reports must be set within a defined time limit to the organisers for assessment and approval (or not). Failure to do so will result in a sanction


OMCL NetworkProficiency Testing Scheme

Initially open to OMCls of the European Union & other OMCLs associated with the European Pharmacopoeia (member & observer states)

Now open to any lab. on a fee paying basis

minimum of 4 tests/year

voluntary scheme


WHO External Quality Assurance Assessment Scheme (EQAAS)

Started in 2001 for selected regional medicines control laboratories

Phase 4 began in 2007


PTS 20Atomic Absorption Spectrophotometry

Incorrect programming of instrument resulting in insufficient significant figures

Calibration 1 significant figure 1200 ppm K

3 significant figure 1290 ppm K

11/28 questionnaire/unsatisfactory results


PTS 025Titration of the conjugate acid of organic bases

* Corrected for water content (not required)A lab. reported a difference of 2% in results between potentiometric & visual end-point. However, incorrect standardisation procedure was employed

* Deterioration in response of the electrode

23/46 questionnaire/unsatisfactory results


Liquid Chromatography Assay of Indapamide

98.33}98.64} Mean = 98.7 RSD = 0.4098.11}

True value:- 99.75 z-score: 2.13

Repeatability of ref. sol. (n=6) :- 0.57


Liquid Chromatography Assay of Indapamide

Mg Area Corrected

Area

Mean ²% Content

1 2

Ref.

Sol.

20.28 1698

1691

1675

1668

1672

1.8

- -

20.32 1669

1668

1643

1642

1643 - -

Test

Sol.

20.79

20.97

20.56

1696

1701

1716

1721

1690

1696

1632

1636

1637

1641

1644

1649

1634

1639

1647

< 1.0

98.33

98.65

99.11

99.1

99.46

99.94


Phase 4 : Procedure 1 - Table 1

Raw data & scoring of participating laboratoriesSemi-micro determination of water


EQAS on water content by Karl FischerNumber & percentage of participating laboratories having shown satisfactory

performance (z-score < 2)

Water content Phase 1 (2001)

Phase 3 (2005)

Phase 4 (2007)

No of Participants 7 33 35

No of labs with satisfactory results

4 19 23

% of labs with satisfactory results

57% 58% 66%


EQAAS

Comments:

- From the 3 participants who failed on the first exercise, 2 of them did not participate in the subsequent studies. Concerning the 3rd one their results slightly improve Phase 3 and were satisfactory in Phase 4.

- From the 14 participants who failed in the second exercise, 8 of them reported satisfactory results on the 3rd one (3 did not participate and 3 didn’t show any improvement)

- There doesn’t seem to be an improvement in the general trend. The overall performance of laboratories using this technique is not very satisfactory and could be improved. However, it has to be pointed out that the determination of water by Karl-Fischer is problematic even for experienced laboratories as we can see from the results reported by the laboratories (including OMCLs) participating in our regular PTS programme.


References

International Organization for S tandardization, ISO/IEC Guide 43-1: 1997, Proficiency testing by interlaboratory comparisons Ń Part 1: Development and operation of proficiency testing schemes.

International Organization for S tandardization, ISO/IEC Guide 43-2: 1997, Proficiency testing by interlaboratory comparisons Ń Part 2: Selection and us e of proficiency testing schemes by laboratory accreditation bodies.

International Organization for Standardization, ISO 5725-2-1994: Statistical methods.

Association of O fficial Analytical Chemists, Guidelines fo r Collaborative Study Procedure to Validate Ch aracteristics of a Method of Analysis, AOAC Arlington, (1990).

The Royal Society of Chemistry, Proficiency Testing in Analytical Chemistry, Teddington, (1997).

International Organisation of Standardisation ISO/IEC Standard ISO13528 : 2005, Statistical Methods for use i n Proficiency Testing by Interlaboratory Comparisons.

qcl training seminar, tanzania | 5-7 dec 07 1 |1 | proficiency tests john h mcb miller laboratory...

Documents

qcl training seminar

laboratory procedures

france slide

laboratory accreditation

reference material slide

iso guide

interlaboratory comparisons

proficiency testing