bmta july 2005: 1 valid analytical measurement studies of proficiency testing scheme performance s...
TRANSCRIPT
![Page 1: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/1.jpg)
BMTA July 2005: 1
Valid Analytical Measurement Studies of Proficiency Testing scheme
performance
S Ellison
LGC Limited, Teddington
The work described in this paper was supported under contract with the Department of Trade and Industry
as part of the Valid Analytical Measurement programme
![Page 2: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/2.jpg)
BMTA July 2005: 2
or63 routes to the wrong result
... and what to do about it
![Page 3: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/3.jpg)
BMTA July 2005: 3
Introduction
• PT in analytical chemistry• Why study mistakes?• How does the UK do?
– PT results compared to international performance
• What goes wrong? (and why)– Web-based study of causes of poor PT scores
![Page 4: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/4.jpg)
BMTA July 2005: 4
PT in analytical chemistry - organisation
• Typical rounds comprise:– test sample preparation, characterisation and distribution– analysis by participants– data collection and processing– preparation and distribution of the report
• Frequency– Typically 6-12 rounds per year
• Analytes (measured quantities)– 1-30 per sample per round
• Participants– Typically 30-100 per round, but strongly scheme-dependent
![Page 5: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/5.jpg)
BMTA July 2005: 5
The aims of proficiency testing
• Primary aim:
“To provide the infrastructure for a laboratory to monitor and improve the quality of its routine analytical measurements”
• Other aims– Provide information on the state-of-the-art in analytical
measurements– Compare performance of analytical methods– Assist a laboratory in the validation of new methods
![Page 6: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/6.jpg)
BMTA July 2005: 6
Principle of performance assessment
• Observed error – difference between
laboratory result (x) and assigned value (X)
• ‘Target range’– usually a standard
deviation ( ) or uncertainty
Compare…..
…..using an acceptability criterion
![Page 7: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/7.jpg)
BMTA July 2005: 7
Performance Scoring: z-scores
x submitted result X assigned valuestandard deviation for proficiency assessment
Z 2 Satisfactory performance
2Z 3 Questionable performance
Z 3 Unsatisfactory performance
Xxz
Interpretation of z is consistent across schemesbut depends on
![Page 8: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/8.jpg)
BMTA July 2005: 8
Typical analytical performance data.
-8 -6 -4 -2 0 2 4 6 8
Z-score
5 values < -8 3 values >8
Collected foodanalysis data:Various analytes
![Page 9: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/9.jpg)
BMTA July 2005: 9
PT data for benchmarking
• Three studies of UK performance– Clinical– Food– Environment
• Clinical: Backed by IMEP-17 study (20 analytes: 35 countries)
• Food: FAPAS PT scheme data (6 representative analytes; 2000 labs; ca. 250 countries and regions)
• Environment: CONTEST and CoEPT project data
![Page 10: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/10.jpg)
BMTA July 2005: 10
UK performance: Clinical
UK performance:Consistent withothers; Rarely poor
![Page 11: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/11.jpg)
BMTA July 2005: 11
UK Performance: Food
Rest of world
UK
-5 0 5
GeMMA z-score by country
z-score
GMO measurement
![Page 12: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/12.jpg)
BMTA July 2005: 12
UK Performance: FoodAflatoxins
![Page 13: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/13.jpg)
BMTA July 2005: 13
UK Performance: FoodPirimphos-Me (pesticide residue)
GB Other
-10
-50
51
0
GB Pirimphos-Methyl Performance for all RoundsZ
-sco
re
UK Other
![Page 14: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/14.jpg)
BMTA July 2005: 14
Problem analytes: ArsenicUK
FAPAS Arsenic data: Rounds 735-753GB results only
Z-score
Fre
qu
en
cy
-10 -5 0 5 10 15 20
01
02
03
0
![Page 15: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/15.jpg)
BMTA July 2005: 15
FAPAS Arsenic data: Rounds 735-753abs(Z)<=20
Z-score
Fre
qu
en
cy
-10 -5 0 5 10 15 20
05
01
00
15
0Problem analytes: Arsenic
All countries
![Page 16: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/16.jpg)
BMTA July 2005: 16
UK Performance: Environment
Total polycyclic aromatics
SO5 SO7 SO6 SO2 SO1 SO3
0
200
400
600
800
TotalPAH
Scheme ID
mg
/kg
UK SchemeOther
![Page 17: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/17.jpg)
BMTA July 2005: 17
UK Performance: Summary
• Broadly comparable to other countries• No problems unique to the UK• Some problems (e.g. Arsenic) shared with other
countries
![Page 18: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/18.jpg)
BMTA July 2005: 18
Part 2: Causes of error
• VAM Project KT2.4/3: Causes of poor PT performance• Aim:
Study "…the principal causes of poor performance in laboratories and ... the effectiveness of the steps taken by Participants in PT to improve the reliability of their results”
• Methodology– Web-based questionnaire– Focussed on documented problems identified via PT scores– Lead questions with follow-up for positive responses
![Page 19: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/19.jpg)
BMTA July 2005: 19
Why study poor scores in PT?
• Why PT?– PT participants are already committed to quality improvement– Participants follow up poor PT scores
• Why only poor scores?– Acceptable scores give poor information about problems– Correlation of scores with general methodology is not very
effective– Every good lab has documented problems and corrective
actions
![Page 20: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/20.jpg)
BMTA July 2005: 20
Top causes of poor scores
Sample preparationEquipment problemHuman error
CalibrationSelection of methodCalculation errorReporting problem
111 respondents230 causes
![Page 21: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/21.jpg)
BMTA July 2005: 21
Top causes of poor scores
Sample preparation
Extraction/recovery
Dilution to volume
![Page 22: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/22.jpg)
BMTA July 2005: 22
Top causes of poor scores
Equipment problem
Equipment failure
![Page 23: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/23.jpg)
BMTA July 2005: 23
Top causes of poor scores
Human error
Training/experience
Transcription error
Reporting error
![Page 24: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/24.jpg)
BMTA July 2005: 24
Top causes of poor scores
Calibration
No reference materialDefective RM
Incorrect procedureCalibration range
Calibration
![Page 25: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/25.jpg)
BMTA July 2005: 25
Top causes of poor scores
Reporting problem
Reporting problems
Value correct but not in customer units
Transcription/typographical error
Incorrect units
![Page 26: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/26.jpg)
BMTA July 2005: 26
Top causes of poor scores
Calculation error
Commercial software problem
Spreadsheet problem
Spreadsheet user errorCalculator error
Arithmetic error
Value mis-enteredSoftware mis-applied
Other
![Page 27: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/27.jpg)
BMTA July 2005: 27
Corrective action
Training
New proceduresRevalidationMethod documentation
New equipment
Additional calibration
Method change
RM change
Other
Detailed information showed problem-specific responses
![Page 28: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/28.jpg)
BMTA July 2005: 28
Corrective action - efficacy
• No significant difference in efficacy across different corrective actions
• Only 50% of actions were marked as ‘fully effective’
• Monitoring of efficacy tended to use local/immediate methods– Monitor QC results– Internal audit
![Page 29: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/29.jpg)
BMTA July 2005: 29
Causes of error: Summary
• Most PT errors were caused by basic lab operations– Incorrect dilution to volume– Transcription and reporting errors– Data and spreadsheet formula entry errors
• Equipment failure is perceived as a problem• Extraction/recovery problems important• Commercial software faults caused no problems
• Corrective actions are problem-specific and ‘multifactor’– More than one action generally required.
![Page 30: BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f045503460f94c17e06/html5/thumbnails/30.jpg)
BMTA July 2005: 30
Conclusions
• UK analytical labs perform similarly to international partners, and share similar problems
• The most common cause of PT performance failures are not technical, but simple human errors such as incorrect volumetric operations and transcription errors
• Time to look harder at human factors?
• Study web page: via http//www.vam.org.uk - surveys link