ISTA Proficiency Test Programme

StatisticStatistic SeminarSeminarHohenheim, 29.08. – 02.09. 2005

- Martina Rösch -ISTA System Auditor

What is Proficiency Testing?

Interlaboratory ring test with several laboratoriesto assess their technical competence

Testing of samples in accordance with determined methods

Statistical evaluation and comparison of a laboratory‘s results with theresults of all participants

Evaluation of the laboratory‘s testing performance

Detection of systematic errors in the laboratory‘s testing system

Detection of trends in testing performance

Part of the continual improvement process!

The ISTA Proficiency Test Programme -Design

Six crop groups

Representative species tested

Three proficiency test rounds per year with three samples

Germination, Purity, Other Seed Determination, Moisture Content, Tetrazolium

Mandatory for accredited laboratories depending on the individualscope of accreditation

Voluntary for ordinary member laboratories

Accreditationmaintained?

The ISTA Proficiency Test Programme -Procedure

The ISTA Proficiency Test Programme –Reporting

The ISTA Proficiency Test Programme –Computation of Germination Test Results

PrinciplesComputation of results follow the Tattersfield method described in SST, 1979

Calculation is based on the assumption that the germination test resultsfollow a Normal distribution if values are between 2 and 98%

Only results of accredited laboratories are used for determining the meanand standard deviation – All laboratories‘ results are checked againstthese values

The mean percentage of the four replicates is determined to two decimalplaces

Step 11. First estimate of the mean of the basic distribution by finding the

median (-> less disturbed by strangers)

The ISTA Proficiency Test Programme –Computation of Germination Test Results

0

2

4

6

8

10

12

85.5 86 86 .5 87 87.5 88 88 .5 89 89 .5 90 90 .5 91 91.5 92 92 .5 93 93 .5 94 94 .5 95 95.5 96 96 .5 97 97.5 98 98 .5 99 99 .5 100

The ISTA Proficiency Test Programme –Computation of Germination Test Results

Median = 93.5

Step 11. First estimate of the mean of the basic distribution by finding the

median (-> less disturbed by strangers)

The ISTA Proficiency Test Programme –Computation of Germination Test Results

2. First estimate of the variance (s2) by calculating the mean squaredeviation of those values which are above the median

0

2

4

6

8

10

12

85.5 86 86 .5 87 87.5 88 88 .5 89 89 .5 90 90 .5 91 91.5 92 92 .5 93 93 .5 94 94 .5 95 95.5 96 96 .5 97 97.5 98 98 .5 99 99 .5 100

The ISTA Proficiency Test Programme –Computation of Germination Test Results

Median = 93.5

t-value*standard error = 4.75

Values to calculate the 1st estimate of the variance (s2)

Step 11. First estimate of the mean of the basic distribution by finding the

median (-> less disturbed by strangers)

2. First estimate of the variance (s2) by calculating the mean squaredeviation of those values which are above the median

The ISTA Proficiency Test Programme –Computation of Germination Test Results

3. Discarding of outliers ((Median - t-value*standard error) < xi< (Median + t*s))

0

2

4

6

8

10

12

85.5 86 86.5 87 87.5 88 88.5 89 89.5 90 90.5 91 91.5 92 92.5 93 93.5 94 94.5 95 95.5 96 96.5 97 97.5 98 98.5 99 99.5 100

The ISTA Proficiency Test Programme –Computation of Germination Test Results

Median = 93.5

t-value*standard error = 4.75

93.5 - 4.75 = 88.75 93.5 + 4.75 = 98.25

Outliers

Values to calculate the 1st estimate of the variance (s2)

Step 11. First estimate of the mean of the basic distribution by finding the

median (-> less disturbed by strangers)

2. First estimate of the variance (s2) by calculating the mean squaredeviation of those values which are above the median

3. Discarding of outliers ((Median - t-value*standard error) < xi< (Median + t*s))

The ISTA Proficiency Test Programme –Computation of Germination Test Results

Step 21. Determination of the (arithmetic) mean (X) and the mean square

deviation (σ) from all remaining values

0

2

4

6

8

10

12

85.5 86 86.5 87 87.5 88 88.5 89 89.5 90 90.5 91 91.5 92 92.5 93 93.5 94 94.5 95 95.5 96 96.5 97 97.5 98 98.5 99 99.5 100

The ISTA Proficiency Test Programme –Computation of Germination Test Results

Median = 93.5

t-value*standard error = 4.75

93.5 - 4.75 = 88.75 93.5 + 4.75 = 98.25

Outliers

Determination of the mean (X) and standard error (σ)

X = 93.3; σ = 1.87

The ISTA Proficiency Test Programme –Computation of Germination Test Results

2. Calculation of Z-Scores: Zi=(xi-X)/σ -> deviation of the individual result from the mean of the distribution, divided by the standard error

Step 11. First estimate of the mean of the basic distribution by finding the

median (-> less disturbed by strangers)

2. First estimate of the variance (s2) by calculating the mean squaredeviation of those values which are above the median

3. Discarding of outliers ((Median - t-value*standard error) < xi< (Median + t*s))

Step 21. Determination of the (arithmetic) mean (X) and the mean square

deviation (σ) from all remaining values

0

2

4

6

8

10

12

85.5 86 86.5 87 87.5 88 88.5 89 89.5 90 90.5 91 91.5 92 92.5 93 93.5 94 94.5 95 95.5 96 96.5 97 97.5 98 98.5 99 99.5 100

The ISTA Proficiency Test Programme –Computation of Germination Test Results

Median = 93.5

t-value*standard error = 4.75

93.5 - 4.75 = 88.75 93.5 + 4.75 = 98.25

Outliers

Determination of the mean (X) and standard error (σ)

X = 93.3; σ = 1.87Zi=(xi-X)/σ

(90.5 – 93.3)/1.87 = -1.50

EXAMPLEEXAMPLE

-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3

95%

2.5%2.5%

5% of the results wouldfall outside ±2.0

The ISTA Proficiency Test Programme –Computation of Germination Test Results

Z-Score = -1.5

-2.23

-1.12

Results outside ±2.0 areconsidered as due not to random events butanalytical problems

In order to determine thelaboratory‘s testingcapabilities, the total sumof absolute Z-scores isdetermined

Total Sum of Z-Scores = 4.85

Problem when results are above 98 or below 2% -> Poissondistribution rather than Normal distribution

Selection of seed lots which have a germination capacity below98%

Calculating the arithmetic mean without discarding outliers whenmedian is above 98 or below 2%

The ISTA Proficiency Test Programme –Computation of Germination Test Results

A, B, C and BMP (Below Minimum Performance) for in-round and overallrating

Overall rating will be determined after 6 rounds of participation

Only results of mandatory test rounds are included in the overall rating

The ISTA Proficiency Test Programme –Rating System

Absolute sum of Z-Scores of normal seedlings is determined; thresholdsdefine the in-round rating

The ISTA Proficiency Test Programme –In-Rating System for GER, PUR, MOI, TZ

Score Sum of absolute Z-Scores

A Sum of absolute Z-Scores ≤ 3.5

B 3.5 < Sum of absolute Z-Scores ≤ 5.3

C 5.3 < Sum of absolute Z-Scores ≤ 7.0

BMP Sum of absolute Z-Scores ≥ 7.0

Example: Germination

Example: Germination

Example: Germination

Example: Germination

The ISTA Proficiency Test Programme –Evaluation of Other Seed Determination

PrincipleSeeds of known number and identity are added to the proficiency test sample which has not been purified beforehand

Laboratories are expected to retrieve and identify all seeds at least to genus level

EvaluationOption 1: determine the retrieval and identification rate and determine the

in-round rating based on thresholds (90/80/70)

Option 2: determine the retrieveal and identification rate based on theactual retrieval rate of the added species. Thresholds are then appliedaccordingly.

The ISTA Proficiency Test Programme –Evaluation of Other Seed Determination

Option 1: DifficultiesNumber of seeds per species is different;

Degree of difficulty to retrieve and identify a particular species isdifferent depending on the species to be tested (Sinapis arvensis in Brassica napus versus Sinapis arvensis in Triticum aestivum);

No distinction between retrieval and identification;

Total number of seeds added affects the impact of each seed missed on the percentage of retrieved and identified seeds.

The ISTA Proficiency Test Programme –Evaluation of Other Seed Determination

Option 2: PrinciplesEach species that was added is assigned a value based on the actualretrieval rate of all seeds added.

Total retrievalrate [%]

Assignedvalue

≥90 3

≥85 2

≥80 1

Number of seeds found by a participant is multiplied by the assignedvalue

The ISTA Proficiency Test Programme –Evaluation of Other Seed Determination

Example: Test round 04-1, Brassica napusSpecies Total retrieval

rate [%]Score

Galeopsis tetrahit 88

93

Sinapis alba 83 2

Sinapis arvensis 61 1

Lotus corniculatus 72 1

Raphanus sativus 81 2

Chenopodium album 84 2

Polygonum aviculare 91 3

79

2

Galium aparine 3

Thlaspi arvense 1

The ISTA Proficiency Test Programme –Evaluation of Other Seed Determination

Example laboratory resultsSpecies # of seeds

addedScore # of seeds x

score# of seeds found

2 2

3

2

2

3

1

1

2

1

3

24

9

8

6

9

3

4

8

2

53

Sinapis alba 4

3

4

3

3

0

0

4

2

21

75

8

Sinapis arvensis 3 0

Thlaspi arvense 2 2

C

Total sum 28 46

Percentage 87

Lotus corniculatus 4 0

Raphanus sativus 4 8

Chenopodium album 3 6

Polygonum aviculare 3 9

Seeds found x score

Galeopsis tetrahit 4

Galium aparine 9

In-round rating B

The ISTA Proficiency Test Programme –Evaluation of Other Seed Determination

Example laboratory resultsSpecies # of seeds

addedScore # of seeds x

score# of seeds found

Galeopsis tetrahit 2 2 4 2 4

Galium aparine 3 3 9 0 0

Sinapis alba 4 2 8 0 0

Chenopodium album 3 2 6 3 6

Polygonum aviculare 3 3 9 3 9

Sinapis arvensis 3 1 3 3 3

Lotus corniculatus 4 1 4 4 4

Raphanus sativus 4 2 8 4 8

Thlaspi arvense 2 1 2 2 2

Total sum 28 53 21 36

Percentage 75 68

In-round rating C BMP

Seeds found x score

Score Number given

A 5

B 4

C 3

BMP 0

Overall Score Threshold

A 28-30

B 21-28

C 16-20

BMP below 16

The same rating system is used for germination, purity, other seeddetermination, moisture content determination and tetrazolium

The ISTA Proficiency Test Programme –Overall Rating

02-3

03-1

03-2

03-3

04-1

04-2

A

A

B

BMP

B

C

ACBBMPBB=

In-round rating Accross 6 rounds

A

The ISTA Proficiency Test Programme –Overall Rating

BMP

COverall

Overall

Warning

Suspension

The ISTA Proficiency Test Programme –Consequences

Example: Germination

Confirmation of good performance

Indication of potential deficiencies in testing procedures

Defective equipment

Level of knowledge of staff

Insufficient documented procedures

Testing material

The ISTA Proficiency Test Programme –Impact

Receipt of testreport

Evaluation of testresults

Results o.k.? Stop

Root causeanalysis

Corrective action

Evaluation ofeffectiveness ofcorrective action

Yes

No

Stop

• Are deficiencies systematic?

• Are trends evident?

• Calculation or transcription error?

• Were all items of equipment appropriately maintained and monitoredat the time of testing (Maintenance and monitoring records, etc.)?

• Is staff trained for the particular type of test, have all proceduresbeen adhered to?

• How are the results of repeat tests?

• Was all testing material appropriate?

• Adjusting maintenance of equipment

• Training of staff

• Update and/or amend test procedures

• Carry out ring tests

The ISTA Proficiency Test Programme –Corrective action

The ISTA Proficiency Test Programme –Evaluation of Purity Test Results

Evaluation system for purity results not yet determined due to thefollowing restrictions:

Values for pure seed are close to 100%, inert matter and otherseed are below 2%

Distribution is not Normal

Number of results per test round is small

Thank you for your attention!