1 can you measure it? planning a successful round robin paul schiffelbein astm committee week 14...

1

Can You Measure It?Planning a Successful “Round

Robin”

Paul SchiffelbeinASTM Committee Week

14 June 2006

2

Overview

• Measurement system analysis (Round robin, Interlaboratory Study (ILS), Interlaboratory Comparison (ILC), Gage Study) description, definitions and motivation

• Round robin guidelines

• Round robin logistics

• Data Analysis

• Precision and bias statement

3

Purpose of Measurement System Analysis

• Ensure the measurement system has adequate accuracy: precision and bias

• Identify sources of measurement variation (and make improvements, if necessary)

• Compare several test devices, test methodologies, test locations (linearity, bias, sensitivity)

• Quantify measurement variability for reference

4

Actual Process Variation

Measurement Variation

Observed Process Variation

Long-term Process

Variation

Short-term Process

Variation

Variation within a Sample

Variation due to Labs,

Operators,

Devices,Time

Variation

due to Gage

Bias

Linearity

“Short-term”

Stability

“Long-term”

Measurement Process Variation

Simple “precision”

“bias”

5

ISO 5725-1, ASTM E177

• Accuracy: The closeness of agreement between a test result and the accepted reference value

– “Accuracy,” when applied to a set of test results, involves a combination of random componentsrandom components and a common systematic error or bias componentbias component

• Bias: the difference between the expectation of test results and an accepted reference value

– Bias is the total systematic error as contrasted to random error. There may be one or more systematic components contributing to the bias.

6

ISO 5725-1, ASTM E177

• PrecisionPrecision: The closeness of agreement between independent test results obtained under stipulated conditions.

– Precision depends only on the distribution of random errors and does not relate to the true value or the specified value.

• Precision can be decomposed into short- and long-term (or narrow and wide) components. RepeatabilityRepeatability and reproducibilityreproducibility are used to quantify this concept.

7

ISO 5725-1

Repeatability: Precision under repeatability conditions

• independent test results • the same test method• identical test items• the same laboratory• the same operator• the same equipment • within short time interval

– Repeatability standard deviation: The standard deviation of test results obtained under repeatability conditions

– Repeatability limit (“r”): the absolute difference between two test results obtained under repeatability conditions should be less than or equal to this value

8

ISO 5725-1

Reproducibility: Precision under reproducibility conditions

• independent test results • the same test method• identical test items• different laboratories• different operators• different equipment • longer time interval

– Reproducibility standard deviation: The standard deviation of test results obtained under reproducibility conditions

– Reproducibility limit (“R”): the absolute difference between two test results obtained under reproducibility conditions should be less than or equal to this value

9

Actual Process Variation

Measurement Variation

Observed Process Variation

Long-term Process

Variation

Short-term Process

Variation

Variation within a Sample

Variation due to Labs,

Operators,

Devices,Time

Variation

due to Gage

Bias

Linearity

“Short-term”

“repeatability”

Stability

“Long-term”

ASTM/ISO Usage

Simple “reproducibility”

“bias”

10

Actual Process Variation

Measurement Variation

Observed Process Variation

Long-term Process

Variation

Short-term Process

Variation

Variation within a Sample

Variation due to Labs,

Operators,

Devices,Time

Variation

due to Gage

Bias

Linearity

“Short-term”

“repeatability”

Stability

“Long-term”“reproducibili

ty”

Gage R&R/Auto Industries Usage

Simple “Gage”

“bias”

11

Round Robin Guidelines

• E 691 “Standard Practice for Conducting an Interlaboratory Study to Determine the Precision of a Test Method”

• D 2904 “Standard Practice for Interlaboratory Testing of a Textile Test Method that Produces Normally Distributed Data”

12

Round Robin Guidelines

• The design should be as simple as possible in order to obtain estimates of within- and between-laboratory variability that are free of secondary effects

• Study should include a minimum of five laboratories

• A minimum of two operators should be used per laboratory

• When multiple instruments within a laboratory are used, tests must be made on all equipment to establish the presence or absence of the equipment effect.

13

Logistics: D 885 Case Study

1 Motivation: The current version of D 885 was written for traditional tensile testing machines. Automated tensile testers are now being used in high-tenacity fiber testing, and need to be included in this standard. The current study will include three types of automated testers, as well as parallel testing on traditional tensile test devices. The objective of this study is to quantify test precision of traditional and automated testers, as well as any bias seen between the device types. The study will only address aramid materials.

14

Logistics: D 885 Case Study

2 Responsibility: Task group D13.19 (Tire Cord and Fabrics) has overall responsibility of the ILS. Dawn Caullwine (chair) will act as overall coordinator for conducting the ILS. The coordinator will supervise the distribution of materials and protocols to the laboratories and receive the test result reports from the laboratories.

15

Logistics: D 885 Case Study

3 Study Design: Nine materials will be tested on each of three types of automated tensile test devices. Two devices of each type will be used. Yarn will be supplied in pre-twisted state for testing. Untwisted yarn will also be provided to Statimat labs, so testing can be performed both on pre-twisted yarn, and yarn automatically twisted by the test machine. Two laboratories will also test the materials using traditional methods for reference. Each of those laboratories will use two operators.

16

Logistics: D 885 Case Study

Automated Testers

Sigma 500 Statimat Uster

Lab AAustria

Lab BAustria

Lab CEurope

Lab DUSA

Lab E Lab F

Pre-twisted Pre-twistedStatimat-twisted

Statimat-twisted

17

Logistics: D 885 Case Study

4 Materials: The ILS will include the following nine materials (All samples are shipped twisted and ready to test, except for Statimat laboratories, which receive both twisted and untwisted samples):– Kevlar®: 195 denier– Kevlar®: 600 denier– Kevlar®: 1140 denier– Nomex®: 200 denier– Nomex®: 1600 denier– Technora®: 550 denier– Twaron: 840 denier– Twaron: 1500 denier– Twaron: 3000 denier

18

Logistics: D 885 Case Study

5 Test Determinations and Test Results: The number of test determinations required for a test result is specified in each individual test method. For the purpose of this study, each laboratory will make one hundred (100) determinations (breaks) for each material.

19

Logistics: D 885 Case Study

5 Test Determinations and Test Results: The following properties (and associated measurement units) will be recorded:

– Break strength (BS) N– Elongation at break (EB) %– Modulus between 300 mN/tex and 400 mN/tex (MOD)

CN/tex– FASE @ 0.3% N– FASE @ 0.5% N– FASE @ 1.0% N

Use nominal linear density for modulus calculation.

20

Logistics: D 885 Case Study

6 Details: The test method being studied is D885-03. Specify the type of equipment used, including manufacturer, model, and software program. Samples should be conditioned as per D885 7.1 to moisture equilibrium in an environmentally controlled room for a minimum of 16 hours, at RH 55 +/- 2% and temperature 24 +/- 1 degrees C (72 +/- 2 degrees F).

Specific for Para-Aramid:– Gauge length: 500mm– Crosshead rate: 250mm/min. (50%)

Etc.

21

Logistics: D 885 Case Study

7 Data: Data should be entered into the Excel workbook provided. Label the workbook with your laboratory name, fill in the test data, and send the completed file to:

Paul SchiffelbeinDuPont Engineering, Quality Management & Technology

paul.a.schiffelbein@usa.dupont.com302/774-2417

22

Logistics: D 885 Case Study

23

Logistics: D 885 Case Study

24

A Better Way?: Internet Entry for F 2130 RR

Figure courtesy of Dr. Anugrah Shaw, UMES

25

Data Structure Guaranteed!

26

Logistics: D 885 Case Study

8 Data Analysis: Material = Kevlar® 1420d

Variability Chart for BF(N)

BF

(N)

250

260

270

280

290

300

310

320

330

340

350S

pru

an

ce

A

Sp

rua

nce

B

TF

A

TT

QR

S

Le

nzin

g

Ma

insite

Sp

rua

nce

TF

A

Sp

rua

nce

TF

A

TT

QR

SInstron Sigma500StatimatStatimat PTUster

Laboratory within Device

27

Logistics: D 885 Case Study

8 Data Analysis: S

td D

ev

0

5

10

15

20

Spr

uanc

e A

Spr

uanc

e B

TFA

TT

QR

S

Lenz

ing

Mai

nsite

Spr

uanc

e

TFA

Spr

uanc

e

TFA

TT

QR

S

Instron Sigma500 Statimat Statimat PTUster

Laboratory within Device

Material = Kevlar® 1420d

Variability Chart for BF(N)

28

Repeatability and Reproducibility

Use between- and within-laboratory variance components:

R = 2L + 2

r

Where:

R is the reproducibility standard deviation

r is the repeatability standard deviation

L is the square root of the inter-laboratory

(device, operator, etc.) variance component

29

REML Variance Component EstimatesRandom Effect Var Ratio Var

ComponentStd Error 95% Lower 95% Upper Pct of Total

Device&Random 0.6823116 66.35 62.607 18.903 1854.81 31.5Laboratory[Device]&Random 0.4806723 46.74 26.848 19.480 224.123 22.2

Residual 97.25 46.2Total 210.36 100.0

Logistics: D 885 Case Study

8 Data Analysis:

Repeatability variance Reproducibility variance(one estimate)

Reproducibility variance(another estimate)

30

Logistics: D 885 Case Study

9 Precision and Bias Statement:

13. Precision and Bias

13.1 Interlaboratory Test Data – An interlaboratory test was conducted in 2006 using

commercially available tensile testers from … . Nine materials were included in

the study, and are listed in table _. The study structure is shown in Figure _.

Variance components were computed for individual tensile determinations, and are

summarized in Table _.

31

Logistics: D 885 Case Study

9 Precision and Bias Statement:

13.2 Precision - Repeatability and reproducibility deal with the variability of test

results obtained under specified laboratory conditions. Repeatability concerns the

variability between independent test results obtained within a single laboratory in

the shortest practical period. Those results are obtained by a single operator with

a specific set of test apparatus using test specimens (or test units) taken at random

from a single quantity of homogeneous material obtained or prepared for the

interlaboratory study (ILS). Reproducibility deals with the variability between

single test results obtained in different laboratories, each of which has applied the

test method to test specimens (or test units) taken at random from a single

quantity of homogeneous material obtained or prepared for the ILS.

32

Logistics: D 885 Case Study

9 Precision and Bias Statement:

Method repeatability is defined as the "maximum difference" that can

"reasonably" be expected between two test results obtained on the same material

when the test results are obtained in the same laboratory. Repeatability standard

deviation is taken to be the square root of the “determination” variance

component, and represents within-operator precision. Method reproducibility is

defined as the "maximum difference" that can "reasonably" be expected between

two test results obtained on the same material when the test results are obtained

from different laboratories. The total, or reproducibility, standard deviation, is

formed by taking the square root of the sum of intra- and inter-laboratory variance

components.

33

Logistics: D 885 Case Study

9 Precision and Bias Statement:

Table _ contains repeatability and reproducibility standard devations and

maximum critical differences for single determinations and specified averages of

determinations for the single operator case (repeatability), within-laboratory case,

and between-laboratory case (reproducibility). Two values or averages of

observed values are considered significantly different at the 95% probability level

if the difference between them exceeds the appropriate critical difference in the

table.

34

Logistics: D 885 Case Study

9 Precision and Bias Statement:

13.3 Bias – The procedure in this test method produces a test value that can be definedonly in terms of a test method. There is no independent referee method by which biasmay be determined. The test method has no known bias.

35

Summary

• Explicitly state objectives for your study, and make sure all participants understand them

• Write protocol covering responsibilities, timing, samples and sample handling, test equipment and set-up, sampling plan, properties, units, etc.

• Details are important! Be specific, explicit, and discuss ahead of time with task group to avoid surprises. Communicate!

• Set up data collection to be as “goof proof” as possible, and to facilitate subsequent analysis

36

Thank you