measurement systems analysis vfp … · measurement systems analysis, study variation statistically...

27
MEASUREMENT SYSTEMS ANALYSIS VFP COLLOQUIUM FRED J. RISPOLI JUNE 29, 2015

Upload: dodat

Post on 18-Jul-2018

228 views

Category:

Documents


4 download

TRANSCRIPT

Page 1: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

MEASUREMENT SYSTEMS ANALYSIS

VFP COLLOQUIUM

FRED J. RISPOLI JUNE 29, 2015

Page 2: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

1.  VFP Overview, 2011-2015 2. The Statapult and Scientific Process 3.  Measurement Systems Analysis

4.  A Measurement Systems Experiment

Outline

Page 3: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Thanks to BNL, DOE and:

Tim Green, Environmental Protection Division Jennifer Higbie, Environmental Protection Division Noel Blackburn, Office of Educational Programs Salvador Gonzalez, Office of Educational Programs John Heiser, Environmental and climate Science Department

Dowling College SULI/VFP Students Suhua Zeng Tom Fasano Raffelle Vesia Matt Bernard

Page 4: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Overview of VFP Projects 1. A Series of experiments to create models that predict

toxicity %

Objective Predict toxicity % when controlling pH, temp, aeration, and copper nanoparticles, The toxicity response is based on E.coli

Method A mixture experimental design was constructed to capture two-way interactions Models Regression model with 15 predictor terms, Ternary Plots Collaborator Vishal Shah, VFP 2010 to 2012

Page 5: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x14 x15

1 0.125 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063

2 0.063 0.125 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063

3 0.063 0.063 0.125 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063

4 0.063 0.063 0.063 0.125 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063 0.063

26 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0 0.071 0.071 0.071 0.071

27 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0 0.071 0.071 0.071

28 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0 0.071 0.071

29 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0 0.071

30 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0.071 0

31 0.067 0.067 0.067 0.067 0.067 0.067 0.067 0.067 0.067 0.067 0.067 0.067 0.067 0.067 0.067

Mixture Design Matrix

Row sums = 1.000 in a mixture design

Page 6: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Ternary plots predicting toxicity

> 2000 < 2000 < 1500 < 1000 < 500 < 0 < -500

0.00

0.25

0.50

0.75

1.00

E. coli

0.00

0.25

0.50

0.75

1.00

pH0.00 0.25 0.50 0.75 1.00

Nano

Page 7: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Follow up experiment Objective Predict toxicity % with 9 input variables: pH, Mg, K, Ca , nano-particles, RPM, E.coli, NOM, and Fructose, and 45 interaction terms Method Augmented Factorial Design Models Regression model with 54 terms, and a Polyline Visualization Rispoli and Shah et. al, Journal of Hazardous Materials, 180, (2011) Rispoli and Shah et. al, PLOS One, (2012)

Page 8: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

A “Polyline” Visualization for 9 inputs

Page 9: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

2. Problem Because of soil contamination at BNL, some of the deer killed on the site roadways were found to have elevated levels of Cs-137. Since the site property is not fenced, this raised concern over the potential radiation doses to those who eat venison from deer harvested on or near the BNL site. BNL Host: Dr. Tim Green

VFP Projects Overview

Page 10: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Objective Determine if the BNL clean-up that took place from 2000 to 2005 was effective in significantly lowering Cs-137 levels Data Collection Data was obtained from deer found dead on roads that are onsite (within 1 mile of BNL), and offsite, and Data was obtained from hunters who allow BNL to remove 2 lbs of flesh of harvested deer to send to a laboratory to determine Cs-137 levels. Statistics Bar Charts, box plots, time series graphs, non-parametric tests, ANOVA

Data Collection and Analysis

Page 11: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

1996

12

10

8

6

4

2

0

Year

Cs-

137

Con

cent

ratio

n (p

Ci/g

)

Onsite, yearly box-plots of the Cesium-137 concentration distribution in the meat of white tailed deer. Rispoli, Fasano, Green, Environmental Science and Pollution Research (2014)

Page 12: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

The distribution of Cesium Levels are not normal Medians and sample size given in table below

Onsite Offsite Pre Clean-up 2.04 pCi/g, 257 0.63 pCi/g, 93 Post Clean-up 1.22pCi/g, 92 0.65 pCi/g, 29 Mann Whitney Statistics Test for Medians: The difference for Onsite is significant, p = 0.014

Formal Analysis

Page 13: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

VFP Projects Overview 3. Are Environmental Scientists Using Statistics Correctly? (Tim Green, and Huan Feng, VFP ) Observations of Some Common Flaws §  OFAT (One Factor At a Time) Experiments are not efficient and do not

capture interactions, Use Statistical DOE such as Factorial Designs §  Primary outcome measures are not clearly defined or reliable, Use

Measurement Systems Analysis, Study Variation Statistically §  Data collection methods are not clear or not reliable, Test methods for

consistency, test for correct sample size §  Statistical assumptions are not met, Test for normality, use non-parametric

tests §  Models are too sensitive to inputs (not robust), Use simulation to test

reliability of models (Energy and Environment Research, Vol 5, 2015)

Page 14: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Current VFP Projects

4. Measuring the effects of environmental changes a) Has the solar farm effected bird diversity? (Tim Green)

b) How has an overabundant deer population effected bird diversity and vegetation at the BNL Site? (Tim Green) c) Is a Wind Farm feasible? If so, how does this effect the environment? (John Heiser)

Page 15: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Using the Statapult to Improve/Teach the Scientific Process

§  Design, Perform and Multi Factor Experiments §  Minimizing Process Variation and Stability §  Measurement Systems Analysis

Page 16: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

0

50

100

150

200

250

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

Statapult Output: Whis team is most stable?

Team 1

Team 2

Team 3

Minimizing Process Variation

Page 17: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

S

top

Ang

le

Hook Position

-1,-1,-1

Tension Pin

Run Hook Tension Stop Inches Number Position Pin Angle Traveled

1 -1 -1 -1 78

2 1 -1 -1 90 3 -1 1 -1 79

4 1 1 -1 85

5 -1 -1 1 92

6 1 -1 1 65 7 -1 1 1 82

8 1 1 1 45

A B C Response

1,1,1

A Factorial Design Example

What factor settings make the ball travel the farthest?

Factorial designs are efficient and allow one to study interactions

Page 18: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Measurement System Analysis, A Typical Study

1 2 3 4 5 6 7 8 9 10

Operator 1

Operator 2

Operator 3

Trial 1

Trial 2

Trial 1

Trial 2

Trial 1

Trial 2

Parts

Measurement Systems Analysis is an experimental study designed to determine how much of the variation in measurements is due to the measurement system, and how much is due to the items being measured.

Page 19: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Accurate but not precise - On average, the shots are in the center of the target but there is a lot of variability

Precise but not accurate - The average is not on the center, but the variability is small

Accuracy vs. Precision

Page 20: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Components of Measurement Variation

All measurement systems have error. If you don’t know how much of the variation you observe is contributed by your measurement system, you can not reach reliable conclusions.

Accuracy Precision

Repeatability Reproducibility

Measurement System Error Item-to-Item (true) Variation

Observed Variation

Stability Bias Linearity

Page 21: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Measurement Systems Experiment 1.  Organize into teams of 5 or 6 people. For each of the Statapult hook

positions (1, 2, 3, 4, 5) launch the Statapult twice. Different hook positions correspond to different items to be measured. Identify 2 or 3 inspectors and have each inspector determine the measure and record the distance. Record the data on the data sheet.

2. Perform a variable Gage R&R study using Minitab. Observe that now, –  Number of items to measure: 5 (Hook positions correspond to items) –  Number of inspectors: 2 –  Number of trials: 2 This is a 5 x 2 x 2 test with 2 inspectors

Page 22: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

%Contribution Source VarComp (of VarComp) Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000 0.00 Operator 0.000 0.00 Part-To-Part 453.045 98.65 Total Variation 459.262 100.00

Minitab Output, Team A

Page 23: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Minitab Output, Team A

Page 24: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Sample Minitab Output, Team B

%Contribution Source VarComp (of VarComp) Total Gage R&R 18.778 4.04 Repeatability 18.120 3.90 Reproducibility 0.658 0.14 Operator 0.658 0.14 Part-To-Part 446.069 95.96 Total Variation 464.847 100.00

Is this acceptable? Which team is better?

Page 25: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Minitab Output, Team B

Page 26: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

VFP carrying out a Measurement Systems Analysis Experiment

Page 27: MEASUREMENT SYSTEMS ANALYSIS VFP … · Measurement Systems Analysis, Study Variation Statistically ! ... Total Gage R&R 6.216 1.35 Repeatability 6.216 1.35 Reproducibility 0.000

Team A vs. Team B in the MSA Experiment