modeling experiments

78
© 2003 Business Process Improvements, LLC. All Rights Reserved. Full Factorial Experiments Process Certification 301

Upload: kashifbutty2k

Post on 19-Jan-2016

13 views

Category:

Documents


0 download

DESCRIPTION

Modeling Experiments

TRANSCRIPT

Page 1: Modeling Experiments

© 2003 Business Process Improvements, LLC. All Rights Reserved.

Full Factorial Experiments

Process Certification 301

Page 2: Modeling Experiments

2Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Objectives

Introduce the concepts of Modeling Experiments2k Factorials

Use Minitab with Standard Order Designs to:ConstructAnalyzeInterpret

Learn how to Improve Standard ErrorReduce the Model

Develop a DOE Analysis Roadmap

Page 3: Modeling Experiments

3Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Where are We?

IdentifyAssignable

CausesIs

Cpk > 1.33?Evaluate Process

Improvements

Implement updated Control

Plan

N

Certify the Process

Prioritize Variables

Y

N Optimize / Model the Process

Implement Control Methods

Create / UpdateProcess FMEA

Create / UpdateBaseline

Control Plan

EvaluateMeasurement

System(s)

ImplementCorrective

Actions

GR&R <20% of

ET?

IsProcess InControl?

ImplementCorrective

Actions

ReviewProcess

Documentation

ReviewProcess

Documentation

Create / updateProcess Map

(KPOs, KPIs, KCs)

Select Process and Charter

Team

Collect & Analyze

Process Data

Select Process and Charter

TeamStart

A

N

Assess ProcessCapabilityB

Maintain the Certification

Go to Next Process

B

A

EstablishProcess Controls

1. Define

2. Measure

3. Assess

4. Improve

5. Implement

6. Certify

Y Y

Page 4: Modeling Experiments

4Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

What’s in Step 4?

IMPROVE the Process and verify

the gains made

IMPROVE the Process and verify

the gains made4IMPLEMENT a control plan

to hold the gainsIMPLEMENT a control plan

to hold the gains

CERTIFY, maintain and standardize the processCERTIFY, maintain and standardize the process

Purpose:Apply P-D-C-A strategies to improve the process by attacking those top priority problems identified in Step 3.

Plan: Plan a well-thought-out set of experiments, tests, and observations meant to address a well-defined problem.

Do: Do the things you planned! Collect the data.

Check: Analyze the results using statistically valid methods.

Act: Take appropriate action based on the results.

Tools:Multi-Vari Studies: Establishing the contribution of input variables to the overall process variability.

Hypothesis testing: Comparing means, variances, and proportions; goodness-of-fit; contingency tables; and others...

Experimentation (DOE): Factorial experiments, Response surface experiments, “steepest assent” strategies, and others...

Modeling: Regression analysis, Analysis of Variance, and others…

DEFINE the processcertification opportunities

DEFINE the processcertification opportunities1

MEASURE and baseline the process key characteristics

MEASURE and baseline the process key characteristics2ASSESS and analyze the

current state of the processASSESS and analyze the

current state of the process3

5

6

Page 5: Modeling Experiments

5Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Design Classification

Design Type Screening Design or Fractional Factorial

Full Factorial

Central Composite

Design

Factors Explored:

3 to 15 2 to 6 2 to 5

Effects Estimated:

Individual Individual, Interaction

Individual, Interaction, Curvature

Result: Identify

Important Factors

Understand System

Behavior

High Quality Prediction,

Optimization

Page 6: Modeling Experiments

6Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Experiments

When we discuss experiments, we are interested in the following:

Number of factors we wish to investigate • Represented by the letter ‘k’

Number of Levels we want to set each Factor to during the experiment

• Represented by the letter ‘n’

Number of runs we need to conduct to look at all combinations of the Factors at their respective Levels

• Represented by the relationship nk

– Note: this is valid for designs where each Factor has the same number of Levels

Page 7: Modeling Experiments

7Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

2k Factorials

A 2k factorial refers to k factors, each with 2 levels

A 22 factorial is also represented as a 2x2 factorial

This design has two factors with two levels and can be done in 2x2, or 4, runs

Likewise a 23 factorial includes 3 factors, each with two levels

This experiment can be done in 2x2x2, or 8, runs

Page 8: Modeling Experiments

8Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Construction

Page 9: Modeling Experiments

9Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Standard Order of 2k Designs

Designs are usually shown in standard orderThe low level of a factor is designated with a “-” or -1

The high level is designated with a “+” or +1

Air Pressure Solids Content-1 -1+1 -1-1 +1+1 +1

Page 10: Modeling Experiments

10Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

23 Design

Air Pressure Solids Content Nozzle Type-1 -1 -1+1 -1 -1-1 +1 -1+1 +1 -1-1 -1 +1+1 -1 +1-1 +1 +1+1 +1 +1

Page 11: Modeling Experiments

11Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Exercise

Create a 24 Factorial Design Matrix on paper

What are the minimum number of runs needed?

Page 12: Modeling Experiments

12Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Experiment Construction Roadmap

Create the Factorial Design

Select type of design

Select Number of Factors

Select design type

Enter names and levels of Factors

Set randomization criteria

Enter response column

Page 13: Modeling Experiments

13Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Creating a Factorial Design

Improve> Modeling> Create Factorial Design

Select type of designSelect type of design

Select number of factorsSelect number of factors

Page 14: Modeling Experiments

14Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Creating a Factorial Design

Select the Design ...Select the Design ...

Page 15: Modeling Experiments

15Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Creating Factorial Design

Enter Factor informationEnter Factor information

Page 16: Modeling Experiments

16Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Create Factorial Design

Deselect randomization for this exercise

Deselect randomization for this exercise

Design OptionsDesign Options

Page 17: Modeling Experiments

17Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

24 Full Factorial Design in Standard Order

Page 18: Modeling Experiments

18Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Exercise

Procedure:Use the following catapult Factors and their respective Levels

• Hinge Point: 1 and 3

• Stop Position: 2 and 4

• Ball Type: Ping Pong and Golf

Response:• Distance to first impact

Design a worksheet using Minitab in Standard Order

Conduct the experiment and load your data into Minitab

Page 19: Modeling Experiments

19Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Full Factorial DesignAnalysis

Page 20: Modeling Experiments

20Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Example Background

The coating thickness across a part is inconsistent

The problem is thought to be associated with the spraying of the material

The aim is to study the effect of 3 Factors on the Response – thickness variation across the product

Response values closer to zero (0) are desiredThis shows no difference across the part

Page 21: Modeling Experiments

21Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

23 Design: 3 Factors, 8 Runs

Assign Factors and Levels

Variable -1 Level +1 LevelAir Pressure 90 PSI 100 PSISolids Content 20% 30%Nozzle Type A B

Page 22: Modeling Experiments

22Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Design with Actual Levels

Run Air Pressure Solids Content Nozzle Type Coating Variation1 90 20% A2 100 20% A3 90 30% A4 100 30% A5 90 20% B6 100 20% B7 90 30% B8 100 30% B

Page 23: Modeling Experiments

23Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Response Values Added

Run Air Pressure Solids Content Nozzle TypeCoating Variation

1 90 20% A 4.302 100 20% A 3.813 90 30% A 2.734 100 30% A 4.005 90 20% B 5.706 100 20% B 4.217 90 30% B 4.708 100 30% B 4.33

Page 24: Modeling Experiments

24Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Design with Coded Levels

Run Air Pressure Solids Content Nozzle TypeCoating Variation

1 -1 -1 -1 4.302 +1 -1 -1 3.813 -1 +1 -1 2.734 +1 +1 -1 4.005 -1 -1 +1 5.706 +1 -1 +1 4.217 -1 +1 +1 4.708 +1 +1 +1 4.33

Page 25: Modeling Experiments

25Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Analysis Mathematics

We will learn how to:Calculate the effect of the variables (main effects)

Calculate coefficients• For variables and interactions

Interpret graphical output from Minitab

There are two ways of analyzing factorial designs:

Using main effects and using coefficients

Minitab does them both!

There are two ways of analyzing factorial designs:

Using main effects and using coefficients

Minitab does them both!

Page 26: Modeling Experiments

26Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Calculating Main Effects

Nozzle TypeCoating Variation

-1 4.30-1 3.81-1 2.73-1 4.00+1 5.70+1 4.21+1 4.70+1 4.33

The main effect, also called the average response, is defined as:(average of response at +1 level) - ( average of response at -1 level)

So the main effect of Nozzle Type is:So the main effect of Nozzle Type is:

4.74 - 3.71 = 1.03

(5.70 + 4.21 + 4.70 + 4.33)/4 = 4.74

For Nozzle Type, the average of the +1 level is:

For Nozzle Type, the average of the +1 level is:

(4.30 + 3.81 + 2.73 + 4.00)/4 = 3.71

And the average of the -1 level is:And the average of the -1 level is:

Page 27: Modeling Experiments

27Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Graphical Representation

The main effect for the Nozzle Type variable is 1.03

This means Coating Variation (Y) increases by 1.03 units when changing from the -1 level (Nozzle A) to the +1 level (Nozzle B)

The Main Effects Plot shows this

Nozzle Type

Mea

n of

Coa

ting

Var

iati

on

BA

4.8

4.6

4.4

4.2

4.0

3.8

3.6

Main Effects Plot (data means) for Coating Variation

Page 28: Modeling Experiments

28Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Other VariablesM

ean

of C

oati

ng V

aria

tion

10090

4.5

4.4

4.3

4.2

4.1

4.0

3.93020

Air Pressure Solids Content

Main Effects Plot (data means) for Coating Variation

Page 29: Modeling Experiments

29Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Exercise

Calculate Main Effects for your catapult Factors (Hinge Point, Stop Position, Ball Type)

Sketch the Main Effects Plots for each

Construct a list of the Factors in descending order of size of effect

Which Factor has the biggest effect?

Which Factor has the smallest effect?

Which Factors have statistically significant effects?

Page 30: Modeling Experiments

30Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Interactions

Sometimes there is a bigger effect when two Factors are changed at the same time than what we would expect from the effect of each Factor on their own

We call this effect an interactioninteraction

We can calculate the interactions from the design matrix by adding some extra columns

Notice that each column has a pattern that is different from all other columns

This is called orthogonality

What would we expect the correlation coefficients to be for all these columns?

Page 31: Modeling Experiments

31Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Design with Interactions

We construct the interaction columns by cross-multiplying the relevant Factors

We construct the interaction columns by cross-multiplying the relevant Factors

Example: for the A*B interaction column multiply the coded values for each Factor

RunAir Pressure

(A)Solids

Content (B)Nozzle Type

(C) A*B A*C B*C A*B*CCoating Variation

1 -1 -1 -1 1 1 1 -1 4.302 +1 -1 -1 -1 -1 1 1 3.813 -1 +1 -1 -1 1 -1 1 2.734 +1 +1 -1 1 -1 -1 -1 4.005 -1 -1 +1 1 -1 -1 1 5.706 +1 -1 +1 -1 1 -1 -1 4.217 -1 +1 +1 -1 -1 1 -1 4.708 +1 +1 +1 1 1 1 1 4.33

This is repeated for all columns and rows

(Minitab does this for us)

This is repeated for all columns and rows

(Minitab does this for us)

Page 32: Modeling Experiments

32Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

What is a Coefficient?

A coefficient is a number that indicates how much the Response (Y) changes for a change of 1 coded unit in the Factor (X)

The coefficient gives the slope of the line

The coefficient gives the slope of the line

Effe

ct

-1 +1

Page 33: Modeling Experiments

33Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

The Coefficients

We use the letter β to indicate our coefficientsA coefficient is the change in Y for a change of one coded unit in X

Variablesβ1 tells us how big is the effect of A individuallyβ2 tells us how big is the effect of B individuallyβ3 tells us how big is the effect of C individually

Interactionsβ12 tells us how big is the effect of changing A and B togetherβ13 tells us how big is the effect of changing A and C togetherβ23 tells us how big is the effect of changing B and C togetherβ123 tells us how big is the effect of changing A AND B AND C together

Page 34: Modeling Experiments

34Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Interpretation

Listed are the Factor names, the mathematical coefficient

symbols and their values

Minitab calculates these values for us

Listed are the Factor names, the mathematical coefficient

symbols and their values

Minitab calculates these values for us

Factor Coefficient ValueConstant β0 4.2225

Air Pressure (A) β1 -0.1350Solids Content (B) β2 -0.2825Nozzle Type (C) β3 0.5125

A*B β12 0.3600A*C β13 -0.3300B*C β23 0.0625

A*B*C β123 -0.0800

Which variable has the biggest effect?Which variable has the smallest effect?

Which variables have statistically significant effects?

Page 35: Modeling Experiments

35Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

DOE Analysis

Page 36: Modeling Experiments

36Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Experiment Analysis Roadmap

Collect Response Data

Determine Response(s) to be analyzed

Select terms to be investigated(Factors and 2nd Order Interactions for

Full FactorialsFactors Only for Fractional Factorials)

Interpret graphical outputPareto and Normal Plot of

Effects

Reduce the model

Page 37: Modeling Experiments

37Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Experiment Analysis

Improve> Modeling> Analyze Factorial Design

Open File: DOE.MPJ

Worksheet: Coating Variation

Open File: DOE.MPJDOE.MPJ

Worksheet: Coating VariationCoating Variation

Select the TermsSelect the Terms

Page 38: Modeling Experiments

38Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Select Graphs

Select Normal and Pareto Effects PlotsSelect Normal and Pareto Effects Plots

We will look at these first to decide which variables and

interactions appear important

We will look at these first to decide which variables and

interactions appear important

Page 39: Modeling Experiments

39Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Pareto Chart of Standardized Effects

We use a Pareto chart of the effects to compare the relative magnitude and the statistical significance of both main and interaction effects

MINITAB plots the effects in decreasing order of the absolute value of the standardized effects and draws a reference line on the chart

Any effect that extends past this reference line is considered tAny effect that extends past this reference line is considered to o be significantbe significant

By default, MINITAB uses an alpha-level of 0.05

Why isn’t anything

important?

Why isn’t anything

important?

Term

Effect

BC

ABC

A

B

AC

AB

C

3.53.02.52.01.51.00.50.0

3.190Factor NameA A ir PressureB Solids C ontentC Nozzle Ty pe

Pareto Chart of the Effects(response is Coating Variation, Alpha = .05)

Lenth's PSE = 0.8475

Page 40: Modeling Experiments

40Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Pareto Chart of Standardized Effects

Hypothesis TestH0 there are no significant effects

Ha there are some significant effects

The red line is a confidence limit based on the alpha level

Any effect to the right of the red line is considered significant

The calculation of the line is based on:The alpha level selected

The standard error in the experiments

Total number of runs in the experiment

The number of terms included in the model

We can change the number of terms in the model and the alpha level and thus change the position of the line

Page 41: Modeling Experiments

41Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Normal Probability Plot

MINITAB draws a line to indicate where the points would be expected to fall if there were no effect of any of the Factors and their interactions

Significant effects are larger and farther from the line than insignificant effects

By default, MINITAB uses an alpha-level of 0.05 and labels any effect that is significant

Effect

Perc

ent

210-1-2

99

95

90

80

7060504030

20

10

5

1

Factor NameA A ir PressureB Solids C ontentC Nozzle Ty pe

Effect TypeNot SignificantSignificant

Normal Probability Plot of the Effects(response is Coating Variation, Alpha = .05)

Lenth's PSE = 0.8475

Looks like nothing is important…

… but let’s look at the Analytical results

before we give up

Looks like nothing is important…

… but let’s look at the Analytical results

before we give up

Page 42: Modeling Experiments

42Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

ANOVA Table

Note there are no degrees of freedom for the error termWithout any DF for Error calculation, we can not get any P values!

We need to remove a term from the modelWe will remove the term with the smallest effect, the B*C interaction

Here are the coefficients calculated

earlier

Here are the coefficients calculated

earlier

Here are the effects

calculated earlier

Here are the effects

calculated earlier

Page 43: Modeling Experiments

43Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Model Reduction

Remove the smallest effect

Examine:Probability plots

Pareto plots ANOVA table

Repeat with next smallest effect

Continue until model is considered “the best”

Tips on “best model” selection:

Do not remove too many terms

Include one insignificant term to ensure that a significant one was not removed by mistake

Tips on “best model” selection:

Do not remove too many terms

Include one insignificant term to ensure that a significant one was not removed by mistake

Analyze Residuals and Fits for “best” model

Page 44: Modeling Experiments

44Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Reducing the Model

We remove the BC Interaction term firstWe remove the BC Interaction term first

Improve> Modeling> Analyze Factorial Design

Page 45: Modeling Experiments

45Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Pareto Chart of “Reduced” Model

Still no variables show to be significant

We can remove the A*B*C interaction term nextIt is unusual to have a third order interaction term that is statistically and practically significant

Term

Standardized Effect

ABC

A

B

AC

AB

C

14121086420

12.71Factor NameA A ir PressureB Solids C ontentC Nozzle Ty pe

Pareto Chart of the Standardized Effects(response is Coating Variation, Alpha = .05)

Page 46: Modeling Experiments

46Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

“Best” Reduced Model

3 terms remain statistically significantTwo interactions have been removed from the model

Notice that two factors appear to not be significant (A and B)

Notice that they are BOTH involved in interactions though!

Term

Standardized Effect

A

B

AC

AB

C

876543210

4.303Factor NameA A ir P ressureB Solids C ontentC Nozzle Ty pe

Pareto Chart of the Standardized Effects(response is Coating Variation, Alpha = .05)

Page 47: Modeling Experiments

47Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

ANOVA Table for “Best” Model

We use the p-values to determine whether the term is significant

Any < 0.05 we will consider as significant

Page 48: Modeling Experiments

48Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Exercise

Conduct the analysis for your catapult Factors (Hinge Point, Stop Position, Ball Type)

Reduce the model as necessary

Identify the significance of the Factors and Interactions

Which Factor has the biggest effect?

Which Factor has the smallest effect?

Which Factors and Interactions have statistically significant effects?

Page 49: Modeling Experiments

49Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Interpretation

Page 50: Modeling Experiments

50Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Analysis Interpretation Roadmap

Examine the ANOVA table

Examine plots:Interactions, Main Effects,

Cube, Variances, Residuals

Determine practical significance of terms (% contribution)

Finalize model

Finalize recommendations next steps

Conduct confirmation runs / validate the model

Page 51: Modeling Experiments

51Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Interpretation

There are many plots to help understand the results

Interaction Plots• Look at significant, highest order interactions first

• Do this before looking at the effects plots

Main Effects Plot• Look at significant factors

Cube Plots• Looks at multiple Factors at the same time

Residuals Plots• Examines the validity of the model

Equal Variance Plot• Tests the underlying assumptions in DOE

Page 52: Modeling Experiments

52Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Interaction Plots

Select Interaction Plot and SetupSelect Interaction Plot and Setup

Insert “Coating Variation”as the Response

Insert “Coating Variation”as the Response

We will look at all the interactions

We will look at all the interactions

Improve> Modeling> Factorial Plots

Page 53: Modeling Experiments

53Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Interaction Plots

A ir Pressure

3020 BA

4.8

4.0

3.2

Solids Content

4.8

4.0

3.2

Nozzle T ype

Air Pressure90

100

SolidsContent

2030

Interaction Plot (data means) for Coating Variation

Look for non-parallel and crossing lines

The plot helps identify interactions we want to look at further (individually) for setting factor levels

Look for non-parallel and crossing lines

The plot helps identify interactions we want to look at further (individually) for setting factor levels

Page 54: Modeling Experiments

54Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Interaction Plot Interpretation

This plot shows a strong interaction between Air Pressure and Solids Content

Notice that the level of Coating Variation “depends” on the levels of the two variables

To achieve the lowest average Coating Variation:Use low Air Pressure and high Solids Content

To achieve the lowest variation in Coating Variation:Use high Air Pressure (especially if Solids Content is hard to control)

Solids Content

Mea

n

3020

5.0

4.8

4.6

4.4

4.2

4.0

3.8

3.6

Air Pressure90

100

Interaction Plot (data means) for Coating Variation

Page 55: Modeling Experiments

55Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Interaction Plot Interpretation

This plot shows a strong interaction between Air Pressure and Nozzle Type

What levels of each variable should be chosen to achieve the lowest average Coating Variation?

What about to achieve the lowest variation in Coating Variation?

Nozzle Type

Mea

n

BA

5.25

5.00

4.75

4.50

4.25

4.00

3.75

3.50

Air Pressure90

100

Interaction Plot (data means) for Coating Variation

Page 56: Modeling Experiments

56Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Main Effects Plot

Select Main Effects Plot and Setup

Select Main Effects Plot and Setup

Improve> Modeling> Factorial Plots

We will plot all Factors

We will plot all Factors

Insert “Coating Variation” as the

response

Insert “Coating Variation” as the

response

Page 57: Modeling Experiments

57Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Main Effects Plot

Main Effects plots should be interpreted after all significant interactions plots have been reviewed

Notice the influence of Nozzle Type versus the other factors

Since we have already chosen levels for our three Factors from the interaction plots, we really don’t need the Main Effects plot

Main Effects plots should be interpreted after all significant interactions plots have been reviewed

Notice the influence of Nozzle Type versus the other factors

Since we have already chosen levels for our three Factors from the interaction plots, we really don’t need the Main Effects plot

Mea

n of

Coa

ting

Var

iati

on

10090

4.6

4.4

4.2

4.0

3.8

3020

BA

4.6

4.4

4.2

4.0

3.8

Air Pressure Solids Content

Nozzle Type

Main Effects Plot (data means) for Coating Variation

Page 58: Modeling Experiments

58Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Cube Plot

Select Cube Plot and SetupSelect Cube Plot and Setup

Insert “Coating Variation” as the

response

Insert “Coating Variation” as the

response

We will plot all Factors

We will plot all Factors

Improve> Modeling> Factorial Plots

Page 59: Modeling Experiments

59Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Cube Plot

B

A

30

2010090

Nozzle Type

Solids Content

Air Pressure

4.33

4.215.70

4.70

4.00

3.814.30

2.73

Cube Plot (data means) for Coating Variation

Cube plots are useful for identifying the combination of Factors that gives the highest or lowest average response

Cube plots are useful for identifying the combination of Factors that gives the highest or lowest average response

Page 60: Modeling Experiments

60Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Factor Level Selection

Based on the Interaction plots, the Main Effects plot, and the Cube Plot we can determine the best factor levels

In this example, we want lower Coating Variation

Therefore, our factor levels should be:Air Pressure: Low

Solids Content: High

Nozzle Type: A

We would expect the Coating Variation to be about 2.73 for this combination

Reference the Cube Plot

Page 61: Modeling Experiments

61Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Residual Plots

Improve> Modeling> Analyze Factorial Design

Page 62: Modeling Experiments

62Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Residual Plots

Residual

Per

cent

0.20.10.0-0.1-0.2

99

90

50

10

1

Fitted Value

Res

idua

l

6543

0.1

0.0

-0.1

Residual

Freq

uenc

y

0.150.100.050.00-0.05-0.10-0.15

4

3

2

1

0

Observation Order

Res

idua

l

87654321

0.1

0.0

-0.1

Normal Probability Plot of the Residuals Residuals Versus the Fitted Values

Histogram of the Residuals Residuals Versus the Order of the Data

Residual Plots for Coating Variation

We expect stability

We expect stability

We expect random

variation, no

patterns

We expect random

variation, no

patternsWe expect

normally distributed

error

We expect

normally distributed

error

Page 63: Modeling Experiments

63Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Test for Equal Variances

An underlying assumption is that the variances within factor levels (of the same factor) are equal

Minitab can not test all three factors at the same time, but it can handle two

Test the variances for Nozzle Type first, then for Air Pressure and Solids Content together

Improve> Test for Equal Variances

Page 64: Modeling Experiments

64Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Equal Variance Plots

Test for Air Pressure and Solids Content

p = 0.454

Variances are equal

Test for Air Pressure and Solids Content

p = 0.454

Variances are equal

Assumptions are met

Assumptions are met

Test for Nozzle Type

p = 0.986

Variances are equal

Test for Nozzle Type

p = 0.986

Variances are equal

Noz

zle

Typ

e

95% Bonferroni Confidence Intervals for StDevs

B

A

3.53.02.52.01.51.00.50.0

Noz

zle

Typ

e

Coating Variation

B

A

6.05.55.04.54.03.53.0

F-Test

0.946

Test Statistic 1.02P-Value 0.986

Levene's Test

Test Statistic 0.01P-Value

Test for Equal Variances for Coating Variation

95% Bonferroni Confidence Intervals for StDevs

Air Pressure Solids Content

100

90

30

20

30

20

200150100500

Bartlett's Test

Test Statistic 2.62P-Value 0.454

Test for Equal Variances for Coating Variation

Page 65: Modeling Experiments

65Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Exercise

Complete the Graphical Analysis of the catapult data

Are there any significant Interactions? How did you interpret the graph?

What level of each Factor would you choose if you are trying to increase Distance?

Page 66: Modeling Experiments

66Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Determining Practical Significance

Statistical significance of the terms / model is only one step – we need to determine the practical significance of each factor

We use the General Linear Model to calculate the SS for each term and the SSTotal for comparison

NOTE: we need to change the display into codedcodedunits first

Improve> Display Design

Page 67: Modeling Experiments

67Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

General Linear Model (GLM)

DOE> Practical Significance> General Linear Model

All terms from our model are included

Note that the Interaction is made by adding an ‘*’

between Factors

All terms from our model are included

Note that the Interaction is made by adding an ‘*’

between Factors

Page 68: Modeling Experiments

68Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

General Linear Model: ANOVA

We need the SS for each Term, and the total SS, to determine the % Contribution of each term

We can copy the data from the ANOVA table to new columns in the worksheet using the Alt-Left Mouse Drag / Copy method

Page 69: Modeling Experiments

69Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Practical Significance

% Contribution is just the proportion of the Total SS for each Factor and Interaction (source)

% Contribution is just the proportion of the Total SS for each Factor and Interaction (source)

Page 70: Modeling Experiments

70Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Practical Results

There are four statistically significant terms

However, Nozzle Type contributes 43% of all the observed variation in our experiment

What action would you take based on these results?

What about the other sources of variation?

Page 71: Modeling Experiments

71Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Mathematical Models

The coefficients are used to produce a mathematical model - an equation

The coefficients are used to produce a mathematical model - an equation

Factor Coefficient ValueConstant β0 4.2225

Air Pressure (A) β1 -0.1350Solids Content (B) β2 -0.2825Nozzle Type (C) β3 0.5125

A*B β12 0.3600A*C β13 -0.3300B*C β23 0.0625

A*B*C β123 -0.0800

The Full Equation

Y = β0 + β1A + β2B + β3C + β12AB + β13AC + β23BC + β123ABC + error

The Factors are in coded values

The Full Equation

Y = β0 + β1A + β2B + β3C + β12AB + β13AC + β23BC + β123ABC + error

The Factors are in coded values

Page 72: Modeling Experiments

72Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Reduced Model

Based on our Factor level selection, the predicted output is then:

Coating Variation (Y) = 4.2225 + (-0.1350)(-1) + (-0.2825)(1) + (0.5125)(-1) + (0.3600)(-1)(1) + (-0.3300)(-1)(-1) + error

Based on our Factor level selection, the predicted output is then:

Coating Variation (Y) = 4.2225 + (-0.1350)(-1) + (-0.2825)(1) + (0.5125)(-1) + (0.3600)(-1)(1) + (-0.3300)(-1)(-1) + error

Predicted Coating Variation = 2.7510 (+ error)

Predicted Coating Variation = 2.7510 (+ error)

The Reduced Model Equation

Y = β0 + β1A + β2B + β3C + β12AB + β13AC + error

The Factors are in coded values

The Reduced Model Equation

Y = β0 + β1A + β2B + β3C + β12AB + β13AC + error

The Factors are in coded values

Factor Coefficient ValueConstant β0 4.2225

Air Pressure (A) β1 -0.1350Solids Content (B) β2 -0.2825Nozzle Type (C) β3 0.5125

A*B β12 0.3600A*C β13 -0.3300

Page 73: Modeling Experiments

73Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Exercise

Using your catapult data, what Factor(s) or Interaction(s) are practically significant?

How much of the total variation is explained by these terms?

What is your final equation?

Calculate the expected maximum distance based on your equation

Take 8 shots at the ‘best’ settings and compare the results to the prediction

What statistical tool should you use here???

Page 74: Modeling Experiments

74Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Summary

Full factorial experiments provide the opportunity to look at all interactions of all Factors included

A predictive model (equation) can be developed based on the results

Practical significance of the terms indicates what variables to control / pay particular attention to first

Residuals analysis can identify missing variables if unusual patterns exist

Page 75: Modeling Experiments

75Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Experiment Construction Roadmap

Improve> Modeling > Create Factorial DesignCreate the Factorial Design

Select type of design

Select Number of Factors

Improve> Modeling > Create Factorial Design> DesignsSelect design type

Enter names and levels of Factors Improve> Modeling > Create Factorial Design> Factors

Set randomization criteria Improve> Modeling > Create Factorial Design> Options

Enter response column

Page 76: Modeling Experiments

76Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Experiment Analysis Roadmap

Collect Response Data

Determine Response(s) to be analyzed

Select terms to be investigated(Factors and 2nd Order Interactions for

Full FactorialsFactors Only for Fractional Factorials)

Improve> Modeling > Analyze Factorial Design> Terms

Interpret graphical outputPareto and Normal Plot of

EffectsImprove> Modeling > Analyze Factorial Design> Graphs

Reduce the model

Page 77: Modeling Experiments

77Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Model Reduction

Remove the smallest effect Improve> Modeling > Analyze Factorial Design> Terms

Examine:Probability plots

Pareto plots ANOVA table

Improve> Modeling > Analyze Factorial Design> Graphs

Repeat with next smallest effect

Continue until model is considered “the best”

Analyze Residuals and Fits for “best” model

Improve> Modeling > Analyze Factorial Design> Graphs

Tips on “best model” selection:

Do not remove too many terms

Include one insignificant term to ensure that a significant one was not removed by mistake

Tips on “best model” selection:

Do not remove too many terms

Include one insignificant term to ensure that a significant one was not removed by mistake

Page 78: Modeling Experiments

78Licensed for use within United Technologies Corporation© 2003 Business Process Improvements, LLC. All Rights Reserved. Module: Modeling Experiments Rev: 1

Analysis Interpretation Roadmap

Examine the ANOVA table Improve> Modeling > Analyze Factorial Design

Examine plots:Interactions, Main Effects,

Cube, Variances, ResidualsImprove> Modeling > Factorial Plots

Determine practical significance of terms (% contribution)

Improve> Practical Significance >General Linear Model

Finalize model

Finalize recommendations next steps

Conduct confirmation runs / validate the model