MotivationAssessment and Scope

ResultsConclusions

Evaluating and Constructing Designs forRobustness to Unusable Observations

Byran Smucker1, Willis Jensen2, Zichen Wu1, and Bo Wang1

1Miami University, Oxford, OH2W.L. Gore and Associates, Flagstaff, AZ

March 2, 2017IFPAC Annual Meeting, North Bethesda, MD

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Satire from The Onion

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Optimality vs. Robustness

A Paradox?

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Box and Draper (1975)

Addressed 14 ways a response surface design can be good.

Generate information in region of interest

Ensure fitted values are close as possible to true values

Detect lack of fit

Allow for transformations

Allow for blocks

Allows for building up of sequential experiments

Provide internal estimate of variability

Robust to wild observations and non-normality

Uses minimum number of runs

Allows for graphical assessment

Simple to calculate

Robust to the factors settings (the x’s)

Do not require a lot of levels in the x’s

Provide check of constant variance assumption

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

15th Way a Design Can Be Good

Robustness to Missing Observations

Note: We distinguish between robustness to outliersand robustness to missing observations.

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

The Problem

How Many Have Ever Had an Experiment WithMissing Observations?

Prevalence of Missing Observations

Siddiqui (2011) suggested 1-10% of observations are wildCo-author’s experience suggests that values of 0-20% arepossible

Assume Observations are Missing at Random

Does not always hold: could be that factor ranges chosenpoorlyAssume the missing values are not a result of the factor levels

Why Not Just Redo the Missing Runs?

Sometimes this is possibleOther times extremely costly or impossible to do

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Generic Example of Modern Industrial Process

These characteristics lead to difficulty in redoing missing runs

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Characteristics of Modern Industrial Processes

Increasing process complexity (more steps and more variables)

Increasing equipment scales (more challenging to useequipment for experiments)

Increasing supply chain complexity (raw materials comingfrom multiple suppliers at multiple places in the process)

Increasing physical distances covered by process (differentsteps in different facilities and geographies)

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Discussion Question

What other characteristics make it more

difficult to redo missing runs?

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Regression

Standard regression model: y = Xβ + �.

Variance-covariance matrix of the least squares estimators is

Cov(β̂) = σ2(XTX)−1

The XTX matrix is called the information matrix.

We consider two types of models:

f T (x)β = β0 +k∑

i=1

βixi

f T (x)β = β0 +k∑

i=1

βixi +k−1∑i=1

k∑j=i+1

βijxixj +k∑

i=1

βiix2i .

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Classical Designs

Fractional factorial designs and Central Composite designs

http://www.itl.nist.gov/div898/handbook/pri/section3/pri3361.htm

Smucker et al. Robustness of Designs to Missing Observations

http://www.itl.nist.gov/div898/handbook/pri/section3/pri3361.htm

MotivationAssessment and Scope

ResultsConclusions

Optimal Designs

Choose design to have desirable statisticalproperties.

1. D-optimal design relates to variance-covariance matrix of leastsquares estimators:

Assume f → Expand to X→ Choose ξn to maximize |XTX|

2. I-optimal designs minimize the average prediction variance ofthe design across design space.

3. Designs constructed to be robust to missing observations.

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Measuring Design Quality for First-Order Models

Use the determinant of the information matrix,∣∣XTX∣∣.

What if m observations are missing?

DF (i ,m) =

∣∣∣XTn−m,iXn−m,i ∣∣∣|(X*)Tn (X*)n|

1/p , i = 1, 2, . . . ,(nm

)

This metric gives a sense, in an absolute way, of how muchinformation is being lost when m runs are missing.

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Measuring Design Quality for Response Surface Models

I-optimal designs because they seek to minimize the averageprediction variance across the design space:

I =

∫Rf T (x)(XTX)−1f (x) dx ,

If m observations are missing?

IF (i ,m) =I *n

In−m,i, i = 1, 2, . . . ,

(n

m

),

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

How to Assess Impact of Missing Runs

Examine how much information classical and optimal designs losewhen runs go missing.

For instance, if 1 run is missing from an n-run design, we willcompute the D-efficiency for each possible (n − 1)-run design andlook at its distribution.

We’ll look at first- and second-order response surface models,various design sizes, and a small number of missing runs(m = 0, 1, 2, 3).

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

First-order Models: k = 5

Figure: D-efficiencies for possible main effects designs, for k = 5 andn = {8, 10, 12, 16}, according to the number of missing runs.

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Second-order Models: k = 3

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Conclusions: Screening Designs

1 Missing observations have relatively little impact on first-orderdesigns unless design is small.

2 Fractional and D-optimal designs are robust to missingobservations.

3 If you are worried about losing a run or two, add a few runs atthe outset.

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Conclusions: Response Surface Designs

1 No evidence that optimal designs are less robust to missingobservations than classical designs

2 Optimal designs are more efficient, and the efficiency holds upwhen a few observations are lost.

3 Missing-robust optimal designs are better when originalnumber of runs is small

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Bottom Line

1. Optimal designs and classical designs have similarrobustness properties, in terms of missing runs.

2. For severely resource-constrained experiments for which youare concerned about missing observations, either (1) add a fewextra runs; or (2) consider using a missing-robust design.

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Some Unanswered Questions

What is the impact on spherical regions (eg. 5 level CCD vs.optimal)?

What is the impact on other types of designs (blocking,split-plots, mixtures, etc)

Are there better metrics to assess impact (ability to detectsignificant effects, width of intervals, bias and variance inpredictions)?

How could we obtain a robustness criteria based onI-optimality?

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Satirical headline: Professor celebrates landmark publication thatwill be carefully read by two people (h/t Maria Weese)

ReferenceSmucker, B.J., Jensen, W., Wu, Z., and Wang, B. (2017+).Robustness of Classical and Optimal Designs to MissingObservations. Computational Statistics & Data Analysis. In press.http://www.users.miamioh.edu/smuckebj/

Smucker et al. Robustness of Designs to Missing Observations

http://www.users.miamioh.edu/smuckebj/

MotivationAssessment and Scope

ResultsConclusions

Quick Plug

community.amstat.org/isvc; contact: Byran Smucker(smuckerb@miamioh.edu)

Smucker et al. Robustness of Designs to Missing Observations

community.amstat.org/isvc

MotivationAssessment and Scope

ResultsConclusions

Questions?

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and Scope

ResultsConclusions

Extra Slide. Second-order Models: k = 5

Smucker et al. Robustness of Designs to Missing Observations

MotivationAssessment and ScopeResultsConclusions