Bayesian Model Robust and Model Discrimination Designs

William Li

Operations and Management Science Department

University of Minnesota

(joint work with Chris Nachtsheim and Vincent Agtobo)

Outline

• Introduction

• Bayesian model-robust designs

• Bayesian model discrimination designs

Part I: Introduction• Main objective: find designs that are efficient over a

class of models– Model estimation: Are all models estimable?– Model discrimination: Can estimable models be

discriminated?

• Brief literature review– Early work: Lauter (1974), Srivastava (1975), Cook and

Nachtsheim (1982)– Cheng, Steinberg, and Sun (1999)– Li and Nachtsheim (2000)– Jones, Li, Nachtsheim, Ye (2006)

More literature review

• Bingham and Chipman (2002): Bayesian Hellinger distance

• Miller and Sitter (2005): the probability that the true model is identified

• Montgomery et al. (2005): application of new tools in model-robust designs

• Loeppky, Sitter, and Tang (2005): projection model space

• Jones, Li, Nachtsheim, Ye (2006): model-robust supersaturated designs

A general framework

• Li (2006): a review on model-robust designs• Framework: three main elements

– Model space: F={f1, f2, …, fu}

– Criterion (e.g., EC, D-, EPD)

– Candidates designs (e.g., orthogonal designs)

• Objective (rephrase): select an optimal design from candidate designs, such that it is optimal over all models in F, with respect to a criterion

Model spaces

• Srivastava (1975): search designs– F = {all effects of type (ii) + up to g effects of type

(iii)}

• Sun (1993), Li and Nachtsheim (2000)– Fg = {all main effects + up to g 2f interactions}

• Supersaturated designs– F = {any g out of m main effects}

• Loeppky, Sitter, and Tang (2005)– Fg = {g out of m main effects + all 2f interactions}

Criteria and candidate designs

• Criteria– Bayesian model-robust criterion (related to EC

and IC of LN)– Bayesian model discrimination criteria (related

to EPD of Jones et al.)

• Candidate designs– Orthogonal designs– Optimal designs

Bayesian optimal designs

• Main elements– Prior distribution: p()– Distribution of data: p(y | )– Utility function: U(d, y)– Design space

Selected literature

• “Bayesian Experimental Design: A Review” – Chaloner and Verdinelli (1995)

• DuMouchel and Jones (1994): Bayesian D-optimal designs

• Jones, Lin, and Nachtsheim (2006): Bayesian supersaturated designs

Part II: Bayesian model-robust designs

• Focus: estimability of designs– Estimation capacity (EC): percentage of estimable

models• Model-robust designs: EC=100%

– Information capacity (IC): average D-criterion value over all models

• Model space– LN (2000): main effects + g 2fi’s– Loeppky et al. (2006): g main effects + all 2fi’s among g factors

Bayesian criterion

Bayesian criterion for model-robust designs

Bayesian model-robust design

• Prior probabilities– Uniform prior– Hierarchical prior

• Chipman, Hamada, and Wu (1997)

• Bayesian model-robust (BMR) criterion

• Bayesian model-robust design (BMRD)

Design evaluations

• Evaluating existing orthogonal designs – 12-, 16-, and 20-run designs (Sun, Li, and Ye,

2002)– Two model spaces– Compute BMR values and rank designs– Compare BMR ranks with generalized WLP

ranks • Generalized WLP: Deng and Tang (1999)• Ranks for GWLP: given in Li, Lin, and Ye (2003)

Rank comparison plot (for 16*7 designs)

Design constructions

• Optimal designs– Balanced (equal # of +’s and –’s)

• CP algorithm of Li and Wu (1997)

– General (unbalanced) optimal designs• Coordinate-exchange algorithm of Meyer and

Nachtsheim (1995)

Part III: Bayesian model discrimination designs

• Issues beyond model estimation– How well can estimable models be distinguished from

each other?

– If true model is known, is it fully aliased with other models through the design?

Criteria

• Atkinson and Fedorov (1975)

• EPD (expected prediction differnce) criterion (Jones et al. 2006)

Expected non-centrality parameter (ENCP) criterion

Bayesian EPD criterion

Design results

• Evaluating orthogonal designs– A comprehensive study of designs– Candidate designs: 12-, 16-, 20-run designs – Model space: both LN and the projected space of

Loeppky et al. (2006)– Criteria: all model discrimination criteria (Bayesian and

non-Bayesian)

• Constructing optimal designs– CP: balanced– Coordinate-exchange: general (unbalanced)

An example

mEPD aEPD mAF aAF mENCP aENCP

----------------------------------------

(n=16, m=5, g=2)

1-3 EC < 100%

4 0.125 0.205 0.347 0.567 16.000 26.182

5 0.063 0.187 0.173 0.520 4.000 22.109

6 EC < 100%

7 0.000 0.184 -9.999 -9.999 0.000 19.806

8 0.094 0.198 0.173 0.495 4.000 19.673

9 EC < 100%

10 0.094 0.198 -9.999 -9.999 8.000 17.673

11 0.058 0.177 -9.999 -9.999 4.667 15.702

THANK YOU!

More information: www.csom.umn.edu/~wli