Dennis D. Leber

National Institute of Standards and Technology

Jeffrey W. Herrmann

University of Maryland

Science of Test Workshop – Springfield, VA

April 4, 2017

Allocating Information Gathering

Efforts for Selection Decisions

of 15

Selecting a Radiation Detection System

1

Performance Operational Impact Cost

Pd SNM (e.g., WGPu, HEU)

Pd Industrial (e.g., 137Cs, 57Co)

Pd Medical (e.g., 131I, 201Tl )

Probability of false alarm

of 15

Challenge

2

How do we allocate our limited and fixed budget for

information gathering to maximize the probability

of selecting the true best alternative?

of 15

Selection Decision Terminology

3

Alternatives

a1: “townhouse”

a2 : “farmhouse”

a3 : “country house”

Attributes (value)

1: House size (mi1)

2: Lot size (mi2)

3: Cost (mi3)

4: Distance to work (mi4)

5: Quality of school (mi5)

Preferences and Decision Model

Mike Mulligan and His Stream Shovel

Virginia Lee Burton (1939)

1 1 1 2 2 2 3 3 3 4 4 4 5 5 5i i i i i iv v v v v m m m m m

of 15

Best Alternative

4

Alternative that provides the largest decision value

Function of:

1. Decision-maker’s preferences

2. True attribute values

Knowledge of decision values is uncertain

Uncertainty in attribute values is a function of amount of information gathered

1 1 1 2 2 2i i i k k ikv v v m m m

1 1 1 2 2 2ˆ ˆ ˆ ˆi i i k k ikv v v m m m

-------------- Estimate --------------

of 15

Selecting an Alternative

5

Probability that alternative ai has largest decision value: 𝑝𝑖 = 𝑃 𝜉𝑖 > 𝜉𝑟, ∀𝑟 = 1,… ,𝑚

Select as, where 𝑠 = argmax𝑖

𝑝𝑖

Define Probability of Correct Selection: PCSDM = 𝑝𝑠

of 15

Information Gathering Layout

6

Assume equal cost for all observations

B is observation budget

Performance

Measure 1

Performance

Measure 2…

Performance

Measure k

Alternative a1 𝑛11 n12 n1k

Alternative a2 𝑛21 n22 n2k

… … … …

Alternative am 𝑛𝑚1 nm2 nmk

of 15

Uniform Allocation

7

Performance

Measure 1

Performance

Measure 2…

Performance

Measure k

Alternative a1 𝑛11 =𝐵

𝑚𝑘𝑛12 =

𝐵

𝑚𝑘𝑛1𝑘 =

𝐵

𝑚𝑘

Alternative a2 𝑛21 =𝐵

𝑚𝑘𝑛22 =

𝐵

𝑚𝑘𝑛2𝑘 =

𝐵

𝑚𝑘

… … … …

Alternative am 𝑛𝑚1 =

𝐵

𝑚𝑘𝑛𝑚2 =

𝐵

𝑚𝑘𝑛𝑚𝑘 =

𝐵

𝑚𝑘

of 15

Proportional Allocation

8

Performance

Measure 1

Performance

Measure 2…

Performance

Measure k

Alternative a1 𝑛11 = 𝜆1𝐵

𝑚𝑛12 = 𝜆2

𝐵

𝑚𝑛1𝑘 = 𝜆𝑘

𝐵

𝑚

Alternative a2 𝑛21 = 𝜆1𝐵

𝑚𝑛22 = 𝜆2

𝐵

𝑚𝑛2𝑘 = 𝜆𝑘

𝐵

𝑚

… … … …

Alternative am 𝑛𝑚1 =𝜆1

𝐵

𝑚𝑛𝑚2 =

𝜆2𝐵

𝑚𝑛𝑚𝑘 = 𝜆𝑘

𝐵

𝑚

1 1 1 2 2 2i i i k k ikv v v m m m

of 15

Sequential Information Gathering

9

of 15

Ranking and Selection

10

Statistics

Bechhofer, R. E. (1954). A Single-sample Multiple Decision Procedure for

Ranking Means of Normal Populations with Known Variances. The Annals of

Mathematical Statistics, 25, 16-39.

Bechhofer, R. E., Santer, T. J., & Goldsman, D. M. (1995). Design and Analysis of

Experiments for Statistical Selection, Screening, and Multiple Comparisons. New

York: John Wiley and Sons, Inc.

Computer Simulation Kim, S.-H., & Nelson, B. L. (2006). Selecting the Best System. In S. G.

Henderson, & B. L. Nelson (Eds.), Handbook in Operations Research and Management Science (Vol. 13, pp. 501-534). Oxford: Elsevier.

Chen, C.-H., & Lee, L. H. (2011). Stochastic Simulation Optimization: An Optimal Computing Budget Allocation. Hackensack, NJ: World Scientific Publishing Co. Pte. Ltd.

of 15

Amalgamation

11

𝑛𝑖𝑗 𝑡

Prior knowledge of

attribute values, 𝜇𝑖𝑗’s Knowledge of

Decision-maker

preferences, 𝜆𝑗Updated knowledge of

attribute values, 𝜇𝑖𝑗’s with collected data, 𝑿 𝒕

Prior knowledge of ability

to measure attribute

values, U Ƹ𝜇𝑖𝑗 = 𝜎𝑗’s

Update knowledge of ability

to measure attribute values,

𝜎𝑗’s with collected data, 𝑿 𝒕

of 15

Sequential Allocation Procedure

12

Data collected thus far for alternative ai and attribute j: 𝐱𝑖𝑗 𝑡 = 𝑥𝑖𝑗1, … , 𝑥𝑖𝑗𝑛𝑖𝑗 𝑡

Knowledge of value of attribute j for alternative ai: 𝑝 𝜇𝑖𝑗|𝐱𝑖𝑗 𝑡

Knowledge of decision value for alternative ai: 𝑝 𝜉𝑖|𝐱𝑖 𝑡

Calculate PCSDM 𝑡 = 𝑃 𝜉𝑠 > 𝜉𝑟 , ∀𝑟 = 1,… ,𝑚|𝐗 𝑡

of 15

Allocation Decision

13

Next sample

Posterior predictive distribution

Allocate sample to 𝑎𝑖, 𝑗 with

largest Expected PCS𝐷𝑀𝑖𝑗

𝑡 + 1

11

ijijDMijn t

x PCS t

1|

ijijijn t

p x t

x

1 11 1 11, , ,

1, ,

1 max , , | , |ij ij ij ij

q r

DM m m iijn t ijn t ijn tq m r q

r m

E PCS t p t x d d p x t dx

X x

of 15

Allocation Procedure Performance

14

Evaluation Experiment:

50,000 decision cases

Concave efficient frontier

𝑚 = 5 alternatives

𝑘 = 2 attributes

100 ≤ 𝜇𝑖𝑗 ≤ 200

19 decision models, 𝜆1, 𝜆2 pairs

𝑣𝑗 𝜇𝑖𝑗 = 𝜇𝑖𝑗 Gaussian measurement error (known)

Bayesian prior on attribute values

𝑝 𝜇𝑖𝑗 = 𝑁 150, 352

Experimental budget 𝐵 = 50

Observed frequency of correct selection

of 15

Summary

15

Allocating a fixed experimental budget across multiple attributes and

alternatives in a selection decision where the results of the experimental

evaluations lead to uncertain estimates of the true attribute values

Allocation approaches:

1. Uniform

2. Proportional

3. Sequential

Allocation does impact the probability of selecting the true best alternative

Importance for projects focused on a selection decision to be managed so

that the decision modeling and the experimental planning are done jointly

rather than in isolation

BACKUP

16

of 15

How should we allocate a fixed budget across multiple attributes

and alternatives to maximize the probability of correct selection?

Assumptions and Problem Statement

17

1. Finite and distinct set of alternatives

attributes; true value

Separate and independent attribute measurement processes

with known

2. Decision model is provided

Linear

3. Fixed experimental budget

B sample measurements; cost equivalent

1, , ma a

1 1, ,

k

i i ik j j ijjf v m m m

2k ijm

ijl ij ijlX m

2~ 0,ijl jN 2

j

j ij ijv m m1

1k

jj

of 15

Bayesian Estimation

18

Estimate by estimating each

Decision-maker’s prior knowledge

Data

Posterior distribution on

Posterior distribution on

ijm

2

0 0,ij ijN m i

1, ,ijij ijnx x

ijm

i

2 2 2 2

0 0 0

1 2 2 2 2

0 0

| , , ~ ,ij

j ij ij ij ij j ij

ij ij ijn

j ij ij j ij ij

n xx x N

n n

m m

2 2 2 2

0 0 02

2 2 2 210 0

| ~ ,k k

j ij ij ij ij j ij

i i j j

j i jj ij ij j ij ij

n xN

n n

m

x

of 15

Results

19