seed center for data farming overview tom lucas and susan sanchez operations research department...

SEED Center for Data Farming

Overview

Tom Lucas and Susan Sanchez

Operations Research Department

Naval Postgraduate School

Monterey, CA.

http://harvest.nps.edu

Mission : Advance the collaborative development and use of simulation experiments and efficient designs to provide decision makers with timely insights on complex systems and operations

Mission : Advance the collaborative development and use of simulation experiments and efficient designs to provide decision makers with timely insights on complex systems and operations

SEED Center for Data Farming Naval Postgraduate School17 April 2007 2

Simulation studies underpin many DoD decisions

DoD uses complex, high-dimensional, simulation models as an important tool in its decision-making process.

– Used when too difficult or costly to experiment on “real systems”– Needed for future systems—we shouldn’t wait until they’re operational

to decide on appropriate capabilities and operational tactics, or evaluate their potential performance

– Investigate the impact of randomness and other uncertainties

Many complex simulations involve hundreds or thousands of “factors” that can be set to different levels.

Many complex simulations involve hundreds or thousands of “factors” that can be set to different levels.


The new view

Appropriate goals are:

(i) Developing a basic understanding of a particular model or system;– seeking insights into high-

dimensional space.– identifying significant factors and

interactions.– finding regions, ranges, and

thresholds where interesting things happen.

(ii) Finding robust decisions, tactics, or strategies;

(iii) Comparing the merits of various decisions or policies

Kleijnen, Sanchez, Lucas & Cioppa 2005

Once you have invested the effort to build (and perhaps verify, validate & accredit) a simulation model, it’s time to let the model work for you!

“Models are for thinking”—Sir Maurice Kendall


An environment for exploration requires…

• Flexible models or tools to build them

• High-performance computing

• Experimental design

• Data analysis and visualization


These goals mean fewer assumptions...

Traditional DOE Assumptions

– Small/ moderate # of factors

– Univariate response

– Homogeneous error

– Linear

– Sparse effects

– Higher order interactions negligible

– Normal errors

– Black box model

Assumptions for Defense & Homeland security Simulations

– Large # of factors

– Many output measures of interest

– Heterogeneous error

– Non-linear

– Many significant effects

– Significant higher order interactions

– Varied error structure

– Substantial expertise exists

“We use simulations to avoid making Type III errors—working on the wrong model”—W. David Kelton

“The idea behind [Monte Carlo simulation]…is to [replace] theory by experiment whenever the former falters—Hammersley and Handscomb

“The idea behind [Monte Carlo simulation]…is to [replace] theory by experiment whenever the former falters—Hammersley and Handscomb


...that, in turn, call for different designs

Few

Many

maximal screening

main effects iid errors

minimal assumptions non-smooth

complex errors

sequential bifurcation

(SB)

folded SBLatin hypercube

(LH)

(nearly) saturated (2k-p FF)

Plackett -Burman

coarse grids (2k factorial)

1st order 2nd order smooth

R5

R4

combined designs

central composite

(CCD)

frequency designs

differential grids

fine grids (mk factorial)

Number of Factors

Response Surface Complexity

Factorial (gridded) designs are most

familiar

We have focused on Latin hypercubes

and sequential approaches

Efficient R5 FF and CCD


Why use experimental designs?

More informative…• consider dozens or hundreds of factors, rather than a handful• a broader range of possibilities

Faster…• e.g., 33 “scenarios” or “design points” vs. 10 billion

More powerful…• get more info from a limited data set

Bottom line --experimental design should be part of EVERY simulation study!


Interpreting the results

• Standard statistical graphics tools (regression trees, 3-D scatter plots, contour plots, plots of average results for a single factors, interaction profiles) can be used to gain insights from the data

• Step-wise regression and regression trees identify important factors, interactions, and thresholds

All Data

Range > 75% of Battlespace

Latency < 1 minute

Sufficient Capacity Insufficient Capacity

Latency > 1 minute

Range < 75% of Battlespace

-0.1

0.2

0.5

0.8

-0.1

0.2

0.5

0.8

-0.1

0.2

0.5

0.8

-0.1

0.2

0.5

0.8

Speed:USV(s) (knots)

120

1

10

36

600

5 15 25 35 45

2

40

Range fromHVU (nm)

110

36

600

5 10 15 20

2

40

120

CameraRange (nm)

36

600

1 2 34 5 6 78 9 11

2

40

1

20

1

10

SimulationLength (minutes)

100 300 500 700

2

3

4

5

6

7

8

9

10

11

12

-33 -31 -29 -27 -25 -23 -21 -19 -16 -14 -12 -10 -8 -6 -4 -2 0 2 4 6 8 10 12 14 17 19 21 23 25 27 29 31 33

Log twd concealment

Mea

n(A

lleg

1Cas

(blu

e))

Each PairStudent’s t0.05

15

20

25

30

35

40

45

50

55

15 20 25 30 35 40 45 50 55

Blue Casualties Predicted

Effect Tests Alternate Tactical 3

SEED Center for Data Farming Mission: Advance the collaborative development and use of simulation experiments and efficient designs to provide decision makers with timely insights on complex systems and operations.

Primary Sponsors:

International Collaborators:

Applications Include: Peacekeeping operations, convoy protection, networked future forces, unmanned vehicles, anti-terror emergency response,urban operations, humanitarian relief, and more

Products Include: New downloadable experimental designs, plus over 40 student thesesand a dozen articles

http://harvest.nps.eduhttp://harvest.nps.edu

Questions?


The “traditional” view

Philosophy: “The three primary objectives of computer experiments are:

(i) Predicting the response at untried inputs,

(ii) Optimizing a function of the input factors, or

(iii) Calibrating the computer code to physical data.”

--Sacks, Welch, Mitchell, and Wynn (1989)

Approach:

• Limit yourself to just a few factors or scenario alternatives

• “Fix” all other factors in the simulation to specified values

• At each design point, run the experiment a small number of times (once for deterministic simulations)

The purpose of computing is insight, not numbers—Hamming

The purpose of computing is insight, not numbers—Hamming

For many (military) applications, these can be problematic!

For many (military) applications, these can be problematic!


Efficiency

• How many runs will you need? A few comparisons…

# factors 10^k factorial 5^k factorial 2^k factorial1 10 5 22 10^2 = 100 5^2 = 25 2^2 = 43 10^3 = 1,000 5^3 = 125 2^3 = 85 10^5 = 100K 5^5 = 3,125 2^5 = 32

10 10^10 = 10 billion 5^10 = 9,765,625 2^10 = 1,02420

Don’t even think about it!

5^20 = 95 trillion 2^20 = 1,048,576

40 5^40 = 9100 trillion trillion 2^40 = 1.1 trillion

In contrast, structured nearly orthogonal Latin hypercube (NOLH designs) require• 17 runs for 2-7 factors (up to 17 levels/factor)

• 33 runs for 8-11 factors (up to 33 levels/factor)



• 257 runs for 23-29 factors (up to 257 levels/factor)Random LH designs can be generated for arbitrary combinations of # factors (k) and # runs (n) as long as n >= k.


So, what is a Latin hypercube?

• In its basic form, each column in an n-run, k-factor LH is a permutation of the integers 1,2,…,n

• The n integers correspond to levels across the range of the factor

• For exploratory purposes, we use a uniform spread over the range (but may round to integer values)

– slightly different designs arise if you force sampling at the low and high values

Factor 1 Factor 2

1 4

5 1

6 2

4 5

3 3

2 6

A 6-run, 2 factor design

Pairwise projection

Factor 1

Factor 2

Low = 0

High = 60

Low = 0

High = 60

5 15 25 35 45 55

0 12 24 36 48 60


Nearly orthogonal and space-filling Latin hypercubes

A

-1.0

0.0

0.5

1.0

-1.0

0.0

0.5

1.0

-1.0

0.0

0.5

1.0

-1.0

0.0

1.0

-1.0

0.0

1.0

B

C

-1.0

0.0

1.0

-1.0

0.0

1.0

D

E

-1.0

0.0

1.0

-1.0

0.0

1.0

F

-1.0

0.0

0.5

1.0

-1.0

0.0

0.5

1.0

-1.0

0.0

0.5

1.0

-1.0

0.0

0.5

1.0

-1.0

0.0

1.0

G

The pairwise projections for a 17-run, 7-factors orthogonal LH show

– Orthogonality (no pairwise correlations)

– space-filling behavior (points fill the sub-plots)

• 17 total runs!

low level 1 1 1 1 1 1 1high level 17 17 17 17 17 17 17decimals 0 0 0 0 0 0 0

factor name6 17 14 7 5 16 102 5 15 10 1 6 113 8 2 5 11 14 174 11 6 17 10 3 13

13 16 8 3 6 1 1417 6 7 14 2 13 1511 4 17 6 15 8 1610 15 13 16 14 11 129 9 9 9 9 9 9

12 1 4 11 13 2 816 13 3 8 17 12 715 10 16 13 7 4 114 7 12 1 8 15 55 2 10 15 12 17 41 12 11 4 16 5 37 14 1 12 3 10 28 3 5 2 4 7 6


• Very large resolution V fractional factorials and central composite designs

– Standard DOE literature: 211-3

– New: an easy way to catalogue and generate up to 2443-423

• Two-phase adaptive sequential procedure for factor screening– New procedure that requires fewer assumptions, improves

efficiency

• Frequency domain experiments– Naturally samples factors at coarser/finer levels

• Crossed/combined designs to identify robust decision factor settings

Other possibilities

-1.01.0

-1.0

1.0

1.0

x1x2

x3

-1.0

-1.0

1.0

-1.0

1.0

1.0

x1x2

x3

-1.0

-1.01.0

-1.0

1.0

1.0

x1x2

x3

-1.0

-1.01.0

-1.0

1.0

1.0

x1x2

x3

-1.0

1 2 3 4 5 6

-1

-0.5

0.5

1


Our portfolio of designs

• Kleijnen, J. P. C., S. M. Sanchez, T. W. Lucas, and T. M. Cioppa, “A User’s Guide to the Brave New World of Designing Simulation Experiments,” INFORMS Journal on Computing, Vol. 17, No. 3, 2005, pp. 263-289.

• Cioppa, T. M. and T. W. Lucas, “Efficient Nearly Orthogonal and Space-filling Latin Hypercubes,” Technometrics, Vol. 49, No. 1, 45-55.

• Sanchez, S. M. and P. J. Sanchez, "Very Large Fractional Factorials and central composite designs," ACM Transactions on Modeling and Computer Simulation, Vol. 15, No. 4, 2005, pp. 362-377.

• Sanchez, S. M., H. Wan, and T. W. Lucas, "A Two-phase Screening Procedure for Simulation Experiments," Invited paper (under review), ACM Transactions on Modeling and Computer Simulation.

• Sanchez, S. M., F. Moeeni, and P. J. Sanchez, "So Many Factors, So Little Time…Simulation experiments in the frequency domain," International Journal of Production Economics, Vol. 103, 2006, pp. 149-165.


Other publications

Sanchez, Lucas, “Agent-based Simulations: Simple Models, Complex Analyses,” Invited paper, Proc. 2002 Winter Simulation Conference, 116-126.

Lucas, Sanchez, Brown, Vinyard, “Better Designs for High-Dimensional Explorations of Distillations,” Maneuver Warfare Science 2002, Marine Corps Combat Development Command, 2002, 17-46.

Vinyard, Lucas, “Exploring Combat Models for Non-monotonicities and Remedies,” PHALANX, 35, No. 1, March 2002, 19, 36-38.

Lucas, McGunnigle, “When is Model Complexity Too Much? Illustrating the Benefits of Simple Models with Hughes’ Salvo Equations,” Naval Research Logistics, Vol. 50, April 2003, 197-217.

Lucas, Sanchez, Cioppa, Ipekci, “Generating Hypotheses on Fighting the Global War on Terrorism,” Maneuver Warfare Science 2003, Marine Corps Combat Development Command, 2003, 117-137.

Lucas, Sanchez, “Smart Experimental Designs Provide Military Decision-Makers With New Insights From Agent-Based Simulations,” Naval Postgraduate School RESEARCH, 13, 2, 20-21, 57-59, 63.

Lucas, Sanchez, ““NPS Hosts the Marine Corps Warfighting Laboratory’s Sixth Project Albert International Workshop,” Lucas, T.W. and S.M. Sanchez, Naval Postgraduate School RESEARCH, 13, 2, 45-46.

Sanchez, Wu, “Frequency-Based Designs for Terminating Simulation Experiments: A Peace-enforcement Example,” Proc. 2003 Winter Simulation Conference, 952-959.

Brown, Cioppa, “Objective Force Urban Operations Agent Based Simulation Experiment,” Technical Report TRAC-M-TR-03-021, Monterey, CA, June 2003.

Cioppa, Brown, Jackson, Muller, Allison, “Military Operations in Urban Terrain Excursions and Analysis With Agent-Based Models,” Maneuver Warfare Science 2003, Quantico, VA, 2003.

Cioppa, “Advanced Experimental Designs for Military Simulations,” Technical Report TRAC-M-TR-03-011, Monterey, CA, February 2003.

Brown, Cioppa, Lucas, “Agent-based Simulation Supporting Military Analysis,” PHALANX, Vol. 37, No. 3, Sept 2004.Cioppa, Lucas, Sanchez, “Military Applications of Agent-based Simulation,” Proc. 2004 Winter Simulation Conference.

Cioppa, Lucas, Sanchez, “Military Applications of Agent-Based Simulations,” Proceedings of the 2004 Winter Simulation Conference, 171-179

Allen, Buss, Sanchez, “Assessing Obstacle Location Accuracy in the REMUS Unmanned Underwater Vehicle,” Proceedings of the 2004 Winter Simulation Conference, 940-948.

Cioppa, “An Efficient Screening Methodology For a Priori Assessed Non-Influential Factors,” Proc. 2004 Winter Simulation Conference, 171-180.

Sanchez, “Work Smarter, Not Harder: Guidelines for Designing Simulation Experiments.” Proc. of the 2005 Winter Simulation Conference, forthcoming.

Wolf, Sanchez, Goerger, Brown, “Using Agents to Model Logistics,” under revision for Military Operations Research.

Baird, Paulo, Sanchez, Crowder, “ Measuring Information Gain in the Objective Force, under revision for Military Operations Research.


Student theses…note the breadth of applications2000 Brown (Captain, USMC)

Human Dimension of Combat2001 Vinyard (Major, USMC)

Reducing Non-monotonicities in Combat Models, MORS/Tisdale Winner, MORS Walker Award

2002 Erlenbruch (Captain, German Army)German Peacekeeping Operations, MORS/Tisdale Finalist

2002 Pee (Singapore DSTA)Information Superiority and Battle Outcomes, MORS/Tisdale Finalist

2002 Wan (Major, Singapore Army)Effects of Human Factors on Combat Outcomes

2002 Dickie (Major, Australian Army)Swarming Unmanned Vehicles, MORS/Tisdale Finalist

2002 Ipekci (1st Lieutenant, Turkish Army)Guerrilla Warfare, MORS/Tisdale Winner

2002 Wu (Lieutenant, USN)Spectral Analysis and Sonification of Simulation Data

2002 Cioppa (Lieutenant Colonel, US Army, PhD)Experimental Designs for High-dimensional Complex Models,ASA 3rd Annual Prize for Best Student Paper Applying Stat. to Defense

2003 Efimbe (Lieutenant, US Navy)Littoral Combat Ships Protecting Expeditionary Strike Groups

2003 Wolf (Captain, USMC)Urban, Humanitarian Assistance/ Disaster Relief Operations,MORSS Best Presentation Award, MORS Barchi Prize Finalist

2004 Milton (Lieutenant Commander, US Navy)Logistical Chain of the Seabase, MORS/Tisdale Finalist

2004 Allen (Lieutenant, US Navy)Navigational Accuracy of REMUS Unmanned Underwater Vehicle, MORS/Tisdale Finalist

2004 Steele (Ensign, US Navy)Unmanned Surface Vehicles

2004 Hakola (Captain, USMC)Convoy Protection

2004 Lindquist (Captain, US Army)Degraded Communication in the Future Force, MORS Tisdale Winner

2004 Aydin (1st Lieutenant, Turkish Army)Village Search Operations

2004 Raffetto (Captain, USMC)UAVs in Support of IPB in a Sea-Viking Scenario, MORS/Tisdale Finalist

2004 Cason (Captain, USMC)UAVs in Support of Urban Operations

2004 Berner (LCDR, US Navy)Multiple UAVs in Maritime Search and Control

2004 Tan (Singapore S&T)Checkpoint Security

2005 Babilot (USMC)DO versus Traditional Force in Urban Terrain

2005 Bain (USMC)Logistics Support for Distributed Ops, MORS/Tisdale Finalist

2005 Gun (Turkish Army)Sunni Participation in Iraqi Elections

2005 McMindes (USMC)UAV Survivability

2005 Sanders (USMC)Marine Expeditionary Rifle Squad

2005 Ang (Singapore Technologies Engineering)Increasing Participation and Decreasing Escalation in Elections

2005 Chang (Singapore DSTA)Edge vs. Hierarchical Organizations for Collaborative Tasks

2005 Liang (Singapore DSTA)Cooperative Sensing of the Battlefield

2005 Martinez-Tiburcio (Mexican Navy)Protecting Mexico’s Oil Well Infrastructure

2005 Sulewski (USA)UAVs in Army’s FCS Family of Systems

2006 Lehmann (Major, German Army)A Discrete, Even-driven Simulation of Peacekeeping Operations

2006 Roginski (Major, US Army)Emergency Response to a Terrorist Attack

2006 Alt (Major, US Army)TTPs for a Future Force Warrior Small Combat Unit

2006 Wittwer (Major, US Army)Non-Lethal Weapons in a Future Force Warrior Small Combat Unit

2006 Nannini (Major, US Army)Dynamic Scheduling of FCS UAVs

2006 Vaughan (Captain, USMC) Force Size Transitions in Stability Operations

2006 Michel (Major, USMC)Evaluating the Marine Corps’ Artillery Triad in STOM Operations

2006 Richardson (Captain, USMC) Distributed Capabilities in a Future Force Warrior Small Combat Unit

2006 Sickinger (Lieutenant, US Navy) Non-Lethal Weapons in a Maritime Environment, MORS/Tisdale Finalist

Coming soon…Many more


Count

Mean

Std Dev

10560

0.2907912

0.1518486

All Rows

Count

Mean

Std Dev

4200

0.193464

0.105083

R Time<6.67

Count

Mean

Std Dev

2080

0.1296

0.0624157

# UAVs<2

Count

Mean

Std Dev

2120

0.2561231

0.1006375

# UAVs>=2

Count

Mean

Std Dev

6360

0.3550638

0.1435538

R Time>=6.67

Count

Mean

Std Dev

3160

0.2666184

0.116969

Routing<2

Count

Mean

Std Dev

1600

0.1920612

0.0911159

# UAVs<2

Count

Mean

Std Dev

960

0.1328437

0.0515426

SW Meters<7087

Count

Mean

Std Dev

640

0.2808875

0.0601411

SW Meters>=7087

Count

Mean

Std Dev

1560

0.3430872

0.0874758

# UAVs>=2

Count

Mean

Std Dev

560

0.2488393

0.0586501

SW Meters<5611

Count

Mean

Std Dev

1000

0.395866

0.0474234

SW Meters>=5611

Count

Mean

Std Dev

3200

0.4424038

0.10998

Routing>=2

Count

Mean

Std Dev

1600

0.3586131

0.0799676

# UAVs<2

Count

Mean

Std Dev

1600

0.5261944

0.0612651

# UAVs>=2

Most Important Factor

Needs to fly over 7 hours

Think like the enemy!

Rt.2 planned with intel

In either case, throw more forces/capabilities at it next… if available

Decision Tree: Time and routing (Raffetto, 2004) MOE: Proportion of enemy classified

JTF Sea Viking

JTF Swashbuckler

Caliphate


Example: Regression analysis (Raffetto, 2004)

• Across the noise factors, the regression models produce R-Square values from .906 to .921 with seven to nine terms for 1-3 UAVs

• Provides a means to compare expected effects of different configurations

• Parameter estimates are put into a simple Excel spreadsheet GUI to allow decision makers to view relative effects of configurations within this scenario

0.01

0.02

0.03

0.04

0.05

0.06

Mean(EClassProp/Hr) Actual

.01 .02 .03 .04 .05 .06

Mean(EClassProp/Hr) Predicted P<.0001

RSq=0.91 RMSE=0.0044

Preferred model for One UAV—7 Terms


Example: Interactions (Steele, 2004) camera range and speed

-0.1

0.2

0.5

0.8

-0.1

0.2

0.5

0.8

-0.1

0.2

0.5

0.8

-0.1

0.2

0.5

0.8

Speed:USV(s) (knots)

120

1

10

36

600

5 15 25 35 45

2

40

Range fromHVU (nm)

110

36

600

5 10 15 20

2

40

120

CameraRange (nm)

36

600

1 2 34 5 6 78 9 11

2

40

1

20

1

10

SimulationLength (minutes)

100 300 500 700

• At low speeds, camera range is unimportant

• At higher speeds, camera range has big impact

• One of several technological challenges for systems design


Example: One-way analysis (Hakola, 2004)

2

3

4

5

6

7

8

9

10

11

12

-33 -31 -29 -27 -25 -23 -21 -19 -16 -14 -12 -10 -8 -6 -4 -2 0 2 4 6 8 10 12 14 17 19 21 23 25 27 29 31 33

Log twd concealment

Me

an

(All

eg

1Ca

s(b

lue

))

Each PairStudent’s t0.05


Example: MART (Ipekci, 2002)

Relative Variable Importance

Relative Variable Importance Red Casualties

Blue Casualties


Example: Contour plot (Allen, 2004)


Resources: Seed Center for Data Farming

http://harvest.nps.edu

Check here for:

• lists of student theses (available online)

• spreadsheets & software

• pdf files for several of our publications, publication info for the rest

• links to other resources

• updates

All models are wrong, but some are useful—George Box