accelerating the analysis of test data using effective and ... · accelerating the analysis of test...

Mark Kiemele

Air Academy Associates

1650 Telstar Drive, Ste 110

Colorado Springs, CO 80920

Phone: 719-531-0777

email: [email protected]

All rights reserved. Do not reproduce. www.airacad.com

Accelerating the Analysis of Test

Data Using Effective and Efficient

Experimentation

ITEA TIW

Las Vegas, NV

May 14, 2019

Copyright © 2018

19-ITEATIWDOE-5A

mailto:[email protected]

© 2018

1|

Introductions

1

• Name

• Position and Company/Organization

• Experience in experimentation and T&E

• Expectations

© 2018

2|

Agenda and Objectives

• Prerequisites for successful experimentation

• Remove excessive variation from the system.

• Ensure the measurement system is capable.

• The Four Pillars of Design of Experiments

• Demonstrate each of these pillars using the Statapult

• Screening

• Modeling

• Performance Verification and Validation

2

© 2018

3|

Statapult ® Catapult

Catapulting Power into Knowledge Gain and T&E

3

© 2018

4|

Simulating a Rapid Improvement Event (RIE)

• Statapult = Delivery Process

• Ball = Service Provided

• Setup = Job prep

• Measurement = Outcome of the service

Using the statapult as a metaphor

4

© 2018

5|

Statapult® Exercise #1(Baselining a Process)

1. Insure all pins are at position #3

2. Pull the arm to 177° and launch the rubber ball

3. Have someone measure your distance

4. Disconnect rubber band between shots

5. Your standard is no more than 15 seconds between shots

6. Record distances; Calculate Range

#1 #2 #3 #4 #5

Range = Longest - Shortest =

Each team member will shoot the Statapult® 3 times using the following steps:

Shot #1

Shot #2

Shot #3

Team Member:

5

#6 #7 #8

© 2018

6|

C = Constants Standard Operating Procedures (SOPs)

N = Noise

X = Experimental

Output(s) and

Specs

(1) PROCESS FLOW (PF) OR PROCESS MAP

(2) CUSTOMER DRIVEN CAUSE AND EFFECT (CE)

How What Who___________

___________

___________

... removes waste, reduces variation, and decreases cycle time

PF/CE/CNX/SOPA Rapid Improvement Event and a Management Tool that

6

© 2018

7|

X = Experimental

• These are key variables that can be controlled and held constant at different

levels or settings for the purpose of determining the effect of this variable on

the CTC.

C = Controlled

• To hold a variable as constant as possible requires controlling the variable via

Mistake Proofing and SOPs to eliminate errors and reduce variation.

• Controlling a variable or holding it as constant as possible doesn't just

happen. It must be “engineered” into the process.

• Mistake Proofing: The process of eliminating conditions that lead to variation

in the CTCs and ultimately cause errors.

N = Noise

• Noise variables are those that are not being controlled or held as constant as

possible

• Mistake Proofing is needed to change an "N" variable to a "C" variable.

(3) Partitioning the Variables into CNX

7

© 2018

8|

(4) Standard Operating Procedures (SOPs)

• Define the interaction of people and their environment when processing

a product or service

• Detail the action and work sequence of the operator

• Provide a routine to achieve consistency of an operation

• Specify the best process we currently know and understand for

controlling variation and eliminating waste

• Provide a basis for future improvements

• Validate mistake proofing in the process

• Strongly impact compliance to QMS such as ISO 9000, CMM, Sarbanes-

Oxley, etc.

8

© 2018

9|

1. Process flow "Shooting the Statapult®"

2. Complete Cause-and-Effect Diagram

3. Label inputs as C or N

4. Use SOPs with mistake proofing to change N’s into C’s

5. Re-shoot Statapult® using the first example instructions (all pins at #3; pull angle = 177°; 15 sec. between shots; etc.)

6. Record data taken after PF/CE/CNX/SOPs and evaluate

7. If improved, develop control plans

Statapult® Exercise #2: Demonstrating Improvement

9

#1 #2 #3 #4 #5

Range = Longest - Shortest =

Shot #1

Shot #2

Shot #3

#6 #7 #8

Team Member:

© 2018

10|

Process Flow

10

Start/Set Up

Place ball in cup

Pull back the arm

177o

?

Spot the Ball/Record Retrieve the Ball Reset the Rubber Band

Final Shot of

Operator ?

Change Operators

No

No

Yes

Yes Final Operator?

Yes

No

Stop

Release the arm

© 2018

11|

Cause-and-Effect Diagram Worksheet

11

Distance

Machine Materials Methods

Manpower Mother Nature Measurement

© 2018

12|

Measurement System Analysis

12


© 2018

13|

Data Collection Template #1

13

© 2018

14|

Measurement is a Process

14

Measure-

ment

Process

My Process

Y (true)

Y (recorded)

Where DOES the variation that we see in Y come from? Is it from the process itself or the

measurement system? Which one of those two variations is stronger?

Is the measurement system –

Accurate?

Precise?

Stable?

© 2018

15|

Partitioning the Measurement System Variability

15

• MSA identifies and quantifies the different sources of variation that

affect a measurement system.

• Variation in measurements can be attributed to variation in the

product/service itself or to variation in the measurement system.

• The variation in the measurement system itself is measurement error.

Product / Service

Variability

Measurement

Variability

Total (recorded) Variability is broken into two major pieces:

This piece of the pie

will be further divided

into two smaller

pieces called

Repeatability and

Reproducibility.

© 2018

16|

Measurement Variability Broken Down Further

16

Purpose:

To assess how much variation is associated with the measurement system and

to compare it to the total process variation or tolerances.

Repeatability:

Variation obtained by the same person using the same procedure on the same

product, transaction or service for repeated measurements (variability within

operator).

Reproducibility:

Variation obtained due to differences in people who are taking the

measurements (variability between operators).

total product measurement= +2 2 2σ σ σ

repeatability reproducibility+2 2σ σ

© 2018

17|

More on Repeatability and Reproducibility

17

For the scenario below, look at the data and indicate which of the following

you think is true:

a. Repeatability appears to be more of a problem than reproducibility

b. Reproducibility appears to be more of a problem than repeatability

c. Repeatability and Reproducibility appear to be about the same

Operator 1 Operator 2

Rep 1 Rep 2 Rep 1 Rep 2

21 23 26 28

19 18 24 24

20 23 27 24

19 22 21 20

▪ MSA will help quantify, more exactly, the capability of the measurement

system and answer questions about repeatability, reproducibility, and

capability with respect to the customer specs.

▪ Rule of Thumb for MSA Sample Size:

- Variables (Continuous) Data: (# operators)*(number of parts) ≥ 20

- Attribute (Binary) Data: (# operators)*(number of parts) ≥ 60

Part 1

Part 2

Part 3

Part 4

© 2018

18|

Measurement System Diagnostics

18

1. Precision-to-Tolerance Ratio (P/TOL)

P/TOL = (Specification Limits are needed)

ROT: If P/TOL .10 : Very Good Measurement System

P/TOL .30 : Unacceptable Measurement System

2. Precision-to-Total Ratio (P/TOT)

P/TOT =

ROT: If P/TOT .10 : Very Good Measurement System

P/TOT .30 : Unacceptable Measurement System

3. Discrimination or Resolution

(# of truly distinct measurements that can be obtained by the

measurement system) ROT: Resolution ≥ 5

total

meas

LSLUSL

6 meas

−

product

meas

.

=

1 41

© 2018

19|

Analysis of Variance (ANOVA) Results

19

• P/TOL and P/TOT are too high.

• Resolution is unacceptable.

• Reproducibility is significantly larger than Repeatability and

appears to be the biggest problem with this measurement

process.

© 2018

20|

Graphical View of Variance Components

20

ReproducibilityRepeatability

Part-to-Part

0

1

2

3

4

5

6

7

8

Z Axis

X Axis Category

Interaction

Operator

© 2018

21|

Graphical View of Operator by Part Interaction

21

Operator By Part

0

5

10

15

20

25

30

1 2 3 4

Part #

Meas

ure

men

t

Operator 1

Operator 2

© 2018

22|

Data Collection Template #2

22

© 2018

The Foundations of Design of Experiments (DOE)

Ort

ho

gon

alit

y

Re

plic

atio

n

Ran

do

miz

atio

n

Blo

ckin

g

Creating a Culture of Effective and Efficient

Experimentation

The Four Pillars of DOE

23

© 2018

Famous Quote

“All experiments (tests) are designed;

some are poorly designed,

some are well designed.”

George Box (1919-2013), Professor of Statistics, DOE Guru

24

© 2018

Design of Experiments (DOEs):

A Subset of All Possible Test Design Methodologies

27

The Set of All Possible Test Design

Methodologies (Combinatorial Tests)

Orthogonal or

Nearly

Orthogonal

Test Designs

(DOEs)

One

Factor

At a

Time

(OFAT)

Best Guess

(Oracle)

Boundary Value Analysis

(BVA)

Equivalence Partitioning (EP)

© 2018

OFAT (One Factor at a Time) Testing

4. One factor at a time results

versus optimal results

3. One factor at a time

results

Chemical

Process

Yield (gr.)X1 = time

Y

80

70

60 90 120 150 180X1

1. Hold X2 constant and vary X1

Find the “best setting” for X1Y

80

70

210 220 230 240 250X2

2. Hold X1 constant at “best setting” and vary X2.

Find the “best setting” for X2.

20060 90 120 150 180

X1

210

220

230

240

250

X2

9080

6070

X2

X1

60 90 120 150 180

• • • • •

•

•

•

•

X2 = temp

200

210

220

230

240

250

25

© 2018

The Good and Bad about OFAT

• Good News

• Simple

• Intuitive

• The way we were originally taught

• Bad News

• Will not be able estimate variable interaction effects

• Will not be able to generate prediction models and thus

not be able to optimize performance or assess risk

26

© 2018

What is a DOE ?

Purposeful and systematic changes of the inputs (factors) in order

to observe corresponding changes in the output (response).

Run

1

2

3

.

.

X1 X2 X3 X4 Y1 Y2 . . . . . . Y SY

Inputs

A = X1

B = X2

D = X4

C = X3

Y1

Outputs

.

.

.

.

.

.

PROCESS Y2

28

© 2018

Statistically Designed Experiments (DOE):

Orthogonal or Nearly Orthogonal Designs

• FULL FACTORIALS (for small numbers of factors)

• FRACTIONAL FACTORIALS

• PLACKETT - BURMAN

• LATIN SQUARES Taguchi Designs

• HADAMARD MATRICES

• BOX - BEHNKEN DESIGNS

• CENTRAL COMPOSITE DESIGNS

• HIGH THROUGHPUT TESTING (ALL PAIRS)

• NEARLY ORTHOGONAL LATIN HYPERCUBE DESIGNS

SIMPLE DEFINITION OF A TWO-LEVEL

ORTHOGONAL DESIGN

Run Actual Settings Coded Matrix Interactions

1

2

3

4

5

6

7

8

(5, 10) (70, 90) (100, 200)

A: Time B: Temp C: Press

(A) (B) (C)

Time Temp Press

5

5

5

5

10

10

10

10

70

70

90

90

70

70

90

90

100

200

100

200

100

200

100

200

-1 -1 -1

-1 -1 +1

-1 +1 -1

-1 +1 +1

+1 -1 -1

+1 -1 +1

+1 +1 -1

+1 +1 +1

Yield

Example:

Chemical

Process

A: Time (5,10)

B: Temp (70,90)

C: Press (100,200)

(AB) (AB)

Uncoded Coded

350

350

450

450

700

700

900

900

+1

+1

-1

-1

-1

-1

+1

+1

Response Surface

Designs

29

© 2018

Correlations Using Actual vs Coded Units

31

Correlation MatrixA B C AB AC BC ABC

A 1.0 0.0 0.0 0.929981 0.688247 0.0 0.661581

B 1.0 0.0 0.348743 0.0 0.348743 0.248093

C 1.0 0.0 0.688247 0.929981 0.661581

AB 1.0 0.640057 0.121622 0.711392

AC 1.0 0.640057 0.961256

BC 1.0 0.711392

ABC 1.0

Summary

Mean 7.5 80.0 150.0 600.0 1125.0 12000.0 90000.0

StDev 2.6726 10.69 53.452 229.907 582.482 4598.14 48476.8

Count 8 8 8 8 8 8 8

A B C AB AC BC ABC

5 70 100 350 500 7000 35000

5 70 200 350 1000 14000 70000

5 90 100 450 500 9000 45000

5 90 200 450 1000 18000 90000

10 70 100 700 1000 7000 70000

10 70 200 700 2000 14000 140000

10 90 100 900 1000 9000 90000

10 90 200 900 2000 18000 180000

A B C AB AC BC ABC

-1 -1 -1 1 1 1 -1

-1 -1 1 1 -1 -1 1

-1 1 -1 -1 1 -1 1

-1 1 1 -1 -1 1 -1

1 -1 -1 -1 -1 1 1

1 -1 1 -1 1 -1 -1

1 1 -1 1 -1 -1 -1

1 1 1 1 1 1 1

Correlation MatrixA B C AB AC BC ABC

A 1.0 0.0 0.0 0.0 0.0 0.0 0.0

B 1.0 0.0 0.0 0.0 0.0 0.0

C 1.0 0.0 0.0 0.0 0.0

AB 1.0 0.0 0.0 0.0

AC 1.0 0.0 0.0

BC 1.0 0.0

ABC 1.0

Summary

Mean 0.0 0.0 0.0 0.0 0.0 0.0 0.0

StDev 1.069 1.069 1.069 1.069 1.069 1.069 1.069

Count 8 8 8 8 8 8 8

© 2018

The Beauty of Orthogonality

(Vertical and Horizontal Balance)

A Full Factorial Design for 3 Factors A, B, and C, each at 2 levels:

30

© 2018

Takata Airbag Defect Findings:

Independent Testing Coalition (ITC) makes key findings

ITC says exposure to heat and humidity, and the use of ammonium nitrateare all required to produce what the commission and the National Highway Traffic

Safety Administration (NHTSA) call an “energetic disassembly.”

“You can’t have the energetic disassembly without all three factors," David Kelly,

leader of the ITC and former chief of the NHTSA told Automotive News Europe.

"You have to have all three.”

In DOE, this is called a significant 3-way interaction effect.

31

© 2018

Design of Experiments (DOE)

▪ An optimal data collection methodology

▪ “Interrogates” the process or product

▪ Used to identify important relationships between inputs (factors)

and outputs (response variables)

▪ Identifies important interactions between input variables

▪ Can be used to characterize and optimize a process

▪ Can be used to asses risk

▪ Changes “I think” to “I know”

▪ Is the science of test and the key connector between test and

evaluation

32

© 2018

Three Major Types of DOEs

• Modeling Designs

- For building functions that can be used to predict outcomes,

assess risk, and optimize performance. These include

the ability to evaluate interaction and higher order effects.

• Performance Verification and Validation Testing

- For confirming that a system performs in accordance with

its specifications/requirements.

• Screening Designs

- For testing many factors in order to separate the vital few

critical factors from the trivial many.

33

© 2018

WHAT IS

THE

OBJECTIVE?

SCREENING VALIDATION or CONFIRMATION

Rules of Thumb for Design Selection

MODELING

Notes:

1. “Mixed” factors means a combination of quantitative and qualitative (categorical)

2. “Mixed” levels means that not all factors have the same number of levels (settings)

3. “K” = Number of Factors and “L” = Number of Levels

4. HTT = High Throughput Testing

5. DSD = Definitive Screening Design

6. “OA” stands for Orthogonal Array; “PAVO” = Pairwise Value Ordering

7. Software such as HD ToolsTM, rdExpertTM Lite, Pro-TestTM and Quantum XLTM

generate some or all of these designs

Mixed Factor/Mixed

Level Designs:

HTT (OA or PAVO)

3-Level Designs:

Full Factorial (K ≤ 3)

CCD, BB, D-optimal

2-Level Designs:

Full Factorial (K ≤ 4)

Fractional Factorial

(K = 5)

D-Optimal

Fixed Number

of Samples:

Descriptive

Sample (DS)*

Latin

Hypercube

Sample (LHS)*

Not a Fixed Number

of Samples:

HTT (all pairs, OA,

PAVO)

* DS and LHS are sampling techniques to generate representative samples

according to a specified distribution and a specified sample size

* Representative samples do not give orthogonal designs. They are often

used for getting test coverage, validating performance/ determining

capability, or creating noise combinations for test

DOE ProTM software is copyright Air Academy Associates, LLC and Digital Computations, Inc.HD ToolsTM is a trademark of Air Academy Associates, LLC and software is copyright SigmaXL.rdExpertTM Lite software is copyright Phadke Associates, Inc.Pro-Test TM software is copyright Digital Computations, Inc.Quantum XLTM software is copyright SigmaZone.com.

2-Level

Designs:

L12 (6 ≤ K ≤ 11)

Fractional

Factorial (res IV)

3-Level

Designs:

L18 (4 ≤ K ≤ 8)

DSD

High Factor/High Level

Designs: (K ≥ 9 and L ≥ 5)

Nearly Orthogonal Latin

Hypercube Designs

(NOLHDs) with K*L runs(NOLHDs require all factors have the

same number of levels)

Mixed Factor/Mixed

Level Designs:

Full Factorial

HTT (OA or PAVO; with

only select interactions)

D-optimal

34

© 2018

36|

Screening DOE on the Statapult

36

• Conduct an L12 Screening Design using the following factors:

Settings

Low (-) High (+)

A: Pull Back Angle 160° 180°

B: Stop Angle 2 3

C: Pin Height 2 3

D: Cup Height 4 5

E: Rubber Band Position 2 3

F: Ball Type 1 2

G: Operator 1 2

• Perform the data collection phase of the experiment and then conduct the analysis as

demonstrated on the subsequent pages of this notebook.

© 2018

37|

DOE PRO Software to Select the Design

37

DOE PRO > Create Design > Non-Computer Aided

© 2018

38|

Naming the Factors, Settings, Responses and Replicates

38

Default setting is for 95% confidence

and power in the standard deviation

assessment.

© 2018

39|

Example Data

39

Factor A B C D E F G

Row #Pull Back

Angle

Stop

Angle

Pin

Height

Cup

Height

Ball

TypeOperator Y1 Y2 Y3 Y4

Rubber Band

Position

1

12

11

10

9

8

7

6

5

4

3

2

160

180

180

180

180

180

180

160

160

160

160

160

2

3

3

3

2

2

2

3

3

3

2

2

2

2

2

3

2

3

3

3

3

2

3

2

4

4

5

4

5

4

5

5

4

5

5

4

2

3

2

2

3

3

2

2

3

3

3

2

1

1

2

1

2

2

1

2

2

1

1

2

1

2

1

1

1

2

2

2

1

2

1

2

31

99.5

106.75

108

101.5

118

112.5

93

89.5

77.5

68.5

33.5

34

99

103

106

103

117

112.5

92.5

90.5

77

63.5

34.75

30.5

100.5

107

110

107

114.5

114.5

93.5

92

78.5

62

35

32.5

100.5

109.5

112

103

117

111.5

92

86.5

76.5

70

33

© 2018

40|

Software Commands to Analyze the Data

40

Analyze the design matrix by selecting

DOE PRO > Analyze Design > Marginal Means Plot…

Analyze the design matrix by selecting

DOE PRO > Analyze Design > Multiple Response Regression

© 2018

41|

Regression and Marginal Means Plots for Standard Deviation (s)

41

© 2018

42|

42

Regression and Marginal Means Plots for the Mean (Y bar)

© 2018

43|

0

5

10

15

20

25

Ab

so

lute

Co

eff

icie

nt

pull back

angle (A)

pin height (C) stop angle

(B)

cup height

(D)

rubber band

position (E)

ball type (F) operator (G)

Effect Name

Y-hat Pareto of Coeffs -

pull back angle (A)

pin height (C)

stop angle (B)

cup height (D)

rubber band position (E)

ball type (F)

operator (G)

Pareto of Coefficients for y-hat and s-hat

43

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Ab

so

lute

Co

eff

icie

nt

operator (G) pin height (C) cup height

(D)

rubber band

position (E)

stop angle

(B)

pull back

angle (A)

ball type (F)

Effect Name

S Pareto of Coeffs -

operator (G)

pin height (C)

cup height (D)

rubber band position (E)

stop angle (B)

pull back angle (A)

ball type (F)

© 2018

44|

Results of the Screening Design

44

• What factors, if any, had a significant effect on the average

distance launched?

• What factors, if any, had a significant effect on the process

standard deviation?

• Any lessons learned?

© 2018

47|

A Theoretical Model

47

)θsinθ(sin)rmgrMg(θdφcosθins)θF(rθI2

10BGF

2

0−+−=

θsin)rmgrMg(φcosθsin)θ(FrθIBGF0

+−= ,cosrd

sinrDtan

F

F

+

−=

).θsinθ(sin)rmgrMg(θdφcosθsin)θF(rθI2

101BGF

2

10−+−=

1B1Bθcosr

2

1tθ

2

πcosvx −

−= .gt

2

1tθ

2

πsinvθsinry 2

1B1B−

−+=

.0

2cos

)cosrR(

V2

g

2tan)cosrR(sinr

2

12

1B2B

11B1B =

−

+−

−

++

−

++

−

+

11B1B12

21B

B

0

2tan)cosrR(sinr

2cos

)cosrR(

r4

gI

θ

θ0

1

0

θ

θ

1

0

θ

θ

).sin(sin)rmgrMg(dcossin)F(r 01BGF −+−=

..

.

.

© 2018

Statapult®

DOE

(The Empirical Approach)

Factor A B C Distance

Row # Pull Back Angle Pin Height Stop Pin Y1 Y2 Y3 Y4 Y bar S

1 160 2 2 33.5 33.75 34 34 33.8125 0.239357

2 160 2 3 54.25 54 55.5 54 54.4375 0.71807

3 160 3 2 44 44 41.5 41.5 42.75 1.443376

4 160 3 3 67 66 67.5 65.5 66.5 0.912871

5 180 2 2 55.5 56 55.5 55 55.5 0.408248

6 180 2 3 73 72 71 71.25 71.8125 0.898494

7 180 3 2 74 73.75 74.5 75 74.3125 0.554339

8 180 3 3 88.5 88.5 90 90 89.25 0.866025

Random Order = 2 ,8 ,4 ,7 ,3 ,5 ,6 ,1

47

© 2018

Regression Analysis on Statapult DOE Data

Y-hat Model

Distance

Factor Name Coeff P(2 Tail) Tol Act

ive

Const 61.047 0.0000

A Pull Back Angle 11.672 0.0000 1 X

B Pin Height 7.156 0.0000 1 X

C Stop Pin 9.453 0.0000 1 X

AB 1.906 0.0000 1 X

AC -1.641 0.0000 1 X

BC 0.21875 0.1494 1

ABC -0.56250 0.0008 1 X

R20.9982

Adj R2 0.9976

Std Error 0.8307

F 1877.9499

Sig F 0.0000

FLOF NA

Sig FLOF NA

Source SS df MS

Regression 9071.9 7 1296.0

Error 16.6 24 0.7

ErrorPure 16.6 24 0.7

ErrorLOF 0.0 0 NA

Total 9088.4 31 48

© 2018

Statapult®

DOE Demo

(The Empirical Approach)

Run

1

2

3

4

A B A B AB Y1 Y2 Y S

Actual

FactorsCoded Factors Response Values

144 2

144 3

160 2

160 3

-1 -1 +1

-1 +1 -1

+1 -1 -1

+1 +1 +1

Percent Confidence

that a term identified

as significant, truly

does belong in [ ]

95% (a = .05)

95% (a = .05)

95% (a = .05)

95% (a = .05)

95% (a = .05)

Percent chance of

finding a significant

variance [average]

shifting term if one

actually exists

40% (b = .60)

75% (b = .25)

90% (b = .10)

95% (b = .05)

99% (b = .01)

Number of Runs in 2 Level Portion of the Design

2

Sample Size per Experimental Condition

4 8 12 16

5 [3]

9 [5]

13 [7]

17 [9]

21 [11]

3 [2]

5 [3]

7 [4]

9 [5]

11 [6]

2 [1]

3 [2]

4 [2]

5 [3]

6 [3]

N/A

2 [1]

3 [2]

4* [2]

5* [3]

N/A

2 [1]

N/A

3 [2]

4* [2]

Simplified Table for Determining Sample Size Based on Confidence and Power

s y

49

© 2018

Value Delivery: Reducing Time to Market for New Technologies

• Total # of Combinations = 35 = 243

• Central Composite Design: n = 30

Modeling Flight

Characteristics

of New 3-Wing

Aircraft

Pitch )

Roll )

W1F )

W2F )

W3F )

INPUT OUTPUT

(-15, 0, 15)

(-15, 0, 15)

(-15, 0, 15)

(0, 15, 30)

(0, 15, 30)

Six Aero-

Characteristics

Patent Holder: Dr. Bert Silich

50

© 2018

Aircraft Equations (predictive models)

CL = .233 + .008(P)2 + .255(P) + .012(R) - .043(WD1) - .117(WD2) + .185(WD3) + .010(P)(WD3) -

.042(R)(WD1) + .035(R)(WD2) + .016(R)(WD3) + .010(P)(R) - .003(WD1)(WD2) -

.006(WD1)(WD3)

CD = .058 + .016(P)2 + .028(P) - .004(WD1) - .013(WD2) + .013(WD3) + .002(P)(R) - .004(P)(WD1)

- .009(P)(WD2) + .016(P)(WD3) - .004(R)(WD1) + .003(R)(WD2) + .020(WD1)2 + .017(WD2)2

+ .021(WD3)2

CY = -.006(P) - .006(R) + .169(WD1) - .121(WD2) - .063(WD3) - .004(P)(R) + .008(P)(WD1) -

.006(P)(WD2) - .008(P)(WD3) - .012(R)(WD1) - .029(R)(WD2) + .048(R)(WD3) - .008(WD1)2

CM = .023 - .008(P)2 + .004(P) - .007(R) + .024(WD1) + .066(WD2) - .099(WD3) - .006(P)(R) +

.002(P)(WD2) - .005(P)(WD3) + .023(R)(WD1) - .019(R)(WD2) - .007(R)(WD3) + .007(WD1)2

- .008(WD2)2 + .002(WD1)(WD2) + .002(WD1)(WD3)

CYM= .001(P) + .001(R) - .050(WD1) + .029(WD2) + .012(WD3) + .001(P)(R) - .005(P)(WD1) -

.004(P)(WD2) - .004(P)(WD3) + .003(R)(WD1) + .008(R)(WD2) - .013(R)(WD3) + .004(WD1)2

+ .003(WD2)2 - .005(WD3)2

Ce = .003(P) + .035(WD1) + .048(WD2) + .051(WD3) - .003(R)(WD3) + .003(P)(R) - .005(P)(WD1)

+ .005(P)(WD2) + .006(P)(WD3) + .002(R)(WD1)

51

© 2018

Cyber Penetration Testing Example*

Putting these factor names and their levels, along with the test constraints, into Pro-Test software yields these 14 optimal test configurations:

* Courtesy of Raytheon

Cyber

Penetration

Testing

Operating System

IS Type

y (did/did not penetrate)

Port

52

© 2018

Command and Control Test Design Example(15 factors each at various levels)Total Combinations: 20,155,392

Variable or Factor Levels (# of levels)

Mission Snapshots Entry, Operations, Consolidation (3)

Network Size 10 Nodes, 50 Nodes, 100 Nodes (3)

Network Loading Nominal, 2X, 4X (3)

Movement Posture ATH, OTM1, OTM2 (3)

SATCOM Band Ku, Ka, Combo (3)

SATCOM Look Angle 0, 45, 75 (3)

Link Degradation 0%, 5%, 10%, 20% (4)

Node Degradation 0%, 5%, 10%, 20% (4)

EW None, Terrestrial, GPS (3)

Interoperability Joint Services, NATO (2)

IA None, Spoofing, Hacking, Flooding (4)

Security NIPR, SIPIR (2)

Message Type Data, Voice, Video (3)

Message Size Small, Medium, Large, Mega (4)

Distance Between Nodes Short, Average, Long (3)

54

© 2018

INPUTS OUTPUTS

CONSTRAINTS

Number of Targets (1,2)

Operator Type (Tier 2 -

male/female)

Time of Activity day/night

Ancillaries

7 configurations

Soldier Combat

Ensemble (i.e. Body

Armour

EF88 and Baseline

(SA2) Ancillary

Testing

Probability-of-hit (Target Hit

Ratio)

Time-to-Engage Target

NOISE

Weather

Target Distance

100, 200, 300, 400, 600

Type of Engagement (deliberate,

snap)

Firing Position –

prone/kneeling/standing

* The description and results of this study appear in the June 2016 ITEA Journal, under the title of “Australia’s First Official Use of Design of Experiments in T&E: User Trials to Select Rifle Enhancements” by Joiner, Kiemele, and McAuliffe.

Using DOE to Select Rifle Enhancements*(trials conducted by Australian Defence Test & Evaluation Office)

56

© 2018

Submarine Threat Detection Test ExampleAll Pairs Testing from Pro-Test generated 15 test cases

Suppose we want to perform a verification test with the following 7 input factors (with their respective settings):

Submarine Type (S1, S2, S3)Ocean Depth (Shallow, Deep, Very Deep)Sonar Type (Active, Passive)Target Depth (Surface, Shallow, Deep, Very Deep)Sea Bottom (Rock, Sand, Mud)Control Mode (Autonomous, Manual)Ocean Current (Strong, Moderate, Minimal)

All possible combinations would involve how many runs in the test?

If we were interested in testing all pairs only, how many runs would be in the test? Pro-Test generated the following test matrix:

1296

57

© 2018

The Big Picture: What can DOE do for us?

1. Infuses power into Systems Engineering (see next page)

2. Provides systematic coverage of the test region

3. Improves the quality of testing, and gives us

▪ Faster detection of problems

▪ Higher probability of detecting faults

▪ Cost and time efficiencies

▪ Ability to quantify risks inherent to any test program

Better testing done faster

at lower cost with less risk

Improved Customer Value and Organizational Success

59

© 2018

DOE Empowers Systems Engineering

FL

OW

ING

RE

QU

IRE

ME

NT

S D

OW

N

Customer Needs

System

Requirements

Sub-system

Requirements

Module

Requirements

Parts

Requirements

Parts

Performance

Module

Performance

Sub-system

Performance

System

Performance

Customer Acceptance

FL

OW

ING

CA

PA

BIL

ITY

UP

Design & Development

60

© 2018

Best Practices for “Operationalizing” DOE(i.e., changing the culture to one of habitually using it)

1. Training plus coaching on projects is an absolute must.

2. A Keep-It-Simple-Statistically (KISS) approach with easy-to-comprehend

materials and easy-to-use software.

3. Gaining and propagating quick-hitting successes.

4. Getting leadership on board and continuously re-invigorating them is

necessary.

5. Developing a culture of continuously generating transfer functions (predictive

models) for the purpose of optimization, prediction, and risk assessment.

62

accelerating the analysis of test data using effective and ... · accelerating the analysis of test...

Documents