multiobjective calibration with padds: testing alternative selection metrics

Multiobjective Calibration with PADDS: Testing Alternative Selection Metrics

Masoud AsadzadehBryan Tolson

2

Outline• Objectives

• PA-DDS algorithm

• Alternative selection metrics

• Experiment to choose proper selection metric

• MO Performance Evaluation with CNHV

• Validation of Selected Metric, MO Model Calibration

• Conclusions and Future Work

3

Objectives• Evaluating PA-DDS performance:

– Solving MOPs with more than 2 objectives– Using alternative selection metrics

• Random (RND)• Crowding Distance (CD)• Hypervolume (HV)

• Choosing proper selection metric• Validating selected metric, comparing modified

PA-DDS against high quality MO algorithms: – AMALGAM vs. ɛ-NSGAII vs. PA-DDS

4

Pareto Archive DDSPerturb current

ND solutionUpdate ND solutions

Continue?STOP

New solution is ND?

Pick the New solution

Pick a ND solution

Initialize starting solutions

YN

Create ND-solution set

YN

5

Alternative Selection Metrics

• Random Selection (RND)

• Crowding Distance (CD)– Deb et al. (2002)

• Contribution to HyperVolume (HV)– Zitzler and Thiele 1999– Used as selection metric in Emmerich et al. (2005) f1

f2

6

Experiment to Choose Selection Metric

PA-DDS

RND CD HV

Mathematical Test Suites1 2 3

• Number of Trials: 50

• Budget: 1,000 and 10,000

• Performance Evaluation: CNHV

7

Mathematical Test Problem, ZDT4Zitzler et al. (2000)

• 10 decision variables

• 2 objectives

• 219 local fronts

• Convex Pareto front

8

Mathematical Test Problem, WFG4Huband et al. (2006)

• Scalable

• 24 decision variables

• 2 and 3 objectives

• Highly Multi-modal

• Concave front

9

Mathematical Test Problem, WFG4Huband et al. (2006)

10

MO Model Comparison• Comparative Normalized Hyper-Volume

1

1

Worst attained front

Best attained front

11

CNHV vs. HV• Same as HV or NHV

– CNHV always prefers dominating solution

– CNHVA > CNHVB : B doesn’t weakly dominate A

– CNHVmax = 1 & CNHVmin = 0

• Compares multiple trials of multiple algorithms

• Measures performance across compared algorithms

12

Results: ZDT4

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RND1,000

CD1,000

HV1,000

CNHV

Prob

abili

ty o

f Fin

ding

Bet

ter

CN

HV

1

11

1

13

Results: ZDT4

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RND10,000

CD10,000

HV10,000

CNHV

Prob

abili

ty o

f Fin

ding

Bet

ter

CN

HV

14

Results: WFG4 Two Objectives

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RND1,000

CD1,000

HV1,000

CNHV

Prob

abili

ty o

f Fin

ding

Bet

ter

CN

HV

15

Results: WFG4 Two Objectives

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RND10,000

CD10,000

HV10,000

CNHV

Prob

abili

ty o

f Fin

ding

Bet

ter

CN

HV

16

Results: WFG4 Three Objectives

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RND1,000

CD1,000

HV1,000

CNHV

Prob

abili

ty o

f Fin

ding

Bet

ter

CN

HV

17

Results: WFG4 Three Objectives

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RND10,000

CD10,000

HV10,000

CNHV

Prob

abili

ty o

f Fin

ding

Bet

ter

CN

HV

18

Validating the Selected MetricPA-DDS

RND CD HV

Mathematical Test Suites

PA-DDSε-NSGAII AMALGAM

Model Calibration

1 2 3

• Number of Trials: 10

• Budget: 10,000

• Performance Evaluation: CNHV

19

• Sub-watershed in Cannonsville– 37 km2

• SWAT2000

• 26 Parameters

• Nash Sutcliffe– Flow, Phosphorus delivery

Model Calibration, Town Brook

Tolson and Shoemaker 2007

20

Model Calibration Results

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

AMALGAM10,000

eNSGAII10,000

PA-DDS10,000

CNHV

Prob

abili

ty o

f Fin

ding

Bet

ter

CN

HV

21

Model Calibration Results

0.52 0.53 0.54 0.55 0.56 0.57 0.58 0.59 0.6 0.61 0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69 0.7 0.710.58

0.6

0.62

0.64

0.66

0.68

0.7

0.72

0.74

0.76Actual Approximate Fronts, Combined 3 Worst CNHV, Budget 10,000

PA-DDSAMALGAMeNSGAIIBest Attained FrontWorst Attained Front

NS Flow

NS

Phos

phor

us T

rans

port

22

Model Calibration Results

0.52 0.53 0.54 0.55 0.56 0.57 0.58 0.59 0.6 0.61 0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69 0.7 0.710.58

0.6

0.62

0.64

0.66

0.68

0.7

0.72

0.74

0.76Actual Approximate Fronts, Combined 4 Average CNHV, Budget 10,000

PA-DDSAMALGAMeNSGAIIBest Attained FrontWorst Attained Front

NS Flow

NS

Phos

phor

us T

rans

port

23

Model Calibration Results

0.52 0.53 0.54 0.55 0.56 0.57 0.58 0.59 0.6 0.61 0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69 0.7 0.710.58

0.6

0.62

0.64

0.66

0.68

0.7

0.72

0.74

0.76Actual Approximate Fronts, Combined 3 Best CNHV, Budget 10,000

PA-DDS

AMALGAM

eNSGAII

Best Attained Front

Worst Attained Front

NS Flow

NS

Phos

phor

us T

rans

port

24

Model Calibration Results

0.52 0.53 0.54 0.55 0.56 0.57 0.58 0.59 0.60 0.61 0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69 0.70 0.710.52

0.54

0.56

0.58

0.60

0.62

0.64

0.66

0.68

0.70

0.72

0.74

0.76Actual Approximate Fronts, 10 Trials Combined, Budget 10,000

PA-DDS

AMALGAM

eNSGAII

Best Attained Front

NS Flow

NS

Phos

phor

us T

rans

port

25

Model Calibration Results

0.52 0.53 0.54 0.55 0.56 0.57 0.58 0.59 0.60 0.61 0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69 0.70 0.710.56

0.58

0.60

0.62

0.64

0.66

0.68

0.70

0.72

0.74

0.76Actual Approximate Fronts, 10 Trials Combined

PA-DDS10,000

PA-DDS1,000

Best Attained Front

NS Flow

NS

Phos

phor

us T

rans

port

26

• PA-DDS inherits simplicity and parsimonious characteristics of DDS– Generates good Pareto approximate front in the modeller's time frame– Reduces the need to fine tune the algorithm parameters– Solves both continuous and discrete problems

• PA-DDS can solve MOPs with more than 2 objectives• HV based selection clearly improves PA-DDS performance• PA-DDS with HV selection is promising compared to two high quality

benchmark algorithms, AMALGAM and ε-NSGAII

Evaluate PA-DDS performance in solving Multi Objective model calibrations with more than 2 objective functions

Implement a more efficient archiving strategy and dominance check (e.g. Fieldsend et al. 2003)

Conclusions & Future Work

27

28

Budget vs. DimensionAlg. Study Type of MOP # DV Budget

AMALGAM

Vrugt and Robinson, 2006 Test problems (ZDT) 10 2,500; 5,000; 7,500; 15,000

Wohling et al. 2008 Soil hydraulic parameter estimation 15 20,000

Huisman et al. 2009 Coupled HYDRUS-2D, CRMOD 12 (?) 10,000

Zhang et al. 2010 SWAT 16 10,000

ε-NSGAII

Kollat, Reed, 2005 Test problems 10, 30 12,000 to 15,000

Kollat, Reed, 2006 Groundwater Monitoring (discrete) 25 200,000

29

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RND1,000 RND10,000

CD1,000 CD10,000

HV1,000 HV10,000

CNHV

Prob

abili

ty o

f Fin

ding

Bet

ter

CN

HV

Results: ZDT4

30

Results: WFG4 Two Objectives

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RND1,000 RND10,000

CD1,000 CD10,000

HV1,000 HV10,000

CNHV

Prob

abili

ty o

f Fin

ding

Bet

ter

CN

HV

31

Results: WFG4 Three Objectives

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RND1,000 RND10,000

CD1,000 CD10,000

HV1,000 HV10,000

CNHV

Prob

abili

ty o

f Fin

ding

Bet

ter

CN

HV

32

Model Calibration Results

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

AMALGAM1,000 AMALGAM10,000

eNSGAII1,000 eNSGAII10,000

PA-DDS1,000 PA-DDS10,000

CNHV

Prob

abili

ty o

f Fin

ding

Bet

ter

CN

HV

33

Model Calibration Results

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

AMALGAM1,000

eNSGAII1,000

PA-DDS1,000

CNHV

Prob

abili

ty o

f Fin

ding

Bet

ter

CN

HV