reliable and objective assessment of the forecast quality ... · 1 reliable and objective...

1

Reliable and objective assessment

of the forecast quality

of meta-models

Thomas Most

Dynardo GmbH

13. Weimarer Optimierungs und

Stochastiktage 23.-24. Juni 2016

2T. Most: Reliable assessment of the forecast quality of meta-models

13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016

Automatic workflow

with a minimum of solver runs to:

• identify the important parameters for each response

• Generate best possible metamodel (MOP) for each response

• understand and reduce the optimization task

• check solver and extraction noise

Sensitivity AnalysisUnderstand the most important input variables!

© Dynardo GmbH



© Dynardo GmbH

Latin Hypercube Sampling



Deterministic DoE

• Complex scheme required to detect multivariate dependencies

• Exponential growth with dimension

• Full factorial:

• Koshal linear:

Advanced Latin Hypercube Sampling

• Reduced sample size for statistical estimates

compared to plain Monte Carlo

• Reduces unwanted input correlation

How to Generate a Design of Experiments

© Dynardo GmbH



• Reduction of correlation errors by simplified Iman &

Connover algorithm (applicable for nSamples > nPar)

• Start designs are not considered

Advanced Latin Hypercube Sampling (oSL 4.x)

© Dynardo GmbH

• 10 variables• 50 samples• Correlation errors

up to 0.04



• Improved Iman & Connover algorithm + optimization

(applicable even for nSamples < nPar)

• Start designs are considered

Advanced Latin Hypercube Sampling (oSL 5.2)

© Dynardo GmbH

• 10 variables• 50 samples• Correlation errors

less than 0.001



• Discrete parameter states are now

considered in the minimization of

the correlation errors

Advanced Latin Hypercube Sampling (oSL 5.2)

© Dynardo GmbH

ALHS for discrete parameters



• Spatial uniform distribution of LHS optimized by discrepancy criterion

Space-filling Latin Hypercube Sampling (oSL 5.2)

© Dynardo GmbH



• Spatial uniform distribution of LHS optimized by discrepancy criterion

• Start design are considered


© Dynardo GmbH



• Optimal space-filling distribution in full space

• Reduced linear and quadratic correlations (efficient up to 10 variables)


© Dynardo GmbH



• Attention: reduced subspace is not space-filling anymore!

Recommended application up to 10 variables or

for adding samples in subspace

2 of 2 parameters 2 of 5 parameters


© Dynardo GmbH



© Dynardo GmbH

Coefficient of Prognosis



• Approximation of response variables as

explicit function of all input variables

• Approximation function can be used for

sensitivity analysis and/or optimization

• Global methods (Polynomial

regression, Neural Networks, …)

• Local methods (Spline interpolation,

Moving Least Squares, Radial Basis

Functions, Kriging, …)

• Approximation quality decreases with

increasing input dimension

• Successful application requires

objective measures of the

prognosis quality

© Dynardo GmbH

Response Surface Method



• CoD (Coefficient of Determination or R²)

• CoP (Coefficient of Prognosis)

with

• Total variation:

• Unexplained variation:

CoP versus CoD

© Dynardo GmbH



© Dynardo GmbH

• Coefficient of Determination quantifies merely the Goodness of Fit.

• Interpolation models (e.g. MLS, Kriging) can reach CoD of 1.00

• But perfect fit does not mean perfect forecast quality!

Measure Goodness of Fit = Coefficient of Determination (CoD)

CoD = 1.00



© Dynardo GmbH

• Split the data in two sets (optimal spatial distribution)

• Build up the meta-model for data set 1 = regression point set

• Evaluate forecast quality on data set 2 = independent test point set

Measure forecast quality = Coefficient of Prognosis (CoP)

CoD_1 = 1.00CoD_1 = 1.00CoP_2 = 0.80



© Dynardo GmbH

• Build up the meta-model for data set 2 = regression point set

• Evaluate forecast quality on data set 1 = independent test point set

CoD_2 = 1.00CoP_1 = 0.59




© Dynardo GmbH

• Coefficient of Prognosis sums up the

errors from both cross validation sets:

• CoP is an objective measure of forecast quality!

CoD = 1.00CoP = 0.74




• CoD (R²) tells how good the

regression model fits through the

sample points, no real prediction

quality

• CoD is too optimistic for a small

number of support points

• CoD=1 for interpolation models

(e.g. Kriging, …)

CoP versus CoD

• CoP measures the true forecast quality of a regression model using an

independent set of test data (cross validation)

• Suitable for any kind of meta-model

• Remark: Leave-one-out may be too optimistic as well!

Labeled as Predictive Coefficient of Determination in oSL post-processing

© Dynardo GmbH



Metamodel of Optimal Prognosis (MOP)

• Objective measure of prognosis quality

• Determination of relevant parameter subspace

• Determination of optimal approximation model

• Approximation of solver output by fast

surrogate model without over-fitting

• Evaluation of variable sensitivities

© Dynardo GmbH



© Dynardo GmbH

• MOP approach yields the best model regarding forecast quality (CoP)

• Smoothing of noisy responses as added benefit

• A difference >3% between CoD and CoP indicates that the model

forecast quality can be further increased by adding more support points.

Metamodel of Optimal Prognosis (MOP)

CoD = 0.87CoP = 0.77



Analytical Nonlinear Function

• 20 input variables

• Additive linear and nonlinear terms, one coupling term and one noise term with normal distribution

© Dynardo GmbH



© Dynardo GmbH

• Anisotropic Kriging in automatically detected four-dimensional subspace

• CoD (100 support points): 98,9% (noise is smoothed)

• CoP (cross validation): 97,0%, predicted RMSE 0,81

• Verification data set (500 samples): 97,8%, verified RMSE 0,70

Metamodel of Optimal Prognosis

Prediction (CoP) Verification



© Dynardo GmbH

• Genetic Aggregation in full space (weighted sum of polynomials,

Kriging, support vector regression and neural networks)

• CoD (100 support points): 100% (noise is interpolated!)

• Predicted CoD: 92,9%, predicted RMSE 1,27

• Verification data set (500 samples): 83,4%, verified RMSE 1,94

“Competitor”



• Smooth approximation function

of MOP enables reliable

optimization on the response

surface

• Interpolating the noise makes

the optimization task much

more difficult and yields to

nonsufficient results

© Dynardo GmbH

Optimization on the Meta-Model

Reference MOP Competitor

Response -12,41 +/- noise -10,94 -9,46

X1 3,14 3,14 1,59

X2 -3,14 -3,14 -3,14

X3 -1,57 -1,56 -1,39

X4 -0,50 -0,36 -0,38



• Global refinement is now

possible with Advanced

(correlation optimized) and

Space-filling Latin Hypercube

Sampling

• Coefficient of Prognosis is a

reliable measure of forecast

quality

• Making meta-models more and more flexible tends to over-fitting with

more flexible results in

low forecast quality!

Further developments: Local refinement of MOP

• Local approximation quality

• Single-objective criteria

• Multi-objective criteria

© Dynardo GmbH

Summary

reliable and objective assessment of the forecast quality ... · 1 reliable and objective...

Documents