L. Mentaschi, G. Besio, F. Cassola, A. MazzinoDipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA),

Università degli studi di Genova

Validation of wave model in the Mediterranean sea

Model Wavewatch IIIArdhuin et al. (2008) source terms.Tolman and Chalikov (1996) source terms, not tuned

to the Mediterranean sea conditions.

O

OSNBI

)( Normalized Bias (NBI)

Statistical indicators used for validation

Normalized Root MeanSquare Error (NRMSE) 2

2)(

O

OSNRMSE

Correlation coeff. (ρ)OS

ii

N

OOSS

)()(

Mentaschi et al. 2013 - Why NRMSE is not completely reliable

Statistics on 17 storms and 23 buoys

ACC350 T & C

NBI 2.1% -11.2%

ρ 0.889 0.883

NRMSE 0.2864 0.2798 (-2.3%)

February 1990 storm

Mentaschi et al. 2013 - Why NRMSE is not completely reliable

In what sense NRMSE is not completely reliable?A numerical example

ρ=0.614 for both of the simulations

NBI = -12% for the red simulation

One would say the best simulation is the blue one.

But …

NRMSE(blue) = 0.384

NRMSE(red) = 0.356 (~ -7.2%)

Mentaschi et al. 2013 - Why NRMSE is not completely reliable

Geometrical decomposition of NRMSE

2

22

22

2

2)]()[(NBI

O

ONRMSE

O

OOSSSI

Oi

ii

NBIO

OBC

O

NRMSE 22

2

222NRMSEBCSINRMSE

Bias component

Scatter component (sometimes called Scatter Index)

O

OSNBI

where

Are SI and BC independent?

Mentaschi et al. 2013 - Why NRMSE is not completely reliable

Example of set of simulations with constant ratio :

In general and are not independent.

.~ constSS

SS /

iiNBI SNBIS 0)1(

where is the unbiased simulation.iS0

S S

Let’s assume this relation:

Mentaschi et al. 2013 - Why NRMSE is not completely reliable

Holds for amplifications

Amplification factor: NBI1

Relation SI – NBI

22

22

2

2

2 2)]()[(

O

OSOS

i

ii

OO

OOSSSI

)1( NBIOS

)1(./ 0 NBIconstS SSS •

22

2200

20

2202 )(2

O

OSOSSS

OSI

•

OS ~0 0~)1(2

~ 2022

2

0

2

SIONBI

SI

O

O

NBI

SI grows linearly in NBI around NBI=0

Mentaschi et al. 2013 - Why NRMSE is not completely reliable

NRMSE

O BCO

OSISI

2

22

0 2

11~

Fits well real world data

Significant wave height Mean period

Mentaschi et al. 2013 - Why NRMSE is not completely reliable

NBISISI

2

11~ 0

The simulation with the minimum value of NRMSE underestimates the average value

Mentaschi et al. 2013 - Why NRMSE is not completely reliable

How to overcome this problem?

Hanna and Heinold (1985) indicator:

SO

OS

OS

OSHH

ii

ii22 )()(

Demonstration that with ρ constantHH is minimum when bias is null

Mentaschi et al. 2013 - Why NRMSE is not completely reliable

2)( 22222

2

OS

OS

ii

ii

OS

OS

OS

OSHH

OS ~0•

0~0

2

NBINBI

HH

02~22

22

0

2

22

O

O

NBIO

O

NBI

HH

HH has a minimumfor null bias

)1( NBIOS

)1(./ 0 NBIconstS SSS •

•

Mentaschi et al. 2013 - Why NRMSE is not completely reliable

Wavewatch III validation on the Mediterranean sea.

ACC350 T & C

NBI 0.021 -0.112

ρ 0.889 0.883

NRMSE 0.2864 0.2798 (-2.3%)

HH 0.3459 0.3634 (+4.8%)

HH has a minimum for null bias

Mentaschi et al. 2013 - Why NRMSE is not completely reliable

Future development

• NRMSE tends to assume smaller values for simulations that underestimate observations, and this is mostly evident when relation holds.

• HH indicator overcomes this problem introducing a different normalization of the root mean square error.

.~/ constSS

Conclusions

• Usage of HH indicator to validate Wavewatch III model on the Mediterranean sea.

Mentaschi et al. 2013 - Why NRMSE is not completely reliable

Thank you!

Mentaschi et al. 2013 - Why NRMSE is not completely reliable

L. Mentaschi, G. Besio, F. Cassola, A. MazzinoDipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA),

Università degli studi di Genova