Lecture 5
Model Evaluation
Elements of Model evaluation
Goodness of fit
Prediction Error
Bias
Outliers and patterns in residuals
Assessing Goodness of Fit for Continuous Data
Visual methods - Don’t underestimate the power of your eyes, but eyes can
deceive, too...
Quantification- A variety of traditional measures, all with some
limitations...
A good review...C. D. Schunn and D. Wallach. Evaluating Goodness-of-Fit in Comparison of Models to Data. Source:http://www.lrdc.pitt.edu/schunn/gof/GOF.doc
Visual evaluation for continuous data
Graphing observed vs. predicted...
obs = 0.62 + 0.95(pred)
0
5
10
15
20
0 5 10 15 20
Predicted
Obs
erve
d
1:1 line
Source: Canham, C. D., P. T. LePage, and K. D. Coates. 2004. A neighborhood analysis of canopy tree competition: effects of shading versus crowding. Canadian Journal of Forest Research.
Examples
Goodness of fit of neighborhood models of canopy tree growth for 2 species at Date Creek,
BC
Red Cedar
y = 1.0022x
R2 = 0.5963
0
0.5
1
1.5
2
2.5
3
0 0.5 1 1.5 2 2.5
Predicted
Hemlock
y = 1.0001x
R2 = 0.3402
0
0.5
1
1.5
2
2.5
3
3.5
0 0.5 1 1.5Predicted
Predicted
Obse
rved
Good fitbiased underestimate
y = 1.5146x
R2 = 0.9755
0
5
10
15
20
25
30
35
0 5 10 15 20
Predicted
Ob
serv
ed
Good fit - no bias
y = 1.0013x
R2 = 0.937
0
5
10
15
20
25
0 5 10 15 20
Predicted
Ob
serv
ed
Poor fit - no bias
y = 1.0167x
R2 = 0.6672
0
5
10
15
20
25
0 5 10 15 20
Predicted
Ob
se
rve
dGoodness of Fit vs. Bias
1:1 line
Poor fitbiased overestimate
y = 0.4508x + 5.2182R2 = 0.4787
0
2
4
6
8
10
12
14
16
18
0 5 10 15 20
Predicted
Ob
se
rve
d
1:1 line
R2 as a measure of goodness of fit
R2 = proportion of variance* explained by the model...(relative to that explained by the simple mean of the data)
N
i
2i
N
iii
)obsobs
exp obs
SST
SSER
1
1
2
2
(
)(11
Where expi is the expected value of observation i given the model, and obs is the overall mean of the observations(Note: R2 is NOT bounded between 0 and 1)
* this interpretation of R2 is technically only valid for data where SSE is an appropriate estimate of variance (e.g. normal data)
R2 – when is the mean the mean?
Clark et al. (1998) Ecological Monographs 68:220
N
i
2ji
S
j
N
iijij
S
j
)obsobs
exp obs
R
11
1
2
12
(
)(
1
For i=1..N observations in j = 1..S sites – uses the SITE means, rather than the overall mean, to
calculate R2
r2 as a measure of goodness of fit
n
yy
n
xx
n
yxxy
r2
22
2 )()(
)((
r2 = squared correlation (r) between observed (x) and predicted (y)
0
5
10
15
20
0 5 10 15 20
Predicted
Ob
se
rve
d
R2 = -59.8r2 = 0.699
1:1 line
0
5
10
15
20
0 5 10 15 20
Predicted
Ob
se
rve
d
R2 = 0.905r2 = 0.912
1:1 line
NOTE: r and r2 are both bounded between 0 and 1
R2 vs r2
y = 1.5003x + 5.3709
0
5
10
15
20
25
30
35
40
0 5 10 15 20
Predicted
Ob
se
rve
d
r2 = 0.81R2 = -0.39
1:1 line
Is this a good fit (r2=0.81) or a really lousy fit (R2=-0.39)?(it’s undoubtedly biased...)
A note about notation...
Check the documentation when a package reports “R2” or “r2”. Don’t assume they will be used as I have used
them...
Sample Excel output using the “trendline” option for a chart:
The “R2” value of 0.89 reported by Excel is actually r2 (While R2 is actually 0.21)
(If you specify no intercept, Excel reports true R2...)
y = 0.9603x + 5.3438
R2 = 0.8867
0
5
10
15
20
25
30
0 5 10 15 20
Predicted
Obs
erve
d
R2 vs. r2 for goodness of fit
When there is no bias, the two measures will be almost identical (but I prefer R2, in principle).
When there is bias, R2 will be low to negative, but r2 will indicate how good the fit could be after taking the bias into account...
Sensitivity of R2 and r2 to data range
Entire Range
y = 1.0035x
R2 = 0.8854
0
10
20
30
40
50
0 10 20 30 40
Predicted
Ob
serv
ed
Range = 0-10
y = 1.4397xR2 = 0.448
0
5
10
15
20
25
0 5 10 15Predicted
Ob
se
rve
d
Range = 30 - 40
y = 1.01xR2 = -0.125
0
10
20
30
40
50
0 10 20 30 40 50
Predicted
Ob
serv
ed
The Tyranny of R2 (and r2)
Limitations of R2 (and r2) as a measure of goodness of fit...
- Not an absolute measure (as frequently assumed),
- particularly when the variance of the appropriate PDF is NOT independent of the mean (expected) value
- i.e. lognormal, gamma, Poisson,
Poisson PDF
0.0
0.1
0.2
0.3
0 5 10 15 20 25 30
x
Pro
b(x
)
m = 2.5
m = 5
m = 10
Gamma Distributed Data...
i
ii
y
eyyf 1
)(
1),|(
i
i
iii
yVar
y of value expected yE
2
)(
)(
The variance of the gamma increases as the square of the mean!...
Gamma PDF ( = 1)
0
0.1
0.2
0.3
0.4
0 5 10 15 20
X
p(x
) u = 1
u = 5
u = 10
So, how good is good?
Our assessment is ALWAYS subjective, because of- Complexity of the process being studied
- Sources of noise in the data
From an information theory perspective, should you ever expect R2 = 1?
Other Goodness of Fit Issues...
In complex models, a good fit may be due to the overwhelming effect of one variable...
The best-fitting model may not be the most “general”
- i.e. the fit can be improved by adding terms that account for unique variability in a specific dataset, but that limit applicability to other datasets. (The curse of ad hoc multiple regression models...)
How good is good: deviance
Comparison of your model to a “full” model, given the probability model.
For i = 1..n observations, a vector X of observed data (xi), and a
vector of j = 1..m parameters (j):
Define a “full” model with n parameters i = xi (full). Then:
)]|()|([2 (D) Deviance XllXll full
n
iixgXL(ll) likelihood-Log
1
)|(ln|ln
Nelder and Wedderburn (1972)
Deviance for normally-distributed data
)2log(2
1)|( 2 nXll full
Log-likelihood of the full model is a function of both sample size (n) and variance (2)
Therefore – deviance is NOT an absolute measure of goodness of fit...
But, it does establish a standard of comparison (the full model), given your sample size and your estimate of the underlying variance...
0
0.2
0.4
0.6
0.8
1
-5 -4 -3 -2 -1 0 1 2 3 4 5
X
Pro
b(x
)
Forms of Bias
N
ii
i
N
ii
exp
)(expobs slope
1
2
1
)(
y = 1.0508x + 8.8405
0
5
10
15
20
25
30
35
40
0 5 10 15 20
Predicted
Ob
se
rve
d
1:1 line
y = 1.48x + 0.9515
0
5
10
15
20
25
30
35
0 5 10 15 20
Predicted
Ob
se
rve
d
1:1 line
Proportional bias (slope not = 1)
Systematic bias (intercept not = 0)
“Learn from your mistakes”(Examine your residuals...)
Residual = observed – predicted
Basic questions to ask of your residuals:
- Do they fit the PDF?
- Are they correlated with factors that aren’t in the model (but maybe should be?)
- Do some subsets of your data fit better than others?
Using Residuals to Calculate Prediction Error
RMSE: (Root mean squared error)
(i.e. the standard deviation of the residuals)
1
)(1
2
n
exp obsRMSE
n
iii
Predicting lake chemistry from spatially-explicit watershed data
At steady state:
decay) inlake rate (flushing volume lake
input ionconcentrat
Where concentration, lake volume and flushing rate are observed,
And input and inlake decay are estimated
k decay inlake
disteEinput iji
C
i
P
ji
1 1
Predicting iron concentrations in Adirondack lakes
Adirondack Lake Iron Concentrations
y = 1.0042x
R2 = 0.563
0
20
40
60
80
100
0 20 40 60 80 100
Predicted
Ob
se
rve
d
-40 -30 -20 -10 0 10 20 30 40 50 60RESIDUAL
-3
-2
-1
0
1
2
3
Ex p
ec t
ed V
a lue
for
No
r mal
Dis
trib
utio
n
-50-40-30-20-10
0102030405060
0 20 40 60 80 100
Predicted
Re
sid
ua
l (O
bs
- P
red
)
Results from a spatially-explicit, mass-balance model of the
effects of watershed composition on lake chemistry
Source: Maranger et al. (2006)
Should we incorporate lake depth?
-50
-40
-30
-20
-10
0
10
20
30
40
50
60
0 5 10 15
Lake Depth (m)
Re
sid
ua
ls (
Ob
s -
Pre
d)
•Shallow lakes are more unpredictable than deeper lakes•The model consistently underestimates Fe concentrations in deeper lakes
Adding lake depth improves the model...
Model with depth term included
y = 1.0082x
R2 = 0.6533
0
20
40
60
80
100
0 20 40 60 80 100
Predicted
Ob
se
rve
d
R2 went from 56% to 65%
-30 -20 -10 0 10 20 30 40 50 60DEPTHRESID
-3
-2
-1
0
1
2
3
Ex p
ecte
d V
a lue
for
Nor
ma l
Dis
trib
ut io
n
It is just as important that it made sense to add depth...
But shallow lakes are still a problem...
Model with depth added
-40
-30
-20
-10
0
10
20
30
40
50
60
0 5 10 15
Predicted
Re
sid
ua
l (O
bs
- P
red
)
Summary – Model Evaluation
There are no silver bullets...
The issues are even muddier for categorical data...
An increase in goodness of fit does not necessarily result in an increase in knowledge…- Increasing goodness of fit reduces uncertainty in the
predictions of the models, but this costs money (more and better data). How much are you willing to spend?
- The “signal to noise” issue: if you can see the signal through the noise, how far are you willing to go to reduce the noise?