Validating the Prognosis DDS model for the Inland Empire

Robert E. Froese Andrew P. RobinsonSchool of Forest Resources Etc. Department of Forest Resources

Michigan Technological University University of Idaho

Forest Biometrics Lab

What is “validation”?

• Evaluation• Verification• Validation• Corroboration• Qualification

• Model testing is optimally the responsibility of the model user, who is in the best position to clearly state goals and objectives (Brand and Holdaway 1983; Robinson and Ek 2000)

• assessment would be simpler if model developers report extensive performance information, rather than leave it to model users to generate for themselves (Brand and Holdaway 1983)

TESTING

X

Purposes for model testing

• Caswell (1976): modelling has two core purposes, prediction and understanding, which can be distinguished by interest in truth or reality

• a simplistic example:– the statement “Y varies in the same direction as X” is embodied in the

model Y = b0 + b1·X (b1>0); a test of the accuracy of the model may show it

to be a poor predictor, while a test of the statement may corroborate it.

• In other words,– does a model user care if the internal structures are truthful, as long as the

model makes accurate predictions?– does the scientist care if the model makes accurate predictions, as long as

the model is useful for testing hypotheses about the underlying system?

• objectives must be clear in design, application and evaluation!

A little bit about Prognosis

• growth engine for the Forest Vegetation Simulator in the inland empire

• collection of models - and a model framework

– Increment

– Mortality

– Regeneration

– Scheduling and many others

The Inland Empire

What’s inside Prognosis?

• trees are grown in three dimensions– basal area growth– height growth– crown ratio

• divided into two classes for modelling– in northern Idaho, large is

> 7.62 cm DBH, or > 3 m tall; the others are small

What’s inside Prognosis?

• Probabilistic – keep track of sampling fraction

• Stochastic – record tripling and random deviates

• No Site Index – uses habitat type and other site descriptors 1/300 acre

Wykoff’s 1990 basal area growth model

The DDS model is the key driver for increment …

because predictions are used directly or indirectly as predictors in other model sub-components

DDS = DBH2t+10 - DBH2

t

but actually..

DDS = DBH2t - DBH2

t-10

BAG = (π/4)·(DBH2t - DBH2

t-10)

DG = (DBH2 + DDS)0.5 - DBH

ln(DDS) = f(SIZE +SITE +COMPETITION)

the 1990 DDS model formulation

• bi – coefficients estimated by ordinary least squares, of which:– b0 depends on habitat type and nearest National Forest

– b2 depends on nearest National Forest

– b12 depends on habitat type

1001ln

sincos

lnln

12112

109

287

26543

2210

CCFbDBHBALbCRbCRb

ELbELbSLbSLbSLASPbSLASPb

DBHbDBHbbDDS

Objectives

1. Produce performance information• bias• Precision

2. Provoke and guide future development• examine performance against individual predictors

3. Examine the model as a scientific statement• does the model behave the way it should based on

biological principles

Not Objectives

• Traditional hypothesis test for model bias– e.g., Ho is of no difference and Ha is of a difference

• Arbitrarily small differences are detectable• Statistical significance is not practical significance

• An alternative: see Andrew Robinson’s talk tomorrow!

FIA Data

• data from the USDA Forest Service -Forest Inventory and Analysis Program (FIA)

• geographically extensive– (now) one National design, all forest land ownerships– but… plot locations are strictly confidential

• unbiased sampling design– systematic random sample– one field location per 2,400 hectares

• estimates of between and within-stand variability– cluster of 5 to 10 (old design) or 4 (new design) plots

• retrospective measurements of growth

Methodology

1. backdate FIA stand conditions following Wykoff’s (1990) rationale1. Find diameter at t-10 for all trees, to calculate competition

variables

2. Find height at t-10 for growth sample trees

2. generate predictions using the 1990 DDS model

3. For growth sample trees1. Calculate a basal area increment prediction residual

2. Estimate volume increment and volume increment prediction residual

Results, overall

• 40,979 trees over 2,632 FIA field locations• for Basal Area Increment

– mean increment is 111.6 cm2dec-1

– mean bias is 13.2 cm2dec-1 or 11.8% underprediction

– bias SD is 76.8 cm2dec-1 or CV is 651%

• for Volume Increment– mean increment is 44.5 m3ha-1dec-1

– mean bias is 1.2 m3ha-1dec-1 or 2.6% underprediction

– bias SD is 11.6 m3ha-1dec-1 or CV is 966%

• this means– 2,632 locations x 2,400 ha·location-1 x 1.2 m3ha-1dec-1

= more than 7.6 million m3dec-1 underprediction

for Basal Area IncrementData Distribution

02000400060008000

100001200014000

AB

GR

AB

LA

LAO

C

OT

HE

PIC

O

PIE

N

PIM

O

PIP

O

PS

ME

TH

PL

TS

HE

No.

Obs

erva

tions

Mean BA Increment

05

101520253035

AB

GR

AB

LA

LAO

C

OTH

E

PIC

O

PIE

N

PIM

O

PIP

O

PS

ME

THP

L

TSH

E

sq.c

m p

er d

ecad

e

Mean Increment Bias

0

1

2

3

4

5

6

AB

GR

AB

LA

LAO

C

OT

HE

PIC

O

PIE

N

PIM

O

PIP

O

PS

ME

TH

PL

TS

HE

sq.c

m p

er d

ecad

e

Bias Relative to Increment

05

101520253035

AB

GR

AB

LA

LAO

C

OT

HE

PIC

O

PIE

N

PIM

O

PIP

O

PS

ME

TH

PL

TS

HE

perc

ent (

%)

By nearest National ForestData Distribution

0

2000

4000

6000

8000

10000

Bitt

erro

ot

Cle

arw

ater

Cd'

A

Fla

thea

d

Kan

isku

Koo

tena

i

Lolo

Nez

Per

ce

St.

Joe

No.

Obs

erva

tions

Mean Increment Bias

05

10152025303540

Bitt

erro

ot

Cle

arw

ater

Cd'

A

Fla

thea

d

Kan

isku

Koo

tena

i

Lolo

Nez

Per

ce

St.

Joe

sq.c

m p

er d

ecad

e

Bias Relative to Increment

05

101520253035

Bitt

erro

ot

Cle

arw

ater

Cd'

A

Fla

thea

d

Kan

isku

Koo

tena

i

Lolo

Nez

Per

ce

St.

Joe

perc

ent (

%)

Mean Increment

020406080

100120140160180

Bitt

erro

ot

Cle

arw

ater

Cd'

A

Fla

thea

d

Kan

isku

Koo

tena

i

Lolo

Nez

Per

ce

St.

Joe

sq.c

m p

er d

ecad

e

for Volume IncrementData Distribution

02000400060008000

100001200014000

AB

GR

AB

LA

LAO

C

OT

HE

PIC

O

PIE

N

PIM

O

PIP

O

PS

ME

TH

PL

TS

HE

No.

Obs

erva

tions

Mean Volume Increment

02468

101214

AB

GR

AB

LA

LAO

C

OTH

E

PIC

O

PIE

N

PIM

O

PIP

O

PS

ME

THP

L

TSH

E

cu.m

. per

ha

per d

ec

Mean Increment Bias

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

AB

GR

AB

LA

LAO

C

OT

HE

PIC

O

PIE

N

PIM

O

PIP

O

PS

ME

TH

PL

TS

HE

cu.m

per

ha

per d

ec

Bias Relative to Increment

-10

-5

0

5

10

15

20

25

AB

GR

AB

LA

LAO

C

OT

HE

PIC

O

PIE

N

PIM

O

PIP

O

PS

ME

TH

PL

TS

HE

perc

ent (

%)

Trends with predictors

Discussion

• FIA data– comparable in size and geographic extent– can’t do (precise) spatial analyses

• BIAS– Practically, 7.5 million m3 is meaningful– relative to SD, perhaps not meaningful

• extrapolation– space, time

• consequences– management for timber– management for non-timber

• LOC is a problem

Conclusions

• The 1990 DDS model is not a particularly accurate predictor of forest growth, but it is relatively robust as a theoretical statement under substantial extrapolation in time and space

• Model users may wish to apply a multiplier to the diameter increment model subcomponent

• Model development in the future should re-evaluate LOC and look for alternatives

Acknowledgements

Funding provided by the USDA Forest Service RMRS-99541-RJVA and the University of Idaho Forest Biometrics Lab.

This research was completed entirely using open source software.

Special thanks to:• My major Professor, Dr. Andrew Robinson• Bill Wykoff, Moscow Forest Sciences Lab• Sharon Woudenberg and John Nelson, FIA Ogden, Utah.