a bayesian hierarchical model for combining several crop ..._presentations_and...anderson, e....

A Bayesian Hierarchical Model for CombiningSeveral Crop Yield Indications

Nathan B. Cruze

National Agricultural Statistics Service (NASS)United States Department of Agriculture

[email protected]

FCSM 2015Washington, D.C.December 1, 2015

“. . . providing timely, accurate, and useful statistics in service to U.S. agriculture.” 1

Goal and technical approach

I Goal: Model sequence of in-season forecasts and estimates ofcrop yield

I NASS Crop Production Report–state and national yieldestimates

I Reproducibility with appropriate measures of uncertainty

I Approach: Bayesian hierarchical model–synthesis of datafrom several surveys

I Enforce physical relationships at two spatial scalesI Incorporate variety of auxiliary data types

Challenge: From data to publication in 3-4 days

FCSM 2015–A Bayesian Hierarchical Model for Combining Several Crop Yield Indications 2

NASS crop yield surveys and reportsYield measures output per area harvested (bushels/acre)

Yield for state j : µj , j = 1, 2, ..., J

Yield for speculative region: µ =∑J

j=1 wjµj

Weights wj ∝ harvested acres for state j

NASS surveys: Objective Yield (OYS), Agricultural Yield (AYS),Acreage, Production, and Stocks (APS)

APS

AYSAYSAYSAYS

OYSOYSOYSOYSOYS

● ● ● ● ●

Crop

Production

Report

Crop

Production

Report

Crop

Production

Report

Crop

Production

Report

Small

Grains

Summary

APS

AYS

OYS

May Jun Jul Aug Sep OctMonth

Survey and Publication Timeline for Winter Wheat


Role of the Agricultural Statistics Board (ASB)Expert panel of commodity specialists

I Current and historical survey ‘indications’I Other information, e.g., weather, crop condition ratingsI Consensus on yield

Publish national and state estimates

OMB Standard 4.1 (2006): “Agencies must use accepted theory and methods when

deriving...projections that use survey data. Error estimates must be calculated and

disseminated to support assessment of the appropriateness of the uses of the estimates

or projections...”

Challenge: Capture expert assessment in a manner that is1) easily reproducible and 2) includes appropriate measuresof uncertainty


Example survey data

●

●

●

● ●

●

●

●●

●● ●

● ●

NASS Yield Survey Indications: Example Winter Wheat State

Year

Yie

ld (

Bu/

Acr

e)

2002 2004 2006 2008 2010 2012 2014

●

●

●

●

●●

●

●

● ●

●

●

●

AYSOYSSept. APS

● MayJuneJuly

AugSept


Bayesian hierarchical model for speculative regionNotationI µt–true yield

I yktm–observed yield

I k ∈ {O,A,Q}–survey index

I t ∈ {1, ...,T}–year index

I m ∈ {months}–survey month

I m∗–forecast month

Region data model

yktm∗|µt ∼ indep N(µt + bkm∗, s

2ktm∗ + σ2

km∗), k = O,A (1)

yQt |µt ∼ indep N(µt , s

2Qt

)(2)

Region process model

µt ∼ indep N(z′tβ, σ

2η

)(3)

Diffuse prior distributions

I Data model parameters: Θd ≡(bkm∗, σ

2km∗)

I Process model parameters: Θp ≡(β, σ2

η

)


Bayesian hierarchical model for speculative regionLikelihood function–assuming conditional independence

[yO , yA, yQ |µt ,Θd ] =∏

k∈{O,A,Q}

[yk |µt ,Θd ] (4)

Posterior distribution

[µt ,Θd ,Θp|yO , yA, yQ ] ∝∏

k∈{O,A,Q}

[yk |µt ,Θd ][µ|Θp][Θd ][Θp] (5)

Full conditional of regional yield, µt

[µt |yO , yA, yQ ,Θd ,Θp] ∼ N

(∆2

∆1,

1

∆1

)(6)

∆1 =∑

k=O,A

1

σ2km∗ + s2kTm∗

+I{Q}s2QT

+1

σ2η

(7)

∆2 =∑

k=O,A

yktm∗ − bkm∗σ2km∗ + s2kTm∗

+I{Q}yQt

s2QT

+z′tβ

σ2η

(8)


Bayesian hierarchical model–state level yield

State-level counterparts indexed by j ∈ {1, 2, ..., J}

Unconstrained State Model–Define µt· ≡ (µt1, µt2, . . . , µtJ),

µt·|y ,Θd ,Θp,∼ indep MVN

(vec

(∆2j

∆1j

), diag

(1

∆1j

))(9)

Constrained State Model–Enforce constraint by conditioning (9)on µt =

∑j wjµtj(

µt1, µt2, . . . , µt(J−1))∼ MVN(µ̄, Σ̄) (10)

µtJ = µt −1

wtJ

J−1∑j=1

wtjµtj (11)


Summary of model outputsSpeculative Region Model Constrained State Model Unconstrained State Model

Region yield and error Benchmarked state yields anderrors

Region forecast decomposition State forecast decompositionsand benchmarking adjustments

Wang et al. (2012) Adrian (2012), Nandram et al.(2014),Cruze (2015)

Kass and Steffey (1989)

State 1 State 2 · · · State J SPECOverall Forecast µ̂Tj x x · · · x xError x x · · · x x

OYS yOTm∗j − b̂Om∗ x x · · · x x

AYS yATm∗j − b̂Am∗ x x · · · x x

Covariates z′T β̂ x x · · · x x

Sept. APS yQTj x x · · · x xBenchmarking Adj. dj x x · · · x

µ̂tj ≈∑

k∈{O,A,Q,Covariates}

ck(SOURCE )k + dj (12)

ck ∝ (variance)−1k


Winter wheat speculative region

1.8%2.5%2.7%

5.1%

8.7% 35.8%

16.9%

11.4%

8.5%6.6%

25

30

35

40

45

50

−120 −110 −100 −90 −80long

lat

I 10 state region–some states geographically isolatedI Kansas has major share of harvested acres (Plotted: wj , 2012)I Four distinct types of winter wheatI Differential planting and harvest


Winter wheat speculative region–types of wheat

0

25

50

75

100

Was

hing

ton

Mon

tana

Col

orad

o

Neb

rask

a

Kan

sas

Okl

ahom

a

Texa

s

Mis

sour

i

Illin

ois

Ohi

o

State

Per

cent

.Pro

duct

ion

TypeRed HardRed SoftWhite HardWhite Soft

State Winter Wheat Production by Percent Type

25

30

35

40

45

50

−120 −110 −100 −90 −80long

lat

I States ‘specialize’

I Soft varieties associatedwith higher yield

I Washington, Missouri,Illinois, Ohio have higheryields

I Confounding with state


Winter wheat speculative region–differential harvest

StartStart

StartStartStart

StartStartStart

StartStart

ActiveActive

ActiveActive

ActiveActiveActive

ActiveActive

Active

EndEnd

EndEnd

EndEnd

EndEnd

EndEnd

TexasOklahoma

MissouriIllinois

KansasColorado

OhioNebraska

WashingtonMontana

Jun 01 Jun 15 Jul 01 Jul 15 Aug 01 Aug 15 Sep 01

Usual Harvest Dates for Winter Wheat Speculative Region States

25

30

35

40

45

50

−120 −110 −100 −90 −80long

lat

I May OYS: only TX, OK, KS

I Southern states complete harvestbefore northern states begin

I Timing of covariates

I Deriving covariates for the region


Winter wheat model–covariatesCovariates reflect conditions approaching active harvest dates

µtj = βj1 + βj2zj2 + βj3zj3 + βj5zj4 + βj5zj5

I State-specific constant

I zj2: Linear time trend

I zj3: Monthly precipitation(NOAA)

I zj4: Monthly avg. temperature(NOAA)

I zj5: Crop condition–% good +% excellent week # (NASS)

State/FIPS

May Covars June Covars July-September Covars

Condition (Week #) Weather (Month) Condition (Week #) Weather (Month) Condition (Week #) Weather (Month)

CO 8 15 April 21 May 21 MayIL 17 15 April 19 May 19 MayKS 20 15 April 19 May 19 MayMO 29 15 April 19 May 19 MayMT 30 15 April 19 May 24 JuneNE 31 15 April 21 May 21 MayOH 39 15 April 21 May 21 MayOK 40 15 April 17 April 17 AprilTX 48 15 April 17 April 17 AprilWA 53 15 April 22 May 22 May


Comparing ASB estimates and model outputs–2012

State 1 State 2 State 3 State 4 State 5 State 6 State 7 State 8 State 9 State 10 Region

●

●●●●●●

●●●

●●●

●●

●

●

●

●

●

●●●●

● ●●●●

●●

●

●

●●

●●

●

●

●

●●

●●

●

●●

●●

●

●

●

●

●

●

●●

●●

●

●●

●

●●

●

●●

●

●

●

●●

●

●

●●

●

●●

●●

●●● ●

●●●●

●

●●

●●

●●

●

●●

●

●●

●

●

●●

●

●●●

●

●●

●

●●

●

●

●

●●●

●

●

●●

●

●

●

●●

●

●

●

●●

●

●

●

●●●

●

●

●●

●

●●●

●

●

●

●

●●

●

●●

●●

●●●●●

●●

●

●●

●

●

●

●●

●●

●

●●●

●●●

●

●●

●

●●

●

●

●

●

●

●●●

● ● ● ●●● ●

● ●●

● ● ●●

●

● ●

● ● ●

May

June

July

Aug

Sep

t

May

June

July

Aug

Sep

t

May

June

July

Aug

Sep

t

May

June

July

Aug

Sep

t

May

June

July

Aug

Sep

t

May

June

July

Aug

Sep

t

May

June

July

Aug

Sep

t

May

June

July

Aug

Sep

t

May

June

July

Aug

Sep

t

May

June

July

Aug

Sep

t

May

June

July

Aug

Sep

t

Month

Yie

ld

Source●

●

●

●

ASBModel95% CI.u95% CI.l

Year 2012 Comparisons: Published Yield and Model−based Yield Indications


Weights applied in wheat forecast decomposition

May June July August September

0.00

0.25

0.50

0.75

1.00C

O IL KS

MO

MT

NE

OH

OK TX

WA

SP

EC

CO IL KS

MO

MT

NE

OH

OK TX

WA

SP

EC

CO IL KS

MO

MT

NE

OH

OK TX

WA

SP

EC

CO IL KS

MO

MT

NE

OH

OK TX

WA

SP

EC

CO IL KS

MO

MT

NE

OH

OK TX

WA

SP

EC

Unit

Wei

ght Source

AYS Covariates OYS Sept. APS

I Early season emphasis on covariates

I Increasing emphasis on OYS in July

I Heavy emphasis on last AYS in August

I Heavy emphasis on quarterly survey in September


Extensions and conclusions

1. NASS yield models (corn, soybeans, winter wheat)capture expert assessment in manner which isreproducible and provide justifiable measures ofuncertainty.

2. This methodology is flexible enough to accommodatemany types of auxiliary data.

I Additional commodities

I Non-spec region states

I New technologies, e.g., soil moisture monitors


Select references

Adrian, D. (2012). A model-based approach to forecasting corn and soybeanyields. Fourth International Conference on Establishment Surveys.

Cruze, N. B. (2015). Integrating survey data with auxiliary sources ofinformation to estimate crop yields. In JSM Proceedings, Survey ResearchMethods Section. Alexandria, VA: American Statistical Association.

Kass, R. and Steffey, D. (1989). Approximate Bayesian inference inconditionally independent hierarchical models (parametric empirical Bayesmodels). Journal of the American Statistical Association, 84(407):717–726.

Nandram, B., Berg, E., and Barboza, W. (2014). A hierarchical Bayesianmodel for forecasting state-level corn yield. Environmental and EcologicalStatistics, 21(3):507–530.

Wang, J. C., Holan, S. H., Nandram, B., Barboza, W., Toto, C., andAnderson, E. (2012). A Bayesian approach to estimating agricultural yieldbased on multiple repeated surveys. Journal of Agricultural, Biological, andEnvironmental Statistics, 17(1):84–106.


Thank you!Questions?

[email protected]


a bayesian hierarchical model for combining several crop ..._presentations_and...anderson, e....

Documents