template u.s. epa’s evaluation of long range transport models ralph e. morris environ...
TRANSCRIPT
TemplateTemplate
U.S. EPA’s Evaluation of Long Range Transport Models
Ralph E. MorrisENVIRON International Corporation
A&WMA CPANS ConferenceUniversity of Calgary
April 23-24, 2012
2773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-
899-0707 www.environcorp.com
ENVIRON’s work for USEPA LRT Model Evaluation Study
• Mesoscale Model Interface (MMIF) tool– Pass WRF/MM5 meteorological model output directly to
CALPUFF with minimal modification (CALMET not needed)• Document evaluation of CALPUFF and other LRT
dispersion models using tracer field study data– EPA performed all of the model simulations
• Perform a Consequence Analysis analyzing CALPUFF and CAMx for LRT AQ and AQRV
• Evaluate CALPUFF, SCICHEM and CAMx using atmospheric plume chemistry data
3
MMIF Tool for CALPUFF Met Inputs
• CALPUFF needs meteorological variables defined at grid cell centers
• WRF uses an Arakawa C grid configuration– U and V winds defined on
cell edges– Other met variables
defined on cell centers– Impossible to pass-through
winds directly to CALPUFF
• Some level of interpolation is needed– MMIF elected to interpolate
winds to cell centers
Arakawa C
(i, j)
T , p , q , K v(i, j)
(i, j)
(i, j-1 )
(i-1 , j) u , Kxu, Kx
v, Ky
v, Ky
4773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-
899-0707 www.environcorp.com
Inert Tracer Experiment Evaluation• LRT transport and dispersion algorithms evaluated using
inert tracer field experiments– Release known amount of inert tracer– Measure at downwind receptor sites– Evaluate LRT models ability to predict tracer concentrations
• Examples of past tracer experiment evaluation studies– 1986 8-model study– 1990 Rocky Mountain Acid Deposition Model Assessment– 1998 EPA evaluates CALPUFF using 2 tracer experiments– 1998 IWAQM Phase II recommendations– 1994 European Tracer Experiment (ETEX)
ATMES = real time model evaluation ATMES-II historical model evaluation
5773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-
899-0707 www.environcorp.com
USEPA LRT Tracer Test Evaluation
• EPA evaluated CALPUFF using five tracer test field study measurements:– 1975 Savannah River Laboratory (SRL75)– 1980 Great Plains (GP80)– 1983 Cross-Appalachia Tracer Experiment (CAPTEX)– 1994 European Tracer Experiment (ETEX)
• Compare CALPUFF GP80/SRL76 with1998 EPA study– Determine how advances in CALPUFF improve performance
• For some experiments compare multiple LRT dispersion models:– CALPUFF; HYSPLIT; FLEXPART; SCIPUFF; CAMx; CALGRID
6
GP80 and SRL 75 Tracer Experiments
GP80• Receptor arcs at 100 km and
600 km• Evaluate CALPUFF separately
on each arc using different configurations
• Compare with 1998 study
SRL75• One 100 km receptor arc• Evaluate CALPUFF using
different configurations• Compare with 1998 study
7
CAPTEX and ETEXCAPTEX (CTEX3 & CTEX5)
• Many CALMET configurations• Evaluate CALMET & MM5 &
CALPUFF• Evaluate LRT models using
default configuration
ETEX• Evaluate LRT models using
default configuration “out-of-the-box”
• Additional CAMx. HYSPLIT and CALPUFF sensitivity tests
8
Number LRT Dispersion Model RunsTest CALPUF
FSCIPU
FFHYSPL
ITFLXPR
TCAMx CALGR
D
GP80 100km 36 0 0 0 0 0
GP80 600km 41 0 0 0 0 0
SRL75 100km 21 0 0 0 0 0
CAPTEX
CTEX3a
25 0 0 0 0 0
CAPTEX
CYEX5a
24 0 0 0 0 0
CAPTEX
CTEX3b
1 1 9 1 16 1
CAPTEX
CTEX5b
1 1 9 1 16 1
ETEX 8 1 9 1 16 0
Total 157 3 27 3 48 3
9
Cross Appalachian Tracer Experiment (CAPTEX)
• 5 tracer releases from Dayton, OH or Sudbury, Canada – CTEX3 – Oct 2, 1983– CTEX5 – Oct 25-26, 1983
• Meteorological Evaluation– Evaluate MM5 and
CALMET– Identify best performing
CALMET settings– Used in to determine
USEPA’s recommended CALMET settings
• Six LRT Models Evaluated– CALPUFF & SCIPUFF– HYSPLIT & FLEXPART– CAMx & CALGRID
10
CTEX3: CALMET Sensitivity Tests Evaluation• 31 CALMET Sens Tests :
– MM5 @ 80, 36 & 12 km How MM5 is used within
CALMET– First Guess– Step 1 Winds– None
– CALMET @ 18, 12 & 4 km– RMAX1/RMAX2 (OA)
A = 500/1000 B = 100/200
– EPA Recommended C = 10/100 D = NOOBS
• 3 MMIF Sensitivity Tests:– MM5 @ 36, 12 & 4 km
• CALMETSTAT to evaluate wind speed and direction– Compare against common
benchmarks used to interpret MM5/WRF evaluation
Wind Speed Root Mean Squared Error (RMSE) Mean Normalized Bias (NMB) Index of Agreement (IOA)
≤ 2.0 m/s ≤ ±0.5 m/s ≥ 0.6
Wind Direction Mean Normalized Gross Error (MNGE) Mean Normalized Bias (MNB)
≤ 30° ≤ ±10°
Temperature Mean Normalized Gross Error (MNGE) Mean Normalized Bias (NMB) Index of Agreement (IOA)
≤ 2.0 K ≤ ±0.5 m/s ≥ 0.8
Humidity Mean Normalized Gross Error (MNGE) Mean Normalized Bias (NMB) Index of Agreement (IOA)
≤ 2.0 g/kg ≤ ±1.0 g/kg ≥ 0.6
11
WS & WD Bias/Error CALMET w/ 12 km MM5EXP4 & EXP6 = 12 & 4 km CALMET
A,B,C,D: RMAX1/RMAX2 = 500/1000, 100/200, 10/100, 0/0
-0.6-0.4-0.20.00.20.40.6
EXP
4A
EXP
4B
EXP
4C
EXP
4D
EXP
6A
EXP
6B
EXP
6C
EXP
6D
MM
5_12
km
WS Bias (ms^-1)IOAWS Gross Error (ms^-1)WS Bias (ms^-1)
-8-6-4-202468
EXP
4A
EXP
4B
EXP
4C
EXP
4D
EXP
6A
EXP
6B
EXP
6C
EXP
6D
MM
5_12
km
WD Bias (deg.)
0
0.5
1
1.5
2
2.5
EXP
4A
EXP
4B
EXP
4C
EXP
4D
EXP
6A
EXP
6B
EXP
6C
EXP
6D
MM
5_12
km
WS Gross Error (ms^-1)
0
5
10
15
20
25
30
35
EXP
4A
EXP
4B
EXP
4C
EXP
4D
EXP
6A
EXP
6B
EXP
6C
EXP
6D
MM
5_12
km
WD Gross Error (deg.)
WS Bias
WD Bias
WS Error WD Error
12773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-
899-0707 www.environcorp.com
Conclusions CTEX3 CALMET Model Performance• CTEX3 MM5 and CALMET wind model performance:
– The EPA-recommended settings of RMAX1/RMAX2 (100/200, “B” Series) produced best wind model performance.
– Use of 4 km grid resolution in CALMET tended to produce better wind model performance than 12 or 18 km. Can’t say anything about finer grid resolution than 4 km.
– The CALMET wind model performance was better using the 12 and 36 km MM5 data as input than using the 80 km MM4 data.
• CTEX5 also found RMAX1/RMAX2 = 100/200 best wind performance
• Note: Not an independent evaluation since WS/WD obs were also used as input to most CALMET tests
13
CTEX3: CALPUFF/CALMET Sens Tests• CALPUFF evaluation using
25 CALMET sensitivity tests– CALPUFF runs for CTEX3
tracer release– Evaluate CALPUFF using
ATMES-II 12 statistical performance metrics
– Determine which CALMET configuration results in best CALPUFF performance
• CALPUFF evaluation using 3 MMIF met inputs
• Evaluate using ATMES-II statistical performance metrics:– Spatial, Temporal & Global– Bias and Error– Scatter– Correlation– Cumulative DistributionStatistical Metric Definition
Perfect Score
Spatial Statistics
Figure of Merit in Space (FMS)
%100
PM
PM
AA
AAFMS
100%
Global Statistics
Normalized Mean Squared Error (NMSE)
21 ii MPPMN
NMSE
0%
Pearson’s Correlation Coefficient (PCC or R)
22
PPMM
PPMMR
ii
iii
1.0
Fractional Bias (FB) MPBFB 2 0%
Kolmogorov-Smirnov (KS) Parameter
kk PCMCMaxKS 0%
14
Two Composite Statistics for Ranking Models
• RANK combines statistics for correlation (R2), bias (FB), spatial (FMS) and cumulative distribution (KS) with perfect model giving score of 4.0
RANK statistics needs to be revised, propose replace FB w/ NMSE
• AVERAGE averages the N model’s rankings from 1 to N across the 11 ATMES-II statistics with the best model being model with lowest score closest to 1.0 – 1.0 = model is highest ranked model across all 11 ATMES-II
statistical metrics
14
%1005.0
N
NFOEX ii NiP
21 ii MPPMN
NMSE
kk PCMCMaxKS
100/1100/2/12 KSFMSFBRRANK
4.0
15
CALPUFF MPE using 12 km MM5EXP4 & EXP6 = 12 & 4 km CALMET
A,B,C,D: RMAX1/RMAX2 = 500/1000, 100/200, 10/100, 0/0
0%5%
10%15%20%25%30%35%40%
EXP
4A
EXP
4B
EXP
4C
EXP
4D
EXP
6A
EXP
6B
EXP
6C
EXP
6D
12K
MM
IF
Figure of Metric in Space (FMS)(Perfect = 100%)
0102030405060
EXP
4A
EXP
4B
EXP
4C
EXP
4D
EXP
6A
EXP
6B
EXP
6C
EXP
6D
12K
MM
IF
Normalized Mean Square Error (NMSE)
(Perfect = 0)
00.20.40.60.8
11.21.4
EXP
4A
EXP
4B
EXP
4C
EXP
4D
EXP
6A
EXP
6B
EXP
6C
EXP
6D
12K
MM
IF
Fractional Bias (FB)(Perfect = 0)
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
EXP4
A
EXP4
B
EXP4
C
EXP4
D
EXP6
A
EXP6
B
EXP6
C
EXP6
D
12KM
_MM
IF
Rank (RANK) (Perfect = 4)
(1-KS/100)
FMS/100
(1-FB/2)
R^2
16773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-
899-0707 www.environcorp.com
CTEX3 CALPUFF Evaluation Conclusions• The CALPUFF MMIF best performing for CTEX3
(worst for CTEX5).• The CALPUFF/CALMET RMAX1/RMAX2 = 100/200
worst performing RMAX1/RMAX2 configuration– Contrast to best performing CALMET configuration for
WS/WD• CALPUFF/CALMET with higher MM5 resolution
(36 and 12 km) performs better than using 80 km MM4 data.
• Generally, CALPUFF/CALMET using 4 km resolution performs better
17773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-
899-0707 www.environcorp.com
CTEX3 Six LRT Model Evaluation
• Use common 36 km MM5 meteorology in all models• Run 6 LRT models out-of-the-box using default
configuration• Evaluate using ATMES-II statistical metrics• 6 LRT dispersion models evaluated:
– Two Lagrangian Puff Models CALMET (w/ MMIF, best performing for CTEX3) and SCIPUFF
– Two Lagrangian Particle Models FLEXPART and HYSPLIT
– Two Eulerian grid models CAMx and CALGRID
20773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-
899-0707 www.environcorp.com
Conclusions of EPA LRT Evaluation (1 of 3)• GP80 Tracer Field Experiment
– Using different valid CALMET configurations, the maximum CALPUFF concentrations vary by factor of 3 Need standardized model configuration for regulatory modeling
– Since less options, less variation using MMIF– CALPUFF “SLUG” near-field option needed to reproduce “good”
model performance on 600 km arc from 1998 EPA study Raises question of validity of 1998 EPA study since SLUG typically not
used for LRT dispersion modeling
• SRL75 Tracer Field Experiment– Fitted Gaussian plume evaluation approach can be flawed
The a priori assumption that the observed plume has a Gaussian plume distribution may not be valid at LRT distances
21773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-
899-0707 www.environcorp.com
Conclusions of EPA LRT Evaluation (2 of 3)• CAPTEX Tracer Field Experiment
– RMAX1/RMAX2 = 100/200 (EPA recommendation) produces best CALMET WS/WD but worst CALPUFF tracer performance EPA needs help from modeling community to identify
regulatory CALMET settings– CALPUFF/MMIF performs better than
CALPUFF/CALMET for CTEX3 but worst for CTEX5– CTEX3: CAMx/SCIPUFF perform best followed by
CALPUF/FLEXPART with HYSPLIT/CALGRID worst– CTEX5: CAMx/HYSPLIT performs best followed by
SCIPUFF/FLEXPART with CALPUFF/CALGRID worst
22773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-
899-0707 www.environcorp.com
Conclusions of EPA LRT Evaluation (3 of 3)• ETEX Tracer Field Experiment
– Objective was to evaluate models run with default configuration using common 36 km MM5 inputs Exception was to use more all hours puff splitting in
CALPUFF rather than default of once per day
• CAMx, HYSPLIT & SCIPUFF perform best• FLEXPART & CALPUFF perform worst
– Sensitivity tests for CALPUFF, CAMx and HYSPLIT CALPUFF more aggressive puff splitting didn’t help CAMx found default configuration best performing HYSPLIT found some hybrid particle/puff configurations
perform better than particle-only default