eric gilleland weather systems assessment program research ...ericg/talks/gilleland2017vcu.pdf ·...
TRANSCRIPT
![Page 1: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/1.jpg)
Verifying NARCCAP Models for Severe-Storm Environments Eric Gilleland
Weather Systems Assessment Program Research Applications Laboratory,
Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University, Richmond, Virginia
3 March 2017 Co-authors: Christopher L. Williams, Melissa Bukovsky, Seth McGinnis, Barb Brown, Linda Mearns, and Caspar Ammann Support for this work provided by the NSF via the Weather and Climate Impacts Assessment Science Program (http://www.assessment.ucar.edu) and Earth System Modeling (EaSM) Grant number AGS-1243030.
![Page 2: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/2.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Photo by Everett Nychka
Study and Visit Opportunities https://www2.ucar.edu/opportunities
But, also talk to Montse about STATMOS!
![Page 3: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/3.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
●
●
●
●
●
●
Estes Park
Boulder
Fort Collins
DenverDIAGolden
NCAR/NCEP reanalysisCCSM3 Global Climate Model
![Page 4: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/4.jpg)
Severe Storm Environments
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
0.00
000
0.00
010
0.00
020
Non−severeSevereSignificant Non−tornadicSignificant Tornadic
0.00
00.
001
0.00
20.
003
0e+00 2e+05 4e+05
0.0
0.2
0.4
0.6
0.8
1.0
0 2000 6000 10000
0.0
0.2
0.4
0.6
0.8
1.0
CAPE × Shear (J kg-1 × m s-1)
Wmax × Shear (WmSh, m2 s-2)
Convective Available Potential Energy
0 – 6 km vertical wind shear
Maximum updraft velocit (Wmax,, ms-1) = (2 * CAPE)1/2
![Page 5: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/5.jpg)
NCEP reanalysis
Community Climate System Model
3rd Generation Coupled Global Climate Model
Hadley Centre Coupled Model, v. 3
abbreviation NCEP CCSM3 CGCM3 HadCM3
Canadian Regional Climate Model (CRCM)
X
X
X
Hadley Regional Model 3 (HRM3)
X
Pennsylvania State University/NCAR mesoscale model (MM5I)
X
X
Weather Research and Forecasting model (WRFG)
X
X
X
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
http://www.narccap.ucar.edu/ http://www.emc.ncep.noaa.gov/mmb/rreanl/
All are interpolated to be on the same grid, which is ≈ 0.5o
![Page 6: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/6.jpg)
Lingo
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
q75: Univariate time series giving the upper quartile of CAPE or WmSh over space at each time point.
WmSh: As before, but set to zero if CAPE < 100 J kg-1 or 5 ≤ Shear ≤ 50 ms-1
High “field energy”: when q75 > its 90th percentile over time.
κ: Frequency of CAPE ≥ 1000 J kg-1 conditioned on the presence of high field energy.
ω: Frequency of WmSh ≥ 225 m2s-2 conditioned on the presence of high field energy.
![Page 7: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/7.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
NARR CRCM−CCSM CRCM−CGCM3
HRM3−HadCM3 MM5I−CCSM MM5I−HadCM3
WRFG−CCSM WRFG−CGCM3 CRCM−NCEP
WRFG−NCEP
0.0 0.2 0.4 0.6 0.8 1.0
κ
![Page 8: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/8.jpg)
NARR CRCM−CCSM CRCM−CGCM3
HRM3−HadCM3 MM5I−CCSM MM5I−HadCM3
WRFG−CCSM WRFG−CGCM3 CRCM−NCEP
WRFG−NCEP
0.0 0.2 0.4 0.6 0.8 1.0UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
ω
![Page 9: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/9.jpg)
Spatial Forecast Verification
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Fig. 1 and Table 2 from Ahijevych et al. (2009, WAF, 24, 1485 – 1497)
All identical measures! • Traditional Verification does not
provide diagnostic information • Often favors coarser scale models
• double penalty • aggregation of small-scale errors
![Page 10: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/10.jpg)
Spatial Forecast Verification
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Fig. 2 from G. et al. (2010, BAMS, 91 (10), 1365 – 1373)
List of papers: http://www.ral.ucar.edu/projects/icp/references.html
• Numerous papers rapidly introduced new methods • image analysis • computer vision • shape analysis • spatial statistics
• ICP invoked to get a handle on the methods • geometric and real
cases • precipitation over
central United States • Most methods fall into
one of 4 categories • MesoVICT continuation of
ICP • complex terrain • More variables • Ensembles (obs and
model)
![Page 11: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/11.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Mean Error Distance
d(x, B | x in A) d(x, A | x in B)
MED(A, B) = Σx d(x, B | x in A) / N
A B
centroid distance
MED(A, B) = Σx d(x, A | x in B) / N N is the size of the domain
= 80 MED(A, B) is the average distance from points in the set B to points in the set A
40 50 60 70 80
![Page 12: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/12.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Baddeley’s Δ Metric
d(x, A)
0 50 100 150 200
d(x, B)
Distance maps for A and B. Note dependence on location within the domain.
![Page 13: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/13.jpg)
0 20 40 60 80
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Baddeley’s Δ Metric Τ= | d(x, A) – d(x, B) |
Δ(B, A) = Δ(B, A) = [Σx in Domain | d(x, A) – d(x, B) |p ]1/p / N N is the size of the domain
• p = 1 gives the arithmetic average of Τ
• p = 2 is the usual choice • p = ∞ gives the max of Τ
(Hausdorff distance) Δ is the Lp norm of Τ
d(x, A) and d(x, B) are first transformed by a function ω. Usually, ω(x) = max( x, constant), but all results here use ∞ for the constant term.
![Page 14: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/14.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Contrived Examples: Circles
0 50 100 150 200
050
100
150
200
1 2 34
Touching the edge of the domain
All circles have radius = 20 grid squares
Domain size is 200 by 200
![Page 15: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/15.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
0 50 100 150 200
050
100
150
200
1 2 34
A B MED(A, B) rank MED(B, A) rank Δ(A, B) rank cent. dist.
rank
1 2 22 2 22 1 29 2 40 2 1 3 62 4 62 3 57 6 80 4 1 4 38 3 38 2 41 5 57 3 2 3 22 2 22 1 31 3 40 2 2 4 22 2 22 1 28 1 40 2 2 1, 3, 4 11 1 22 1 29 2 13 1 3 4 38 3 38 2 38 4 57 3
Contrived Examples: Circles
![Page 16: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/16.jpg)
0 50 100 150 200
050
100
150
200
A B
MED(A, B) = 32.30MED(B, A) = 27.68
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Circle and a Ring
MED(A, B) = 32 MED(B, A) = 28 Δ(A, B) = 38 centroid distance = 0
![Page 17: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/17.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Mean Error Distance
Missed Areas
MED(ST2, ARW) ≈ 15.42 is much smaller than MED(ARW, ST2) ≈ 66.16
Fig. 2 from G. (2016 submitted to WAF, available at: http://www.ral.ucar.edu/staff/ericg/Gilleland2016.pdf)
High sensitivity to small changes in the field! Good or bad quality depending on user need.
![Page 18: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/18.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Geometric ICP Cases
Table from part of Table 1 in G. (2016, submitted to WAF) Fig. 1 from Ahijevych et al. (2009, WAF, 24, 1485 – 1497)
Case MED(A, Obs) rank MED(Obs, A) rank
1 29 2 29 1
2 180 5 180 5
3 36 3 104 3
4 52 4 101 2
5 1 1 114 4
Values rounded to zero decimal places
Avg. Distance from green to pink
Avg. Distance from pink to green
![Page 19: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/19.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Geometric ICP Cases
Table from part of Table 1 in G. (2016, submitted to WAF) Fig. 1 from Ahijevych et al. (2009, WAF, 24, 1485 – 1497)
Case MED(A, Obs) rank MED(Obs, A) rank
1 29 2 29 1
2 180 5 180 5
3 36 3 104 3
4 52 4 101 2
5 1 1 114 4
Values rounded to zero decimal places
Case Δ(A, Obs) rank 1 45 1 2 167 5 3 119 3 4 106 2 5 143 4
![Page 20: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/20.jpg)
• Magnitude of MED tells how good or bad the “misses/false alarms” are.
• Miss = Average distance of observed non-zero grid points from forecast. § Perfect score: MED(Forecast, Observation) = zero (no misses at all)
• All observations are within forecasted non-zero grid point sets. § Good score = Small values of MED(Forecast, Observation)
• all observations are near forecasted non-zero grid points, on average.
• False alarm = Average distance of forecast non-zero grid points from observations. § Perfect score: MED(Observation, Forecast) = zero (no false alarms at all)
• All forecasted non-zero grid points fall overlap completely with observations. § Good score = Small values of MED(Observation, Forecast)
• all forecasts are near observations, on average.
• Hit/Correct Negative § Perfect Score: MED(both directions) = 0 § Good Value = Small values of MED(both directions)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Mean Error Distance
![Page 21: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/21.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
κ
Misses
Fals
e A
larm
s
0 2 4 6 8 10 14
02
46
8
CRCM−CCSM3
●
0.0 0.5 1.0 1.5 2.0 2.5
0.0
0.5
1.0
1.5
2.0
CRCM−CGCM3
●
0.00 0.02 0.04 0.06
0.0
0.4
0.8
1.2
HRM3−HadCM3
●
0 2 4 6 8 10 12
02
46
810
MM5I−CCSM3
●
0.0 0.5 1.0 1.5 2.0
0.0
0.4
0.8
1.2
MM5I−HadCM3
●
0.0 0.5 1.0 1.5 2.0 2.5 3.0
01
23
WRFG−CCSM3
●
0 1 2 3 4 5 6 7
0.0
0.5
1.0
1.5
WRFG−CGCM3
●
0.0 0.2 0.4 0.6 0.8
0.0
0.4
0.8
1.2
CRCM−NCEP
●
0.0 0.5 1.0 1.5
0.0
0.4
0.8
1.2
WRFG−NCEP
●
Note the Scales
Most models are closer to the NARR on average than the NARR is to them (more “misses” than “false alarms”). HRM3-HadCM3 and CRCM-NCEP are exceptions, but both have very small average distances in both directions
![Page 22: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/22.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
κ
Misses
Fals
e A
larm
s
0 2 4 6 8 10 14
02
46
8
CRCM−CCSM3
●
0.0 0.5 1.0 1.5 2.0 2.5
0.0
0.5
1.0
1.5
2.0
CRCM−CGCM3
●
0.00 0.02 0.04 0.06
0.0
0.4
0.8
1.2
HRM3−HadCM3
●
0 2 4 6 8 10 12
02
46
810
MM5I−CCSM3
●
0.0 0.5 1.0 1.5 2.0
0.0
0.4
0.8
1.2
MM5I−HadCM3
●
0.0 0.5 1.0 1.5 2.0 2.5 3.0
01
23
WRFG−CCSM3
●
0 1 2 3 4 5 6 7
0.0
0.5
1.0
1.5
WRFG−CGCM3
●
0.0 0.2 0.4 0.6 0.8
0.0
0.4
0.8
1.2
CRCM−NCEP
●
0.0 0.5 1.0 1.5
0.0
0.4
0.8
1.2
WRFG−NCEP
●
0.95 quantile threshold
![Page 23: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/23.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
κ
Misses
Fals
e A
larm
s
0 2 4 6 8 10 14
02
46
8
CRCM−CCSM3
●
0.0 0.5 1.0 1.5 2.0 2.5
0.0
0.5
1.0
1.5
2.0
CRCM−CGCM3
●
0.00 0.02 0.04 0.06
0.0
0.4
0.8
1.2
HRM3−HadCM3
●
0 2 4 6 8 10 12
02
46
810
MM5I−CCSM3
●
0.0 0.5 1.0 1.5 2.0
0.0
0.4
0.8
1.2
MM5I−HadCM3
●
0.0 0.5 1.0 1.5 2.0 2.5 3.0
01
23
WRFG−CCSM3
●
0 1 2 3 4 5 6 7
0.0
0.5
1.0
1.5
WRFG−CGCM3
●
0.0 0.2 0.4 0.6 0.8
0.0
0.4
0.8
1.2
CRCM−NCEP
●
0.0 0.5 1.0 1.5
0.0
0.4
0.8
1.2
WRFG−NCEP
●
0.9 quantile threshold
![Page 24: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/24.jpg)
MED Summary • Mean Error Distance
§ Useful summary when applied in both directions § New idea of false alarms and misses (spatial context) § Computationally efficient and easy to interpret
• Properties § High sensitivity to small changes in one or both fields § Does not inform about bias per se
• Could hedge results by over forecasting, but only if over forecasts are in the vicinity of observations!
§ No edge or position effects (unless part of object goes outside the domain) § Does not inform about patterns of errors § Does not directly account for intensity errors (only location) § Fast and easy to compute and interpret
• Complementary Methods include (but not limited to) § Frequency bias (traditional) § Geometric indices (AghaKouchak et al 2011, doi:10.1175/2010JHM1298.1)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
![Page 25: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/25.jpg)
Baddeley’s Δ Metric Summary
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
• Sensitive to differences in size, shape, and location • A proper mathematical metric (therefore, amenable to
ranking) • positivity (Δ(A, B) ≥ 0 for all A and B) • identity (Δ(A, A) = 0 and Δ(A, B) > 0 if A ≠ B) • symmetry (Δ(A, B) = Δ(B, A)) • triangle inequality (Δ(A, C) ≤ Δ(A, B) + Δ(B, C))
• Sensitive to position within the domain, edge effects, and orientation between two objects (so, when ranking, need to be careful if values are close) • For single object comparisons, perhaps could be overcome by
centering and rotating (the pair of objects together) and calculating within a bounding box. Future work!
• Unbounded upper limit! (i.e., Δ(A, B) in [0, ∞)) • Can be alleviated by proper normalization (as is done here). • Need to take care when ranking anyway because of above issues.
![Page 26: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/26.jpg)
Centroid Distance Summary • Is a true mathematical metric. So, conducive to
rankings. • Not sensitive to position within a field (or orientation
of A to B; i.e., if A and B are rotated as a pair, the distance does not change)
• No edge effects • Gives useful information for translation errors
between objects that are similar in size, shape and orientation.
• Not as useful otherwise. • Should be combined with other information.
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
![Page 27: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/27.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Spatial Forecast Verification Image Warping
UCAR Confidential and Proprietary. © 2014, University Corporation for Atmospheric Research. All rights reserved.
Graphic by Johan Lindström
Observed Image (O(s)) Forecast Image (F(s))
Warped Image (F(W(s)))
![Page 28: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/28.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Pair of thin-plate spline transformations Φ(s) = (Φ1(s), Φ2(s))T = a + Gs + WT Ψ(s – p0)
x-coordinate y-coordinate
affine transformation
Nonlinear transformations
Ψ(h) = ||h||2 log ||h||
Image Warping
Columns of coefficients in W and the sum of products of W times p0 both constrained to sum to 0.
![Page 29: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/29.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Pair of thin-plate spline transformations Φ(s) = (Φ1(s), Φ2(s))T = a + Gs + WT Ψ(s – p0)
Image Warping
LA =
Ψ k1 0pkT1 0 0
0
Tp 0 0
⎡
⎣
⎢⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥⎥
WTaTG
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
=1p00
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
Want L-1. The upper k × k matrix of L-1, call it L11, gives the bending energy matrix. And W = L11p1. The bending energy is given by trace( p1
T L11p1).
![Page 30: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/30.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Image Warping
k parameters of interest are the locations p1.
Q1p( ) = 1
2Nσ2
Z W (s)( )−Z(s)( )s=1,N∑ +
βT
1,xp − 0p( ) 11L 1,xp − 0p( ) +T
1,yp − 0p( ) 11L 1,yp − 0p( )⎡
⎣⎢
⎤
⎦⎥
Found by numerically optimizing the objective function:
Ideally, want to find the optimal deformation without hand-selecting control points!
RMSE of deformed Forecast against observation
Penalty for too much warping and too much bending
User-chosen penalty parameter. Controls how much bending and deformation can happen
![Page 31: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/31.jpg)
Image Warping
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
![Page 32: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/32.jpg)
Image Warping
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
![Page 33: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/33.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
% error reduction ≈ 40% minimum bending energy = 2.0042
RMSE0 = 0.2665 RMSE1 = 0.1605
Image Warping
−120 −100 −90 −80 −70
3035
4045
0−energy field
0.0 0.2 0.4 0.6 0.8 1.0
−120 −100 −90 −80 −70
3035
4045
1−energy field
0.0 0.2 0.4 0.6 0.8 1.0
−120 −100 −90 −80 −70
3035
4045
Error Field
−0.8 −0.4 0.0 0.2 0.4
−120 −100 −90 −80 −70
3035
4045
Distance Travelled
5 10 15 20
−120 −100 −90 −80 −70
3035
4045
Deformed 1−energy field
0.0 0.2 0.4 0.6 0.8 1.0
−120 −100 −90 −80 −70
3035
4045
Error Field(after warping)
−0.8 −0.4 0.0 0.2 0.4
MM5I-CCSM3 κ
Important for later
![Page 34: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/34.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Image Warping
RMSE0 RMSE1 RMSE Reduction
Minimum Bending Energy
CRCM-CCSM3 0.214 0.139 35% 0.96 CRCM-CGCM3 0.147 0.103 30% 1.07 HRM3-HadCM3 0.157 0.110 30% 0.25 MM5I-CCSM3 0.267 0.161 40% 2.00 MM5I-HadCM3 0.148 0.084 43% 0.69 WRFG-CCSM3 0.249 0.096 61% 3.27 WRFG-CGCM3 0.241 0.092 62% 3.32 CRCM-NCEP 0.214 0.173 19% 0.25 WRFG-NCEP 0.171 0.092 46% 0.43
![Page 35: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/35.jpg)
Spatial Prediction Comparison Test
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
D1 D2 Hering and Genton (2011, Technometrics, 53, (4): 414—425)
No significant results for these verification sets using standard SPCT.
![Page 36: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/36.jpg)
Spatial Prediction Comparison Test
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
AE + distance map loss G. (2013, MWR, 141 (1), 340 – 355) Case 16 Distance Map Case 18 Distance Map
0 50 100 150
Absolute Difference of Distance Maps
0 10 20 30 40 50 60 70
5 10 15 20
020
4060
80
Variograms of Distance Maps
distance (grid squares)
Vario
gram
●●●●●●●●●●●●
●●●●●●●●●●
●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●Case 16 distance map
Case 18 distance mapAbsolute Differences
![Page 37: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/37.jpg)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
Model 1 Model 2 SPCT Statistic
p-value
CRCM-CCSM3 CRCM-CGCM3 -1.24 0.21 CRCM-CCSM3 HRM3-HadCM3 1.15 0.25 CRCM-CGCM3 HRM3-HadCM3 1.66 0.10 HRM3-HadCM3 MM5I-CCSM3 -1.71 0.09 HRM3-HadCM3 WRFG-CCSM3 -1.45 0.15 HRM3-HadCM3 WRFG-CGCM3 -3.06 0.002 MM5I-CCSM3 WRFG-CGCM3 -2.12 0.03 MM5I-HadCM3 WRFG-CGCM3 -1.45 0.15 WRFG-CGCM3 WRFG-NCEP 1.42 0.16
AE + deformation loss G. (2013, MWR, 141 (1), 340 – 355) ω
![Page 38: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/38.jpg)
Conclusions • Models generally agree with NARR about spatial location and
overall pattern of high severe storm frequencies (κ and ω). • They tend to under-project the spatial extent of high frequency
areas compared to NARR. • HRM3-HadCM3 is by far the closest to NARR for both κ and ω. • WRFG configurations not coupled with NCEP (i.e.,
“observations”) have the least agreement with NARR. • Climate models should reproduce observed distributional
properties for the current-period climate, making spatial forecast verification methods particularly useful, and easy to implement in this context.
• Full analysis including many other spatial methods in G. et al. (submitted to ASCMO, available at http://www.ral.ucar.edu/staff/ericg/GillelandEtAl2016.pdf)
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.
![Page 39: Eric Gilleland Weather Systems Assessment Program Research ...ericg/Talks/Gilleland2017VCU.pdf · Seminar at: Statistical Sciences and Operations Research, Virginia Commonwealth University,](https://reader034.vdocument.in/reader034/viewer/2022051607/603417901c99c3688b6aab6a/html5/thumbnails/39.jpg)
Thank you. Questions?
• http://www.ral.ucar.edu/staff/ericg • Test cases for part 2 of ICP (MesoVICT)
§ http://www.ral.ucar.edu/projects/icp § Ensembles of models § Ensembles of observations § Precipitation, wind § complex terrain § point observations + re-analysis product
UCAR Confidential and Proprietary. © 2017, University Corporation for Atmospheric Research. All rights reserved.