(8 pages) IAVWOPSG.8.IP.003.6.en.docx

INTERNATIONAL AIRWAYS VOLCANO WATCH OPERATIONS GROUP

(IAVWOPSG)

EIGHTH MEETING

Melbourne, Australia, 17 to 20 February 2014

Agenda Item 6: Development of the IAVW (Deliverables 06, 07 and 10)

6.1: Improvement of tools for detecting and forecasting volcanic ash (Deliverable 06)

QUANTITATIVE VERIFICATION OF VOLCANIC ASH DISPERSION MODEL

(Presented by Japan)

SUMMARY

This paper presents a method being developed by VAAC Tokyo to verify the accuracy of volcanic ash dispersion models including quantitatively. This method enables to quantify the accuracy of a model and will contribute to its verification and/or parameter tuning. To achieve the quantification, indices such as threat scores are calculated by comparing model outputs and corresponding satellite imageries. Action by the IAVWOPSG is in paragraph 4.

1. INTRODUCTION

1.1 It is important to improve the accuracy of volcanic ash dispersion models. In this regard, the group will recall the discussion at the seventh meeting of the International Airways Volcano Watch Operations Group (IAVWOPSG/7) in Bangkok on dispersion model output uncertainty as noted in the final report. The group formulated the following conclusion:

Conclusion 7/18 — Reducing dispersion model output

uncertainty That in an effort to improve volcanic ash dispersion forecasts, the WMO-IUGG VASAG be invited to:

a) encourage research to quantify and reduce dispersion model output uncertainty in view of supporting operational decision making within the framework of the international airways volcano watch (IAVW);

IAVWOPSG/8-IP/3 15/1/14

IAVWOPSG/8-IP/3

- 2 -

b) evaluate the list in Appendix K to this report in order to prioritize which task(s) would provide the greatest benefit in mitigating the uncertainty in modeling; and

c) report back to the IAVWOPSG/8 meeting.

1.2 To underpin these efforts, quantitative verification of the model output is essential since it enables to assess the improvement objectively.

1.3 To this end, volcanic ash advisory centre (VAAC) Tokyo has been developing a quantitative verification method. Details of the method and its anticipated application are presented in the following sections.

2. DISCUSSION

2.1 Verification Method

2.1.1 Stunder et al. calculated the threat score (TS), or a statistic representing the degree of overlap, of some pairs of forecast-analysed ash clouds to evaluate the reliability of HYSPLIT model1.

2.1.2 Based on this study, VAAC Tokyo has been developing its new verification method where polygons depicted in satellite imageries by forecasters are regarded as observed ash clouds while the distribution of the tracers in the dispersion model is regarded as forecast ash clouds.

2.1.3 Figure 1 illustrates an observed and forecast ash cloud emitted from the Mt. Sarychev Peak that started to erupt on 12 June 2009 with significant emission of volcanic ash. The satellite image was taken at 2300 UTC on 16 June 2009, and a red polygon is an observed ash cloud identified by a forecaster. Coloured dots in the right-side image represent a forecast ash cloud with valid time T+12 hours.

Figure 1: Observed and forecast ash cloud emitted from the Mt. Sarychev Peak

1 Stunder et al., 2007: Airborne Volcanic Ash Forecast Area Reliability. Wea. Forecasting, 22, pp. 1132--1139.

IAVWOPSG/8-IP/3

- 3 -

2.1.4 To verify the accuracy of the model output, the image is divided into grids around 20 km on a side by applying Reduced Gaussian Grids. While each grid is regarded as “Forecast” where one or more tracers are located, it is regarded as “Observed” where a part of the depicted polygon is included.

2.1.5 Thus every grid is categorized into four types: “Forecast and Observed”, “Forecast and Not Observed”, “Not Forecast and Observed” and “Not Forecast and Not Observed” as shown in Table 1, where the number of grids categorized as each type is represented by FO, FX, XO and XX, respectively.

Observed

True False

Forecast True FO FX

False XO XX

Table 1 : Categorization of grids

2.1.6 In accordance with this categorization, Figure 1 can be converted into the grid mapping image as illustrated in Figure 2, where red, blue, purple and white grids represent “Forecast and Observed”, “Forecast and Not Observed”, “Not Forecast and Observed” and “Not Forecast and Not Observed”, respectively.

Figure 2: Categorization of each grid

2.1.7 By using FO, FX and XO from Table 1, following four types of scores can be defined:

a) Threat Score (TS): Ratio of “Forecast and Observed” to “Forecast or Observed” running from 0 through 1, where the larger, the more accurate the dispersion model;

Red: Forecast and Not Observed Blue: Not Forecast and Observed Purple: Forecast and Observed White: Not Forecast and Not Observed

IAVWOPSG/8-IP/3

- 4 -

b) Undetected Error Rate (UR): Ratio of “Not Forecast and Observed” to “Observed” running from 0 through 1, where the smaller, the better;

c) False Alarm Ratio (FAR): Ratio of “Forecast and Not Observed” to “Forecast” running from 0 through 1, where the smaller, the better; and

d) Bias Score (BS): Ratio of “Forecast” to “Observed” running from 0 through 1 when forecast too small whereas from 1 through infinity when forecast too large, where the closer to 1, the better.

2.1.8 In the case of Figure 2, where FO = 2836, FX = 419 and XO = 1642, the scores are calculated to be TS = 0.58, UR = 0.37, FAR = 0.13 and BS = 0.73 respectively.

2.2 Application

2.2.1 This quantitative verification could be applied to a wide range of issues regarding the development of the international airways volcano watch (IAVW). For example, when updating a dispersion model it could allow for a preliminary assessment of the new model. The verification method would also be beneficial to evaluate different models when conducting a benchmark test.

2.2.2 In addition, this method can be useful to discuss issues of greater interest regarding the IAVW, such as volcanic ash information beyond T+18 hours. An example of the verification of such information is available in the appendix to this paper.

2.2.3 Further work is needed to take into account the vertical distribution of forecast/observed volcanic ash.

3. CONCLUSION

3.1 This information paper has introduced the quantitative verification method to evaluate the accuracy of volcanic ash dispersion models, which has been under development by VAAC Tokyo. That can be achieved by calculating scores such as TS from the categorization of grids according to whether they are forecast and/or observed as those contaminated by ash. This method could contribute to the enhancement of the model accuracy, allowing for the parameter tuning or comparison of models.

IAVWOPSG/8-IP/3

- 5 -

4. ACTION BY THE IAVWOPSG

4.1 The IAVWOPSG is invited to note the information contained in this paper.

— — — — — — — —

IAVWOPSG/8-IP/3 Appendix

APPENDIX

CASE STUDY ON QUANTITATIVE VERIFICATION OF

VOLCANIC ASH FORECAST BEYOND T+18 HOURS

Test Case

Volcanic ash emitted from Sarychev Peak at 1100 UTC on June 16 2009 taken by MTSAT-1R, or JMA’s geostationary satellite, is selected as an initial location (T+0) of the test case. The forecast movement of the ash cloud is computed by JMA Global Atmospheric Transport Model (JMA-GATM).

Figures on the left and right sides are satellite imageries and model output images at every 6 hours up to 2300 UTC on June 17 2009 respectively. Colored dots indicate forecast ash tracer distribution, and red polygons represent observed ash clouds from satellite imageries depicted by a forecaster. Each pair of figures is followed by Threat Score (TS), Undetected Error Rate (UR), False Alarm Ratio (FAR) and Bias Score (BS).

At the end of this appendix, the time series variation of these scores is provided. A drastic increase of FAR and BS reveals JMA-GATM’s tendency to forecast “too much” as the forecast range extends.

T+0: 2009/06/16 1100Z (initial location)

T+6: 2009/06/16 1700Z

TS=0.53 UR=0.42 FAR=0.13 BS=0.67

IAVWOPSG/8-IP/3 Appendix

A-2

T+12: 2009/06/16 2300Z

TS=0.58 UR=0.37 FAR=0.13 BS=0.73 T+18: 2009/06/17 0500Z

TS=0.54 UR=0.28 FAR=0.32 BS=1.06 T+24: 2009/06/17 1100Z

TS=0.45 UR=0.32 FAR=0.43 BS=1.21

A-3

IAVWOPSG/8-IP/3 Appendix

T+30: 2009/06/17 1700Z

TS=0.40 UR=0.32 FAR=0.51 BS=1.39 T+36: 2009/06/17 2300Z

TS=0. 40 UR=0. 28 FAR=0. 53 BS=1.53 Time series variation of scores

— END —

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

6 12 18 24 30 36

Sco

re

Time Step (hr)

TS

UR

FAR

BS