Modeling Ground Delay Program Incidence using Convective and Local Weather Information Presenter: Yi Liu (Amazon) Co-authors (UC Berkeley): Mark Hansen, Danqing Zhang, Yulin Liu, Alexey Pozdnukhov

Funded by NASA under Award No NNX14AJ79A

ATM Seminar 2017

Outline

• Background • Data • Methodology • Results • Conclusions

2

3

Background • Similar Days is a NASA funded project

– 3 years, starting from 9/14 – Team members: UCB, UMD, and ATAC

• Project Goal: Support TMI decision making by finding and presenting pertinent information about historical days that are “similar” to a given reference day

• Similar Days tool identifies the similar days and presents information about daily weather, capacity, TMIs, and performance

• Applications include – Post-operational analysis – Day-of-operations decision making – Research

Objective in this research

• Predict the incidence of GDPs based on weather conditions – Help flight operators predict GDPs, which can

significantly impact their operations – Generate analytics for “significant” convective

weather used for assessing similarity between convective weather conditions

4

Contribution to the literature

• Previously, researchers have investigated the links between weather and GDP occurrence (Wang & Kulkarni 2011; Mukherjee et al. 2014; Bloem and Bambos 2015; Kuhn 2016)

• Our analysis is unique in its attention to the spatial patterns of convective weather

5

Scope of this analysis

• Our work focuses on the relation between current realized weather and GDP occurrence

• Use of our models with weather forecasts, rather than realized weather, will reduce their reliability to some extent, but we will not address this problem in this research

6

Outline

• Background • Data • Methodology • Results • Conclusions • Future Research

7

Data: hourly GDP label

FAA’s National Traffic Management Log

8

• Dependent variable

Data: airport local weather

• Aviation System Performance Metrics – visibility (statute mile) – ceiling (feet) – wind magnitude (knots) – wind direction – instrument meteorological conditions (IMC)

indicator

9

Data: airport local weather

• Aviation System Performance Metrics – visibility (statute mile) – ceiling (feet) – wind magnitude (knots) – wind direction – instrument meteorological conditions (IMC)

indicator

Tailwind/headwind Crosswind

5 airports: EWR, LGA, JFK, ATL, PHL Reference runways are: 22, 31, 31, 27, 27

10

Data: convective weather

• National Convective Weather Forecast (NCWF), providing coverage at national airspace system level

https://www.aviationweather.gov/products/ncwf/info 11

= 1

https://www.aviationweather.gov/products/ncwf/info

Data: convective weather

• Polygonal representations of convective areas provided in NCWF are discretized into geo-referenced image spanning the areas around the selected five airports

• A range of area sizes and resolutions were evaluated experimentally, resulting in a squared area of 200x200 square miles centered at the airport of interest, with a resolution of 200x200 pixels and one pixel per 1x1 square mile

12

200x200 sq miles

Data range

• The data range covers 2012 to 2014 with occasional gaps due to missing or incomplete data in one of the data sources for any of the airports

13

EWR

JFK

LGA

PHL

ATL

NO GDP

20305

(82.1%)

23160

(91.9%)

21588

(86.6%)

22521

(90.3%)

24355

(98.6%)

GDP

4418

(17.9%)

2055

(8.1%)

3354

(13.4%)

2426

(9.7%)

355

(1.4%)

Outline

• Background • Data • Methodology • Results • Conclusions • Future Research

14

Data mining techniques

• We use two data mining techniques: – Support Vector Machine (SVM); to understand

and quantify the impact of convective weather on GDP incidences where convective weather is represented using geo-referenced image

– logistic regression; to model GDP incidences using SVM output, airport local weather variables and airport busy hour (7 am to 10 pm; Gorripaty et al. 2016) indicator variable

15

SVM

16

200x200 GDP initiation

(C)

Tuning parameters

17

Performance metrics (AUC, F1-score and precision)

The samples of GDP and no GDP can overlap heavily in the feature space, favoring use of lower values of C, and, hence models that tolerate misclassification

Average Area under ROC Curve over 5-fold Cross-validation for EWR

Tuning parameters

18

Performance metrics (AUC, F1-score and precision)

Map overlay of the convective weather weight vector w for EWR model at C=1e-4

Average Area under ROC Curve over 5-fold Cross-validation for EWR

Spatial pattern of weights, C = 1e-4

Tuning parameters

19

Spatial pattern of weights, C =1

✔

Spatial pattern of weights, C = 1e-4

✗

Methodology (II)

• Estimate Binary Logit Model of GDP Occurrence – Dependent variable: GDP in effect in a given hour – Explanatory variables

• Ceiling, Visibility, IMC Dummy • Tailwind and Crosswind • Busy hour dummy (=1, if local departure hour is

between 7 am and 10 pm ) • Convective weather score from SVM

20

Why not modeling all features in SVM? Lead to imbalance in the dimensionality of the feature space Complicate the usability and interpretability of the weights from SVM

Outline

• Background • Data • Methodology • Results • Conclusions

21

SVM output— convective weather heatmap

22

One can observe spatially contingent pattern centered on the EWR and spread along the main east coast corridors.

23

1. Consistency of the observed pattern for all the airports of New York metroplex. 2. While the comparison confirms that weather closer to the airport (located at the center of the figure) is more important, it also reveals that the area of high impact is elongated along the northeast/northwest axis. This belies the assumption that weather impact depends only on the distance from the airport.

Logit Results—Dependent Variable: GDP in Effect

24

For the three airports within New York area, the magnitude of the convective weather variable is comparable ATL has the largest coefficient for convective weather variable indicating strong impact of convective weather on GDP at ATL

Variable

Airport

Est./ (Std.)

EWR

JFK

LGA

PHL

ATL

3.893***

4.511***

3.369***

2.773***

9.660***

(0.12)

(0.13)

(0.11)

(0.10)

(0.41)

0.789***

0.798***

1.366***

1.526***

0.23

(0.07)

(0.09)

(0.06)

(0.08)

(0.20)

-0.280***

-0.750***

-0.716***

-1.627***

-0.903***

(0.05)

(0.08)

(0.07)

(0.09)

(0.23)

-0.104***

-0.044***

-0.090***

-0.178***

0.061

(0.01)

(0.01)

(0.01)

(0.01)

(0.03)

0.017***

0.096***

0.024***

-0.034***

-0.058**

(0.01)

(0.01)

(0.01)

(0.01)

(0.02)

0.046***

0.065***

0.056***

0.044***

-0.038*

(0.00)

(0.01)

(0.00)

(0.01)

(0.02)

0.088***

0.033***

0.054***

0.071***

-0.047**

(0.00)

(0.01)

(0.01)

(0.01)

(0.02)

2.284***

2.267***

2.466***

3.767***

2.546***

(0.07)

(0.12)

(0.08)

(0.16)

(0.37)

Constant

-0.942***

-1.354***

-1.746***

-2.489***

2.399***

(0.14)

(0.20)

(0.15)

(0.20)

(0.60)

Observations

24723

25215

24942

24947

24710

Logit Results—Dependent Variable: GDP in Effect

25

Increased ceiling and visibility reduces the likelihood of GDP incidence, while the presence of IMC increases the likelihood of GDP

Variable

Airport

Est./ (Std.)

EWR

JFK

LGA

PHL

ATL

3.893***

4.511***

3.369***

2.773***

9.660***

(0.12)

(0.13)

(0.11)

(0.10)

(0.41)

0.789***

0.798***

1.366***

1.526***

0.23

(0.07)

(0.09)

(0.06)

(0.08)

(0.20)

-0.280***

-0.750***

-0.716***

-1.627***

-0.903***

(0.05)

(0.08)

(0.07)

(0.09)

(0.23)

-0.104***

-0.044***

-0.090***

-0.178***

0.061

(0.01)

(0.01)

(0.01)

(0.01)

(0.03)

0.017***

0.096***

0.024***

-0.034***

-0.058**

(0.01)

(0.01)

(0.01)

(0.01)

(0.02)

0.046***

0.065***

0.056***

0.044***

-0.038*

(0.00)

(0.01)

(0.00)

(0.01)

(0.02)

0.088***

0.033***

0.054***

0.071***

-0.047**

(0.00)

(0.01)

(0.01)

(0.01)

(0.02)

2.284***

2.267***

2.466***

3.767***

2.546***

(0.07)

(0.12)

(0.08)

(0.16)

(0.37)

Constant

-0.942***

-1.354***

-1.746***

-2.489***

2.399***

(0.14)

(0.20)

(0.15)

(0.20)

(0.60)

Observations

24723

25215

24942

24947

24710

Logit Results—Dependent Variable: GDP in Effect

26

The tailwind seems to have mixed effects on GDP for different airports, but headwind and crosswind both increase the likelihood of GDP, except for ATL

Variable

Airport

Est./ (Std.)

EWR

JFK

LGA

PHL

ATL

3.893***

4.511***

3.369***

2.773***

9.660***

(0.12)

(0.13)

(0.11)

(0.10)

(0.41)

0.789***

0.798***

1.366***

1.526***

0.23

(0.07)

(0.09)

(0.06)

(0.08)

(0.20)

-0.280***

-0.750***

-0.716***

-1.627***

-0.903***

(0.05)

(0.08)

(0.07)

(0.09)

(0.23)

-0.104***

-0.044***

-0.090***

-0.178***

0.061

(0.01)

(0.01)

(0.01)

(0.01)

(0.03)

0.017***

0.096***

0.024***

-0.034***

-0.058**

(0.01)

(0.01)

(0.01)

(0.01)

(0.02)

0.046***

0.065***

0.056***

0.044***

-0.038*

(0.00)

(0.01)

(0.00)

(0.01)

(0.02)

0.088***

0.033***

0.054***

0.071***

-0.047**

(0.00)

(0.01)

(0.01)

(0.01)

(0.02)

2.284***

2.267***

2.466***

3.767***

2.546***

(0.07)

(0.12)

(0.08)

(0.16)

(0.37)

Constant

-0.942***

-1.354***

-1.746***

-2.489***

2.399***

(0.14)

(0.20)

(0.15)

(0.20)

(0.60)

Observations

24723

25215

24942

24947

24710

Factor contribution

27

0%

20%

40%

60%

80%

100%

JFK LGA EWR PHL ATL

Factor Contributions

Wx IMC C Vis TW HW CW

• We construct a counter factual scenario in which each factor is set to the value in the dataset that would minimize the GDP probability, and use the logit model to predict the corresponding GDP probability for each record

• Contribution: % change between the predicted and observed GDP probability

Convective weather information is indispensable to predicting GDPs

Conclusions

• We have modeled GDP occurrence using convective weather in region surrounding airport and local airport weather conditions

• We find that the SVM, when properly tuned, provides a reasonable and spatially coherent picture of where convective weather is important

• The importance of convective weather depends on both its distance and direction from the airport

• Local and regional convective weather both have significant impact on GDP probability, but the latter is most important

28

Modeling Ground Delay Program Incidence using Convective and Local Weather Information�OutlineBackgroundObjective in this researchContribution to the literatureScope of this analysisOutlineData: hourly GDP labelData: airport local weather Data: airport local weather Data: convective weather Data: convective weather Data rangeOutlineData mining techniquesSVMTuning parametersTuning parametersTuning parametersMethodology (II)OutlineSVM output—�convective weather heatmapSlide Number 23Logit Results—Dependent Variable: GDP in EffectLogit Results—Dependent Variable: GDP in EffectLogit Results—Dependent Variable: GDP in EffectFactor contributionConclusions