Using Program Generation to Support V&V in the Next Generation Air

Transportation System Ewen Denney

(joint work with Xavier Bouyssounouse

Johann Schumann

Karen Tung-Cate)

August, 2007

NGATS

• US air transportation system (airports and airspace) rapidly reaching capacity

• Major effort to develop integrated solutions for a safe, efficient, and high-capacity airspace system– Dynamic Airspace Configuration– Traffic Flow Management– Separation Assurance – Safety analysis

Trajectory Prediction

Trajectory Prediction predicts the future progress of individual aircraft on the basis of

• the current aircraft state,• estimates of pilot and controller intent,• expected environmental conditions, and• models of aircraft performance and procedures

• What is a trajectory?– Actual flight path (“ground truth”) seldom known– Tracking data: radar, altitude, weather– Intent: flight-plan (pilot/ATC), way points, jet-routes– Route constraints

TP Client Applications

• Flight Plan Distribution and Sector Loading: prediction of the Center Sequence (i.e., Air Traffic Control facilities to be transited)

• Flight Data Processing Systems: estimation of hand-over conditions, collision avoidance

• Basic Decision Support Tools: monitor flight plan and tracking data to advise controllers of suggested metering/sequencing and potential conflicts

• Advanced Decision Support Tools: directly suggest solutions (e.g., instructions for conflict resolution, accurate prediction of cruise-descent and climb-cruise profiles in presence of tactical maneuvers)

Sources of Uncertainty

• Data errors: incomplete, inaccurate, corrupted or untimely data (e.g., flight plan and radar track data inputs, airspace/navigation database)

• Modeling errors: simplifications, omissions and uncertainty in the mathematical models used in the system (e.g., aircraft performance modeling database)

• Intent errors: unpredictability of pilot and controller intent

Tracking Database

• Trajectory info in NAS– Track reading (radar

position and barometric altitude): noisy measurement of actual AC movement

– Flight plan: “intended path” of the AC

• Initial flight plan• Amendments to flight

plan (often not in system)

actual flightpath

flightplan

radar reading

CM_SIM database assembles aircraft trajectory data from multiple noisy sources: raw, cleaned, and derived data, weather, pilot intent, etc.

Tracking Data

• AC_DATA: radar tracking data– Noisy radar readings (every 12 secs)

• ADD_FLIGHT_PLAN: flight plan data• AMEND_FLIGHT_INFO: additional intent data• weather and waypoint data are separate

AMEND_FLIGHT_INFO 15 FDX1169/AFW.0015 000000002000 -NS- F -NS- AFW/1043 GRABE/1049 OKMX/1049 GRABE012038/1052 TIKYS260015/1055 EAKER/1057 ATOKA313002/1059 MLC304033/1104 LOBOE085022/1104 LOBOE058028/1106 69194/1106 69195/1107 TUL200033/1107 69184/1109 TUL/1111ADD_FLIGHT_PLAN 15 AAL8/PHNL.0016 ZFW AAL8 0016 -NS- -NS- 1114 H/B763/QHNL./.GATES247077..3254N/9702W..KDFW TXO221069 1100 390 455 -NS- ESTIMATED_FP ATC_ARRIVALAC_DATA 52 BKA302/ATL.0002 806.6335 540.9157 333850.00N 0913712.00W 28100 0.00 400.65 0.00 259.26 0.00 41.29 1 -NS- 28 N N F F ZFW 0.0000 403.0000 259.2621AC_DATA 52 AAL8/PHNL.0016 217.1337 539.1776 333634.00N 1032302.00W 39000 0.00 450.52 0.00 97.61 0.00 52.69 2 93 93 N N F F ZFW 0.0000 510.0000 97.6074ASSIGN_CONFIG 15 AAL8/PHNL.0016 CM KDFW SOUTH_4_VFR

Trajectory Synthesis

• Mature discipline with complex algorithms to– integrate equations of motion, aircraft models– analyze weather– interpret pilot intent

• Can represent trajectories using Generalized Altitude Profile– Domain-specific language for trajectories: GenAlt– Input request constrains possible trajectories– TS tool outputs parametric trajectory and point sequence of

states

V&V Challenge

• TS tool highly safety-critical but difficult to test– Complex algorithms– Lots of domain knowledge– Many different “kinds” of trajectories– Many different “kinds” of requests

• Huge database of raw data– US national airspace tracking data– Weather data– Aircraft manufacturer models– Field tests

• Need flexible data selection and analysis mechanism• Want to write analysis queries at high-level• Goal:

– develop DSL for trajectory analysis– use program synthesis to generate analysis code

V&V Goals

• Validate TS algorithm– Test with representatives and edge-cases of important trajectory

classes • Populate trajectory database (based on tracking data, etc.)

– Contractor building database infrastructure– Use for testing/assessment of trajectory prediction algorithms

• Determine faulty assumptions in models• Compare trajectory predictors• Initial analysis techniques

– Clustering wrt.• Point data• Trajectory data

– Change-point detection• Decompose trajectories into segments• Can use as features for clustering

Clustering

• Given: data-points that originate from 3 different sources/processes

• Data points are:– iid– Gaussian distributed

Problem: you don’t see the colors

Clustering

• Clustering task: Estimate the parameters µ, such that

P(x| µ,) is maximal

Clustering also produces most likely class-assignment (“coloring”) of data points and class frequency .

Usually, an iterative machine-learning algorithm (EM, k-means) is used

AutoBayes• Customizable Bayesian data

analysis code generator• Synthesizes custom algorithms

– C/C++, Matlab, …– Different model domains:

MLE, MAP, mixture models, change-point detections, etc.

– Different PDFs– Different algorithms (EM, k-

means, kd-trees)– Analysis or sampling (Monte

Carlo simulation)– Can be extended with new

algorithms

• Input is statistical model of analysis problem

Example Data Model

• Specify:– num. classes– num.

datapoints– distribution of

data– goal function

Trajectory Features• Each trajectory (i.e., sequence of AC

locations over time) can be characterized by a set of scalar features: – % in level flight, climbing, descent– Endpoints– Min distance to other AC– Min/max/mean v/speed, g/speed, g/accel,

Mach/CAS/TAS– Diff raw and cleaned data– Turning maneuvers:

• Total, max, sum• Altitude and heading• Could smooth wrt. turbulence

Clustering Analyses

• Simple clustering– Cluster into disjoint sets of ACs: each represents

set of AC trajectories– Cluster into disjoint time segments: each

represents all AC info during that segment– Cluster into disjoint sets of ACs at specific time

point

• Cluster on– Tracking data– GenAlt requests – Synthesized trajectories

Example Analysis

• Approx. 300x300nm around DFW

• Approx. 1 hour of data

• 215 AC trajectories• Cluster wrt. several

features• Blue and cyan are

landing and approach to DFW

Example: Endpoints

• DFW forms one cluster (cyan)

• Other points not obviously clustered– through flights– cut-off by time

Example: Close AC-distance

• AC in descent to/departure from DFW: small distance to airport (?) (d~1.2nm)

• En route flights which exhibit small minimal distances to other AC (d~0.97nm)

Change-point Detection

• Data analysis technique for time series• Detect (rapid) changes in

– Intensity– Noise– etc.

Mach/CAS Transition

• Mach number– Velocity relative to sound

(M = v/a)– Depends on altitude

• Calibrated airspeed (CAS)– Indicated airspeed,

corrected for errors• Mach/CAS transition point

– During ascent: constant CAS then constant Mach

Climb/Descend Profiles

mach

cas

alt

Generate code to find transition points: Mach/CAS, climb/cruise, …

• Accurate AC models for climb profiles are necessary for accurate trajectory prediction

• Mach/CAS profiles can be extracted from radar trajectory data – Convert ground speed to

Mach/CAS profile– Cluster according to

Mach/CAS values for different aircraft types and maneuvers

– Change-point detection in climb/descend maneuvers

Finding Transition Points

• Pre-process to find “climbs”

• Find where transition from slope to constant occurs

• Noisy data• Should also find

best Mach/CAS transition

const nat N, t_0; const double sigma_sqdouble a,b,c.data double mach(0..N-1).mach(t) ~ N( if(T < t_0)? a+b*t : c, sigma_sq).max pr(mach|{a,b,c}) for {a,b,c}.

a = 0.69428b = 0.0072091c = 0.44532t_0 = 35Red: actual trajectory

Blue: estimated profile

Next Steps• Characterize important trajectory features• Generate test cases for TS from clusters• Combine extracted transitions and parameters with other

features• Covariance analysis wrt. AC types• Extension of statistical model by domain knowledge

(“priors”)– e.g., airport locations

• Incorporation of other data– flight plans– weather data (e.g., for Mach/CAS)

• Relating physical trajectories (tracking data), synthesized trajectories, and TS requests

• Recursive clustering algorithms (large data sets)

Longer Term

• Domain-specific language to describe– analysis types – trajectory features– data filters

• weather conditions• changes of flight plans• specific airports, flight types, flight segments

• Integrate generator with– trajectory database– TS tool