development of a collision avoidance ... - atm seminar › seminarcontent › seminar13 ›...

Sybert Stroeve, Henk BlomCarlos Hernandez Medel, Carlos Garcia Daroca

Alvaro Arroyo Cebeira, Stanislaw Drozdowski

13th USA/Europe ATM R&D Seminar, Vienna, 17-21 June 2019

Development of a Collision Avoidance Validation and Evaluation Tool (CAVEAT)

Addressing the intrinsic uncertainty in TCAS II and ACAS X

2

Contents

• Context

• Agent-based sociotechnical model

• CAVEAT software tool

• Illustrative simulation results

• Conclusions

ContextUncertainty in encounter scenarios for ACAS design

3

4

TCAS II & ACAS X

Airborne Collision Avoidance System (ACAS)– Provides advisories to pilots for avoiding potential collisions– Traffic advisories (TAs): support pilot situation awareness on nearby traffic– Resolution advisories (RAs): Level off, Climb, Descend, Do not climb, etc.

Traffic Collision Avoidance System II (TCAS II)– Current commercial implementation, version 7.1 required by ICAO Annex 10– Filtering of transponder data and a fixed set of rules for alert generation

Recent development of ACAS X– Modular architecture for various surveillance sources– Optimization of logic tables for alert generation– Variants: ACAS Xa / ACAS Xo / ACAS Xp / ACAS Xu

5

Evaluation of ACAS designs

USEurope

ACAS advisories TCAS II v7.1 ACAS XaPilot response Standard 20% no response

P(NMAC) ACAS type Altitude Pilot responseP(RA types) Reversal Crossing Strengthening

1

encN

1

encN

6

Elements in an encounter scenario

7

Intrinsic uncertainty (each aircraft)

Pilot performance variability• Probability of response• Time of response• Strength of response

Sensor errors• Static errors (bias)• Variable errors (jitter)

8

Resulting uncertainty in ACAS alerting

As a result of uncertainty in ACAS surveillance, ACAS alerting is a stochastic process• No single unique RA for a scenario• Probability density for RA timing • Probability distribution RA types

P(RA)

9

Uncertainty in outcomes of an encounter scenario

10

ACAS evaluation with Monte Carlo simulation

MCN

1 MCN

1

encN

1

encN

ACAS typeSensor error models Ownship state estimation Othership measurementPilot response models Stochastic

Agent-based Sociotechnical Model

11

12

Agent-based sociotechnical model

13

Pressure altimetry system

0

0

0

,, , ,,

, , ,, , ,

( ;0, ) if { }

( ;0, ) if

N z bias biasPAS i PAS iz bias

t PAS i L z bias biasPAS i PAS i

f Normal

f Laplaceτ

ττ

σ κε

σ κ≥

⋅ =

⋅ =

Static errors (bias)• Normal or Laplace distribution• Altitude dependent variance

Dynamic errors (jitter)• First-order auto-regressive noise process

,

, , 0,,, ,

, , 0, ,

if if sample

PAS i

noiset PAS iz jitterauto z jitter noiset PAS iPAS t PAS it T PAS i

tt

ε τε α ε ε τ

−

== + >, 2

, , ,{ } (0, 1 ( ) )noise N z jitter autot PAS i PAS PAS ifε σ α−

, ,, , , , , , , ,z z z bias z jittert PAS i t AC i t PAS i t PAS is s ε ε= + +

Resulting pressure altitude

14

Slant range measurement

ACAS MOPS• RMS jitter ≤ 50 ft (mode S or C)• RMS bias ≤ 125 ft (mode S) or 250 ft (mode C)

0 ,

, ,, , , ,

{ } ( ;0, )jTS i

r bias N r biasTS i TS m i

fτε σ⋅

, , ,, , , , ,, ,sample

TS

r jitter auto r r jitter noiset TS i TS i t TS it T TS iε α ε ε

−= +

,

, , 2, , ,, ,

{ } ( ;0, 1 ( ) )jTS i

noise N r jitter auto rt TS i TS iTS m i

fε σ α⋅ −

, ,, , , , , , , , , ,j r bias r jitter

t TS i t AC i t AC j t TS i t TS ir ε ε= − + +s s

Error modelStatic and variable noise processes with mode dependent variance

• Gaussian bias

• 1st order autoregressive jitter

• Resulting slant range

15

ACAS algorithms in CAVEAT

TCAS II• Versions 7.0, 7.1, 7.2 stem from InCAS 3.3, based on MITRE

implementation (v7.0) and adjustments by Evosys (v7.1, v7.2)

ACAS Xa• Presented results: R15V2 in Julia with interface to C++ • Current implementation: R15V4 in C++ code from Honeywell

ACAS Xu• Expected in 2020/2021

16

Pilot model: response probability

ICAO standard model• Response probability: 100%• Delay

5 s (initial RA) 2.5 s (modified RA/COC)

• Acceleration 0.25g (normal) 0.35g (reverse / increase)

CAVEAT response probability model• P(initial RA | context)

altitude rate reversal parallel approach

• P(modified RA | previous response)• P(COC)• 13 conditional probabilities

Response probability measurements• Londner and Moss (2017)• P(response) <<100%• P(response | context)

rate reversal / parallel approach

17

Pilot model: delay

ICAO standard model• Response probability: 100%• Delay

5 s (initial RA) 2.5 s (modified RA/COC)

• Acceleration 0.25g (normal) 0.35g (reverse / increase)

CAVEAT delay model• Log-normal probability distribution• Advisory dependent parameters

initial / modified / COC

Delay measurements• Cloerec (2005)

18

Pilot model: acceleration

ICAO standard model• Response probability: 100%• Delay

5 s (initial RA) 2.5 s (modified RA/COC)

• Acceleration 0.25g (normal) 0.35g (reverse / increase)

CAVEAT acceleration model• Log-normal probability distribution• Linear relation between mean

acceleration and vertical speed

Acceleration measurements• Cloerec (2005)• Relation between acceleration

and vertical speed

1 1 1

, , ,, , , , , ,

a RA z RA a zPF i PF i PF iv vτ τ τµ α β= + −

CAVEAT Software Tool

19

20

High-level design of CAVEAT software

21

HMI allows setting of all agent parameters

Pilot response Ownship state estimation

22

Horizontal / vertical views of a simulation run

23

Statistics report (for MC simulation)

Illustrative Simulation Results

24

25

Encounter scenarios

Encounter Horizontal Vertical HMD (ft) VMD (ft)

E1 Crossing AC1 climbsAC2 is level 3038 200

E2 Crossing AC1 is levelAC2 is level 0 0

Scenario Sensor errors

Pilot Dynamic Variability

Pilot Response Probability

D none none (standard) 100%

S1 MOPS none (standard) 100%

S2 MOPS delay, rate, acc. 100%

S3 MOPS delay, rate, acc. 80%

Stochastic

Deterministic

26

Encounter scenario E1-D with TCAS II v7.1Vertical viewHorizontal view

HMD: 3038 ft VMD: 200 ft (original) / 522 ft (modified)

27

Encounter scenario E2-D with TCAS II v7.1Vertical viewHorizontal view

HMD: 0 ft VMD: 0 ft (original) / 975 ft (modified)

DE

CL

28

Timing of advisories (TCAS II v7.1) Encounter scenario E1-S1 Encounter scenario E2-S1

29

Number & types of advisories (TCAS II v7.1)

AC RADeterministic Stochastic (S1)

Sense P(RA) P(Sense | RA)

1

RA-1 Level off 100% Level off 97%Descend 3%

RA-2 - 6% Level off 53%Descend 47%

RA-3 - 3% Level off 100%

2

RA-1 Climb 100%Climb 94%Do not

descend 6%

RA-2 Level off 94% Level off 94%Climb 6%

RA-3 - 1% Level off 1%

Encounter E1 (climb x level)AC RA

Deterministic Stochastic (S1)

Sense P(RA) P(Sense | RA)

1RA-1 Descend 100% Climb 51%

Descend 49%

RA-2 Level off 100% Level off 100%

2RA-1 Climb 100%

Descend 49%

Climb 51%

RA-2 Level off 100% Level off 100%

Encounter E2 (same level)

30

VMD empirical PDF of Encounter E1 (climb x level)

Sensor errors only (S1) Sensor errors & pilot var. (S3)522 723

Deterministic: TCAS II v7.1 ACAS Xa V15R2

TCAS II v7.1mean: 519 ftSD: 111 ft

ACAS Xa V15R2mean: 691 ftSD: 96 ft

522 723

TCAS II v7.1mean: 466 ftSD: 143 ft

ACAS Xa V15R2mean: 639 ftSD: 162 ft

31

VMD empirical PDF of Encounter E2 (same level)

Sensor errors only (S1) Sensor errors & pilot var. (S3)

Deterministic: TCAS II v7.1 ACAS Xa V15R2

975 1025 975 1025

TCAS II v7.1mean: 845 ftSD: 85 ft

ACAS Xa V15R2mean: 906 ftSD: 78 ft

TCAS II v7.1mean: 916 ftSD: 230 ft

ACAS Xa V15R2mean: 916 ftSD: 230 ft

Conclusions

32

33

Conclusions

Evaluation of ACAS designs requires assessment of all relevant sources of variability and uncertainty

– Variability in encounter geometries / altitude layers– Variability in equipage types– Uncertainty due to pilot response– Uncertainty due to sensor errors

Development of an agent-based sociotechnical model for Monte Carlo simulation of encounter scenarios

– Various sensor error (bias & jitter) models– Pilot performance models– Supports deterministic (single run) simulations– Systematic evaluation of the impact of uncertainty in encounter scenarios– Extendible for new agents: ACAS Xu, UAV, ATC,...

34

Conclusions

Illustrative simulation results– Variability in timing and types of RAs, and dispersion in VMD

• Considerable differences with deterministic simulation results – Mean MC simulation results can differ considerably from deterministic simulation results

Use for prospective analysis: What will be the impact of changes? – Impact on RA types / VMD / P(NMAC)– MC simulation of sets of encounter scenarios – Evaluation of ACAS designs / ACAS related systems / system compatibility– Evaluation of changes in airspace, ATM, and regulations

Use for retrospective analysis: What happened / could have happened? – MC simulation of a single encounter scenario– Likelihood of observed RAs (type / timing)– Likelihood of observed CPA

35

Conclusions

CAVEAT software development for EUROCONTROL– Extensive HMI gives wide control over parameters of the agent-based model– TCAS II v7.0, v7.1, v7.2, ACAS Xa R15V4– Detailed views for incident investigation of single encounters– Powerful simulation core for analysis of large sets of encounters – Integration of ACAS Xu (when available)

Future research options– Required sizes of encounters sets and number of MC simulation runs– Sensitivity analysis for understanding the impact of the range of uncertainty sources– Detailed analysis of the impact of new ACAS designs (ACAS Xa, ACAS Xu)

36

Acknowledgements

The R&D in this paper has been conducted with support from EUROCONTROL in the scope of SESAR 2020 Project 11

We thank MITRE and Honeywell for making available their TCAS II and ACAS Xa/Xo libraries, respectively

We thank Carmelo Javier Villanueva Cañizares, Daniel Rubio Garcia, Cristina Midori Fukuda Leon, Eduardo Pablos Ruiz, Ángela Merino Pérez, Volker Huck, David Phu, Garfield Dean, Busso Gellert, Frank Bussink and Bert Bakker for their support of the CAVEAT development

NLR AmsterdamAnthony Fokkerweg 21059 CM AmsterdamThe Netherlands

p ) +31 88 511 31 13 e ) info@nlr.nl i ) www.nlr.org

NLR MarknesseVoorsterweg 318316 PR MarknesseThe Netherlands

p ) +31 88 511 44 44 e ) info@nlr.nl i ) www.nlr.org

Fully engagedRoyal Netherlands Aerospace Centre