Vision-guided Planning and Control for Autonomous Taxiing via Convolutional Neural NetworksPresenter: Chang LiuAdvisor: Silvia FerrariLaboratory for Intelligent Systems and Controls (LISC)Cornell University

AIAA SciTech 2019GNC-18/IS-10, Advances in Adaptive Control Systems I

San Diego, CA, January 8, 2019

1

2

Outline

Motivation and Background Problem Formulation Technical Approach:

Object Recognition using Mask-RCNN ATC-based Path Planning Hybrid Control of Autonomous Taxiing Aircraft

Simulation Results Conclusions

3

Motivation and Background

Aerodromes are becoming increasingly complex and crowded

Dangerous situations seriously affects aerodrome safety

Motivation

Crowded airports Runway incursion incidents

Year2018

Operational Incident

Pilot Deviation

Vehicle Pedestrian Deviation

Other Total

Totals 345 1142 335 10 1832

Incidents of runway incursions in 2018 (from FAA)

4

Autonomous Taxiing:

Automating aircraft taxiing process without human intervention

Take Air Traffic Control (ATC) commands in the loop

Detect obstacles and unforeseen conditions in situ

Generate corrective planning and control to guarantee aircraft safety

Motivation

ATC tower Unforeseen objects

5

Surface Operation Scheduling: schedule the use of taxiways, runways, and gates to reduce the overall travel time and maximize throughput (Morris 2016)

Aircraft Path Planning and Control: Generate energy-efficient and collision-free trajectories based on aircraft motion models (McGee 2007, Coetzee 2011, Chen 2016, Zhang 2018)

Related Work

Situation Awareness for Planning: Uses machine learning algorithms for taxiway feature extraction and unknown obstacle identification (Lu 2016, Lu 2018)

Limitations: Lack of ability to incorporate ATC commands Planning and control ignores environmental perception Unable to handle unexpected dangerous situations

6

7

Problem Formulation

Continuous controller

Problem Formulation

Goal: Develop a vision-guided path planning and control approach for autonomous taxiing under both normal conditions and unforeseen conditions

8

)](),([)1( kkk ξξ usfs =+ )()1( kk µξ =+

)(kξu

Hybrid system modeling:

System discrete mode System continuous state

Objective function Continuous dynamics

Planning horizon Stage cost

sJ

∑−

=

Ψ+=1

))(),(),(())((fk

kjf jjjkJ µϕ ξξ uss

Objective function

,ϕf

],[ fkk

Discrete controller

Normal conditions

Unforeseen conditions

Ψ

Yaw angle Speed

Steering angle Acceleration

Sampling interval Front-rear axle distance

Aircraft Motion Model

The aircraft uses a simple car model, defined as,

Coordinate of the rear-axle center

𝑥𝑥, 𝑦𝑦

𝜃𝜃

𝜙𝜙

𝑣𝑣

9

The simple car model of an aircraft

T

k

kLk

kkkk

kkk

kkk ee

∆

+

=

+++

)(

)(tan)()(sin)()(cos)(

)()()(

)1()1()1(

β

φυθυθυ

υθ

υθ

qq

θ υ

β

LT∆

[ ]Te kkkk )()()()( υθqs =Aircraft stateT

e yx ],[=q

Aircraft control input

Camera Measurement Model

Camera measurement model

Recognizing three semantic classes of incursion objects

“People”

“Animals”

“Ground Vehicles”

objects classification

camera-to-object distance

10Animal VehiclePeople

Onboard RGB-D Camera

Camera View

},2,1,0{)( ∈kl

0)( ≥kdl

Tl kdklk )](),([)( =z

1)( =kl

2)( =kl

3)( =kl0)( =kl means no detected object

Taxiway region

Runway region

Terminal region

Airport Modeling

Taxiway label set

Runway label set

Terminal set

Airport Diagram Airport Satellite Map11

},{ , VA, E, P, FL =A

}33152810{ , , , L =P

},{ 2210310 ,, g, g, gggL =G

,2RWa ⊆ ALa∈

,2RWp ⊆ PLp∈2

0RWg ⊆

,2,1, =igi represents the ith gate

Airport Modeling

A

B

B E

A

hB

h(A,B) h(G,28)

h28

G

28

h(D,g0)

D

Example nodes:

g11

12

Nodes , where connection of two regions terminal gates aircraft current position

},,{ EBAvi =},{ BAv j =}28,{Gvl =

},{ 011 ggvm =},{ 0gDvn =

Taxiway centerline Runway centerline

,0)( =qah,0)( =qph

Taxiway connecting arc ,0)( =qAh AA LLa ×∈

Taxiway-runway arc ,0)( =qPh PA LLP ×∈

Taxiway-terminal arc ,0)( =qTh }{ 0gLT ×∈ A

ALa∈

PLp∈

vi

vj

),( qγ=vGPA LLL ××∈γ

vl

vmvn

Airport Modeling

13

Node set

Edge set Terminal gate region

Runway

Taxiway ATaxiway B

Taxiway C

2002

Simulated airport

Airport Graph: a topological graph representing the connectivity of different airport regions.

Partial graph of the airport

}{vV =

},|{ Vvvvv jiji ∈∩=Ξ

),( Ξ= VG

Nodes, centerlines, and connecting arcs

ATC Commands

Command Category Instruction ATC Command Examples

Cruising Move along certain taxiways to a runway.

• “Runway Three-Six Left, taxi via Taxiway Alpha, hold short of Taxiway Charlie.”• “Cross Runway One-Six Left and Runway One-Six Right at Taxiway Bravo.”

Traffic Following Follow the traffic. • “Follow (traffic), cross Runway Two-Seven Right, at Taxiway Whiskey.”

Holding Hold short of a runway or hold in position.

• “Hold short of runway Two-Seven.”• “Hold in position.”

14

c

15

Technical Approach

Object Recognition using Mask-RCNN

Mask-RCNN: state-of-the-art object detector

16

Consists of a region proposal network and a binary mask classifier

Uses RGB images as input

Outputs a class label, bounding box, segmentation mask, and a confidence level The Mask-RCNN structure (He 17’)

Recognition results using Mask-RCNN

reference waypoint

ATC command sequence

ATC-based Path Planning

ATC command where and

• Example: “Runway Two-Zero, taxi via Taxiway Alpha and Bravo”

Example: node sequence

Path Generation from ATC Commands• Connect graph nodes in accordance with ATC commands

• Turn node sequence into waypoints

),,,,( 121 nn ψψψψ −=Ψ

ALn ∈−121 ,,, ψψψ GPA LLLn ∪∪∈ψ

)20,,( BA=Ψ⇒

),,,,,( 15141398 vvvvvvs

]))(),((,,))1(),1([(* Trr

Trr NyNxyx =q

),( rr yxΨ 17

Hybrid Control of Autonomous Aircraft

The optimization problem in mode is defined as

ξxmin

..ts 0)( ss =k1,,)),(),(()1( −==+ fkkjjjj ξusfs

1,,)( −=∈ fkkjSj s

1,,)( −=∈ fkkjUj ξξu

( ) ( )∑−

=

Ψ+=1

)(),(),(),()(),(),(fk

kjfff jjjyjxkkykxJ ξξ υυϕ u

Reference waypoint Aircraft initial state

State feasible set Control feasible set

),( rr yx 0sS ξU

T

f

TTkkkk )]1(),(),(,),([ −= ξξξ uussx is the optimization variable

Five aircraft modes

18

{cruising (1), traffic following (2), holding (3), incursion (4), idle (5)}

ξ

where:

Continuous State Control

Cruising mode: Follow the reference path and maintain a desired cruise speed

∑−

=

+−+

−=1

2

21

2

2

2

21

)(]),(),([)](),(),([

]),(),([)](),(),([fk

kj

Tcrr

T

Tcfrfr

Tfff

jjyjxjjyjx

kykxkkykxJ

uυυ

υυcυ

Traffic following mode: Follow the reference path and maintain a speed similar to the aircraft in front

∑−

=

+−+

−=1

2

22

2

2

2

22

)(]),(),([)](),(),([

]),(),([)](),(),([fk

kj

Tfrr

T

Tffrfr

Tfff

jjyjxjjyjx

kykxkkykxJ

uυυ

υυ

fυ

Holding mode: Decelerate to stop at the hold-short position

∑−

=

+−+

−=1

2

23

2

2

2

23

)(]0),(),([)](),(),([

]0),(),([)](),(),([fk

kj

Trr

T

Tfrfr

Tfff

jjyjxjjyjx

kykxkkykxJ

uυ

υ

with the additional constraint 0)( =fkυ19

Continuous State Control

Incursion mode: Decelerate to avoid collision with the incursion object

∑−

=

+−+

−=1

2

24

2

2

2

24

)(]0),(),([)](),(),([

]0),(),([)](),(),([fk

kj

Trr

T

Tfrfr

Tfff

jjyjxjjyjx

kykxkkykxJ

uυ

υ

Idle mode: Remain current state

1,,],[)](),([2

2−=≥− fs

Too

T kkjdyxjyjx

with the additional collision avoidance constraint

Object position

Safety distance

),( oo yx

sd

0u =5

20

Discrete State Control Definition

Cruising mode

Traffic following mode

==∈

====

=

otherwise0)( and 0)( if }3,2,1{)( if

)( and 0)( if)( and 0)( if

)(

3

5

4

22

11

µυµ

µµµ

µkkl

klckcklckckl

k

Holding mode

Incursion mode

====

=

otherwise0)( if

)( if)( if)( if

)(

4

5

33

22

11

µυµ

µµµ

µk

ckcckcckc

k

==

=otherwise

)( if)( if

)(

5

22

11

µµµ

µ ckcckc

k

Idle mode

ATC command category

Object classification)(kl321 ,, ccc

∈====

=

otherwise }3,2,1{)( if

)( and 0)( if)( and 0)( if

)(

1

4

33

22

µµµµ

µkl

ckcklckckl

k

21

==∈

====

=

otherwise0)( and 0)( if }3,2,1{)( if

)( and 0)( if)( and 0)( if

)(

2

5

4

33

11

µυµ

µµµ

µkkl

klckcklckckl

k

where: and ii =µ

Visualization of Discrete State Control

),( 5usfs =

5=ξ

),( 1usfs =

1=ξ

),( 2usfs =

2=ξ

),( 3usfs =

3=ξ

),( 4usfs =

4=ξ

2)( and 0)( if ckckl ==

3)( and 0)( ifckc

kl=

=

}3,2,1{)( if ∈kl}3,2,1{)( if ∈kl

}3,2,1{)( if ∈kl

1)( and 0)( ifckc

kl=

=

0)( and 0)( if=

=k

klυ

22

23

Simulation Results

Runway

Taxiway ATaxiway B

Taxiway C

2002

Simulation Setup

A simulated small-sized airport modeled in UnrealEngine®

Simulated airport

Runway and taxiways Ground markings and signs Terminal area 24

Results – Normal Taxiing

Taxis along Alpha and Bravo to Runway Two-Zero following ATC commands

25

Trajectory

y(m

)

x(m)Velocity Profile

v(m

/s)

t(s)

26

Results –Incursion Events

Original plan: taxi along Alpha and Charlie to Runway Two-Zero Incursion object detected and classified as car. New path generated based on ATC commands.

Velocity Profile

v(m

/s)

t(s)

Object Detection

27

Conclusions

Vision-guided Autonomous Taxiing:

Airport modeling using airport diagrams and geographical information.

A systematic approach that enables real-time aircraft perception, obstacle avoidance, and feedback control with ATC commands in the loop

Future Work:

Robustness analysis of the proposed approach

Incorporate action recognition and environmental semantic understanding for airport environments

Conclusions

28

Thank you

Questions?

29

Lu, W., Zhu, P., and Ferrari, S., “A hybrid-adaptive dynamic programming approach for the model-free control of nonlinear switched systems,” IEEE Transactions on Automatic Control, Vol. 61, No. 10, 2016, pp. 3203–3208

Zhu, P., Isaacs, J., Fu, B., and Ferrari, S., “Deep learning feature extraction for target recognition and classification in underwater sonar images,” IEEE 56th Annual Conference on Decision and Control (CDC), 2017, pp. 2724–2731.

Lu, W., Zhang, G., and Ferrari, S., “A randomized hybrid system approach to coordinated robotic sensor planning,” 49th IEEE Conference on Decision and Control (CDC), 2010, pp. 3857–3864.

References

30