TRAFFIC VISUALIZATION IN HELMET-MOUNTED DISPLAYS INSYNCHRONIZATION WITH NAVIGATION DISPLAYS

Ferdinand Eisenkeil, University of Konstanz, Konstanz, GermanyJohannes Ernst, Technische Univ. München, Munich, Germany

Ralf Stadelhofer, Airbus Defence and Space, Immenstaad, GermanyUwe Kühne, Airbus Defence and Space, Immenstaad, GermanyOliver Deussen, University of Konstanz, Konstanz, Germany

AbstractOne of the major challenges for helicopter pilots

are low level flights and landings in degraded visualenvironments. Without proper assistance systems, thepilots are prone to lose their situational awarenesswhen fog, heavy precipitation, limited sunlight andstirred-up sand or snow degrades their view. In recentyears, various synthetic and enhanced vision systemswere developed so as to assist the pilots in thesedemanding situations. We enhance the existing sys-tems by proposing a concept for the visualizationof traffic information in head-mounted displays. Theintuitive representation provides additional cues aboutthe environment and decreases the pilots’ workload,especially during flights in offshore windparks orwhile search and rescue operations with many othervehicles operating within a small range.

IntroductionIn modern helicopter flight decks, Enhanced

Vision Systems (EVS) incorporating head-trackedHead-Mounted Displays (HMDs) are used to improveflight safety, especially during operations in DegradedVisual Environments (DVEs) such as brownout land-ings and flights during night-time or in adverseweather conditions [1], [2]. These displays decreasethe pilot’s head-down time and increase his situationalawareness by displaying important information virtu-ally superimposed on the out-the-window view. Ascan be seen in Figure 1, this involves simple two-dimensional symbols like the speed and altitude tapesbut also three-dimensional symbology. The latter con-forms with the real world behind in order to highlightterrain contours or a desired landing location.

Within the framework of the Next Generation AirTransportation System (NextGen)[3] and the Single

European Sky ATM Research (SESAR)[4], a newaeronautical surveillance system called AutomaticDependent Surveillance-Broadcast (ADS-B) has beenintroduced. This satellite-based technology allows forprecise determination of an aircraft’s, ground vehi-cle’s or ship’s position, speed and supplemental data,which is then broadcast to be received by air trafficcontrollers and other vehicles in the vicinity.

Our key idea is to combine the benefits of thesetwo existing technologies by visualizing receivedADS-B data of surrounding traffic on an HMD. Sucha system will assist helicopter pilots flying in offshorewindparks or taking part in search and rescue oper-ations where many other vehicles operate within asmall range. Example situations are given in Figure 2.

Since the pilots have to cope with high workloadduring such scenarios, the traffic visualization hasto present the required information in a way that iseasy and intuitive to use and does not unnecessarilydistract the pilot. To achieve this and to conform withadditional requirements of head-mounted see-throughsymbologies, we developed an integrated traffic visu-alization concept: a head-down and a head-mounteddisplay complement each other in their roles so asto combine the advantages of both representations.For the synthetic ADS-B input data, we adapt aclustering algorithm to analyze the high-dimensionaltraffic data in order to identify groups of similartraffic that can be visualized by a single symbol inthe HMD. This cluster information is used to avoiddisplay clutter by decreasing the number of renderedsymbols. In addition, we present a special symbologythat offers supplemental information about the cuedtraffic group. This includes parameters such as thenumber and type of the contained vehicles as well asa measure of the formation’s threat potential, basedon the predicted closest point of approach. To further

(a) (b)Figure 1. The head-mounted symbology of the SFERION R© pilot assistance system by Airbus Defenceand Space [2]. (a) 3D conformal symbology (terrain grid with contour lines) overlaid with 2D symbology.(b) 3D conformal landing symbology with doghouse.

(a) (b)Figure 2. Exemplary scenarios where a head-mounted traffic display increases the situational awareness.(a) A close formation of Coast Guard helicopters [5]. (b) Two helicopters operating inside an offshorewindpark [6].

decrease the pilot’s workload by avoiding distractingchanges in the visual representation of clusters, asmooth transition visualization algorithm is developedfor splitting and merging clusters.

Related WorkRelated work includes an explanation of the

ADS-B technology, a presentation of the TCAS anda brief overview of cluster analysis. Furthermore,work related to head-mounted displays for increasingsituational awareness in helicopters is presented.Automatic Dependent Surveillance-Broadcast:According to ICAO Doc 9924 [8], an AeronauticalSurveillance System “provides the aircraft positionand other related information to Air Traffic Manage-ment (ATM) and/or airborne users”. In its simplest

realization, it provides only the aircraft’s position ata certain time. More advanced designs allow the userto get information on the identification, the speed,the intent and other characteristics of the aircraft.Depending on the technologies used, it can be dis-tinguished between three major types of aeronauticalsurveillance systems: Independent Non-CooperativeSurveillance, Independent Cooperative Surveillanceand Dependent Cooperative Surveillance [7].

While Independent Non-Cooperative Surveil-lance systems do not rely on the co-operation ofthe aircraft, which means that an aircraft does nothave to be equipped with any on-board surveillanceequipment to be detect-able, Independent CooperativeSurveillance systems communicate with the aircraftto be identified. Therefore, it is possible to request

Table I. Overview of aeronautical surveillance systems [7].

Category Technology

Independent Non-Cooperative Primary Surveillance Radar (PSR)Multi-Static Primary Surveillance Radar (MSPSR)

Cooperative Secondary Surveillance Radar (SSR) (Mode A, C, S)Wide Area Multilateration (WAM)Multi-LATeration (MLAT)

Dependent Cooperative Automatic-Dependent Surveillance Broadcast (ADS-B)

supplemental data such as the identification or thecurrent airspeed. To do so, each aircraft hast tobe equipped with a radio receiver and transmitter,referred to as transponder. Examples of independentcooperative surveillance technologies are SecondarySurveillance Radar (SSR), Wide Area Multilateration(WAM) and Multi-LATeration (MLAT) [7], [9].

The ADS-B system we require for our trafficvisualization approach is a Dependent CooperativeSurveillance system. This kind of surveillancesystems depend on a system that enables the on-board determination of the vehicles position, forinstance a GPS receiver. This information is thenbroadcast together with supplemental data. Thedata provided by the ADS-B system and used forour visualization is the identification ID, position,velocity, air/ground state and heading of the vehicle.

Traffic Alert and Collision Avoidance System:TCAS is an Airborne Collision Avoidance Sys-tem that works independently of airborne navigationequipment and ground-based systems, only basedon the direct communication of aircraft via theirtransponders.

The central TCAS unit processes all responsesand applies a special logic to the classification ofeach intruder into three categories: Proximate Traffic,Traffic Advisory (TA) and Resolution Advisory(RA). The remaining aircraft are declared OtherTraffic. According to this classification, an intruderis highlighted on the Cockpit Display of TrafficInformation, triggers an aural annunciation (TA) oreven initiates the display of instructions on how tosolve the conflict (RA). The TCAS threat detection isbased on the concept of the warning time τ , which isan approximation of the time to the Closest Point of

Approach (CPA) between ownship and intruder. Thelower this measure, the higher is the threat potentialof the target aircraft. The full TCAS description aswell as the definition of the different dimensions ofthe spaces for RA and TA and the threat resolutionlogic is given in the ACAS Guide [10].

Cluster analysis:Cluster analysis is one of the central methods usedin the field of data mining, which is a key stepwithin the discovery of knowledge in databases [11].It has been utilized in various fields of scienceincluding image segmentation, artificial intelligenceresearch, big data analysis on data derived fromdifferent fields such as geology or biology. This vastnumber of applications with different prerequisitesand requirements yields numerous algorithms for theidentification of clusters in large data sets. For ourwork, the DBSCAN algorithm introduced by Esteret. al [12] is applied to the classification of trafficinformation. For a more thorough description ofexisting clustering methods one can consider [13],[14], [15] and [16].

Head-Mounted Displays for Increasing SituationalAwareness:Melzer [17] describes HMDs as “powerful tools thatcan unlock the pilot from the interior of the cockpit orthe forward line of sight of the Head-up Display”. Asa result, the displays “can enable the pilots to do theirjob more effectively while simultaneously decreasingworkload” [17]. One major goal in the development ofHMD symbologies is the improvement of the pilot’ssituational awareness.

A particular challenge is the development of pilotassistance systems that increase the pilot’s situational

(a) (b) (c)Figure 3. Illustration of a possible traffic situation causing overlapping symbols on the HMD. (a) A basictraffic visualization on the HMD - visible and invisible traffic highlighted with circles. (b) Head-downDisplay - top view. (c) Head-mounted Display - overlapping symbols.

awareness during low level flights and landings inDVE. Degraded vision due to fog, heavy precipitation,limited sunlight and stirred-up sand or snow can causespatial disorientation and lead to fatal accidents. TheGerman Aerospace Center (DLR) as well as AirbusDefence and Space conduct research within this fieldso as to enhance the pilot’s performance in suchsituations [1], [2]. Both follow a three-dimensional,visual-conformal visualization approach since thisscene-linked presentation of information is a proofedconcept that facilitates the mental processing andfusion of both impressions, the outside scene and thesymbology [18]. It implies that, for instance, obstaclesymbols on the HMD are virtually superimposed onthe underlying real world scene so that they appearat the location where the pilot would see the realobstacle if the view was not degraded.

OverviewFor the visualization of our cluster approach, we

use renderings on a head-mounted display as wellas on a head-down display (HDD). Target aircraftare cued by displaying a symbol superimposed onthe real aircraft in the outside view. In Figure 3asuch a visual-conformal symbology is sketched thathighlights other aircraft with a framing circle. Forthe Head-Down Display rendering every vehicle witha certain symbol is sufficient (Figure 3b), while inthe Head-Mounted Display such visualizations couldlead to cluttering effects (Figure 3c).

Both display types, HMD and HDD, playcomplementary roles in the integrated trafficvisualization concept:

Head-Down Display:• Provides an overview of the traffic situation,

from which the pilot can easily estimate hori-zontal distances and tracks of other aircraft.

• Incorporates an interface through which the pilotcan select nearby aircraft so as to receive ad-ditional information and highlight them in thehead-down and in the head-mounted visualiza-tion.

• Adapts the aircraft symbol according to thethreat potential of the intruder.

Head-Mounted Display:• Supports quick locating of relevant traffic.• Includes as much important information as pos-

sible in order to decrease the pilot’s head-downtime, but not more than needed to avoid clutter-ing.

• Allows the pilot to de-clutter the display byexcluding irrelevant traffic.

• Applies a clustering algorithm and an advancedsymbology in order to avoid overlapping trafficsymbols and to densely pack available trafficinformation.

System Architecture: The ADS-B traffic visualiza-tion software is designed as an add-on to AirbusDefence and Space’ advanced synthetic vision sys-tem [19]. The ADS-B traffic visualization implemen-tation comprises five major modules: Traffic Input,Traffic Processing, Head-Down visualization, Basic

Figure 4. The system architecture of the ADS-Btraffic visualization software.

HMD rendering and Advanced HMD visualization.The relations between these subsystems as well as anadditional user input are sketched in Figure 4.

The Traffic Input module receives and convertsADS-B messages broadcast by surrounding aircraftand ground vehicles. One current ADS-B messagefor each traffic participant is generated and sent tothe subsequent module. Within the Traffic Processingsubsystem, the received ADS-B messages are furtherprocessed and the parameters required by the trafficdisplays are derived. The output comprises all quan-tities included in the synthetic ADS-B data as well asadditionally computed parameters such as the distanceto the ownship. The values can either be baseddirectly on last message received or be derived froman interpolation between the last two stored messages.An interpolation of these values yields the possibilityof a smooth visualization. For different displays types,varying visualizations are provided: The HDD trafficvisualization is inspired by current Cockpit Display ofTraffic Information designs, with additional featuresso as to support the concept of the integrated head-mounted and head-down traffic visualization. For theHMD we implemented a straight-forward traffic vi-sualization approach as given in Figure (3a) and (3c)as reference for our more sophisticated visualization.This more advanced HMD subsystem is the keypart of our work. This subsystem incorporates allprocessing required to realize a head-mounted trafficsymbology that presents valuable information to thepilot. This includes clustering as well as the gener-ation of a de-cluttered, smooth traffic representation.The user is able to adapt the appropriate parametersfor the Traffic Processing as well as the configurationfor the visualizations.

Advanced Traffic Visualization in Head-Mounted Displays

This section describes the development andimplementation of the advanced traffic visualizationin HMDs. As stated above, this module is integratedwith the head-down traffic display. Recalling therole definition of the head-mounted module withinthe overall traffic visualization concept, the centralrequirement is that the advanced traffic visualizationin head-mounted displays should include as muchimportant information as possible in order todecrease the pilot’s head-down time, but not morethan needed to avoid cluttering. Thus, the definitionof an advanced visualization strategy is started withthe identification of methods for de-cluttering theHMD while presenting all relevant informationin a way that conforms with the requirements ofhead-mounted see-through symbologies.

De-Cluttering of the Display:Rendering the received position data of all surround-ing ADS-B-equipped vehicles can lead to many over-lapping traffic symbols and unnecessary informationon the HMD. Figure 5 shows such a traffic situation,where 11 other ADS-B equipped vehicles are visiblefor the pilot: two groups of ground vehicles below,three aircraft flying towards the ownship, three intrud-ers crossing the own flightpath from left to right andone single aircraft departing from the ownship on theleft side. For head-mounted renderings, we propose togroup intruders that operate close to each other andin a similar way and represent them by a single HMDtraffic symbol instead of overlapping icons.

Due to the movement of the own and otheraircraft, groups of similar traffic will be split upinto new clusters, others will fuse as well as variousgroups will overlap on the screen occasionally. Thisemphasizes the need for a strategy for merging anddividing different clusters and visualizing that processsmoothly for the pilot not to get distracted.

In summary, the realization of a visualizationmethod for groups of similar traffic can be dividedinto three steps:

1) Identification of traffic groups2) Graphical representation of these clusters3) Smooth visualization of cluster fusions and

splittings

headingdirection

∆y

∆x

Hdg ψ

o

i

y

x

ψ = atan2(∆y,∆x)−Hdg

(a) Azimuth ψ . (top-down view)

√x2 + y2

∆zθ

o

i

xy

z

θ = atan(

∆z√x2+y2

)(b) Elevation θ . (side view)

Figure 6. The calculation of the three-dimensional bearing (ψ,θ ) of an intruder i relative to ownship obased on their positions (x,y,z) in a world-fixed North-East-Down frame of reference.

(a) (b)Figure 5. Example of a traffic situation resultingin cluttering. (a) Cluttered visualization of twogroups of ground vehicles (squares), two groupsof airborne traffic and one single aircraft (circles).(b) Head-down traffic display showing a top-downview of the same situation as in (a).

Clustering to Identify Groups of Similar Traffic:In general, clustering algorithms try to group a set ofpatterns, here the feature vector of an aircraft, intodifferent clusters based on their similarity. The resultis that objects which are similar to each other belongto one cluster while objects that are dissimilar areassigned to different clusters. It can be stated thata group of similar traffic should contain vehicles thatoperate close to each other and in a flight formation.Additionally, further investigations revealed that thesevehicles must also appear close to each other on the

HMD. The features we use can be derived from thebasic received ADS-B data:

• Spatial proximity:Geometric position

• Similar flight formation:Track angleSpeedAir/Ground state

• Closeness on the HMD:Relative three-dimensional bearing

All aircraft belonging to one group operatenearby the other members of their group. However,this must not be the only feature to be comparedbecause different aircraft formations fly close byeach other, for example when their routes cross.Thus, solely spatial proximity does not form a groupof similar traffic. Similar behavior of the groupmembers is ensured by clustering vehicles based ontheir track angle and speed. Moreover, the air/groundstate broadcast via ADS-B is a suitable feature toseparate airborne from surface traffic. Finally, theproximity of the cluster members on the HMDis covered through the relative three-dimensionalbearing (ψ,θ ). The calculation of this feature, whichrepresents the azimuth and elevation angle of avehicle as seen from the ownship, is illustratedin Figure 6. The features used for clustering canindividually changed.

Clustering Procedure:For the clustering of the vehicles into groups ofsimilar traffic the DBSCAN [12] algorithm is adapted.The main reasons for choosing this algorithm are thatit does not require prior knowledge about the numberof clusters, that it relies on one input parameter onlyand that it is able to detect clusters of arbitrary shapeas well as noise. Furthermore, its density-based notionof clusters allows for the development of an adaptedalgorithm which is intuitively to use. We adaptedthe Euclidean distance function that is used by theDBSCAN to be able to handle different scales thanmetric scales.

The selected features comprise a binary value,the air/ground state, and many values with continuousdata types measured on different scales, for instancethe speed expressed in knots or the track angle indegrees. To derive a composite dissimilarity measure,one can normalize all continuous values to a rangefrom 0 to 1 and combine the results based on a chosenweighting to one single dissimilarity measure. TheGower similarity coefficient is a well-known exampleof such a composite measure [20]. Nevertheless, suchan approach implicates the additional challenges ofhow to weight the different features and how tochoose a threshold for this complex distance measure.

To avoid these problems, a different idea ispursued by our work. Instead of a single distance, theregion query checks the dissimilarity of all selectedcontinuous features i separately against an associatedthreshold εi. This means that for the assessment ofthe spatial proximity, the Euclidean distance betweenthe three-dimensional ownship and intruder positionsis compared with the chosen threshold εPosition. Like-wise, the differences between the track angles, thespeed and the bearing angles are checked. Only ifevery test succeeds, the respective vehicle belongsto the ε-neighborhood. To separate airborne fromground traffic, both types of vehicles are clusteredindividually.

The minPts parameter of the DBSCAN is setto two, implying that only single vehicles without anyother vehicle in their ε-neighborhood are labeled asnoise. In a post-processing step each of these vehiclesreceives a unique cluster ID which is a requirementfor the input of the smooth transition visualization.

Traffic Group SymbologyIn order to represent the identified groups of

similar traffic on the HMD screen, we developed avisualization with two-dimensional symbols. Thesesymbols are used to support the pilot in the detectionof surrounding traffic by indicating the position ofthe cluster. Figure 7 shows a few approaches whichwere developed to fulfill this task. Each sub-figuresketches the same traffic situation in window coor-dinates (X ,Y ), while the projected positions of thesix vehicles, represented by red dots, are framed bya different symbol in each case.

The straight-forward approach in Figure 7aapplies a rectangle enclosing all vehicles. However,the box, which is aligned with the axes of thewindow coordinate frame, also covers areas whereno vehicles reside. The convex hull, sketched inFigure 7b, overcomes this drawback by findingthe surrounding polygon, for instance by means ofthe gift wrapping or Jarvis march algorithm [21].This results in a great many symbols of various,continually changing shapes, which depend on thenumber and arrangement of the involved vehicles.Therefore, defined symbols like the ellipse and therectangle presented in Figure 7c are favored for thevisualization of traffic in HMDs. In order to resolvethe problems encountered in the first approach, thesymbol is turned in the direction of the principal axesX ′,Y ′ of the traffic group. To clearly differentiateairborne from ground traffic, we decided to visualizeair traffic by an ellipsoidal and ground traffic by arectangular symbol. As can be seen in Figure 7c,the parameters of the ellipse and the rectangle aredefined in the principal coordinate system. Thisdirection, where the data shows the largest variance,can be found by performing a PCA [22].

Smooth Merging and Splitting of Traffic Groups:With the knowledge about the clustering and thegroup symbology, all methods needed for the re-alization of an advanced traffic visualization thatdecreases the occurrence of overlapping symbols areavailable. By rendering the ellipse or rectangle foreach cluster every time, new ADS-B information isretrieved, yields abrupt changes of the traffic symbolsif two or more clusters merge to one single clusteror if a cluster splits up into several smaller trafficgroups. This poses the risk of distracting the pilot by

X

Y

(a)

X

Y

(b)

X

Y

c

c′X ′Y ′

(c)Figure 7. Different approaches to the visualization of a traffic group. (a) A framing rectangle alignedwith the window coordinate frame. (b) The convex hull of a traffic group. (c) An ellipse and a rectangleoriented to the principal axes of the traffic group.

unnecessarily drawing his attention to these events. Toavoid these undesired sudden changes of the clusterrepresentation, we present a procedure for the smoothvisualization of the transitions between partitionedand merged traffic.

The smooth visualization is conducted in inter-vals of a certain length or number of iterations N.In the first iteration of each interval, when i = 0, thecurrent traffic is clustered and the fusion and splittingprocesses to be visualized within this interval are de-termined. During all following iterations (0 < i < N)until the end of the interval no clustering is per-formed again. Instead, the unchanged group symbolsare rendered and the identified transitions betweenthe traffic groups are visualized incrementally. Thisstepwise change from the initial to the final symbolsof the particular fusion or splitting is controlled bythe Smooth Transition Progress, which represents aratio of elapsed to total time of the interval.

The scenario comprises three aircraft representedby red dots (Figure 8), which constituted two clustersafter the previous visualization interval. The clus-tering performed in the first iteration of the currentinterval delivered a new cluster comprising all threeaircraft. The corresponding symbol, which shall bethe final result of the transition process, is illustratedby the black ellipse in the left image of Figure 8. Thestandard visualization of the clustering results wouldinstantaneously draw this new cluster, the smoothvisualization proposed here, however, visualizes asuccessive transition from the initial to the final trafficsymbols.

Figure 8 sketches a timeline of the step-wisetransformation of the traffic symbols, from the initial

ellipse and circle on the left side to the final ellipseon the right side. At the beginning of the visual-ization interval the two input symbols are openedup. Afterwards, during the visualization iterations ofthe current interval, the resulting curves are adaptedincrementally until the final symbol shape is reached.Figure 9 shows an excerpt of the same merging pro-cess as above, however with additional information onimportant details of the smooth transition algorithm.

Each traffic symbol is composed of several linesegments connecting two adjacent vertices. Duringthe creation of the traffic group symbols, these cornerpoints are determined in the local principal coordi-nates (X ′,Y ′) of each symbol and afterwards trans-formed back to window coordinates (X ,Y ) for draw-ing. In Figure 9, the vertices of the two input symbols,S1, j and S2, j, an ellipse and a circle, are illustrated bythe light blue dots, while black dots show the verticesof the output ellipse, S0,k. Furthermore, the cornerpoints of the currently drawn symbols, S′1,i and S′2,i,are illustrated by the dark green dots on the greenlines. Finally, the geometric centers C1, C2 and C0 ofthe symbols S1, S2 and S0 are depicted as dark bluedots. The algorithm for the computation of the currentvertices to draw can be divided into three tasks:

1) Find the points where the two input symbolsare opened up (magenta dots).

2) Determine a starting point for the mapping ofthe input to the output vertices (orange dot)and assign each point of the input symbols toa vertex of the output symbol (gray lines).

3) Interpolate the resulting points of the currentlydrawn shape (green dots) based on the previ-ously determined mapping.

ν = 0.0 ν = 0.5 ν = 1.0

Figure 8. The step-wise transformation of the traffic symbols beginning with a clustering iteration (v= 0.0)followed by several visualization iterations (v = 0.5 and v = 1.0). The convex hull of the detected clusteris depicted as a black line.

The fusion and splitting algorithm presentedabove has two constraints that need to be consideredfor the determination of the transitions to be visu-alized in the subsequent visualization interval. First,only two symbols can be merged at once and a clustercan only be split up into two distinct parts during asingle visualization interval. Second, a symbol canonly be involved in one transformation process si-multaneously; it can not be part of a fusion and asplitting in the same interval. The method we proposeto generate such transitions between the previous andthe current clusters is shown by the example scenariosin Figure 10.

C0

C1

C2

Figure 9. Details of the smooth transition algo-rithm.

By comparing the initial constellation with thedesired final result, three separate transition processescan be identified:• fusion of the previous cluster p0 and the left part

of the previous cluster p1 to the current clusterc0.

• splitting of the previous cluster p1.• merging of the previous clusters p2, p3 and the

right part of p1 to the current cluster c1.The required symbol transformations conflict with

both constraints of the smooth transition visualizationstated above. First, the final cluster c1 is a result ofthe fusion of more than two input symbols. Second,the previous cluster p1 is part of two fusions and onesplitting process at the same time. Consequently, thefinal traffic group symbols detected by the clusteringcan not be reached within this visualization interval.Instead, all fusions and splittings which are permittedsimultaneously will be determined and forwarded fordrawing. This status is saved as initial constellationfor the following visualization interval, which impliesthat all transformations that are not performed nowwill be made up later if the clustering does not changein the meantime. In conclusion, the whole transitionprocess of the symbols in this example spans overtwo visualization intervals with an intermediate stepas depicted in the right half of Figure 10.

initial

final

intermediatei3

c0 c1

p0 p1 p2 p3

i0 i1 i2

Figure 10. Example of a multiple fusion andsplitting process with intermediate visualizationstep.

The whole process is predicated on the compar-ison between the resulting situation of the previousvisualization interval and the new desired constella-tion determined by the current clustering. These initialand final situations are contrasted by means of theassignment of the vehicles to the clusters. For theimplementation it must be noted that the clustering

does not ensure that the same traffic groups receivethe same Cluster ID in two consecutive visualizationintervals, which means that even vehicles that remainin the same cluster can have different Cluster IDs inthe previous and the current situation.

With this information at hand, the algorithmcan be divided into two consecutive parts for theidentification of splittings and fusions. Splittings areprocessed first since this decreases the number ofrequired intermediate steps in many situations. Thesmooth visualization of the situation in Figure 10, forinstance, would necessitate two additional intermedi-ate steps if the order was reversed.

For the detection of splittings, each previouscluster with more than one member is checked. Asplitting occurs if the current cluster IDs are not thesame for all vehicles of the respective previous cluster.In other words, the vehicles of this traffic group arenow part of at least two different clusters, whichrequires the visualization of a splitting. To avoid thatthis previous cluster will be used for a fusion later, itis marked as processed before the two output clustersare identified. Finally, the whole splitting process isstored for visualization.

In the example scenario this part of the algorithmrecognizes that vehicle 1 and 2 in the previous clusterp1 received different cluster IDs c0 and c1. Thus, ittriggers the splitting of the cluster into i1 and i2 andmarks p1 as processed.

The consecutive identification of fusions worksin a similar way by checking which final cluster isconstructed of at least two previous symbols thatwere not part of a splitting before. However, thedetermination of the input and output symbols isdifferent as the algorithm must account for situationswere the current cluster is composed of three ormore previous clusters. In this case, not all incomingsymbols can be fused at once but pairs of two clusterscan iteratively be merged until fewer than two areremaining. The same problem is coming up, if acluster splits up in more than two new clusters asgiven in Figure 11.

Since only one of the three occurring splittingscan be executed in the first interval, it is important tochoose a proper partitioning of the four involved vehi-cles into the two intermediate clusters. The favorablesubdivision for this exemplary case is depicted on theright side of Figure 11. First, the algorithm avoids

overlapping symbols because it keeps the adjacentvehicles 0 and 1 as well as 2 and 3 merged. Second,the procedure allows for a simultaneous execution ofthe two remaining splittings as it distributes the fourvehicles evenly over the two intermediate clusters.If three aircraft were grouped into one intermediatecluster, two instead of one additional visualizationinterval would be needed to reach the desired finalstate. Without going into the exact details, the basicidea of the algorithm is to first find the two finalclusters with the greatest distance in between, here c0and c3. Subsequently, the chosen clusters are used asseeds for constructing the two intermediate symbolsas the respective nearest remaining final clusters, herec1 and c2, are alternately distributed between them.

initial

final

intermediate

p0

i0 i1

c0 c1 c2 c3

Figure 11. Splitting a cluster into more than twodistinct traffic groups.

Visual Threat PotentialThe second major part of the advanced traffic

visualization in HMDs besides the de-cluttering is thevisual coding of important traffic parameters whichare not directly visible from a simple perspectiveprojection of the traffic positions. A suitable trafficvisualization on the HMD must employ additionaltechniques to present the required information to thepilot: The threat potential measure.

One could label every traffic symbol withadditional information such as altitude or verticalspeed trend as it is done in the head-down trafficdisplay. Such an approach was chosen by Wong etal. [23]. Moreover, the size of the symbol could beincreased as the intruder comes closer to the ownshipor the flight direction could be visualized by usingarrows as traffic symbols. However, for our approachit was decided to condense all this information toone single measure, the threat potential, so as toreduce display clutter. Supplemental data on everytarget aircraft shall be retrieved from the HDD.

This approach follows the maxim that only the mostimportant information is visualized head-mounted.

Threat Potential Measure The main question for apilot facing an intruder in the vicinity of his aircraftis, if the other aircraft poses a potential threat and ifhe is required to take any corrective action or if he cancontinue his flight as planned. This is influenced bymany parameters such as positions, speeds and tracksof both the ownship and the intruder. Only certaincombinations of these parameters make a nearbyaircraft a potential threat. However, the pilot needsto decide in short time if he must react or if he canignore a target aircraft. Therefore, TCAS provides aThreat Potential Measure calculation method that isbased on the Closest Point of Approach (CPA). Theapproach taken by the TCAS is highly affected bythe restricted knowledge of the encounter geometry.Because of ADS-B providing complete position andvelocity vectors, a three-dimensional vector analysisof the scenario can be performed instead of using theTCAS approach of dividing the range by the rangerate.

To simplify the computations, the positions andvelocities of the involved aircraft (ownship o, intruderi) are transformed to a relative frame of reference,whose static origin is located at the intruder position(Figure 12). The point of the predicted closest ap-proach of intruder and ownship is represented by theposition vector p. In terms of the known relative po-sition x = xo−xi and the relative velocity v = vo−vi,it can be expressed as

p = x+ tcpa ·v (1)

where tcpa is the unknown time at which the closestpoint of approach is reached.

In order to derive an equation for the determina-tion of this parameter, the vector w is introduced. Ascan be seen in Figure 12, tcpa can now be describedas the ratio between the lengths of the vectors w andv

tcpa =|w||v|

. (2)

Furthermore, |w| is related to |x| via basictrigonometry |w| = |x| · cosφ , where cosφ can beexpressed through the scalar product of the adjacentvectors as cosφ = −x◦v

|−x|·|v| . Combining these twoexpressions, substituting the result into Equation 2

dlim

o

iyrel

xrel

x

v

pCPA

φw

Figure 12. Definition of the parameters used bythe Euclidean vector algebra method (w parallelshifted for better visibility).

and reducing the resulting fraction yields a simpleequation for the time of the closest point of approach:

tcpa =−x◦v|v|2

(3)

As it turns out, this definition of tcpa is similar to theone derived by [24], however via a different approach.

Finally, the calculated tcpa is substituted into 1in order to compute the miss distance of both aircraftat CPA

dcpa = |p|= |x+ tcpa ·v|. (4)

The final Threat Potential Measure is thenormalized value of tcpa, that is color-coded forvisualization purposes.

ResultsIn summary, the preceding chapter presented

several methods for the de-cluttering of the HMD andfor the graphical representation of important infor-mation in a way that fulfills the head-mounted see-through requirements. The former includes a rangeand altitude filter function, a relevance filter via thethreat potential and a method for identifying groups

of similar traffic which can be visualized by a singlesymbol. The latter includes reference lines to supportthe qualitative assessment of an intruder’s relativeposition as well as the graphical representation of athreat potential, which constitutes an intuitive way toquickly recognize the most relevant intruders. Thissection illustrates how all these features are combinedto the final prototype version of the advanced trafficvisualization where the HMD symbology is connectedto the head-down visualization.

Figure 13. The traffic visualization as seen by thepilot wearing an HMD.

Figure 13 shows a screenshot of the developedsymbology. The image was recorded in a flight sim-ulator at Airbus Defence and Space, into which thesoftware is integrated. For demonstration purposes,the outside view is overlaid with the head-mountedsymbology in order to simulate the pilot’s view of thescene through the combiner of the HMD. In general,airborne traffic is highlighted by ellipsoidal shapeswhereas ground traffic is represented by rectangularsymbols. The head-mounted visualization of bothvehicle types can be switched off separately on theimplemented HDD. In this exemplary situation agroup of airborne traffic with a high threat potentialis visible in the upper left corner. To the right ofthis cluster, a single aircraft with a medium threatpotential measure can be seen. Finally, the fusion oftwo ground vehicles is captured on the right side ofFigure 13.

As mentioned above, the head-mounted trafficrepresentation is connected to the implemented head-down traffic display. Figure 14 illustrates the realiza-tion of this connection via a selection mechanism.A selected vehicle is visualized by a magenta HMDsymbol. At the same time, it is highlighted magenta

Figure 14. The connection between HMD andHDD.

within the traffic list and the heading rose on theHDD. Supplemental information about the selectedvehicle can be retrieved from the head-down info box.

(a) (b)Figure 15. An exemplary traffic situation showingthe effects of different clustering strategies forground and airborne traffic. (a) The HDD withground vehicles (tan diamonds) and airborne traf-fic (cyan arrows). (b) The situation seen throughthe HMD. (traffic positions indicated by magentadots for demonstration purposes only).

Figure 15 presents the effects of different clus-tering strategies for ground and airborne traffic. Thefeatures on which the clustering is based can beselected easily and combined in many ways. In thedepicted case, the ground traffic is grouped subjectto its position on the HMD only. Thus, the threesurface vehicles in the foreground and the vehicle inthe background, which appear clearly separated onthe HDD, are represented by one rectangular head-up symbol. By contrast, the three visible aircraft arevisualized by two distinct icons on the HMD, despitetheir overlapping. The reason for this behavior is

the clustering strategy checking not only the screenposition but also the behavior (track angle, speed) aswell as the spatial proximity, which is not given inthis case.

For this Figure 15, the symbology designed forcolor HMDs is illustrated. The monochrome visual-ization applies a decreased transparency to aircraftwith a lower threat potential. Furthermore, the numberof vehicles represented by a group symbol can becoded visually via the line width.

EvaluationThe number of primitives, that is the amount of

traffic symbols, drawn by the advanced traffic visu-alization can never exceed the quantity of primitivesin the basic approach, which indicates every vehicleby a single symbol. Again, the extent to which theprimitive count is reduced depends on the specificsituation. For example, displaying a cluster of twoaircraft with one group symbol instead of two singlesymbols decreases the number of primitives by 50%while in case of four similar aircraft the quantity canbe reduced by 75%. For each cluster comprising Nvehicles, the relation is given by

nAdv = nBas ·1N

(5)

where nAdv and nBas are the number of primitivesdrawn by the advanced and the basic traffic visual-ization respectively. Figure 5 shows a scenario wherethe number of rendered icons is lowered by 55%, from11 to 5. Similarily, the number of rendered pixels canbe reduced by our advanced approach. Additionally,the computation time measurements revealed that theproposed algorithms are capable of running in real-time.

ConclusionIn our work we present a prototype implemen-

tation of an integrated traffic visualization concepton both head-down and helmet-mounted displays.The main focus of our work is placed on the de-velopment of methods for de-cluttering the HMDand increasing the information content of the head-mounted symbology by coding important parametersvisually. The realized functions are integrated into aflight simulator, the proposed algorithms are evaluatedin terms of their effectiveness and run-time efficiency.

The central idea of the developed traffic visual-ization concept is to combine the strengths of both in-volved display types – head-mounted and head-down– while simultaneously diminishing their individualweaknesses. The role of the HMD is to support thepilot in seeing and avoiding other aircraft even if theview is degraded. Additionally, it shall help to keepthe head-mounted and the eyes out during the flight.The head-down traffic display complements the roleof the HMD by providing an overview of the trafficsituation and presenting additional information thatcan not be displayed head-mounted.

In order to automatically de-clutter the HMD,a technique for clustering the nearby vehicles intogroups of similar traffic is developed. To do so,the existing DBSCAN algorithm is expanded to dealwith non-metric scales. Furthermore, a graphical rep-resentation of the identified traffic clusters on theHMD is suggested. To achieve time coherency and toavoid distracting, sudden changes of the symbology,the cluster representation is dynamically adapted tothe varying clustering results. The developed algo-rithm generates a smooth visualization of the sym-bol transitions when clusters merge or split. Also,our work deals with the visual coding of importantparameters on head-mounted displays. The relevanceof an aircraft is expressed by one single measurethat is intuitive to use: the threat potential. For thecomputation of this property, the well-known TCAS IIlogic [24] is further developed so that it makes full useof the comprehensive traffic data provided by ADS-B. The clustering and the group symbol calculationsof the advanced traffic visualization method requiremore computation time than the basic visualizationapproach. However, the realized de-cluttering meth-ods are effective and the proposed algorithms are stillfast enough to be real-time capable even for highnumbers of vehicles to be processed.

In conclusion, our approach provides a completeprototype of a traffic visualization system, which isin many ways adaptable to the pilot’s needs andcomprises newly developed algorithms that basicallyfulfill the requirements for a potential certification.Future work will focus on integration of the algo-rithms into an operational synthetic vision system soas to conduct further tests with real ADS-B data. Fur-thermore, a thorough validation of the visualizationconcept by means of a simulator study is required.

References[1] H.-U. Doehler, S. Schmerwitz, and T. Lueken,“Visual-conformal display format for helicopter guid-ance,” in Proc. SPIE 9087, Degraded Visual Envi-ronments: Enhanced, Synthetic, and External VisionSolutions, Baltimore: SPIE, 2014, 90870J.[2] T. Münsterer, T. Schafhitzel, M. Strobel, P.Völschow, S. Klasen, and F. Eisenkeil, “Sensor-enhanced 3D conformal cueing for safe and reli-able HC operation in DVE in all flight phases,” inProc. SPIE 9087, Degraded Visual Environments:Enhanced, Synthetic, and External Vision Solutions,Baltimore: SPIE, 2014, p. 90870I.[3] Federal Aviation Administration, NextGen Imple-mentation Plan. 2014.[4] SESAR, http://www.sesarju.eu, 29.10.2014.[5] C. Lagan, 3 Coast Guardsmen lost in helicoptercrash off Washington, http://coastguard.dodlive.mil/2010/07/3-coast-guardsmen-lost-in-helicopter-crash-off-washington/, 24.04.2015.[6] Radio Bremen, Notfälle auf hoher See: Bergungmit dem Heli, http : / / www . radiobremen . de /wissen / themen / rettungsleitstelle - offshore100 . html,24.04.2015.[7] EUROCONTROL, Surveillance: Summary of Ter-minology, http : / / www . eurocontrol . int / articles /summary-terminology, 20.04.2015.[8] ICAO, Doc 9924: Aeronautical Surveillance Man-ual. 2010.[9] P. Vabre, Air Traffic Services Surveillance Sys-tems, http://www.airwaysmuseum.com/Surveillance.htm, 20.04.2015.[10] EUROCONTROL, ACAS II Guide: AirborneCollision Avoidance System II (incorporating version7.1). 2014.[11] U. Fayyad, G. Piatetsky-Shapiro, and P. Smyth,“From Data Mining to Knowledge Discovery inDatabases,” AI Magazine, vol. 17, no. 3, pp. 37–54,1996.[12] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “ADensity-Based Algorithm for Discovering Clusters inLarge Spatial Databases with Noise,” in Proceedingsof the Second International Conference on KnowledgeDiscovery and Data Mining (KDD-96), Portland:AAAI Press, 1996, pp. 226–231.[13] C. C. Aggarwal and C. K. Reddy, Eds., DataClustering: Algorithms and Applications. Boca Raton:CRC Press, 2013, ISBN: 1466558210.

[14] A. K. Jain and R. C. Dubes, Algorithms forClustering Data. Englewood Cliffs: Prentice Hall,1988.[15] A. K. Jain, M. N. Murty, and P. J. Flynn, “DataClustering: A Review,” ACM Computing Surveys, vol.31, no. 3, pp. 264–323, 1999.[16] R. Xu and Wunsch II, D. C., “Survey of Clus-tering Algorithms,” IEEE Transactions on NeuralNetworks, vol. 16, no. 3, pp. 645–678, 2005.[17] J. E. Melzer, “HMDs as enablers of situationawareness, the OODA loop and sensemaking,” inProc. SPIE 8383, Head- and Helmet-Mounted Dis-plays XVII; and Display Technologies and Applica-tions for Defense, Security, and Avionics VI, Balti-more: SPIE, 2012, 83830F.[18] S. Schmerwitz, H. Többen, B. Lorenz, T. Iijima,and A. Kuritz-Kaiser, “Investigating the benefits of‘scene linking’ for a pathway HMD: From laboratoryflight experiments to flight tests,” in Proc. SPIE 6226,Enhanced and Synthetic Vision, Orlando: SPIE, 2006,62260Q.[19] T. Schafhitzel, M. Hoyer, and P. Völschow,“Increasing Situational Awareness in DVE with Ad-vanced Synthetic Vision,” in Proc. SPIE 8737, De-graded Visual Environments: Enhanced, Synthetic,and External Vision Solutions, Baltimore: SPIE, 2013,p. 87370C.[20] J. C. Gower, “A General Coefficient of Similar-ity and Some of Its Properties,” Biometrics, vol. 27,no. 4, pp. 857–871, 1971.[21] R. A. Jarvis, “On the identification of the convexhull of a finite set of points in the plane,” InformationProcessing Letters, vol. 2, pp. 18–21, 1973, ISSN:00200190.[22] H. Abdi and L. J. Williams, “Principal compo-nent analysis,” Wiley Interdisciplinary Reviews: Com-putational Statistics, vol. 2, no. 4, pp. 433–459, 2010.[23] D. T. Wong, L. J. Kramer, and R. M. Norman,“Two Aircraft Head-Up Traffic Surveillance Symbol-ogy Issues: Range Filter and Inboard Field-of-ViewSymbology,” 2004.[24] C. Munoz, A. Narkawicz, and J. Chamberlain,“A TCAS-II Resolution Advisory Detection Algo-rithm,” in AIAA Guidance, Navigation, and Control(GNC) Conference, American Institute of Aeronauticsand Astronautics, 2013.

34th Digital Avionics Systems ConferenceSeptember 13–17, 2015