advancing multi-vehicle deployments in oceanographic field …folk.ntnu.no/torarnj/advancing...

Advancing Multi-Vehicle Deployments in Oceanographic Field Experiments

António Sérgio Ferreira · Maria Costa · Frédéric Py · José Pinto · Mónica A.Silva · Alex Nimmo Smith · Tor Arne Johansen · João Borges de Sousa · KannaRajan

Received: 26 February 2018 / Accepted: 3 October 2018

Abstract Our research concerns the coordination andcontrol of robotic vehicles for upper water-columnoceanographic observations. In such an environment,operating multiple vehicles to observe dynamic oceano-graphic phenomena, such as ocean processes and ma-rine life, from fronts to cetaceans, has required thatwe design, implement and operate software, methodsand processes which can support opportunistic needsin real-world settings with substantial constraints. Inthis work, an approach for coordinated measurementsusing such platforms, which relate directly to task out-comes, is presented. We show the use and operationalvalue of a new Artificial Intelligence (AI) based mixed-initiative system for handling multiple platforms alongwith the networked infrastructure support needed to

António Sérgio Ferreira · Maria Costa · Frédéric Py∗, · JoséPinto · João Borges de SousaFaculdade de Engenharia da Universidade do Porto, PortugalE-mail: {asbf,mcosta,zepinto,jtasso}@lsts.pt,[email protected]ónica A. SilvaIMAR-Açores & MAREMarine and Environmental Sciences Center, Horta, PortugalE-mail: [email protected] Nimmo SmithSchool of Biological & Marine Sciences,Plymouth University, UKE-mail: [email protected] Arne JohansenCenter for Autonomous Marine Operations and Systems,Department of Engineering Cybernetics,Norwegian University of Science and Technology (NTNU),Trondheim, NorwayE-mail: [email protected] RajanCenter for Autonomous Marine Operations and Systems,Department of Engineering Cybernetics,NTNU, Trondheim, Norway &Faculdade de Engenharia da Universidade do Porto, PortugalE-mail: [email protected]* – now at SINTEF Digital, Trondheim, Norway

conduct such operations in the open sea. We articulatethe need and use of a range of middleware architec-tures, critical for such deployments and ground this inthe context of a field experiment in open waters of themid-Atlantic in the summer of 2015.

Keywords Mixed-Initiative control, Marine Robotics,Control, Artificial Intelligence, Ocean science, Opera-tional Oceanography, Upper water-column biology

1 Introduction

Oceanographic field experiments, targeting large-scaledynamic phenomena, typically require the coordina-tion of multiple manned and unmanned assets to dealwith the spatio-temporal variability of the upper water-column in the ocean. The coordinated execution ofthese multiple assets, critical for co-temporal observa-tions, is a difficult challenge due to a myriad of op-erational issues ranging from limited communicationrange, bandwidth and uptime, to difficulty in creatingan accurate global view for asset location, supervision,awareness and operation. In the past, we have under-taken a series of field experiments with unmanned vehi-cles under such challenging unstructured conditions inthe context of inter-disciplinary applications not onlyof scientific interest to oceanographers [12,18] and bi-ologists [39,52], but also to first responders [9] and themilitary [36]. Uncertainty and unpredictability are of-ten the norm, thus making operations extensively chal-lenging. This is further compounded when the targetapplications involve responding to dynamic and un-predictable phenomena in such unstructured environ-ments, through the combined use of autonomous un-derwater vehicles (AUVs) and unmanned aerial vehicles(UAVs).

2 António Sérgio Ferreira et al.

Fig. 1 AUV and UAV operations from the NRP Gago Coutinhoin the Azores in July 2015.

Coordination implies the ability to envision taskcompletion in light of unpredictability while dealingwith all remaining operational constraints. Typically,coordination in marine robotics has been viewed as ameans to demonstrate nominal engineering principles[17,27,2]. Our experiments use networked robotic plat-forms tied together with a mature set of tools for deci-sion support, situational awareness, control, planning,data visualization and archiving for inter-disciplinaryexperimentation [20,40].

While the scenarios described have yield sciencedata the focus of the work presented here is not on it’sanalysis, but rather to showcase advances in operationalmethodologies for marine robotics for inter-disciplinary

experiments at sea, while drilling down to the feasibil-ity of successfully applying such methods. Furthermore,it highlights the changing nature of oceanographic ex-periments, by articulating the tools, methods and op-erational artefacts which comprise a portable labora-tory, including heterogeneous robotic platforms andtheir infrastructure. In addition, we articulate the needfor a novel mixed-initiative constraint-based plannerwhich aids operational robotic field experimentation.This tool, EUROPtus, surfaces as an extension to [9]by using temporal planning methods for operationalcontrol of vehicles, while assigning them specific coor-dination tasks. The EUROPtus planner is a ship/shoresituated temporal constraint-based automated plannerfor field experiments. The planner does not attempt tomodel all constraints comprehensively and is used pri-marily for task assignment, information gathering andsituational awareness of AUVs, all the while keeping asimple resource model of UAV operations.

We do so in the context of an actual field experimentin the deep ocean, off the Azores in the mid-Atlantic1,where UAVs were used to spot cetaceans (Fig. 1) pro-viding a GPS fix on their coordinates, as a means fortargeted oceanographic measurements with AUVs inthe water column. With this operational model, we aimto understand how far automation of field operationsinvolving multiple coordinated assets can aid or hinderoperational effectiveness. While EUROPtus has beenfield tested only in the specific operating environmentfor our experiment, we believe the principles behind itare general.

This paper is organized in the following manner:first we introduce the problem and challenges of multi-vehicle oceanographic deployments in Section 2; next,we briefly place the context of this work in Section 3and describe the tools, techniques and process we use inSection 4. Section 5 explains the approach in automatedplanning we use in this work and, in Section 6, we doc-ument experimental results from the field deploymentin the Azores. We conclude with Section 7, discussingthe results and future work.

2 Scientific Motivation and Challenges

Oceanographic field experiments typically combine re-mote sensing with in situ observations using researchvessels off of which sensors and robotic platforms aredeployed. The research vessel is guided towards an op-erational area of interest, where opportunistic needsdrive what and how the operations on-board are con-ducted. More recently, as mobile robotic platforms havebecome more robust and able to carry a diverse and use-ful scientific payload, experiments are driven by scien-tific hypothesis, typically covering the meso-scale (i.e.

1http://rep15.lsts.pt/

http://rep15.lsts.pt/

Advancing Multi-Vehicle Deployments in Oceanographic Field Experiments 3

Fig. 2 Typical setup of operators on ship/shore for vehicle con-trol in oceanographic field operations.

> 50km2) regions repeatedly for weeks with such as-sets. Campaigns like the Autonomous Ocean SamplingNetwork (AOSN-I/-II) [10,47] and the Controlled Agileand Novel Observation Network (CANON) field pro-gram [13,11,12] are prominent examples, with a largenumber of robotic and/or manned assets for samplingand observation.

The traditional approach in operational oceanogra-phy, is for the use of individual robotic vehicles con-trolled by a planning tool with decision-support capa-bility, with humans making all decisions a priori, totake measurements in a very uncertain, harsh and dy-namic setting (Fig. 2). In these scenarios, situationalawareness is often opaque, increasing operator workload[24] while prompting an increased need for operator-to-operator coordination. Our objective in this work, is tonot only augment such methods with automated plan-ning [30], but also attempt to allow a single operator tocommand multiple heterogeneous vehicles while keep-ing humans in the loop and provide maritime domainawareness.

While tele-presence techniques have had a visibleimpact in the ocean science community [53,32], a keyissue continues to be that of situational awareness,namely where a sensor or platform is, and what itis observing, over space and time. Unpredictable con-ditions over and below the sea surface and its typi-cally harsh environment, preclude full knowledge at alltimes. Adaptability and high-level guidelines are thuspreferred, so as to allow accommodation to local per-ceived phenomena, while still fulfilling the experiment’sobjectives and enforcing safety constraints.

As robotic platforms become more ubiquitous, thecomplexity of dealing with multiple assets for simulta-neous, coordinated and co-temporal observations, hasadded to this challenge. This is prevalent in the op-

erational scenario addressed in this paper, of cetaceantracking that underpins the focus of this work.

In such a domain, understanding the processeswhich influence the distribution and ecology of animalsis a fundamental problem with important implicationsfor our understanding of ecosystem function and dy-namics, and for conservation and management of nat-ural resources. Most studies investigating the environ-mental drivers of marine animal distribution have re-lied on remote-sensed data or existing oceanographicmodels. However, a more comprehensive understandingof the movements and behavior of marine animals andhow they interact with the environment requires morethan averaged observations, often collected at coarsespatial and temporal scales – animal derived data needsto be integrated with more synoptic oceanographic ap-proaches [19]. Standard methods to collect concurrentobservations of oceanographic conditions and of the dis-tribution and behavior of cetaceans use large, expensiveocean going vessels. As a consequence, oceanographicsampling is limited to a few stations providing but asnapshot of oceanographic parameters.

In addition, operation of large vessels may inter-fere with natural animal behavior. Recent advancesin electronic transmitters and data-storage tags havemade it possible to collect 3D high-resolution biologicaland oceanographic observations from animals in theirhabitats. However, because of satellite availability andtransmission costs, tag recovery is usually necessary inorder to access collected data, limiting the use of thesetags to animals that reliably return to the same lo-cation or remain within a small geographic area, andshortening deployment durations. Moreover, as thesetags remain relatively large they cannot be attachedto most free-ranging cetaceans without the potentialto impact their behavior. Because of these limitations,research into the oceanographic drivers of cetacean dis-tribution continue to rely on the analysis of sightingdata or low-resolution satellite tags and remote sensedvariables [5,48]. These data can be used to correlateestimated positions from surfacing cetaceans with re-mote sensed or model-derived oceanographic variables,usually available at sparse temporal resolutions.

Using unmanned assets, deployed at sea, enablecost-effective near real-time monitoring of ocean pro-cesses and cetacean movements and behavior. UAVs canbe deployed in the areas of interest and tasked to fol-low cetacean positions in an autonomous manner tak-ing into account detection, classification and trackingmethods [28]. AUVs can then be deployed to sample thewater column on the surfacing locations, in order to pro-vide complete synoptic observations around and abovethe target individual(s) – AUVs are also less prone tointerfere with normal behavior of animals due to theirreduced footprint. Finally, if AUVs and UAVs can per-form co-temporal observations, a portable observatory


can be created where animals and surrounding condi-tions are observed continuously in their natural habitat.

However, the current generation of AUVs can facechallenges of either speed or endurance when trackingfast moving cetaceans. Typically, vehicles that can stayin the water for extended periods of time are only capa-ble of speeds below 1 m/s [54,31]. Moreover, communi-cation with these vehicles is also limited: only satellitecommunications are available at remote locations andonly when vehicles are at the surface. Equally, acous-tic communications are noise prone and have limitedbandwidth and range. Thus the current capabilities ofvehicles can only provide sparse location informationor engineering telemetry at best. In addition, althoughUAVs can be continuously connected to control stationsby using short-range (tens of kilometres) point-to-pointcommunications, or alternatively by using expensive in-termediate (satellite) gateways, their endurance rangesto a few hours at best.

Finally, coordination of UAVs and AUVs carries itsown challenges, requiring the fusion of two very dif-ferent vehicle types, with specific communication, lo-comotion and sensor payloads, into a unified domainwhere they share common objectives. Due to the highvariability of ocean conditions, it may be required toadapt plan execution on board the vehicles according tothe perceived environmental conditions. The computa-tional power available on board these vehicles, however,is limited for running on board planning algorithms.

3 Related Work

Coordination of multiple heterogeneous robots requiresa common communication infrastructure that is usedto share the state of the different intervening assets toperform collaborative behavior. There are various opensource robotic frameworks that can be used to coordi-nate multiple robots, but usually these target controlledenvironments, where reliable and continuous communi-cation are available.

The Mission-Oriented Operating Suite (MOOS)[38]relies on a robust communications library to share databetween applications, using a centralized node calledMOOSDB. This node acts as a knowledge informa-tion base where all other nodes can send informa-tion to and receive information updates. The widelyused Robotic Operating System (ROS), uses a pub-lish/subscribe communication system that relies on acentral node service for node registration and look-up,called the master node [43]. As such, whenever sepa-rate robots are to be coordinated, they should use acommon master node running in either of the two orin another accessible network location turning it into acentral point of failure. More recently, the possibility touse ROS in a multi-master architecture was introduced

where multiple master nodes co-exist and exchange in-formation being, however, still required to be awareof each other prior to their execution. The recently-developed GAMS middleware supports collaborativeautonomy amongst heterogeneous teams of agents us-ing non-blocking, best-effort communications protocols,which allows operation even in communications con-strained environments [15]. All nodes under a GAMScollaboration system have their own knowledge basewhich, using MADARA, synchronizes relevant infor-mation with peers when that information is changed[16]. The LSTS Toolchain [40,20], used in this work,was developed to support different robot and com-munications hardware, mission purposes and operatorneeds. Reference implementations are freely availablefor AUVs, ASVs, ROVs and UAVs1. In its case, theInter-Module Communication protocol (IMC) definestransport-agnostic discovery mechanisms, allowing it tobe used in unreliable and heterogeneous networks usingany available links such as acoustic, Iridium or Wi-Fi.Dealing with uncertainty in communications, especiallyin the context of oceanographic field experiments, istherefore key to the tool-chain. For instance, IMC tar-gets unreliable communication environments and pro-vides dynamic discovery algorithm implementations.All messages in IMC are time-tagged and include infor-mation about its origin and destination sub-systems.

Besides considering the communication needs forcoordinated operation of these systems, vehicle auton-omy is also a factor. Onboard autonomy can vary fromthe very basic tele-operation to fully intelligent systemsthat generate and adapt plans on board, in response toobjectives and environment. In oceanographic experi-ments, simultaneous operation of multiple vehicles istypically along institutional lines, with a group of indi-viduals focusing on one specific robotic platform [47].Our work is unusual in that a small group is focusednot only on a collection of vehicles, but that they arealso heterogeneous; in such cases, the amount of atten-tion that can be devoted to each entity is often limited.Compounding this, autonomous operation of UAVs isusually done by defining a sequential list of waypointsthat the vehicle must follow. This is a very efficient ap-proach for static and collision-free environments whilerequiring very limited onboard autonomy. Moreover,some execution environments also have triggers that usecontingency waypoints whenever errors are diagnosed.However, in more dynamic and unknown scenarios orwhenever the phenomena of interest is unpredictable,a more deliberative planning is required. There are anumber of approaches to deliberation.

MOOS-IvP Helm is a behavior-based architecturethat uses multi-objective optimization [3]. In MOOS-IvP, different behaviors may have different priorities

1http://github.com/LSTS

http://github.com/LSTS


Fig. 3 The software components used to control unmanned ve-hicles with the LSTS toolchain.

and triggering mechanisms and can be simultaneouslyactive, in which case they compete to define the ongo-ing actions of the vehicle. If all behaviors have differ-ent priorities, the resulting architecture can be equiva-lent to Brooks’ subsumption architecture [8]. However,whenever two competing behaviors become active, theHelm selects an action by maximizing a resulting value.NASA developed MAPGEN (Mixed Initiative ActivityPlan Generator) combining the EUROPA planner [26]with a user interface that allows human operators tointeractively define plans, by providing an initial state,prioritized goals, standard constraints and daily con-straints [1,6,7]. Such Artificial Intelligence (AI) basedmixed-initiative methods for planning, continue to benovel in the oceanographic domain. Typically, a map-based user interface is used as a planning tool, wherethe ’plan’ is essentially a sequence of waypoints care-fully constructed by the human operator, dispatched toa vehicle out at sea. Such software systems offer basicdecision-support capability, with little in the way of ma-chine intelligence with human decision-making drivingrobotic activity. NEPTUS [40], which we later describe,is one such mature tool in this category and we buildon its functionality.

In this work we extend the shore-based mixed-initiative planner demonstrated in [9] for controllingmultiple AUVs, where the underlying automated plan-ner did not allow re-planning to deal with environmen-

tal uncertainty or dynamic events, nor reasoning abouttime and resources, as this work does. Doing so, al-lows us to use active simulation on shore or ship tobe coupled with the proscribed execution trace whilekeeping the operator situationally aware. Furthermore,in this work, we use the same temporal formalism onboard our AUVs as we used in T-REX in the contextof upper water-column exploration [42,46]. Finally, weare informed by efforts on the command/control of theOpportunity rover on Mars [1,6] using a similar plan-ning approach we use here. The operating domain atsea however, is far harsher and substantially more con-strained with a far more dynamic pace and operationalfluidity.

In summation, the application deals with coordi-nated measurements using robotic platforms, where thecoordination is related to task outcomes and not spe-cific navigational needs. While [49] also demonstratesthe use of multiple robotic assets in the water-column,albeit in shallow near-shore waters, to the best of ourknowledge using automated planning [30] as a meansto provide abstraction in control over single or multi-ple vehicles in the oceanographic domain, as this workdoes, continues to be novel.

The work in this manuscript builds upon the LSTStoolchain which has extensive operational use in re-curring large scale exercises [34,51], is open-sourceand with a growing group of users and contributorssuch as Norwegian University of Science and Technol-ogy (NTNU), Monterey Bay Aquarium Research In-stitute (MBARI), Laboratory for Underwater Systemsand Technologies (LABUST), Natural EnvironmentResearch Council (NERC) and Centre for Maritime Re-search and Experimentation (NATO-CMRE), amongothers. Furthermore, the use of the LSTS toolchain asa baseline proves advantageous for scalable use of thetool due to the ongoing adaptation of the toolchainto other communication protocols (e.g., MAVLink,JANUS, MOOS) and compatibility with vehicles fromother manufacturers (e.g., LRAUV, Waveglider, IVER,SPARUS II). These adaptation and integration effortsopen up the range of what can be done, operationallywith this work.

4 Technical Approach

In order to control a set of heterogeneous platforms,a number of components needed to be developed oradapted. Our software architecture is built on top of theLSTS Toolchain [40], and uses the DUNE onboard soft-ware for control and navigation of all UAVs and AUVs.IMC is used across all systems to exchange informationand NEPTUS has been extended for providing globalsituational awareness and supervision for human oper-ators (Fig. 3).


4.1 The Command and Control System

The DUNE (DUNE Uniform Navigation Environment)onboard software has been developed to support all itsunmanned vehicles and other embedded systems suchas communication gateways and data loggers [40]. Itprovides a uniform navigation environment that is irre-spective of sensors used to estimate a system position(localization) or actuators used to interact with the en-vironment; there exist standardized structures that canbe used for abstract representations of estimated anddesired system states. This allows sharing most of thecode between all vehicles and being required to adaptonly low level actuation and sensing.

DUNE uses a publish/subscribe system for message-passing between modules that run concurrently in a lo-cal system or distributed over the network. All messagesflowing within and outside a DUNE system are definedin IMC . A conceptual view of a DUNE based system,where device drivers and transport components (whichhandle communication with the world) is illustrated inFig. 3.

NEPTUS is a software infrastructure for develop-ing user applications that interface Networked VehicleSystems [14]. It is currently used for operating fleetsof heterogeneous autonomous vehicles by diverse end-users such as biologists, oceanographers, archaeologists,students or the military. It is used not only duringplanning, but also to supervise execution and revisedata after mission execution [41]. It uses an extensi-ble architecture which supports different plug-in types:Console Widgets, Console Daemons, Interactive MapLayers, Interactive Data Visualizations, Plots, etc. In-side the software, all plug-ins interact with each othervia Google’s Guava Event Bus1, a library that en-ables loosely coupled communication between compo-nents via asynchronous message-passing, similarly towhat happens in DUNE. Operation consoles, for in-stance, can be configured to match the needs of specificmission objectives or operator preferences by definingactive plug-ins and their configurations . Even thoughDUNE and NEPTUS were developed in two differentprogramming languages (C++ vs Java, respectively),the two systems share most of the design patterns anddevelopment approach.

4.2 Onboard Plan Deliberation

In order to cope with uncertainty and environment in-terference during execution, vehicles need to adapt theiractions according to the perceived environment. In ad-dition, they also need to potentially change the orderand set of actions to execute, in order to maintain ve-

1https://github.com/google/guava

hicle safety while attaining the most important missionobjectives.

The problem of running computationally-intensivedeliberation systems on board cpu-constrained AUVshas been approached by developing a new mission ex-ecution system named Teleo-Reactive EXecutive (T-REX) [35,42,45,46]. T-REX 2 allows creation of de-liberative agents by having simultaneous execution ofmultiple planners with heterogeneous latencies (timeto produce a new plan based on new observations), aswell as different planning windows (time span of pro-duced plans). This “divide-and-conquer” strategy forproblem-solving is encased in this decomposition of theT-REX agent into different software components (reac-tors) making it possible to have planning and reactivebehaviors running in parallel and continuously.

T-REX does this by providing a software frameworkwhere different reactors interact with each other usingtimelines: a flexible representation to describe the evo-lution of a given state variable over time. Reactors pro-vide internal (controlled) timelines where they publishstate updates (observations) and use external timelinescontrolled by other reactors in order to read its stateand post desired goal states. Deliberative reactors pro-duce a plan to attain the desired goal states if theyfall within their planning window, and do so with theunderlying use of the EUROPA planner.

EUROPA is an AI-based planning system used togenerate plans executed onboard NASA’s 1999 DeepSpace One probe with the Remote Agent Experiment(RAX) [4,37,26,44] and on the ground as a mixed-initiative planner for the 2003 Mars Exploration Roversmission with the MAPGEN system [1,6,7]. EUROPA isa vastly re-factored, higher performance open-source3

version of the systems used here and uses a domainmodel written in NDDL (New Domain Description Lan-guage), together with initial conditions and goals (alsoin NDDL), to construct a set of temporal relations thatmust be true at start time. These models include asser-tions about the physics of the vehicle, i.e., how it re-sponds to external stimulus and internally driven goals.An in-depth description of how domain models are de-fined and plans deliberated is provided in Section 5.

A simplified version of our AUV’s domain modelis depicted in Fig. 4. The “Medium” timeline may becontrolled by an Interface reactor that simply trans-lates sensor readings into observations of the vehicle,being either underwater or at the surface. Since theAUVs are positively buoyant, whenever the vehicle is’Idle’ it will float towards the surface. Moreover, a de-liberative reactor controls the “Navigator” and “Yoyo”timelines. Whenever the vehicle needs to reach a des-tination x, the Navigator reactor will produce a plan

2https://github.com/fredpy/trex2-agent/3https://github.com/nasa/europa

https://github.com/fredpy/trex2-agent/

https://github.com/nasa/europa


Fig. 4 An example of timelines (with their names on the left)and token relationships used on the domain model for AUVdeliberative control.

Fig. 5 The principal reactors running on the LSTS AUVs insidethe T-REX framework for the July 2015 off of the Azores.

that projects the vehicle being at x at a future time.For that, the vehicle should start descending and theniteratively switch between Ascend and Descend modes,defined in the “Yoyo” timeline. The semantics of “ar-riving” at the destination implies stopping, in order tofloat to the surface. The undulating “Yoyo” behavioritself, is important for all underwater vehicles to cap-ture the variability in the upper water-column acrossthe vertical rather than the horizontal dimension.

Fig. 5 shows how different reactors can interact byexchanging goals and observations. From bottom totop, the reactors further abstract the environment andtend to have larger planning windows and latencies.The “Survey” reactor will produce plans that have sev-eral waypoints that the vehicle should travel to. Thesewaypoints are passed as objectives to the “Navigator”reactor that, as described previously, will trigger a yoyobehavior in the vehicle.

Fig. 6 Communication links between consoles and vehicles.

To integrate T-REX, running on an auxiliary CPU,with the LSTS Toolchain, a special DUNE task wasdeveloped that translates DUNE data into T-REX ob-servations and parses incoming requests into vehicle ac-tions. When T-REX receives a goal, its objective is tosynthesize partial plans and dispatch these to DUNE,while simultaneously tracking their execution onboarda vehicle.

In order to translate T-REX requests into DUNEactions, DUNE was extended to allow reception ofexternal control references such as desired speed,depth/altitude, target position and/or heading via a“back-seat driver” API. The API accepts both com-plete or incomplete reference definitions. For instance,if so desired, the external controller can specify a speedto be maintained while travelling towards the targetpose or choose to not specify it. As a result, the back-seat driver will use an optimal (and possibly varying)speed while going to the target. Something similar oc-curs with altitude, depth and hovering radius1 actions.The flexibility so provided allows for the developmentof external controllers, which are not tied to specific ve-hicle hardware, allowing DUNE to use provided slackto improve vehicle safety, navigation or battery opti-mization.

4.3 Communications Infrastructure

A cornerstone of multi-vehicle operations is its commu-nications infrastructure. In the case of oceanographicfield operations, heterogeneous and opportunistic linksusing a mix of Wi-Fi, satellite communications, GSMand acoustics are critical. Our approach has been touse the IMC protocol across all systems, irrespectiveof the available underlying transport mechanism. Thisprovides additional flexibility, in that functional partsof the software can be either implemented on board thevehicles or distributed on the network and accessed us-

1The acceptable distance to the desired end point


Fig. 7 The Ripples HTML interface visualized in a mobilebrowser (left) and the same data being accessed by Neptus(right)

ing any of the available communication methods. More-over, the portability provided by DUNE allows it to runnot only on vehicles but also on communication gate-ways, which can be used to extend the network andbridge different communication methods. These gate-ways allow operator consoles to establish a connectionvia Wi-Fi or Ethernet and then use its acoustic, Iridiumor GSM modems to communicate IMC data.

IMC embraces heterogeneous means of communi-cation by allowing nodes to be discovered over differ-ent interfaces and announcing its provided capabilitiesand names using transport-agnostic identifiers. For in-stance, a vehicle may be discovered by another usingUDP multicast but afterwards, communication can bemaintained between the two using TCP, UDP, HTTP,acoustic modem, etc. Deciding which communicationmethod to use is the responsibility of the software andnot the protocol; however, efficient implementations areavailable in NEPTUS and DUNE that can try all differ-ent reliable protocols (in order of degrading bandwidth)until the message is successfully delivered at its desti-nation [33].

Vehicle operations must ingest the data coming frommultiple sources and allow variable latency for incom-ing data and outgoing commands. For aggregating datafrom multiple sources we use a centralized communica-tions system, named Ripples, running in the cloud anddeveloped using the Google App Engine. This cloud-based application ingests data pushed through Irid-ium, Globalstar and Argos satellite communications,as well as several web entry points, for posting real-time and/or historical information (Fig. 6). All incom-ing data is stored in the cloud and can be forwarded toother recipients, via Iridium or Web Sockets. Ripplesalso provides a web interface where the telemetry andreal-time logs of field events are accessible via a browser(Fig. 7).

To allow operations even in situations where the In-ternet is not available, such as the open sea, Ripplesallows on-demand subscription of Iridium updates. Aspecial entry point is used to subscribe/unsubscribe viaIridium. Whenever a message is sent to all systems (spe-cial IMC broadcast destination), it is forwarded to allsubscribed systems.

Irrespective of how data arrives at the Manta gate-way – Wi-Fi, Web, acoustic modem or Iridium – it isdispatched to all connected NEPTUS consoles. The re-ceived information is then processed by any NEPTUSplug-ins that have subscribed to that message type. Todistinguish the messages arriving from multiple sourcesand means, the IMC message header includes informa-tion such as identity of the generating system and timeof message generation. The Mantas can then be usedto send commands back to the vehicles using the avail-able links. In practice, message flow from the vehiclesto consoles using Iridium is as follows:

1. Vehicle arrives at surface and is not connected toany console;

2. Vehicle sends state update via Iridium not address-ing it to any system;

3. An Iridium service provider forwards Iridium mes-sages to Ripples using a POST request;

4. Ripples verifies the list of active Iridium subscribersand forwards the message to them, serially;

5. The message arrives at an Iridium device inside theconsole’s WLAN (manta gateway);

6. The gateway forwards incoming Iridium messagesto all connected consoles.

A similar (in reverse) route is used to send com-mands to the vehicles but, in this case, messages are ad-dressed to the target system or else they are forwardedto all Iridium subscribers as before.

4.4 The EUROPtus Mixed-Initiative Planner

Maritime field experiments usually require having mul-tiple vehicles disconnected for substantial periods oftime (in excess of 15−30 minutes). In order to improvethe operators’ situational awareness of vehicle positionsand progress, we used a number of simulated vehiclesrunning off of commands sent to the actual platforms.This was done by running DUNE and T-REX simu-lators side-by-side with NEPTUS. NEPTUS was con-figured to forward any commands sent to the actualvehicles also to their respective shore-side simulators.As a consequence, the simulated vehicles execute thesame set of commands albeit in an idealized environ-ment. While doing so and in periods of loss of contact,operators can determine with clarity, what each asset isexpected to be doing – clearly in off-nominal situations,this simulation does not reflect reality.


Fig. 8 NEPTUS operator console during the cetacean-trackingexperiment, July 2015. (1) is the location of the control station(onboard ship), (2) is the real-time location of a flying UAV,(3) is an AUV position received via acoustic modem 3 minutesbefore, (4) is the simulated position of the same AUV and (5)is a plot of data received via acoustic modem from the AUVs.

In the case of AUVs, any incoming updates (receivedover any available communication channels) are used toreset the simulated vehicle’s localization filter, similarlyto what is done when the vehicles arrive at the surface,as shown in Fig. 8.

In order to coordinate the actions of multiple vehi-cles, operators can use NEPTUS to override the currentbehavior on the vehicles using any of the available com-munication means. However, manual control and super-vision of multiple heterogeneous vehicles becomes infea-sible as the number of vehicles and latency increases.This happens due to the increased uncertainty and dis-continuous changes regarding the perceived state of thesystem, in addition to information overload on the op-erator.

In order to help operators coordinate fleets ofrobotic vehicles, we have designed, built and testedEUROPtus, a ship/shore situated temporal constraint-based automated planner which deals with operationalconstraints for field experiments. EUROPtus leveragesexisting work in fully-autonomous AUV operations andcouples that with the need to operate multiple (of-ten heterogeneous) robotic vehicles. It has a simpleresource model which is integrated into the plannerto enable as complex a coordination model as neces-sary; by using EUROPtus, the operator offloads a por-tion of the planning task, which is quite relevant givennetworks with very high variability of vehicle configu-rations and capabilities (while interacting operationalconstraints and substantial operational telemetry fromvehicles with varying speeds). The operator, however,is still in charge of providing high-level goals, supervis-ing the plans sent to the vehicles and override them ifnecessary.

A special NEPTUS plug-in was developed to actas a routing device for incoming and outgoing mes-sages. The plug-in redirects data from EUROPtus to

controlled/simulated AUVs and, at the same time, re-ports all received updates to EUROPtus; converselythis plug-in also notifies UAV operators about new in-coming requests – a new deploy request or survey areafrom EUROPtus, for instance.

When EUROPtus determines so, it can request newobjectives for vehicles deployed in the field or prompthumans behind an operation console. For instance, op-erators are expected to execute some task and provideinput (e.g., inspect collected data and determine a listof waypoints to be visited) or, EUROPtus generatesa high-level objective that is sent to an autonomousvehicle to replan in situ accordingly. If a new plan isfound, this is reported to the operator from which theycan both provide new objectives and inputs, and/or re-call existing objectives. This is the focus of such mixed-initiative interaction between vehicle(s) and operator.

5 Planning and Execution Approach

Managing a mission with interdependency of multipleassets, some human operated, is often a challenge. Itbecomes even more problematic when assets like AUVsare often out of communication range – either becausethey are underwater or simply because they have tooperate at a safe distance from a control station – andacting for the most part autonomously. Coupled withdynamic operational constraints, including the impactof weather and sea state, a substantial part of the op-eration requires human decision-making. Equally, syn-chronizing the operation of multiple robotic assets withship-based operational constraints can overwhelm theoperator. For instance, in our experiment, our researchvessel had to be oriented to have the aft deck facing atail wind for stable UAV take-offs; but UAV landingson a net facing starboard, required the ship to pivottowards the wind coming from the port side. Unify-ing such complex constraints (which require ship move-ment, and coordination with the bridge, for instance)while robotic vehicles are in operation, continues to re-quire human judgement and involvement.

We used a mixed approach to autonomous platformoperation, where a planner and its executive on the shipare used to track and supervise as much as the execu-tion of the mission as possible, to alleviate these oper-ational constraints. This calls for an approach differentfrom fully autonomous embedded planning/execution.Equally, in structured laboratory experiments, it is as-sumed that execution can be tracked and every observa-tion from the agent(s) are fully observed and in a timelyfashion; this is often not so in real-world environments,especially at sea.

We believe that planning and execution interact,and tightly so. Such interaction goes beyond the classi-cal conception of interleaving planning and execution,


such as in [29], which continues to regard planning as anoff-line process interrupting execution until it identifiesa complete plan. We see execution as an integral partof the overall deliberation process that both interruptsand enriches the planning as the world evolves. Thiskey concept provides the backbone of how EUROPtusis designed.

5.1 EUROPA Planning Essentials

In EUROPtus, a plan is composed of a tempo-rally scoped predicate called a token. A token canbe defined as a property – described as a first-order logic predicate – with its associated tempo-ral scope (start, duration, end), using flexible intervalarithmetic. All the attributes of a token are describedas a domain of possible values for this token in theplan context. In order to be part of the plan, a tokenneeds to be associated with one of the plan timelines.A timeline is a sequence of tokens describing the evolu-tion of a state variable. Concurrency between timelinesand therefore between tokens on separate timelines, isthe basis for concurrent state variable evolution. To-kens are causally linked by rules in a domain modelthat describe temporal relations and/or causality linksbetween tokens [22,42] (see Fig. 4). Finally, a token canbe marked optionally as either being a Fact or Goal;while a Fact requires no justification, a Goal will neednot only to be inserted in the plan but necessarily havea causal chain connecting it to one or more Facts. LikeT-REX, the underlying planner for EUROPtus is EU-ROPA [26,22]; while EUROPtus’s concepts are not tiedto this specific planner, our implementation relies heav-ily on both its flexible and rich representation, as alsothe basic principle in the way search is implemented inplan-space planners [30,23].

The planner works by continuously repairing flawsin a plan until no more flaws are present. Prior to execu-tion, all partial plans must not have any flaws. Typicallywe deal with two types of flaws 1:

– Open condition: A token is not yet associated witha timeline. It can be resolved by either inserting thetoken into a specific timeline or merging it with acompatible token;

– Threat: Once a token has been inserted it may im-pact other tokens indirectly through possible over-lapping requirements. A timeline by definition, en-forces a strict sequence of tokens with no concur-rency within the timeline. The solver then needs toenforce a scheduling constraint on those potentially

1There can be additional flaw types depending on the solverbeing used; but these are the minimum needed to converge toa solution.

conflicting tokens, so they cannot overlap; for exam-ple, by enforcing that one should occur before theother.

The solver resolves these flaws until either it reachesan inconsistency i.e., a situation where constraints ofthe plan cannot be satisfied), in which case the plan-ner will backtrack to explore alternate solutions; or aconsistent solution is found and the plan presents nofurther flaws generating a valid solution.

Note that while the plan might be complete, it doesnot have to commit to the value of its variables. Forexample, the start time of a token can be left to be theinterval [1, 10] as long as it does not present a threatto the partial-plan. This leaves the decision of the starttime to the executive, which is critical for operating inuncertain real-world environments.

5.2 The Cetacean Tracking Domain

We can briefly describe the cetacean tracking domainprimarily as a way to motivate the use of EUROPtusand thus, as a way to ground the methodology in thissection and experiments in Section 6.

In this domain we consider two types of assets: aUAV that is externally operated (with human-in-the-loop waypoint-based control) and for which the onlyobservable outcome is a cetacean sighting; and AUVs(two in our case) which have in situ goal-oriented com-manding using T-REX and can receive survey objec-tives via a timeline goal. The UAV launch and recoveryoperations involve substantial human involvement andits operation currently requires close monitoring. Con-sequently, the basis of interaction with EUROPtus wassimplified as follows:

– EUROPtus can request a new deployment throughNEPTUS directed at the UAV operator;

– the maximum UAV operation time is assumed to be30 minutes, given available battery life;

– recharging a UAV battery on shore/ship takes ap-proximately 15 minutes 2;

– a cetacean position update is expected to be theobserved outcome of a successful UAV survey and,when available, generates a goal within EUROPtuswhich is instantiated on a timeline.

As a result, EUROPtus approximately models UAVoperational constraints and its interactions are indirectas it only sends an IMC message to the UAV operatorsconsole to trigger the operators response; and its obser-vations similarly are driven by a human operator’s eventwhen s/he manually identifies and “marks” a cetaceanposition on NEPTUS.

2Both the launch/recovery operation and charging timeswhere approximate, yet reasonable estimates.


The EUROPtus AUV model was based on the fol-lowing:

– a timeline representing the vehicle position updatedwhenever a position update was received and placedin the timeline according to its observation times-tamp;

– A set of possible surveys both parameterized withtheir scale in meters (representing the outer boxsurrounding the survey), its centroid (representedby a latitude and longitude) and its orientation (arotation angle) [12];

– The high level operational state of the vehicle be-ing either Inactive, Operating and Survey, the lat-ter which takes as an argument a fully instantiatedsurvey as above.

Typically within EUROPtus the AUV’s overall statecycles between Inactive, Operating and Survey whileexecuting a survey. The duration of Operating dependson the scale of the survey and the distance from thesurvey start point (which should be the last positionobserved when the vehicle was Inactive and on the sur-face). Survey being a goal state, its duration is veryshort (equivalent to 1 sec on the embedded T-REX)since it is a feedback confirming the successful comple-tion of the survey request. This timeline – along withthe AUV position – effectively produced by T-REX on-board, were the means of interaction between the AUVand EUROPtus on shore/ship. Time-stamped messagesfrom the AUVs, as noted, come with significant time de-lays. This means that the executive should not only beable to dispatch the actions in a timely manner, butalso to integrate observations from the past and, whenrequired by the model, delay the execution of a specificaction until its conditions are effectively observed.

5.3 Planning and Execution with Asynchronism

Typically in fully autonomous systems such as T-REX,planning and execution are intertwined. Both manipu-late the same plan representation and plan execution isrequired to occur at every clock cycle. This influencesin turn, the outcome of the planning process while en-suring that any plan produced by the planning agent(EUROPtus in our case) is taking into account worldstate evolution. And so, it assumes that the agent iswithin a synchronous and fully observable world. Suchdesign, while appropriate for embedded systems, is in-compatible with the oceanographic domain where op-erations are asynchronous and where coordination onlaunch and recovery operations can be complex. Fur-thermore, in our case, observations from AUVs arrivesporadically, and observability is limited by the avail-ability of an acoustic channel to the AUVs.

For EUROPtus, computing power is not a criticalissue, but communication limitations and observationalupdates need to account for the fact that world evolu-tion will often be observed later than occurrence, if atall. Execution can still be integrated into a delibera-tion process as planning and we go a step further inconsidering that execution is a part of planning [23].

5.4 Tracking Execution within Deliberation

Execution feedback is integrated into the the delib-eration process by taking full advantage of the waythe planner works. Specifically, the planner will stopsearching as soon as no more flaws are found; the intro-duction of a new token (including Facts) in the plan,creates new flaws in the plan.

Let’s consider that the planner has no more flawsand that the next command to be executed has beenalready dispatched. Because of communication disrup-tion, we receive feedback from AUVa that indicates thatits state changed from Inactive to Operating an hourago. Our approach is then to create the Fact token Op-erating for AUVa starting at the time corresponding toan hour ago. This token is added to the plan generat-ing a new Open condition flaw that the solver needs toresolve.

The resolution can be either as simple as a merge,if the token reflects exactly what was planned, or it isconflicting with the partial plan requiring in turn, theplanner to backtrack. The insertion of such a token as aFact however, is akin to plan recognition in the contextof the currently maintained plan. As the planner op-erates continuously keeping its search state alive, thisrecognition can impact the search by forcing the plan-ner to backtrack over past decisions until it finds analternative new solution, given the injected Fact. Aslong as we assume that the decisions impacted by thenew observation are not close to the root of the searchtree, it allows for a plan resolution in few steps with-out an adverse impact to performance. Further, suchan assumption is reasonable as often past observationsarrive in relative chronological order rarely impactingthe distant past in plan history. These steps also high-light how execution tracking can be a pure deliberativetask within the same planning engine.

The remaining problem is to decide which part ofthe plan is ready to be executed and sent to NEPTUS.Many actions in a partial plan can be conditioned bythe need to observe a situation. Actions are specific to-kens introduced in EUROPA, that instantiate a causalrelationship between their condition and effect tokens.It reflects the classical approach to describe a plan do-main (similar to STRIPS [21] but substantially moreexpressive) and allows its use as additional semanticsfor our need. In EUROPA, an action is a special kind of


token with temporal relations expressed either as con-ditions necessary for its execution, or expected effectsof this actions [46].

To command the AUVs for a survey we needed EU-ROPtus to observe the following:

– Both AUVs are ready and in the Inactive state;– We have a cetacean position update that is at most30 minutes old – this heuristic was imposed to en-sure “freshness” of cetacean tracks.

When deciding on dispatching a partial plan for anAUV survey, EUROPtus does an analysis of the causal-ity structure of an action with its tokens. We do so byintroducing the notion of a Justified token as follows: atoken is Justified if either it is a Fact token, the condi-tion of a Justified action or it is an action for which allthe effects are Justified.

In a complete plan, an action in the plan can be dis-patched for execution when all its conditions are Jus-tified and its start time interval contains the currenttime. While action justification is reasonable, we needto ensure that we do not dispatch the action before itsvalid start time. However, we could potentially be ina situation where an action does not have all its jus-tified conditions due to one or more missing messagesand yet its start time is before the current time (i.e., itshould have been dispatched for execution in the past).Dispatch is then postponed and we address this as aPending action flaw, which applies to any action of theplan that could start at the current time but does nothave all of its conditions Justified nor is it Justified it-self. Its default resolution is to restrict the start timeto be postponed. Consequently the planner needs topostpone dispatching this action until either new ob-servations justify the action or the start time can nolonger be pushed; the latter triggers a backtrack for analternate solution.

Any pending action that has all of its conditionsJustified is dispatched to NEPTUS for execution, whicheventually will receive the observation of its completion(from the vehicle) and report it to EUROPtus. This re-sults in a control loop that is managed as a pure con-tinuous deliberation process governed by the principlesjust described.

The position update then comes into play for theAUV operations model. As the AUVs are driven by anembedded T-REX agent, EUROPtus could be furtherextended to directly leverage such a positional updatefor “direct” control of the vehicles. Yet the limitation interms of communication had to be taken into account.Our AUVs can communicate with the ship only if theyare at the surface and either in WI-FI range (whichmight also be not desirable as the ship can present athreat if operated too close to the vehicle) or is at thesurface long enough to initiate a satellite connection.

Fig. 9 The location of the cetacean tracking experiment off ofthe island of Pico in the Azores in the mid-Atlantic.

Fig. 10 Architectural block diagram of coordinated observa-tions of a UAV with two AUVs in the water-column in July2015 off of the Azores.

When the vehicle is underwater, it was assumed thatthere is no means to communicate 1.

6 Experiment Domain and Setup

In 2015, a key objective of our annual inter-disciplinaryfield experiment was to perform co-temporal surveys forsynoptic observations on the habitat of sperm whales.Cetacean tracking experiments were conducted in openwaters south of Pico Island in the Azores (Fig. 9).

1Note: acoustic communication was only used for vehicletelemetry.


(a) Launching the LAUV-Xplore-1 off of a pier in the is-land of Horta for testing. Inset image shows the nose coneof the AUV with an experimental miniature holographiccamera.

(b) A test flight of the X8 Skywalker UAV prior to off-shore operations.

Fig. 11 Tests of platforms on-shore prior to the cruise.

Operations were conducted on board the PortugueseNavy research vessel NRP Almirante Gago Coutinho1,which was used for launching and recovering AUVs andUAVs from the aft deck (Fig. 1). All operators and pilotswere on board the vessel which had multiple antennaswhich provided 802.11 Wi-Fi coverage via Ubiquiti ra-dios. The antennas were connected to multiple “Manta”communications gateways that relayed all data betweenunmanned vehicles and NEPTUS consoles. Inside theship, a local area network was created to which all sys-tems (mantas and consoles) were connected to. Thisnetwork setup allowed operators to be aware of theAUV and UAV operations simultaneously (Fig. 10).

Two upper water-column Light AUVs [50] (LAUV-Xplore-1 & LAUV-Xplore-2) were used, both equippedwith WHOI acoustic modems and Iridium SBDmodems. LAUV-Xplore-1 was additionally equippedwith an RBR XR620 CTD (Conductivity, temperatureand Density) and an experimental holographic particle

1http://goo.gl/hEVmFU

Fig. 12 Video feed captured by X8-03 UAV with infrared im-agery on the right.

imaging system developed at Plymouth University (Fig.11(a)) [25]. LAUV-Xplore-2, which is a similar vehiclein size and endurance, included only a Turner DesignsCyclops-7 wet-probe with a fluorometer. Both these ve-hicles can travel at a nominal speed of 2 knots (∼1 m/s)for up to 24 hours. Moreover, while deployed in the opensea, they can receive new commands and report theirstate over Iridium SBD messages (up to 250 bytes usingRockBlock 2).

Additionally, there were multiple Skywalker X8UAVs, seen in Fig. 11(b), at our disposal. These low-cost vehicles are adaptation of COTS (Commercial Off-The-Shelf) units which LSTS has augmented with spe-cific components including System-On-a-Chip (SOC)CPUs, communications hardware and specific sensorpayload. These frames, normally used for RC-flying, areinexpensive and made a resilient styrofoam which canbe be further customized if needed. The system has amaximum flight time of around 45 minutes at an aver-age cruise speed of ∼17 m/s giving it a useful (roundtrip) range of almost 25km which are typically not uti-lized in our kind of scenario. For this experiment, theUAVs carried both video cameras providing real-timefeed to operators and still cameras taking periodic im-agery of the water at high resolution. While some cam-eras captured visible light, others sampled in the LWIRspectrum (Fig. 12). We also deployed a new light-weighthyper-spectral imager on a separate UAV for purposesof testing; the results of that test are not part of thiswork.

As noted earlier, the key scientific objective of thisfield experiment was to characterize the upper watercolumn properties within sperm whale foraging areasto understand what drives their foraging behavior. Ourscenario called for either an experienced human ob-server on a separate boat, or airborne UAVs with real-time imagers, to be able determine target cetaceans onthe surface. Researchers on board the ship or small ves-sels acted as spotters.

2http://www.rock7mobile.com/products-iridium-sbd

http://goo.gl/hEVmFU

http://www.rock7mobile.com/products-iridium-sbd


Fig. 13 A NEPTUS console snapshot taken during the exper-iment showing a submerged AUV (1), an AUV at the surface(2), a UAV (3), the ship position (4) and the simulation timefor the AUV (5).

As individual sperm whales surfaced, a referencepoint was determined either from imagery or from thespotter, which then was communicated to the opera-tors on the vessel. If AUVs were not already in the wa-ter, they were launched to start autonomous execution.Repetitive AUV surveys around this targeted spot wereto commence, using two AUVs to measure spatial vari-ations in the water column.

Both T-REX-enabled AUVs were tasked by EU-ROPtus to synthesize yoyo based survey patterns [12]in two concentric square patterns of 400X400 meters2and 800X800 meters2 diving to a depth of 50 meters.The vehicles also surfaced on the corners of the squaresin order to correct its localization with a GPS fix and(tentatively) report its progress via Iridium. The largerpattern was used to sample outside the sperm whalemovement path, as a measure of understanding the vari-ability in the upper water-column. The smaller surveywas expected to take 25 minutes, while the larger 50minutes, at about 2.5 knots speed over ground for thevehicles. Consequently, the inner surveys were executedtwice to provide a dense coverage co-temporal to thesingle outer survey, (Fig. 14(a)). At the end of the sur-vey, the next target spot was to be determined eithervia the spotter or a UAV, and the vehicles re-tasked byEUROPtus.

One extended objective was to attempt not only co-temporal AUV surveys, but to coordinate the survey ofthe sampling area with UAV overflights with Far-IR andthe hyper-spectral imagers. In doing so, it was thought,we could obtain additional data of the ocean surface tobe merged and subsequently studied to understand thebio-geochemical composition in the light of any effluentsfrom the passing whales. This was successfully achievedby sharing of mission plans between consoles and bycoordinating the execution of plans via EUROPtus .Vehicles operating in the same area can be seen in Fig.

14(b), one airborne, one at the surface and one AUVunderwater (connected acoustically).

At the same time, multiple NEPTUS consoles werereceiving and controlling the different vehicles from theship, as seen in Fig. 13. One of the consoles was explic-itly used for AUV operations and had the EUROPtusplug-in activated while also synchronizing all data re-ceived via Iridium and Wi-Fi from the two AUVs con-nected to their respective shipboard simulators. Thedata from the simulators was, however, available to allconsoles, providing situation awareness on the positionof the vehicles underwater, even for the UAV operators.

At the UAV consoles, operators could receive im-agery in real-time and a plug-in was used to geo-locatepoints in the video stream. Whenever a frame of interestwas detected in the video feed, the operator could pausethe feed and click a point in the image. After attachinga name to the clicked point, these names, together withtheir coordinates were disseminated to the remainingconsoles, so that identified cetaceans or other pointsof interest were available in all operator consoles. If thedisseminated position was a cetacean to be tracked, theAUV operator could select it as a valid target which re-sulted in forwarding this location to EUROPtus. Thisadditional step was added so that human operators canselect preferred targets. Preference would generally begiven to targets closer to current AUV locations.

In EUROPtus we tied the AUVs operational modelwith that of the UAV to ensure coordination; UAVs ifnot in the air were launched from the aft deck, to besurveying the same area overhead. For this experiment,launch was based primarily on readiness of the vehicle.

A cetacean position was considered “fresh” by EU-ROPtus if and only if it was received at most 30 min-utes before the vehicles were to start a new survey. Thesurveys’ relative scales between the two AUVs was iden-tified in order to have both vehicle finishing the surveyapproximately at the same time for ease of recovery orrelocation. All of this could be determined by EUROP-tus including, for example, that having a “fresh” whaleposition would require an UAV launch and for a validsurvey both AUVs are required to be available and suf-ficiently close to the survey area. While this domainappears to be straight-forward, for a human operatorit is quite difficult to properly follow the execution ofmultiple robots and still make ahead-of-time decisionsof what the vehicles should be doing next. EUROPtus,on the other hand, tracks execution automatically anddeals with hard to predict and often very large latencyof communications.

Fig. 15 shows EUROPtus deliberation steps anddepth along the first hour of the July 19th mission.A single deliberation ’step’ is associated with the sys-tem making a single choice of a binding for a variablein order to satisfy a constraint, which ideally movesthe system towards a final plan (or outcome). This fig-


(a) Two AUV trajectories with CTD profiles (red: warmersurface waters, blue: colder deeper waters) executingtwo consecutive co-temporal surveys around a targetedcetacean position.

(b) UAV trajectory (green) superimposed on AUV track-lines (red and blue) from 14(a). UAV tracks show severalsearch trajectories and long-distance tests.

Fig. 14 Coordinated observations of a UAV with two AUVs in the water-column on July 19th, 2015.

Fig. 15 Number of deliberation steps in EUROPtus and itssearch tree depth for a subset of July 19th operations. EU-ROPtus was running on a Linux Virtual Machine on a 2.8GHzIntel Core i7.

ure shows the number of steps increasing monotoni-cally reflecting the integration of new observations asthey were received from NEPTUS. The depth growsslower showing that backtracking occurs during execu-tion. Except for the small downward spike, the depthtends to either remain flat or climb, indicating that ingeneral backtracking was only impacted by chronologi-cally recent observations and hence the planner recov-ered gracefully. The downward spike was due to erro-neous timestamps on some AUV location messages be-fore they reached EUROPtus. These could have beencaused because of improper clock settings. Despite thelarge jump in steps (≈ 700) the planner recovered inless than 3 seconds.

In tandem with all these deployments, the exper-imental holographic particle imaging system recordedhigh-resolution images of suspended particles (25µm to5mm size) within a 2ml sample volume at 1 Hz. As vis-ible in Fig. 11(a) the system was mounted on the nose-cone of LAUV-Xplore1 to sample the particles in anundisturbed state. Digital reconstruction of the holo-

Fig. 16 Example particles imaged by the miniature holographicsensor on-board LAUV-Xplore-1. A) Phytoplankton – diatomchains. B) Zooplankton – crustaceans and fish larvae. C) De-tritus – flocculated marine snow

grams provides focused images of the particles allow-ing them to be categorized, sized and counted. Fig. 16shows example images of particles recorded during theexperiment, comprising phytoplankton, zooplankton aswell as detritus (marine snow). The distributions of therelative abundance of each of these particle categorieswill be related to the physical and biological propertiesof the water column (temperature, salinity, chlorophyllfluorescence). Such data presents an interesting oppor-tunity, and serves to highlight the capacity such coordi-nated autonomous systems have to collect co-temporaland diverse data, which can then be cross-calibrated ifneeded.

7 Discussion and Conclusions

We present an infrastructure to supervise coordinatedoperations of multiple heterogeneous assets, while deal-


ing with unpredictability of the environment and spo-radic communications in the open sea. Many of thesetools are the result of years of work iterating towards abalance between autonomous decision making and mis-sion execution, as well as feedback for operator super-vision.

The operation of different vehicles was carried outby a mix of operators and automated planners on boardthe vehicles and at the control station. The use of theLSTS Toolchain and its unified IMC protocol allowedcoordination of systems with very different capabilitiesover heterogeneous links. The DUNE software porta-bility and its small footprint allowed it to run acrosscommunication gateways, vehicles and shore-side sim-ulators. NEPTUS was used simultaneously by differ-ent operators targeting different mission responsibil-ities. Despite using different user interfaces, relevantdata could be shared between the consoles over Wi-Fi,which allowed for an effective localization and trackingflow having human operators in the loop.

EUROPtus was designed and first deployed in thecontext of this operation. Its ability to handle observa-tions with temporal delays and to dispatch commandsonly when all their conditions are observed proved tobe effective. Operators consequently, could focus onhigh level operational concerns and rely on NEPTUSto maintain situational awareness for understanding thecurrent status of the mission.

The recent addition of EUROPtus relieves opera-tors of the intricacies of synchronization between as-sets and focuses more on higher level issues such as thescientific goals and the safety of the assets. To addressthe latter, NEPTUS was extended to not only dispatchcommands from EUROPtus to the assets but also to in-tercept them, in order to give an overview of expectedbehavior to the users. This ability to maintain situa-tional awareness, proved to be critical for our at-seaoperations.

Additionally, EUROPtus allowed for the reductionin logistics outside of the operational team. The sim-plification came in the streamlining of spotting tasksfor cetaceans since systems coordinated by EUROP-tus were also tasked to fulfill that role. This allows foradded scalability when both the operation area and mis-sion objectives increase.

Further work remains. Among those is the way weresolve Pending action flaws. The only way the systemcan resolve them currently, is by postponing the actionuntil it has all of its conditions Justified. An interestingextension would be for the planner to actively enrichthe plan by proactive search (for example by polling)for such a justification, perhaps even asking for operatorinput. This ’inquisitive’ approach would then preventthe system from replanning just because there was nofeedback before a certain deadline.

Another aspect we would like to explore is allow-ing EUROPtus to forget part of its search when it nolonger impacts the plan. A novelty of EUROPtus, whencompared to classical approaches, is that it keeps all ofits search history. One benefit is that EUROPtus canrevisit the past and restore a partial plan fragment inthe light of a delayed observation or justification. Aside effect, is that the receding planning horizon canresult in computational search being slower as time ad-vances. A solution to this potential issue would be toprune nodes from the search tree as they are justified byobservations. This would require replacing the chrono-logical backtracking search, allowing the system to runfor a long periods without performance impacts.

Acknowledgements Johansen was supported in part by the Re-search Council of Norway through the Centers of Excellencefunding scheme, grant number 223254 – NTNU AMOS, andby grant number 235348. Rajan and Py were supported by theUS Office of Naval Research, ONR Grant # N00014-14-1-0536.Silva was supported by FEDER and COMPETE funds andby FCT grant # IF/00943/2013. Silva was also supported byFRCT from the Government of the Açores. Authors are grate-ful to the Cetacean Ecology Research Group including PabloChellavard Navarro, Rui Prieto, Cláudia Oliveira, Marta To-bena, and skipper Renato Bettencourt, and to NTNU’s UAVteam, including Lars Semb, Krzysztof Cisek, Frederik Leira andJoão Fortuna. This work was partially supported by the SUN-RISE - ”Sensing, monitoring and actuating on the UNderwaterworld through a federated Research InfraStructure Extendingthe Future Internet”, project #611449, funded by the EuropeanUnion’s Seventh Framework Programme for Research and Tech-nological Development - Large scale integrating project (IP).We are grateful to the captain and crew of NRP Gago Coutinho.Finally, we are grateful to the entire LSTS team that partici-pated in the REP-15 exercise in the Açores.

References

1. Ai-Chang, M., Bresina, J., Charest, L., Chase, A., Hsu, J.,Jonsson, A., Kanefsky, B., Morris, P., Rajan, K., Yglesias,J., Chafin, B., Dias, W., Maldague, P.: MAPGEN: MixedInitiative Planning and Scheduling for the Mars’03 MERMission. IEEE Intelligent Systems 19(1) (2004)

2. Arrichiello, F., Das, J., Heidarsson, H.K., Sukhatme, G.S.,Chiaverini, S.: Experiments in autonomous navigation withan under-actuated surface vessel via the Null-Space basedBehavioral control. In: Conference on Advanced IntelligentMechatronics, pp. 362–367 (2009)

3. Benjamin, M.R., Schmidt, H., Newman, P.M., Leonard,J.J.: Nested autonomy for unmanned marine vehicles withMOOS-IvP. Journal of Field Robotics 27(6), 834–875(2010)

4. Bernard, D.E., Dorais, G.A., Fry, C., Jr., E.G., Kanfesky,B., Kurien, J., Millar, B., Muscettola, N., Nayak, P., Pell,B., Rajan, K., Rouquette, N., Smith, B., Williams, B.: De-sign of the Remote Agent experiment for spacecraft auton-omy. In: Proceedings of the IEEE Aerospace Conf. Snow-mass, CO (1998)

5. Block, B.A., Jonsen, I.D., Jorgensen, S.J., Winship, A.J.,Shaffer, S.A., Bograd, S.J., Hazen, E.L., Foley, D.G.,Breed, G.A., Harrison, A.L., Ganong, J.E., Swithenbank,A., Castleton, M., Dewar, H., Mate, B.R., Shillinger, G.L.,Schaefer, K.M., Benson, S.R., Weise, M.J., Henry, R.W.,


Costa, D.P.: Tracking apex marine predator movementsin a dynamic ocean. Nature 475(7354), 86–90 (2011).DOI 10.1038/nature10082. URL http://www.nature.com/doifinder/10.1038/nature10082

6. Bresina, J., Jonsson, A., Morris, P., Rajan, K.: ActivityPlanning for the Mars Exploration Rovers. In: Interna-tional Conference on Automated Planning and Scheduling(ICAPS). Monterey, California (2005)

7. Bresina, J., Jonsson, A., Morris, P., Rajan, K.: Mixed-Initiative Planning in MAPGEN: Capabilities and Short-comings. In: Proceedings of the workshop on Mixed-Initiative Planning and Scheduling, ICAPS. Monterey, Cal-ifornia (2005)

8. Brooks, R.: A robust layered control system for a mobilerobot. IEEE Journal on Robotics and Automation 2(1),14–23 (1986)

9. Chrpa, L., Pinto, J., Ribeiro, M., Py, F., Sousa, J., Ra-jan, K.: On Mixed-Initiative Planning and Control for Au-tonomous Underwater Vehicles. In: Intelligent Robots andSystems (IROS), IEEE/RSJ International Conference on.Hamburg, Germany (2015)

10. Curtin, T.B., Bellingham, J.G., Catipovic, J., Webb, D.:Autonomous oceanographic sampling networks. Oceanog-raphy 6(3), 86–94 (1993)

11. Das, J., Maughan, T., McCann, M., Godin, M., O’Reilly,T., Messie, M., Bahr, F., Gomes, K., Py, F., Bellingham, J.,Sukhatme, G., Rajan, K.: Towards mixed-initiative, multi-robot field experiments: Design, deployment, and lessonslearned. In: Proc. Intelligent Robots and Systems (IROS).San Francisco, California (2011)

12. Das, J., Py, F., Maughan, T., Messie, M., O’Reilly, T.,Ryan, J., Sukhatme, G.S., Rajan, K.: Coordinated Sam-pling of Dynamic Oceanographic Features with AUVs andDrifters. Intnl. J. of Robotics Research 31, 626-646 (2012).April

13. Das, J., Rajan, K., Frolov, S., Ryan, J.P., Py, F., Caron,D.A., Sukhatme, G.S.: Towards Marine Bloom TrajectoryPrediction for AUV Mission Planning. In: IEEE Interna-tional Conference on Robotics and Automation (ICRA).Anchorage, Alaska (2010)

14. Dias, P.S., Fraga, S.L., Gomes, R.M., Goncalves, G.M.,Pereira, F.L., Pinto, J., Sousa, J.B.: NEPTUS- a frame-work to support multiple vehicle operation. In: Proc OceansMTS/IEEE Conference, pp. 963–968. Brest, France (2005)

15. Dukeman, A., Adams, J.A., Edmondson, J.: Extensible col-laborative autonomy using gams. In: Proceedings of the31st Annual ACM Symposium on Applied Computing, pp.281–283. ACM (2016)

16. Edmondson, J., Schmidt, D.: Multi-agent distributed adap-tive resource allocation (madara). International Journal ofCommunication Networks and Distributed Systems 5(3),229–245 (2010)

17. Encarnaçao, P., Pascoal, A.: Combined trajectory trackingand path following: an application to the coordinated con-trol of autonomous marine craft. In: Decision and Control,2001. Proceedings of the 40th IEEE Conference on, vol. 1,pp. 964–969. IEEE (2001)

18. Faria, M., Pinto, J., Py, F., Fortuna, J., Dias, H., Leira,R.M.F., Johansen, T.A., Sousa, J.B., Rajan, K.: Coordi-nating UAVs and AUVs for Oceanographic Field Experi-ments: Challenges and Lessons Learned. In: IEEE Inter-national Conference on Robotics and Automation (ICRA).Hong Kong (2014)

19. Fedak, M.: Marine animals as platforms for oceanographicsampling: a “ win / win ” situation for biology and opera-tional oceanography. Memoirs of the National Institute forPolar Research, Special Issue 58, 133–147 (2004)

20. Ferreira, A.S., Pinto, J., Sousa Dias, P., de Sousa, J.B.:The LSTS software toolchain for persistent maritime oper-ations applied through vehicular ad-hoc networks. In: 2017

International Conference on Unmanned Aircraft Systems(ICUAS), pp. 609–616. IEEE (2017)

21. Fikes, R., Nilsson, N.: STRIPS: A new approach to the ap-plication of theorem proving to problem solving. ArtificialIntelligence 2, 189 – 205 (1971)

22. Frank, J., Jónsson, A.: Constraint-based Attribute and In-terval Planning. Constraints 8(4), 339–364 (2003)

23. Ghallab, M., Nau, D., Traverso, P.: Automated Planningand Acting. Cambridge University Press (2016)

24. Gonçalves, G., Gonçalves, R., Ferreira, A.S., de Sousa, J.B.,Pinto, J.: Supervisory Control: Optimal Distribution ofWorkload Among Operators for Mixed Initiative Controlof Multiple UAVs. In: Dr. Leo van Breda (ed.) Supervi-sory Control of Multiple Uninhabited Systems – Method-ologies and Enabling Human-Robot Interface Technologies,chap. 10, pp. 153 – 163. STO/NATO (2012)

25. Graham, G.W., Nimmo Smith, W.A.M.: The applicationof holography to the analysis of size and settling velocityof suspended cohesive sediments. Limnology and Oceanog-raphy: Methods 8(1), 1–15 (2010)

26. Jónsson, A., Morris, P., Muscettola, N., Rajan, K., Smith,B.: Planning in Interplanetary Space: Theory and Practice.In: Artificial Intelligence Planning and Scheduling (AIPS)(2000)

27. Lapierre, L., Soetanto, D., Pascoal, A.: Coordinated motioncontrol of marine robots. Proceedings of the 6th IFACMCMC, Girona, Spain (2003)

28. Leira, F.S., Johansen, T.A., Fossen, T.I.: Automatic de-tection, classification and tracking of objects in the oceansurface from uavs using a thermal camera. In: 2015 IEEEAerospace Conference, pp. 1–10 (2015). DOI 10.1109/AERO.2015.7119238

29. Lemai-Chenevier, S., Ingrand, F.: Interleaving TemporalPlanning and Execution in Robotics Domains. In: As-soc. for the Advancement of Artificial Intelligence, NationalConference (AAAI) (2004)

30. M. Ghallab and D. Nau and P. Traverso: Automated Plan-ning: Theory and Practice. Elsevier Science (2004)

31. Manley, J., Willcox, S.: The wave glider: A persistent plat-form for ocean science. In: OCEANS 2010 IEEE-Sydney,pp. 1–5. IEEE (2010)

32. Martinez, C., Keener-Chavis, P.: NOAA Ship Okeanos Ex-plorer: Telepresence in the Service of Science, Educationand Outreach. In: OCEANS 2006, pp. 1–5. IEEE (2006)

33. Martins, R., de Sousa, J.B.: An extensible networking ar-chitecture for autonomous underwater vehicles. In: 2013OCEANS - San Diego, pp. 1–5 (2013)

34. Martins, R., de Sousa, J.B., Afonso, C.C., Incze, M.: REP10AUV: Shallow water operations with heterogeneous au-tonomous vehicles. In: OCEANS 2011 IEEE - Spain, pp. 1–6. IEEE (2011). DOI 10.1109/Oceans-Spain.2011.6003568.URL http://ieeexplore.ieee.org/document/6003568/

35. McGann, C., Py, F., Rajan, K., Thomas, H., Henthorn, R.,McEwen, R.: A Deliberative Architecture for AUV Con-trol. In: IEEE International Conference on Robotics andAutomation (ICRA). Pasadena (2008)

36. McGillivary, P., Sousa, J., Martins, R., Rajan, K., Leroy,F.: Integrating Autonomous Underwater Vessels, SurfaceVessels and Aircraft as Persistent Surveillance Componentsof Ocean Observing Studies. In: IEEE Autonomous Under-water Vehicles. Southampton, UK (2012)

37. Muscettola, N., Nayak, P., Pell, B., Williams, B.: RemoteAgent: To Boldly Go Where No AI System Has Gone Be-fore. Artificial Intelligence 103, 5–48 (1998)

38. Newman, P.: A mission oriented operating suite. Tech. rep.,Technical Report OE2007-07. MIT Department of OceanEngineering (2003)

39. Pinto, J., Faria, M., Fortuna, J., Martins, R., Sousa, J.,Queiroz, N., Py, F., Rajan, K.: Chasing Fish: Tracking andcontrol in a autonomous multi-vehicle real-world experi-ment. In: MTS/IEEE Oceans. San Diego, California (2013)

http://www.nature.com/doifinder/10.1038/nature10082

http://www.nature.com/doifinder/10.1038/nature10082

http://ieeexplore.ieee.org/document/6003568/


40. Pinto, J., Martins, P.S.D.R., Fortuna, J., Marques, E.,Sousa, J.: The LSTS toolchain for networked vehicle sys-tems. In: MTS/IEEE Oceans, pp. 1–9. IEEE (2013)

41. Pinto, J.Q., Dias, P.S., Gonçalves, R., Gonçalves, G.M.,Sousa, J.T.d.F.B., Pereira, F.L., et al.: Neptus a frameworkto support a mission life cycle. In: Proceedings of the 7thConference on Manoeuvring and Control of Marine Craft(2006)

42. Py, F., Rajan, K., McGann, C.: A Systematic Agent Frame-work for Situated Autonomous Systems. In: 9th Interna-tional Conf. on Autonomous Agents and Multiagent Sys-tems (AAMAS). Toronto, Canada (2010)

43. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T.,Leibs, J., Wheeler, R., Ng, A.Y.: ROS: an open-sourcerobot operating system. In: ICRA workshop on open sourcesoftware, vol. 3, p. 5. Kobe (2009)

44. Rajan, K., Bernard, D., Dorais, G., Gamble, E., Kanefsky,B., Kurien, J., Millar, W., Muscettola, N., Nayak, P., Rou-quette, N., Smith, B., Taylor, W., Tung, Y.: Remote Agent:An Autonomous Control System for the New Millennium.In: Proceedings Prestigious Applications of Intelligent Sys-tems, European Conf. on AI. Berlin (2000)

45. Rajan, K., Py, F.: T-REX: Partitioned Inference for AUVMission Control. In: G.N. Roberts, R. Sutton (eds.) FurtherAdvances in Unmanned Marine Vehicles, pp. 171–199. TheInstitution of Engineering and Technology (IET) (2012)

46. Rajan, K., Py, F., Berreiro, J.: Towards Deliberative Con-trol in Marine Robotics. In: M. Seto (ed.) Autonomy inMarine Robots, pp. 91–175. Springer Verlag (2012)

47. Ramp, S.R., Davis, R.E., Leonard, N.E., Shulman, I., Chao,Y., Robinson, A., Marsden, J., Lermusiaux, P., Fratantoni,D., Paduan, J.D., et al.: Preparing to predict: the secondautonomous ocean sampling network (AOSN-II) experi-ment in the Monterey Bay. Deep Sea Research Part II:Topical Studies in Oceanography 56(3), 68–86 (2009)

48. Roberts, J.J., Best, B.D., Mannocci, L., Fujioka, E.,Halpin, P.N., Palka, D.L., Garrison, L.P., Mullin, K.D.,Cole, T.V.N., Khan, C.B., McLellan, W.A., Pabst, D.A.,Lockhart, G.G.: Habitat-based cetacean density modelsfor the U.S. Atlantic and Gulf of Mexico. Scientific Re-ports 6(1), 22,615 (2016). DOI 10.1038/srep22615. URLhttp://www.nature.com/articles/srep22615

49. Shkurti, F., Xu, A., Meghjani, M., Higuera, J.C.G., Gird-har, Y., Giguere, P., Dey, B.B., Li, J., Kalmbach, A., Pra-hacs, C., et al.: Multi-domain monitoring of marine envi-ronments using a heterogeneous robot team. In: IntelligentRobots and Systems (IROS), 2012 IEEE/RSJ InternationalConference on, pp. 1747–1753. IEEE (2012)

50. Sousa, A., Madureira, L., Coelho, J., Pinto, J., Pereira, J.,Sousa, J., Dias, P.: LAUV: The man-portable autonomousunderwater vehicle. In: Navigation, Guidance and Controlof Underwater Vehicles, pp. 268–274 (2012)

51. de Sousa, J.B., Pereira, J., Alves, J., Galocha, M., Pereira,B., Lourenço, P., Portuguesa, M.: Experiments in multi-vehicle operations: The Rapid Environmental Picture At-lantic exercise 2014. In: OCEANS 2015 - Genova,pp. 1–7. IEEE (2015). DOI 10.1109/OCEANS-Genova.2015.7271761. URL http://ieeexplore.ieee.org/document/7271761/

52. Sousa, L.L., López-Castejón, F., Gilabert, J., Relvas, P.,Couto, A., Queiroz, N., Caldas, R., Dias, P.S., Dias, H.,Faria, M., Ferreira, F., Ferreira, A.S., J.Fortuna, Gomes,R.J., Loureiro, B., Martins, R., Madureira, L., Neiva, J.,Oliveira, M., Pereira, J., J.Pinto, Py, F., Queiroz, H., Silva,D., Sujit, P.B., Zolich, A., Johansen, T.A., Sousa, J., Rajan,K.: Integrated monitoring of mola mola behaviour in spaceand time. PLOS one (2016). DOI http://dx.doi.org/10.1371/journal.pone.0160404

53. Stoker, C.R., Burch, D., Hine, B.P., Barry, J.: Antarctic un-dersea exploration using a robotic submarine with a telep-resence user interface. IEEE Expert 10(6), 14–23 (1995)

54. Webb, D.C., Simonetti, P.J., Jones, C.P.: Slocum: An un-derwater glider propelled by environmental energy. IEEEJournal of oceanic engineering 26(4), 447–452 (2001)

António Sérgio Fer-reira is a Ph.D stu-dent and holds a Mas-ter’s degree in Infor-matics and ComputingEngineering from Uni-versity of Porto. Hisresearch interests in-clude unmanned vehi-cle systems, communi-cations and coordina-tion of multiple dy-namic systems, hybridsystems, systems engi-neering, and manage-

ment architectures for multi-vehicle systems. Since 2011he has been part of the Underwater Systems and Tech-nology Laboratory (LSTS) where his work focused onfurther developing the Neptus Command and ControlInfrastructure for UAS as well as being a UAS fieldoperator.

Maria Costa is aRobotics Engineer in arobotics research labo-ratory (LSTS). She isa main software de-veloper (onboard soft-ware) and is also a partof both the AUV andUAS field operationalteams (where she is themain UAS Operator:fixed-wing and multi-rotor vehicles). Duringthe last four years shegained significant expe-

rience in field operations, mostly maritime environ-ments (operating from ship) in cooperation with thePortuguese Navy. She also had the opportunity toparticipate in several national and European projects(Seagull, PITVANT, Safeport, NetOcean, NETMAR,Fire-RS, URready4OS, E-URready4OS) and annualbig-scale exercises (REP14, REP15, REP16, REP17).Maria Costa holds an Electrotechnical and ComputerEngineering degree and a Masters in Autonomous Sys-tems from School of Engineering Polytechnic of Porto(ISEP). Her master thesis is entitled “Assisted Maneu-vers for Underactuated ROVs”. Prior to college edu-cation, she also graduated top of her class as a EULevel 3 Technician in Industrial Electronics and Au-tomation. Throughout her college education she workedas an Electronics Technician and also integrated the Ise-Porto robotic football team (participating in a World

http://www.nature.com/articles/srep22615




Cup Middle Size League competition (RoboCup 2009)and in a couple of National Middle Size League compe-titions - Robótica 2009 and Robótica 2010).

Frédéric Py holdsa doctorate degree inRobotics and ArtificialIntelligence from Uni-versity Paul Sabatier,Toulouse. He has pio-neered the use of ad-vanced statistical meth-ods using Clusteringalgorithms for detect-ing dynamic events inthe water-column in theocean and has lead thedevelopment and de-sign of the T-REX soft-

ware agent for high level planning and execution alongwith its application on scientific operations. Moreover,he has worked on automated model learning techniquesfor in situ analysis of physical oceanographic processes(such as Intermediate Nepheloid Layer, plume, ...) Hehas collaborated with the LSTS team during the last2 REP projects, where we worked together integrat-ing the T-REX architecture which he developed whileworking at MBARI and still supports up to this day.

José Pinto was bornin Matosinhos, Portu-gal, in 1982. He re-ceived the B.S. de-gree in computer sci-ence from Porto Uni-versity, Porto, Portu-gal, in 2005. He joinedthe Underwater Sys-tems and TechnologyLaboratory (LSTS) in2004 still as an under-graduate student whenhe started the devel-opments leading to thecreation of the NeptusCommand and Control

Infrastructure as well as other parts of the LSTS Soft-ware Toolchain. During his stay at LSTS he has beenworking towards the autonomous operation of water-column AUVs from satellite communications to under-lying control layers, cloud services and user interfaces.Since 2006 he is an Assistant Professor at Porto Univer-sity, teaching classes on Programming and EmbeddedComputers. Currently he is finishing his PhD (ABD)on Coordination Infrastructures for Networked Vehi-cle Systems, under the supervision of João Borges deSousa.

Mónica A. Silvagraduated in Biologyfrom the University ofLisbon in 1996 and re-ceived my PhD degreefrom the University ofSt. Andrews (Scotland)in 2007. From the timeof graduation untilenrolling in a PhD,she developed research

on marine mammal biology and ecology and MarineProtected Area design at the Nature Conservation In-stitute (Lisbon), Ministry of the Environment; workedas scientific consultant for the World ConservationUnion in Guinea-Bissau, developing a ConservationPlan for the West African manatee; and was tem-porarily assigned (3 years) to the Department ofOceanography and Fisheries/University of the Azores(DOP/UAz), to work as a Research Assistant in thescope of an EC LIFE-Nature Project. She started apost-doc in 2007 at the Centre of IMAR of the Uni-versity of the Azores (IMAR/UAz) and at the WoodsHole Oceanographic Institution (WHOI) (USA). In2013 she was awarded a 5-year contract as SeniorResearch Associate (Investigador Auxiliar under FCTInvestigator Programme) at IMAR.

Alex is an AssociateProfessor (Reader) inMarine Physics at Ply-mouth University, andis part of the Ma-rine Physics ResearchGroup (MPRG). Hisresearch interests arefocussed around sus-pended particle dynam-ics, both in terms ofhow particles (plank-ton, sediment, bubbles,droplets) behave in anatural, turbulent envi-

ronment, and also in the use of particles as tracers ofturbulent flows. He has developed several in situ opti-cal systems to directly image particles for measurementand identification, and visualise 3D turbulence struc-ture. He’s member of the Challenger Society for MarineScience, of the American Geophysical Union and fellowof the Higher Education Academy.


Tor Arne Johansenreceived the MSc de-gree in 1989 and thePhD degree in 1994,both in electrical andcomputer engineering,from the NorwegianUniversity of Sci-ence and Technology,Trondheim, Norway.From 1995 to 1997, heworked at SINTEF asa researcher before hewas appointed Associ-

ated Professor at the Norwegian University of Scienceand Technology in Trondheim in 1997 and Professorin 2001. He has published several hundred articles inthe areas of control, estimation and optimization withapplications in the marine, aerospace, automotive,biomedical and process industries. In 2002 Johansenco-founded the company Marine Cybernetics AS wherehe was Vice President until 2008. Prof. Johansenreceived the 2006 Arch T. Colwell Merit Award of theSAE, and is currently a principal researcher withinthe Center of Excellence on Autonomous MarineOperations and Systems (NTNU-AMOS) and directorof the Unmanned Aerial Vehicle Laboratory at NTNU.

João Borges deSousa is with theElectrical and Com-puter EngineeringDepartment of theFaculty of Engineeringfrom Porto Univer-sity and he is alsothe director of LSTS.His research interestsinclude unmannedvehicle systems, con-trol and coordinationof multiple dynamicsystems, hybrid sys-

tems, systems engineering, and control architectures

for multi-vehicle systems. Since 1997 he has beenleading the design, implementation and deployment ofadvanced unmanned vehicle systems in projects fundedby the Portuguese Foundation for Science and Tech-nology, Portuguese Innovation Agency, Luso-AmericanFoundation, NATO, Office of Naval Research andDARPA. He had several Visiting Scholar appointmentsat the University of California at Berkeley since 1997.In the last 20 years he has been involved in fosteringand growing a world-wide research community in thisfield with yearly conferences and workshops in the ar-eas of Hybrid Systems, Intelligent Control, NetworkedVehicle Systems and Multi-Agent Systems. He hasbeen lecturing and delivering seminars on networkedrobotics in universities in the United States, Europe,Asia and Africa. In 2006 he received the nationalBES Innovation National Award for the design of theLight Autonomous Underwater Vehicle. In 2008 hereceived an outstanding teaching award from PortoUniversity. He was the chair of several IEEE and IFACconferences and workshops. He has authored morethan 300 publications.

Kanna holds anInternational Chairadjunct professorshipat NTNU’s Dept. ofEngineering Cybernet-ics and affiliated to theCenter for AutonomousMarine Operations andSystems. His back-ground is in ArtificialIntelligence Planningand Scheduling andhas applied his work inthe domains of space-flight while at NASAand marine roboticswhile at MBARI, allin California. He isalso a Visiting Profes-

sor at the Faculty of Engineering, Univ. of Porto’sUnderwater Systems and Technology Laboratory.

advancing multi-vehicle deployments in oceanographic field …folk.ntnu.no/torarnj/advancing...

Documents