PANDORA: Persistent Autonomy throughLearning, Adaptation, Observation and

Re-planning ?

David M. Lane ∗ Francesco Maurelli ∗ Tom Larkworthy ∗

Darwin Caldwell ∗∗ Joaquim Salvi ∗∗∗ Maria Fox ∗∗∗∗

Konstantinos Kyriakopoulos †

∗Ocean Systems Lab, Heriot-Watt University, Edinburgh, UK, contact:d.m.lane@hw.ac.uk

∗∗Advanced Robotics Department, Italian Institute of Technology(IIT), Genova, Italy

∗∗∗Depart. d’Arquitectura i Tecnologia de Computadors, CampusMontilivi, Universitat de Girona, Spain

∗∗∗∗Department of Informatics, King’s College London, UK† Control Systems Laboratory, National Technical University of Athens

(NTUA), Greece

Abstract: Autonomous robots are not very good at being autonomous. Operating in realenvironments, they easily get stuck, often ask for help, and generally succeed only whenattempting simple tasks in well-known situations. PANDORA is a three year project thatwill develop and evaluate new computational methods to make underwater robots PersistentlyAutonomous, significantly reducing the frequency of assistance requests. The key to this is anability to recognise failure and respond to it, at all levels of abstraction and time constant.Under the guidance of major industrial players, validation tasks of Inspection, cleaning andvalve turning will be trialled with partners’ AUVs in Scotland and Spain.

1. INTRODUCTION

Whilst humans and animals perform effortlessly doingcomplicated tasks in unknown environments, our human-built robots are not very good at being similarly in-dependent. Operating in real environments, they easilyget stuck, often ask for help, and generally succeed onlywhen attempting simple tasks in well-known situations.We want autonomous robots to be much better at beingautonomous for a long time (persistent autonomy), andto be able to carry out more complicated tasks withoutgetting stuck, lost or confused. Following the Deep WaterHorizon disaster in the BP Macondo oilfield in the Gulf ofMexico in 2010, Oil Companies are developing improvedways to cost effectively and safely carry out more frequentinspection, repair and maintenance tasks on their subseainfrastructure. This is particularly challenging in deepwater. To date, Autonomous Underwater Vehicles (AUVs)have been deployed very successfully for various forms ofseabed and water column transit survey. First commercialunits will soon be applied to simple hovering inspectiontasks, with future units expected to address much harderintervention where contact is made to turn a valve orreplace a component. Because these vehicles reduce orremove the need for expensive ships, their adoption isexpected to grow over the next 5 to 10 years.

? The research leading to these results has received funding fromthe European Union Seventh Framework Programme FP7/20072013Challenge 2 Cognitive Systems, Interaction, Robotics under grantagreement No 288273 PANDORA

To be successful commercially, these hovering AUVs mustoperate for extended periods (12-48 hours +) without thecontinual presence of a surface vessel. They must thereforedemonstrate persistent autonomy in a challenging envi-ronment. We therefore choose this application focus toevaluate the projects research, with guidance from BP,Subsea7 and SeeByte Ltd. on the projects Industrial Advi-sory Group. We have identified three essential areas wherewe believe core research is required in order to provide theessential foundations for Persistent Autonomy:

• Describing the World• Directing and Adapting Intentions• Acting Robustly

1.1 Describing the World

Whilst current generations of autonomous robots use se-mantic descriptions of themselves, their environment andtheir tasks as a basis for directing their actions, thesedescriptions are typically fixed and certain. We wish todevelop and evaluate semantic representations (using on-tologies) that are explicitly uncertain, and that continuallyevolve, change and adapt based on sensor data that ob-serves the environment and the robot as it acts. This buildson low level perception detecting features in sensor data,and on status assessment that estimates the actual stateof the robot and task execution as it progresses. It alsoinvolves directing the sensors during execution to makeobservations appropriate to the task and situation (focus

Fig. 1. PANDORA: Computational architecture to develop and study persistent autonomy

attention). As a result, a robot can be aware of task failureand its context.

1.2 Directing and Adapting Intentions

By observing the effects of actions in this changing prob-abilistic world above, we wish to develop and evaluatemechanisms to create and refine task plans and primi-tives. By maintaining and modifying models of actionsand their effects, the autonomous robot formulates plans,updates them or repairs them on the fly when they don’twork, accounting for uncertainty and within time andresource budgets. This ability to model cause-and-effectand reason about state and goal satisfaction relies on theseadaptive representations of the world. Thus equipped, theautonomous robot can change its strategic approach, itstactical plan and perhaps eventually its purpose to remainsafe, operate efficiently and survive over an extended pe-riod without recourse to a human operator.

1.3 Acting Robustly

Noisy sensor data and disturbances on the robot and theworld produces uncertainty and errors in the location ofobjects and the robot itself. Attempts to execute atomicactions from planning above will be prone to failuretherefore, unless local real-time adaptation of motionsand contacts can compensate for these errors. Further,robots have to be instructed on how to execute theseactions, and how to respond to these local uncertaintiesand disturbances.

2. ARCHITECTURE

Figure 1 outlines the computational architecture designedfor development and study of persistent autonomy. Keyis the notion that the robots response to change and theunexpected takes place at one or a number of hierarchicallevels. At an Operational level, sensor data is processed inPerception to remove noise, extract and track features, lo-calise using SLAM, in turn providing measurement valuesfor Robust Control of body axes, contact forces/torquesand relative positions. One of the goals is to further exploresome of the current approaches (Aulinas et al. [2011], Leeet al. [2012]) and integrate them on a real vehicle. In thecases where a map is given, localisation techniques will beused (Petillot et al. [2010]), with a specific attention to ac-tive localisation (Maurelli et al. [2010]). Relevant work onrobust control can be found in Panagou and Kyriakopoulos[2011], Karras et al. [2011]. At a Tactical Level, StatusAssessment uses status information from around the robotin combination with expectations of planned actions, worldmodel and observed features to determine if actions areproceeding satisfactorily, or have failed. Alongside this, re-inforcement and imitation learning techniques are used totrain the robot to execute pre-determined tasks, providingreference values to controllers. Fed by measurement valuesfrom Perception, they update controller reference valueswhen disturbance or poor control causes action failure.The learning block will be lead by IIT, with relevantexpertise in the field (Kormushev et al. [2011], Calinonet al. [2010]) Finally at a Strategic level, sensor featuresand state information are matched with geometric dataabout the environment to update a geometric world model.These updates include making semantic assertions about

Table 1. Relevance of the elements ofPANDORA’s computational architecture to

these challenges

Id ChallengesDescribingTheWorld

DirectingandAdaptingIntentions

ActingRobustly

1 Coupled Dynamics X2 Noisy Sensors X X3 Currents X

4Limited Communi-cation

X X

5 Reaction Forces X6 Object Motion X7 Finite Energy X X

8Partially KnownEnvironments

X X

9 Training X10 Failures X X

the task, and the world geometry, and using reasoning topropagate the implications of these through the world de-scription. Task Planning uses both semantic and geometricinformation as pre-conditions on possible actions or actionsequences that can be executed. When Status Assessmentdetects failure of an action, Task Planning instigates aplan repair to asses best response, if any. Where there isinsufficient data to repair, Task Planning specifies FocusAreas where it would like further sensor attention directed.These are recorded in the World Model and propagatedthrough Status Assessment as Focus of Attention to di-rect the relevant sensors to make further measurements.Relevant work on Planning has been performed by Foxet al. [2011, 2012].

3. DOMAIN AND TEST SCENARIOS

We have chosen autonomous underwater inspection andintervention in the oilfield as our application focus. Thespecific domain challenges that require these advancedfeatures to realise persistent autonomy are (Table 1):

1.Coupled Dynamics: Underwater vehicles have dynamicsthat are coupled across the axes, significantly time vary-ing, with large inertias (stored energy). Adequate controlacross a range of payloads and operating conditions (e.g.pitch angles of the vehicle) remains challenging. This isessential for inspection where imaging sensor position andorientation relative to the structure must be maintained.

2.Noisy Sensors: Much sensor data is acoustically derived(e.g. navigation, imaging). It is therefore inherently noisyand of low resolution with some latency resulting from thespeed of sound in water. This leads to uncertainties in theposition of the vehicle and the objects around it.

3.Currents: The medium the vehicle moves through (thewater) also moves in unpredictable ways and disturbs itsmotion. This causes navigation to drift, and perturbs ineddies around structures.

4.Limited Communication: E/M waves do not propagateunderwater at useful frequencies. Communication is there-fore acoustic, with severely limited bandwidth and noisecorruptions. AUVs cannot be tele-operated from the sur-face. Removal of the umbilical cable (c.f. a tethered Re-

motely Operated Vehicle ROV) enforces the requirementfor autonomy therefore.

5.Reaction Forces: During contact, the vehicle can onlycompensate reaction forces by thrusting. It therefore be-haves like a passively compliant system subject to distur-bances in applications where precision is required e.g. forgrasping. Docking first to a subsea structure assists this,but docking bars or other fittings are not always availableon the structure, and are expensive to install.

6.Object Motion: Subsea objects to be inspected movewith time constant slower than the vehicle. In some cases(e.g. cleaning marine growth) they move as result of thevehicles actions. The vehicle must compensate for theseunexpected movements

7.Finite Energy: AUVs carry batteries for power. Themission and the vehicle systems have to be managedeffectively to keep the vehicle safe and complete usefultasks. Where energy is being consumed more quickly thanexpected, the mission plan must adapt.

8.Partially Known Environment: Whilst environmentsshould be known a priori (subsea infrastructure, ship hull),the geometric world model data is often incomplete orincorrect, through previously undocumented maintenance,changes from as built drawings just before installation, anddamage (mapping the debris after the Deep Water Horizonexplosion was a serious challenge before intervention couldcommence). The AUV must therefore be ready to mapunexpected structures, and adapt missions accordingly.

9. Training: Skilled underwater vehicle pilots are in shortsupply. They have skills of motion control and task execu-tion that must be readily captured into the vehicle withoutextensive programming.

10.Failures: While all the symptoms of success may bepresent in the immediately available sensing and controldata, in fact the task may have failed e.g. servo lockonto the wrong anchor chain or riser for inspection. Thismust be detected and addressed without recourse to theoperator. With these specific challenges in mind. Thefollowing validation scenarios were selected for testing theability of the PANDORA technology to deal with complextasks.

3.1 Task A: Autonomous inspection of a submerged structuree.g. a ship hull (FPSO) or manifold (Fig. 2)

A hover capable autonomous underwater vehicle is equippedwith a forward looking sonar, a video camera and deadreckoning navigation system. The structure is partiallyknown, but there are inconsistencies between it and thegeometric world model. The vehicles high-level goal is toautonomously inspect the entire structure with no dataholidays, and bring back a complete data set of video andsonar for mosaicking and post processing. There may bea current running, and the optical visibility may be verypoor. In some cases, the sonar inspection sensors must bekept at a constant grazing angle relative to the structure,for best performance. In the absence of a pan and tiltunit, the vehicle must dynamically pitch, yaw and roll tomaintain this orientation.

Fig. 2. Task A: Autonomous Inspection of Ship Hull or Subsea Structure

Fig. 3. Task B: (a) Marine growth (b) Anchor Chain (c) FPSO, Anchors and Risers

Fig. 4. Task C: Valve Turning: (a) Docked ROV (b) AIV for Prior to Launch

3.2 Task B: Autonomous location, cleaning and inspectionof an anchor chain (Fig. 3)

A hover capable autonomous underwater vehicle is equippedas above, but in addition carries a high-pressure water jet.Its goal is to locate the correct anchor chain of an FPSOand traverse it to remove the marine growth on all sidesusing the water jet. Thereafter it revisits the chain andbrings back complete video inspection data for subsequentpost processing. The reaction forces from the water jetintroduce significant forces and moments onto the vehicle,and also disturb the anchor chain. Both are therefore inconstant disturbed motion. The optical visibility dropsto zero during jetting as the marine growth floats in thewater. There may be sea currents moving over the anchor,creating minor turbulence downwind of the chain. Thechain is located adjacent to flexible risers of slightly largerdimension bringing oil to the surface.

3.3 Task C: Autonomous grasping and turning of a valvefrom a swimming, undocked vehicle (Fig. 4)

A hover capable autonomous underwater is equipped asin Task A, with a simple robot arm at the front. Its goalis to locate the correct valve panel of a subsea manifoldand open the correct valve. On each panel a selection ofvalve heads are exposed, each with a T bar attached forgrasping. The vehicle must identify the state of the valves(open, close, in-between) from the T bar orientations, andif appropriate, use the robot arm to grasp the correct valveand open it. The vehicle does not dock, because thereare no docking bars on the panel. It must therefore hoverby swimming, counteracting any reaction forces from theturning. It must also ensure that the gripper position andorientation of the gripper after grasping does not causesignificant shear forces in the T bar, and break it off.The visibility is generally good, but there may be seacurrents running and minor turbulence down current fromthe manifold.

4. VEHICLE ASSETS

The testbed assets for these integrations and trials includeHWU’s Nessie AUV, UdG’s Girona 500 AUV and HWU’sCartesian robot mounted over a test tank. It can beanimated with a 6DOF AUV dynamic model, and will beused to initially test the Perception and Status Assessmenttechniques in the project.

For in water trials with the real vehicles, two trials loca-tions will be employed. For Task A, HWU’s Nessie vehiclewill be used, featuring its unique bow and stern thrusterarrangements enabling 360 rotation in yaw pitch and roll,and hovering in surge, sway and heave. Initial trials willtake place in the small test tank at HWU. Thereafter,facilities at the Underwater Centre in Fort William, Scot-land will be used, where larger structures are installed foroffshore ROV pilot and diver training. For Tasks B and C,UdG’s Girona 500 vehicle will be used in the underwatertest facilities that form part of the Girona laboratory.Some of the demonstration platforms are showed in Fig.5.

5. VALIDATION

A comprehensive list of validation metrics have been devel-oped to test the ability of each system to meet its criteria.Ultimately the best all-encompassing metrics of persistentautonomy is quantitative analysis of robot engineer inter-actions, which will be greatly reduced by PANDORA’ssuccess. So overall, we measure our success by the reduc-tion in the number of times the operator is called to assista stuck robot during execution of tasks and sequenceswith noisy sensor data. We also count the number ofsuccessful automatic recoveries the robot achieves from anexecution failure. These require world modelling to detectthe failure, task planning to indicate corrective action, andreinforcement learning/adaptive control to successfully ex-ecute once more.

Beyond this, the overarching performance indicators withineach core theme are:

• Describing the World:· Trends in numerical errors of vehicle location

and object location/geometries (or other indirectmeasures such as residuals, covariance, OSPAerror, Hellinger distance), and probabilities insemantic relationships (ontologies).

· Number of correct and incorrect diagnoses of taskexecution failure.

• Directing and Adapting Intentions: Comparison ofthe number of failing actions in an automatedplanning/re-planning system with the number of fail-ures produced by hand-built plans, assuming givenworld information (efficiency and resource utilisationare therefore implicit)

• Acting Robustly: Position and orientation error normper unit distance, and the average wrench (force andtorque) error norm.

6. SYSTEM INTEGRATION

Given the project is expected to further the state-of-the-art in a number of different research areas, we deemed

it appropriate to adopt an Agile system developmentmethodology. Agile methods promote a project manage-ment process that encourages frequent inspection andadaptation, a leadership philosophy that encourages team-work, self-organization and accountability, a set of engi-neering best practices that allow for rapid delivery of high-quality solutions. Conceptual foundations of this frame-work are to be found in modern approaches to operationsmanagement and analysis such as lean manufacturing,soft systems methodology, speech act theory (network ofconversations approach), and Six Sigma.

Agile methods choose to do things in small incrementswith minimal planning, rather than long-term planning.Iterations are short time frames (known as ’timeboxes’)which typically last from one to four weeks. Each itera-tion is worked on by a team through a full developmentcycle, including planning, requirements analysis, design,coding, unit testing, and acceptance testing when a work-ing prototype is demonstrated to stakeholders. This helpsto minimize the overall risk, and allows the project toadapt to changes quickly. Documentation is produced asrequired. Multiple iterations may be required to releasenew features.

Team composition in an agile project is usually cross-functional and self-organizing without consideration forany existing hierarchy or formal roles of team members.Team members normally take responsibility for tasks thatdeliver the functionality of an iteration. They decide forthemselves how they will execute during an iteration.Agile methods emphasize face-to-face communication overwritten documents, when working in the same location,or in different locations but having video contact daily,communicating by videoconferencing, voice, e-mail etc.

Agile methods emphasize working prototypes as the pri-mary measure of progress. Combined with the preferencefor face-to-face communication, agile methods usually pro-duce less written documentation than other methods. Inan agile project, documentation and other project arte-facts all rank equally with working product. Stakeholdersare encouraged to prioritize them with other iterationoutcomes based exclusively on application value perceivedat the beginning of the iteration.

Specific tools and techniques such as continuous integra-tion, automated or unit tests, pair programming, testdriven development, design patterns, domain-driven de-sign, code refactoring and other techniques are often usedto improve quality and enhance project agility.

We will apply this methodology outside it’s normal busi-ness application to an open ended research problem. Ourcontribution will be evaluation and evolution of prescribedmethods above to the practicalities of realising and ex-perimenting a research challenge of some significance andcomplexity.

7. CONCLUSION

This paper has presented the challenges to be addressedby the FP7 Project PANDORA on persistent autonomy.Current existing autonomous systems require frequent op-erator intervention. The focus of Pandora is to enhance thelong-term autonomy of AUV missions, through increased

Fig. 5. Demonstration Platforms: (a) top left: HWU Nessie; (b) top right: UdG Girona 500; (c) HWU AugmentedReality Testbed

cognition, at all the levels of abstraction. An agile de-velopment approach will used which will allow frequentmeasurements of development metrics to rapidly optimizethe system on the three main experimental scenarios. Thisfrequent feedback will allow drive the expected increase ofautonomy over the project’s lifetime.

ACKNOWLEDGEMENTS

The research leading to these results has received fund-ing from the European Union Seventh Framework Pro-gramme FP7/20072013 Challenge 2 Cognitive Systems,Interaction, Robotics under grant agreement No 288273PANDORA

REFERENCES

Josep Aulinas, Marc Carreras, Yvan R. Petillot and-Joaquim Salvi andXavier Llado andRafael Garcia, andRicard Prados. Feature extraction for underwater visualslam. In IEEE/OES OCEANS 2011, 2011.

S. Calinon, F. D’halluin, E. L. Sauser, D. G. Caldwell, andA. G. Billard. Learning and reproduction of gestures byimitation: An approach based on hidden Markov modeland Gaussian mixture regression. IEEE Robotics andAutomation Magazine, 17(2):44–54, 2010.

Maria Fox, Derek Long, and Daniele Magazzeni. Auto-matic construction of efficient multiple battery usagepolicies. In Proceedings of the 21st International Con-ference on Automated Planning and Scheduling, ICAPS2011, 2011.

Maria Fox, Derek Long, and Daniele Magazzeni. Plan-based policy learning for autonomous feature tracking.In Proceedings of the 22nd International Conference onAutomated Planning and Scheduling, ICAPS 2012 (toappear), 2012.

George C. Karras, Savvas G. Loizou, and Kostas J. Kyr-iakopoulos. Towards semi-autonomous operation ofunder-actuated underwater vehicles: sensor fusion, on-line identification and visual servo control. Auton.Robots, 31(1):67–86, 2011.

Petar Kormushev, S. Calinon, and D. G. Caldwell. Imita-tion learning of positional and force skills demonstratedvia kinesthetic teaching and haptic input. AdvancedRobotics, 25(5):581–603, 2011.

Chee Sing Lee, Daniel E. Clark, and Joaquim Salvi. Slamwith single cluster phd filters. In Proceedings of the 21stInternational Conference on Robotics and Automation,ICRA 2012, 2012.

F. Maurelli, A. Mallios, S. Krupinski, Y. Petillot, andP. Ridao. Speeding-up particle convergence with prob-abilistic active localisation for auv. In IFAC IAV, 2010.

Dimitra Panagou and Kostas J. Kyriakopoulos. Controlof underactuated systems with viability constraints. InProc. of the 50th IEEE Conference on Decision andControl and European Control Conference, pages 5497–5502, Orlando, Florida, December 2011.

Y. Petillot, F. Maurelli, N. Valeyrie, A. Mallios, P. Ridao,J. Aulinas, and J. Salvi. Acoustic-based techniques forauv localisation. submitted to Journal of Engineeringfor Maritime Environment, 2010.