paper development of a flexible command and control...

P A P E R

Development of a Flexible Commandand Control Software Architecturefor Marine Robotic Applications
A U T H O R SBrian S. BinghamDepartment of MechanicalEngineering, Universityof Hawaii at Manoa
Jeffrey M. WallsDepartment of MechanicalEngineering, University of Michigan

Ryan M. EusticeDepartment of Naval Architectureand Marine Engineering,University of Michigan

A B S T R A C T
This paper reports the implementation of a supervisory control framework and
modular software architecture built around the lightweight communication andmarshalling (LCM) publish/subscribe message passing system. In particular, weexamine two diverse marine robotics applications using this modular system:(i) the development of an unmanned port security vehicle, a robotic surface platformto support first responders reacting to transportation security incidents in harborenvironments, and (ii) the adaptation of a commercial off-the-shelf autonomousunderwater vehicle (the Ocean-Server Iver2) for visual feature-based navigation.In both cases, the modular vehicle software infrastructures are based around theopen-source LCM software library for low-latency, real-time message passing. Toelucidate the real-world application of LCM in marine robotic systems, we presentthe software architecture of these two successful marine robotic applications andillustrate the capabilities and flexibilities of this approach to real-time marine robot-ics. We present benchmarking test results comparing the throughput of LCM withthe Mission-Oriented Operating Suite, another robot software system popular inmarine robotics. Experimental results demonstrate the capacity of the LCM frame-work tomake large amounts of actionable information available to the operator and toallow for distributed supervisory control. We also provide a discussion of the quali-tative tradeoffs involved in selecting software infrastructure for supervisory control.Keywords: autonomous vehicles, maritime domain awareness, supervisory control,interprocess control
ingly broad set of users. The Depart-
ment of Defense, the energy industry,

Introduction

The role of unmanned systems inthe maritime domain continues togrow as robotic tools continue tomature and evolve, serving an increas-

and the scientific community havebeen early adopters of this technology,guiding the development of newcapabilities to satisfy the needs of ap-plications such as ocean mine counter-measures (Alt, 2003; Schnoor, 2003;Eickstedt & Schmidt, 2003; Rishet al., 2001), seafloor survey for oiland gas exploration (Chance et al.,2000; Vestgard et al., 1999), andstudying deep-sea hydrothermal vents(Yoerger et al., 2007a, 2007b; Sohn,et al., 2008; Kunz, et al., 2009). Morerecently, autonomous vehicles havebeen used for disaster response such asresponding to oil spills similar to the

recent Deepwater Horizon incident(Camilli et al., 2010; Bhattacharyaet al., 2011) and natural disasterssuch as hurricanes (Murphy et al.,2008, 2009).

The focus of this article is on thedevelopment and implementation ofa flexible software architecture basedaround the lightweight communica-tions and marshalling (LCM) messagepassing protocol. We report on thedevelopment of the command andcontrol software developed for twomarine robot applications: supervisorycontrol of a maritime domain aware-ness (MDA)mission and themodifica-

May/J

tion of an Ocean-Server Iver2 au-tonomous underwater vehicle (AUV)for real-time visual feature-based navi-gation. This software is based on theopen-source LCM project initiatedto support the Defense AdvancedResearch Projects Agency (DARPA)UrbanGrandChallenge. The architec-ture we propose has distinct advan-tages over some of the available softwaretools for robotics, but it is not a uniquesolution. We describe a generic frame-work for this software along withbenchmarking tests to compare theperformance of this system with otherrobotics operating tools.

une 2011 Volume 45 Number 3 25

Background and CloselyRelated WorkReal-Time Commandand Control

A fundamental aspect of roboticsoftware is the use of modularity tocope with the complexity of real-timeoperation. For example, each hardwaresensor typically has a software driverthat manages the sensor communi-cation and passes the data from eachsensor to estimation, control, andplanning software modules. Generi-cally, the process of communicatingbetween modules is called interprocesscommunication (IPC). The choice ofIPC method is the key to the per-formance of the real-time roboticssoftware.

The LCM system is a relatively newalternative for implementing IPC forreal-time robotics in the marine envi-ronment (Huang et al., 2009, 2010).LCM was developed specifically forlow-latency, high-throughput com-munication to support the MIT en-trant in the DARPA Urban GrandChallenge (Leonard et al., 2008).The authors of LCM have comparedtheir design with similar systems cur-rently in use by the robotics commu-nity including the Player project, theCARMEN robotics package, theJoint Architecture for Unmanned Sys-tems, the Robotics Operating System(ROS), Microsoft’s Robotics Devel-opment Studio, and the Mission Ori-ented Operating Suite (MOOS)(Huang et al., 2009). LCM providesa set of software tools for message1

passing between modules (processesor threads). These tools provide the

1A “message” is any digital data that can be en-capsulated into a C-like data structure. For ex-ample, messages could be simple text, images,files, or binary data.

26 Marine Technology Society Journa

following functions for the develop-ment of a real-time system:■ message type specification for

l anguage - i ndependen t da t astructures;

■ marshalling (encoding and decod-ing) of LCM messages via softwarelibraries for C/C++, Java, Python,MATLAB, and C#;

■ s ca l ab le communica t ion v iauser datagram protocol (UDP)multicast.

The design of LCM emphasizes thehigh-throughput, low-latency needsof distributed robotics applicationsand accomplishes this by providingthe infrastructure that holds togetherthe modules of a software solutionrather than prescribing the structureof any of the processes or threads.The result is an extremely generalsoftware architecture that is amenableto the needs of a supervisory controlscheme described below.

The LCMmessages are transmittedacross the network via multicast UDPpackets. In this method, a process pub-lishes the data on a particular channel,defined by a unique name. Any otherprocess on the network can subscribeto the same channel and receive theUDP message. The marshalling ca-pabilities encode and decode thestructured data transmitted over thenetwork. The fingerprinting functionsof LCM guarantee that the messagesare type-safe, i.e., that the communi-cating processes/threads agree exactlyabout how to interpret the messages.This design allows for transparent dis-tribution of the system across a stan-dard Ethernet or wireless network.

The MOOS software system isbeing applied to a growing numberof marine robotic applications. Incontrast to the LCM libraries, whichprovide a minimal message passingfunctionality, MOOS is a more com-

l

plete set of communicating applica-tions including both the libraries andexecutables for a real-time platform(Newman, 2003). Communication isdone in a publish/subscribe model, re-ferred to as a star topology where eachclient subscribes to a central databasewhere all messages are stored. Messagepassing is done using human-readabletext strings containing one or morename/value pairs via transmission con-trol protocol (TCP). Typically pro-cesses and threads that interact withthe database are derived from a com-mon application. The central databasecan then control the message passingand execution of software developedfor the particular platform. MOOShas been used for a variety of ocean ro-botics applications including AUVsand unmanned surface craft (Curcioet al., 2008). The MOOS softwarehas more recently been extended tosupport cooperative control of multi-ple underwater platforms (Benjaminet al., 2010; Tye et al., 2009). Oneof the goals of this research is to qual-itatively and quantitatively comparethe MOOS system with an LCM im-plementation of real-time roboticcommand and control (Table 1).

Unmanned Vehicles for Port andHarbor Security

Unmanned marine vehicles (bothsurface vehicles and underwater vehi-cles) continue to evolve as importantcomponents of defense, industry, andscientific ocean operations. As theseplatforms have matured, they havefound application in an increasinglywide variety of operational scenarios.The research presented here is a partof a burgeoning impetus to apply ro-botic technology to MDA.

One aspect of MDA that has seenconsiderable attention is the need forunmanned systems to assist in disaster

response, including search and rescue.A number of commercial platforms(e.g., the iRobot Packbot and Foster-Miller Talon) are available for urbansearch and rescue (USAR). This tech-nology has aided a number of im-portant USAR incidents, the WorldTrade Center disaster being a particu-larly dramatic example (Casper &Murphy, 2003). Unmanned aerial ve-hicles (UAVs) have also been taskedwith disaster response such as survey-ing structural failures like the BerkmanPlaza II collapse in 2007 (Pratt et al.,2008). Similarly, UAVs have been ap-plied to wilderness search and rescue(Goodrich, et al., 2008). These ex-amples demonstrate the capability ofunmanned systems for security appli-cations using terrestrial and aerial ap-plications, but this potential is yet tobe realized by the maritime securitycommunity.

In the marine environment, re-searchers have leveraged the robotictechnologies developed for defense formaritime event response. The Centerfor Robot-Assisted Search and Rescuehas deployed both Unmanned SurfaceVehicles (USVs) and UAVs for post-disaster port and littoral inspectionafter hurricanes Ike andWilma (Steimleet al., 2009;Murphy et al., 2008, 2009).

2The authors’ use of the acronym “UPSV” isnot meant to add a new acronym to the longlist of acronyms in the literature, but thisacronym is used for brevity in this article.

Supervisory ControlSupervisory control explicitly in-

cludes the human operator in the feed-

back loop of an automated system.Thomas Sheridan coalesced much ofthe early thought on this paradigm,“supervisory control means that oneor more human operators are intermit-tently programming and continuallyreceiving information from a com-puter that itself closes an autonomouscontrol loop through artificial effectorsto the controlled process or task envi-ronment” (Sheridan, 1992).

The framework we have developedis inspired by Sheridan’s emphasis oncareful attention to the capabilities ofthe human and robotic componentsof such a system. Figure 1 illustratesthe supervisory architecture where thehuman supervisor can interact with thesystem at a variety of levels. By build-ing this flexibility into the software,we can provide users with an interfacethat allows them to simultaneouslyautomate tasks and take direct controlof the platform depending on thesituation.

Case Study I: Universityof Hawaii FieldRobotics Laboratory

We present the design of an un-manned system for port and harbor se-curity as part of the Department ofHomeland Security’s (DHS) initiativeto enhance MDA. At the Field Robot-ics Laboratory (FRL) at the University

May/J

of Hawaii, we have designed, built,and tested the unmanned port securityvessel (UPSV2) to investigate theunique challenges of the homelandsecurity mission.

The UPSV concept is designedaround the need for port resilienceafter a transportation security incident(TSI). A TSI can be the result of a nat-ural disaster (e.g., hurricane, earth-quake, or tsunami), environmentaldamage (e.g., chemical spill), or terror-ist attack. As an example of the poten-tial impact, U.S. West Coast portssupport 8 million U.S. jobs as 14 mil-lion TEUs (20-foot equivalent units)are moved through these ports (An-nual Report, 2009). Because of theeconomic and security risks associatedwith the impact of such an event, there

TABLE 1

Comparison of LCM and MOOS communication implementations.

LCM
MOOS
Topology
Broadcast: Each process/threadsends message throughout thenetwork.
Star: Centralized database. Eachclient allowed to publish andsubscribe.

Transport
UDP multicast TCP
Message types
User defined, type-safe binary Text strings
FIGURE 1

Conceptual flowchart of the automatic andsupervisory control tasks for the UPSV plat-form. The tasks on the left are automated bythe use of computational elements for real-time feedback. The tasks on the right representthe various ways the human supervisor mayintervene and redirect the system.


is considerable interest in a variety oftechniques to minimize the downtime of port facilities affected by suchan event. The UPSV is designed toprovide information to decision mak-ers that can help speed the process ofbringing the port back to operation.In particular, the UPSV is capable ofbathymetric mapping to identify ob-structions and to perform acousticand optical surveys of coastal infra-structure such as piers and pilings.

The FRL Vehicle Software (FVS) isa general purpose real-time supervisorycommand and control system for ro-botic development. This software is aderivative of the LCM-based softwareinfrastructure developed by the Per-ceptual Robotics Laboratory (PeRL)at the University of Michigan (dis-cussed in Case Study II: University ofMichigan PeRL). The FVS system isspecifically designed to support thesupervisory control necessary to includethe operator as an explicit componentin the vehicle command and control.

Unmanned Port Security VesselOur initial prototype platform is

shown in Figure 2. In working withthe DHS first responders, we have dis-covered the following concepts thathave led us toward the supervisorycontrol framework for the UPSV.


■ Port and harbor environments areexceedingly complex environ-ments. Fully autonomous controlis not suitable for such an environ-ment because of the changing na-ture of the operation and the needof first responders to provide real-time commands.

■ The decision making structure forhomeland security during an in-cident in a port or harbor is dis-tributed. Actionable informationmust be relayed to the decisionmakers in near real-time and thetask of first responders is constantlychanging, resulting in dynamicgoals for a robotic platform.

■ To successfully support homelandsecurity’s MDA, robotic toolsmust augment the multiagencyteams. This requires close collabo-ration between first respondersand the unmanned system.

■ First responders need real-timeaccess to all the data being collectedas well as an interface to seamlesslyretask the unmanned system with-out requiring the full attention ofthe operator.

One particularly important facet ofthese concepts is the need for un-manned surface systems to obey the“rules of the road” and avoid colli-sions. Current thinking suggests thatunmanned marine vessels must com-ply with the International Regulationsfor Preventing Collisions at Sea(COLREGS) while on the surface(Showalter & Manley, 2009). A num-ber of research projects have ad-dressed this requirement by seekingto advance the technology to autono-mously comply with such a mandate(Benjamin et al., 2006; Larson et al.,2006). The missions of the UPSV(explained below) are intended totake place immediately after an inci-dent that would close a port, so the

l

vehicle would be operating in scenarioswhere the United States Coast Guard(USCG) has complete control of sur-face traffic. However, there is sig-nificant interest from these users tomaintain authority over any vessel,even if it had the ability to autono-mously follow the rules of the road.

FRL Vehicle SoftwareFigure 3 illustrates the FVS soft-

ware design. The sensors are generi-cally categorized as either navigationor payload sensors. The difference isthat the navigation sensors are used inreal-time by the onboard autonomouscontrol to guide the vehicle state. Thepayload sensors provide the environ-mental measurements and are archivedonboard and transmitted to the super-visor to provide feedback for interven-tion and tasking. All onboard softwareis running on a singleboard PC runningthe Ubuntu distribution of Linux.

Each sensor has a correspondingsoftware driver (written in standardC), which maintains the commu-nication interface to the vehicle. Eachdriver is a standalone process that deli-vers the sensor data to the software es-timator and LCM logger via LCMmessages. The estimator and controller(written in Python) are two threads ofthe same process. Using threads sim-plifies the asynchronous coordinationso that the estimator can provide afresh state estimate to the control-ler at a predetermined rate (typically10 Hz). The controller provides thelow-level feedback control that re-sponds to the state estimate and gener-ates the thrust commands for the portand starboard thrusters. These com-mands are then sent to another driverthat manages the interface with themotor controllers.

To illustrate the function of theLCM messaging, two examples of

FIGURE 2

Photograph of the first prototype UPSV vehicledesigned for software and instrument evaluation.

LCMdata type specification are shownin Figure 4. The LCM toolbox pro-vides executables for automaticallygenerating the necessary source codeso that all the processes and threads(in C, C#, C++, Python, and Java)can make use of these custom datastructures. The gps_rmc_t data typeis for transmitting GPS position infor-mation from the driver to the estima-tor. The message is composed of atimestamp (in microseconds), latitude,longitude, and the speed-over-ground.Each of these numerical components isspecified by an operating system, lan-

guage, and processor-independenttype specification (such as 64-bit inte-ger or double). The encoding and de-coding of the message is accomplishedby the LCM generated software and istransparent to the FVS processes thatactually manipulate the informa-tion. The pose_t data type is a sim-ple container for a timestamp and a12-element array of doubles that con-tain the estimated position, attitude,velocities, and attitude rates. This sim-ple message is used extensively by therobot command and control softwareas an output of any of the estimation

May/J

algorithms and input to any of thecontrol algorithms. This example illus-trates how simple it is to asynchro-nously pass arrays of data betweenvarious processes and threads writtenin different languages. Processes andthreads can seamlessly be distributedon any machine throughout the net-work. This is of critical importancefor supporting supervisory controlsince the human supervisor needs tointeract directly with both the naviga-tion and payload.

Case Study II: Universityof Michigan PeRL

The PeRL Software Library (PSL)is a general purpose robotics toolkitinitially developed by the PerceptualRobotics Laboratory at the Universityof Michigan for use withOcean-ServerTechnology, Inc., Iver2 AUVs. PSL isbased around the LCM messagingparadigm (Huang et al., 2010) al-lowing for modular development andoperation. Multiple processes can

FIGURE 3

Flowchart of the FVS LCM-based software infrastructure for supervisory control.

FIGURE 4

Two examples of LCM type definitions. The definition on the left is associated with the recom-mended minimum data (RMC) sentence from a GPS receiver. The definition on the right is as-sociated with the estimated state broadcast from the estimator thread.


communicate by both publishing mes-sages and assigning callback functionsthat execute upon receiving messagesover a given LCM channel. A suiteof useful tools are also included withLCM such as logging, playback, andreal-time data visualization. These util-ities ease the process of debugging ex-isting software as well as writing newfunctions. This system lends itself toa clean organizational structure thatcan easily partition jobs such as hard-ware interfacing, state estimation andcontrol tasks.

Major thrusts of current devel-opment wi th in PeRL focus onperception-driven control and real-timevisually augmented navigation (VAN)(Brown, Kim, & Eustice, 2009).Perception-driven control refers to avehicle autonomously adapting its tra-jectory, either returning to a knownlandmark or continuing exploration,in order to optimally cover a surveyarea with minimal pose uncertainty.The VAN framework uses perceptualdata from onboard cameras to correctdrift inherent in dead-reckoned navi-gation. The following section providesadditional detail through an exampleof a real-world implementation of thePSL architecture on an Iver2 AUV.

Iver2 ImplementationThe basic system design is illus-

trated in Figure 5. PSL software handlesall sensor processing and communica-tion with a separate Ocean-Servervehicle controller called UnderwaterVehicle Console (UVC). The modu-larity provided by LCM allows theuser to launch any one of a variety ofsensors or use different fi lteringschemes to process raw data, such asvehicle attitude or velocity, sent to thevehicle controller. Furthermore, PSLinterfaces with the vehicle controllerthrough a library of “backseat-driver”


commands to execute perception-driven control (Figure 6).

The Iver2 AUV is a commercialoff-the-shelf underwater platformproviding great flexibility in terms ofhardware and software that can beadded by the end user. Two Iver2AUVs currently serve as real-worldtestbeds for simultaneous localizationand mapping (SLAM) research withinPeRL. The vehicles were chosen fortheir small size (easing transportationand launch/recovery constraints)while still providing capable perfor-mance. Each vehicle was integratedwith additional navigation and per-ceptual sensors to perform SLAM aswell as multi-AUV communicationsand operation. A 32-bit embeddedPC104 CPU payload was also addedfor data logging and real-time controlthrough the vehicle controller.

In the Iver2 software configuration,PSL accomplishes two main tasks:

l

publishing sensor data and interfacingwith the lower level vehicle controller.

Sensor DriversIndependently running sensor driv-

ers publish all sensor informationover the network through LCM.Table 2 lists available sensors on thePeRL Iver2 AUVs. Other processesthat have subscribed to the sensordata are then able to execute callbackfunctions based upon sensor publishevents.

Backseat DriverAll low-level vehicle control such as

waypoint and depth tracking is carriedout by the proprietary Ocean-Serversoftware, UVC, on an embeddedCPU. Additionally, UVC can processa large set of commands from a payloadCPU. This architecture is designated afrontseat/backseat scheme where in-telligent control can be relayed fromthe backseat to a frontseat processorhandling dynamic low level control.Commands are formatted ASCIINMEA strings exchanged over a serialconnection between the two CPUs.Table 3 lists some of the availablebackseat driver messages and theircorresponding LCM structures. Thebackseat driver has the ability to sendservo and primitive commands aswell as higher level waypoint informa-tion. Additionally, the backseat drivercan update the vehicle state as per-ceived by UVC. UVC also transmitsacknowledgement of received com-mands with an error flag when appro-priate. Finally, the backseat driver canquery UVC for basic state informationincluding mission status with a datarequest message.

All commands sent to UVC fromPSL are sent serially through theos-conduit process running on thebackseat CPU. In addition to UVC

FIGURE 5

Iver2 System Architecture illustrating informa-tion exchange onboard an Iver2 AUV: sensor-level information is passed via LCM towardbackseat processes that in turn communicateserially with the low-level vehicle controller.

FIGURE 6

PeRL Iver2 AUV.

commands, the backseat CPU can for-ward vehicle state information. Theos-conduit process listens to the fullLCM stream and processes sensorinformation and backseat driver com-mands by correctly packaging messagesand sending them to the serial bus.

A h i ghe r l e v e l p r o c e s s , o s -remotehelm, tracks mission statusand publishes LCM backseat drivercommands that os-conduit later relaysto UVC. This process performs taskssuch as loading, starting and abortingmissions. Furthermore, os-remote-helm is able to control other devicesvia LCM, such as cameras, which it

only powers between designated mis-sion waypoints. Through this process,we have begun to implement percep-tion driven control. Future work willadaptively alter waypoint parametersin order to aid optimal exploration.

Visually Augmented Navigation(VAN)

The real-time VAN library is cur-rently an active area of developmentwithin the PSL. VAN is a real-timefeature-based navigation and mappingframework, which augments onboarddead-reckoned navigation with con-straints derived from visual data. Pair-

May/J

wise navigation constraints betweenoverlapping images are fused within apose-graph SLAM framework. ThisVAN methodology improves upontraditional navigation techniques bybounding pose uncertainty and hasthe ability to scale to large environ-ments capturing thousands of imagekeyframes. For a detailed descriptionof the VAN algorithm, the reader is re-ferred to Eustice (2008) and Kim andEustice (2009).

Figure 7 depicts algorithmic andcommunication flow in the VANframework. VAN is a multithreadedprocess exploiting LCM as well asshared memory for communicationwhen necessary for reduced bandwidthor increased speed. For example, staticinformation, such as the camera cali-bration matrix, is required across allthreads and therefore uses sharedmemory. Threads also communicatevia LCM to share dynamic informa-tion such as an image feature vector.The use of LCM allows the refinednavigation estimate to be easily passedto other processes, such as the backseatdriver detailed in the previous section.

Within VAN, the feature threadfirst extracts feature keypoints fromeach image. Currently, fast feature ex-traction (up to 10 Hz) is achieved by

TABLE 2

PSL available sensors.

Sensor
State Property Update Rate (Hz) LCM Structure
une 2011 Vo

LCM Channel

Ocean-Server OS5000 Compass
Attitude 0.01-20 os_compass_t OS_COMPASS
USGlobalSat EM-406a GPS
XYZ position 1.0 gpsd_t GPSD
Prosilica GC1380H(C) Camera
gray/color image 1-5 prosilica_t PROSILICA_M PROSILICA_C
Teledyne RDI 600kHz Explorer DVL
body velocity 7 pdi_pd5_t RDI
KVH DSP-3000 single-axis FOG
Yaw rate 100 kvh_dsp3000_t KVH
Desert Star SSP-1 300PSIG DigitalPressure Transducer

Depth
0.0625-4 dstar_ssp1_t DSTAR
Microstrain 3D-GX1 AHRS
Attitude, body rotation rate 1.0-100 ms_gx1_t MICROSTRAIN
TABLE 3

Backseat driver commands.

UVC ASCII Command
Action LCM Structure
$ACK
command acknowledgment ack_t
$OMP
primitive control mode omp_t
$OMS
servo control mode oms_t
$OMP
primitive control mode omp_t
$OMW
mission control mode omw t
$OMSTART/STOP/LOAD
mission start/stop/load omstart/stop/load_t
$OSD
data request osd_t
$OSI
state information osi_t
$OJW
jump to waypoint ojw_t
lume 45 Number 3 31

using a feature detector running on aGraphics Processing Unit (GPU) (Wu,2007). The hovering AUV (HAUV)thread is then responsible for assign-ing a camera pose prior to the time ofeach image event. This thread also pre-pares bathymetric information fromDoppler Velocity Logger (DVL) rangeobservations that is later used for esti-mating scene depth in the camera frame.Our state estimation engine is thenable to hypothesize potential matchingpairs. The final task of the image regis-tration engine is to recover an optimalestimate of relative poses betweencorresponding camera pairs. The two-


view thread first performs a pose-constrained correspondence search(Eustice, 2008), which rejects geomet-rically inconsistent feature matchesbased upon the uncertainty in the rel-ative pose prior. A robust correspon-dence method rejects outliers from theputative set of feature correspondencesand refines the initial relative pose esti-mate for bundle adjustment. The two-view thread thenminimizes a two-viewbundle adjustment cost function com-posed of the sum squared reprojectionerror in order to accomplish this.

VAN constraints are fused under apose-graph SLAM framework, referred

l

to as the state estimation server. Thisserver communicates via LCM withthe state estimation engine. Currently,we use an open-source implementa-tion of iSAM (Kaess et al., 2008;Kaess & Dellaert, 2009), although an-other estimation framework could beused. This state estimation server alsocommunicates with the image registra-tion engine by providing hypothesizedcandidate overlapping image pairsbased upon an estimated overlap be-tween pairwise camera nodes in thepose-graph.

Real-time VAN performance iscurrently being field-tested for ship

FIGURE 7

(a) Van systems-level approach to image registration. (b) VAN software architecture. Overview of VAN framework illustrating algorithmic and com-munication flow, solving for the relative pose between the camera pair corresponding to image Ii and Ij. Bold boxes denote each thread while innerboxes designate sub processes within each thread. Arrows depict the direction of communication with the structure name to the LCM stream. SIFTfeatures detectors are used for extraction. Prior knowledge of relative-pose is derived from the state estimate, which is then used in the two-viewregistration framework. The final output from this framework is an optimal 5-DOF camera constraint.

hull inspection applications using theBluefin HULS3 HAUV. Preliminaryreal-world experiments have shownthe ability of this software to handlehigh bandwidth data and correctpoor dead-reckoned navigation bymatching visual features in the envi-ronment in real-time running on stan-dard laptop hardware. LCM hashelped to provide a basic set of func-tions and tools leading to a successfulreal-time VAN implementation.

Results: SoftwareBenchmarking

For supervisory control, the soft-ware system must be scalable to allowlarge amounts of data to be transmittedthrough the system, which includesthe human supervisor. For example,in addition to updates on vehiclestate, the UPSV operator needs to re-view chemical data (from the massspectrometer) and bathymetry data(from the multibeam sonar) to providehigh-level intervention and tasking ofthe vessel while under operation. Theauthors of LCM have characterized

the bandwidth performance of theLCM software by comparing the per-formance of an LCM system (in Cand in Java) with other robotics soft-ware packages such as IPC, Player,and ROS (Huang et al., 2009, 2010).We have extended this characteriza-tion to include a Python implementa-tion of LCM (because we makeextensive use of Python in the FVSsoftware) and to compare the perfor-mance with the MOOS system,which is currently in use in a varietyof marine robotic applications.

The experimental procedure is sim-ilar to that discussed in the originalLCM comparisons by the authors ofLCM where fixed size messages weretransmitted from a source node to one,two, or four clients simultaneously.We followed the same protocol as re-ported previously (Huang et al.,2009) so that our analysis would beconsistent with their tests and providenew comparisons. The results areshown in Figure 8. For each datapoint in the figures 100 MB of datais split into 800 byte messages at afixed rate. The 800 byte message is ofsimilar size to a single multibeam scan

May/J

from the UPSV Imagenex sonar. Allthe tests were run on an Intel Core 2Duo CPU running Ubuntu 10.04with 2 Gb of RAM. Three different im-plementations were run for comparison:(1) the LCM sending and receivingnodes written in standard C, (2) theLCM sending and receiving nodeswritten in Python, and (3) the MOOSclients written in C++.

The results in Figure 8 illustratesome important conclusions aboutthe performance of these three sys-tems. First, performance of the LCMimplementations in C and Pythonwere comparable. The maximumthroughput for these implementationsshowed that the Python implemen-tation achieved only slightly lessthroughput than the C implementa-tion: 29.3 MB/s for C and 22.1 MB/sfor Python. The MOOS implementa-tion is qualitatively and quantitativelydifferent. Because MOOS is a full-operating suite of software tools theupdate rate of the source node wasspecified indirectly through MOOSat a fixed rate. Setting this updaterate above 90 Hz (which equated to0.144 MB/s for one client) resulted

FIGURE 8

(a) Results for one client. (b) Results for two clients. (c) Results for four clients. Message passing throughput test results with one, two, and four clients.In each comparison, three implementations are illustrated as indicated in the legend. LCM-C is a C implementation of LCM. LCM-Py is a Pythonimplementation of LCM. MOOS-C is a C implementation of MOOS.


in MOOS running at its maximumupdate rate. This resulted in a fixed,maximum throughput for any targetthroughput above this threshold.This is illustrated in the horizontalportions of the throughput curves inFigure 8. For the single source, singleclient case the throughput achievablewith MOOS was found to be roughly6 MB/s, or about 27% of the through-put using the Python implementationof LCM under similar circumstances.

DiscussionThe quantitative results presented

above demonstrate the ability of theLCM-based software architecture tomove large amounts of data through-out a distributed supervisory controlsystem. However, there are qualitativereasons for our choice of using LCMcompared with other tools availablefrom the robotics community. Oneimportant consideration is that LCMis based on a common set of librariesthat encapsulate the important task ofmessage passing. This is in contrast tomany of the other robotics toolkits thatprovide a full infrastructure by includ-ing libraries, pre-defined data typesand prescribed event loops. The devel-opers of LCM have taken a minimalis-tic approach, they have termed an “a lacarte” system, that serves the needs ofthe UPSV project because it is scalable,affording the ability to simultaneouslydevelop low-level feedback control andto provide high-level supervisory inter-faces with the same set of tools.

Another important design aspect isthe choice of binary message passingrather than using plain text or a text-based protocol such as XML. Theobvious tradeoff is between the band-width efficiency of binary messagesand the readability of text-based com-munication. LCM includes both an


API for developers and a set of debug-ging, monitoring and playback toolsthat successfully manage the additionalcomplexity of binary messages. Ourexperience with LCM binary messageshas been seamless and the added func-tionality of passing rich, user-defineddata structures has eased our develop-ment tremendously.

Lastly, a key aspect of the LCM de-sign is the choice of a UDP multicastfor communication that allows all cli-ents to see all messages rather than in-cluding a centralized hub to mediatecommunication between clients. Thisdesign choice emphasizes the need forlow-latency communication, which iscritical for real-time implementationat the expense of dropped messages.(In the two applications discussedabove, the number of dropped mes-sages was near zero during field appli-cations.) This also eliminates the needfor the added complexity of a cen-tralized hub for communication. Ourexperience with this feature is very pos-itive as demonstrated by the data pre-sented above. The data presentedabove suggests that the minimalisticapproach taken by the LCM providesfor a leaner software implementationwith a greater capability for movingmessages through the network.

ConclusionsThis paper has described the devel-

opment of a distributed, LCM-basedsoftware architecture for robotic sys-tems. This architecture leverages severaladvantages of LCM for real-time low-latency robotic command and control.In order to emphasize the advantagesof a modular architecture, the success-ful implementation of this software de-sign for both an UPSV and within thePSL were reported.

l

We first presented the software in-frastructure for supervisory control ofan UPSV to augment MDA for home-land security operation. Homelandsecurity operations present new chal-lenges to unmanned marine systemsbecause of the complexity of both theoperating environment (ports andharbors) and the mission, which canbe multifaceted because of the distrib-uted nature of decision making in amultiagency incident response. Wepresented a supervisory control frame-work that provides first responderswith a tool to increase their situationalawareness during the response to aTSI.We argue that supervisory controlbalances the need for extending theabilities of first responders with theirneed for high level intervention andtasking of assets during a response.

The PSL used for real-time visualfeature-based navigation was also pre-sented. An overview of the PSL hasdemonstrated the ability of the LCMframework to promote a clean, modu-lar architecture. Furthermore, a reviewof the backseat driver and VAN imple-mentation has shown that LCM-basedsoftware can easily interface with bothexisting software and handle largebandwidth data in real time.

The quantitative results of thestudy illustrate the performance ofthe LCM framework for implementingsuch a command and control system.We have extended the comparison ofLCM with other robotic control soft-ware by comparing it to the MOOSsystem, which is common in marinerobotics applications. We have shownthat the LCM framework can pro-vide data throughput on the order of25 MB/s, which is more than suffi-cient to support real-time feedbackwhile providing the human super-visor with actionable remote dataand the interface to interact with the

UPSV for intervention and reactivetasking.

AcknowledgmentThe work on the UPSV was sup-

ported in part by a grant from theDepartment of Homeland SecurityScience and Technology Directorate(2009-ST-061-MD001); the workon PSL was supported in part by grantsfrom the National Science Foundation(IIS-0746455) and through a grantfrom the Office of Naval Research(N00014-07-1-0791).

Lead Author:Brian S. BinghamDepartment of MechanicalEngineering, Universityof Hawaii at Manoa,2540 Dole Street,Holmes Hall 302, Honolulu, HIEmail: [email protected]

ReferencesBenjamin, M., Curcio, J., & Newman, P.

2006. Navigation of unmanned marine

vehicles in accordance with the rules of the

road. In: Proceedings 2006 IEEE International

Conference on Robotics and Automation.

3581-7. doi: 10.1109/ROBOT.2006.1642249.

Benjamin, M.R., Schmidt, H., Newman,

P.M., & Leonard, J.J. 2010. Nested auton-

omy for unmanned marine vehicles with

MOOS-IvP. J Field Robot. 26(6):834-74.

doi: 10.1002/rob.20370.

Bhattacharya, S., Heidarsson, H.K.,

Sukhatme, G.S., & Kumar, V. 2011.

Cooperative control of autonomous surface

vehicles for oil skimming and cleanup.

In: Proceedings of the IEEE International

Conference on Robotics and Automation.

Brown, H.C., Kim, A., & Eustice, R.M.

2009. An overview of autonomous underwater

vehicle research and testbed at PeRL. Mar

Technol Soc J. 43(2):33-47. doi: 10.4031/

MTSJ.43.2.4.

Camilli, R., Reddy, C.M., Yoerger, D.R.,

Van Mooy, B.A.S., Jakuba, M.V., Kinsey,

J.C., … Maloney, J.V. 2010. Tracking

hydrocarbon plume transport and biodegra-

dation at Deepwater Horizon. Science.

330(6001):201-4.

Casper, J., & Murphy, R. 2003. Human-

robot interactions during the robot-assisted

urban search and rescue response at the World

Trade Center. IEEE Trans Syst Man Cybern

B Cybern. 33(3):367-85.

Chance, T., Kleiner, A., & Northcutt, J.

2000. The autonomous underwater vehicle

(AUV): A cost-effective alternative to deep-

towed technology. Integr Coastal Zone

Manage. 2(7):65-9.

Curcio, J., Schneider, T., Benjamin, M., &

Patrikalakis, A. 2008. Autonomous surface

craft provide flexibility to remote adaptive

oceanographic sampling and modeling.

In: Proc MTS/IEEE OCEANS Conf. 1-7.

IEEE. doi: 10.1109/OCEANS.2008.5152078.

Eickstedt, D., & Schmidt, H. 2003. A

low-frequency sonar for sensor-adaptive,

multistatic, detection and classification of

underwater targets with AUVs. In: Proc

MTS/IEEE OCEANS Conf. 3:1440-7.

Eustice, R.M. 2008. Toward real-time

visually augmented navigation for autono-

mous search and inspection of ship hulls

and port facilities. Intl. Symposium on

Technology and the Mine Problem. Mine

Warfare Association (MINWARA).

Eustice, R.M., Brown, H.C., & Kim, A. 2008.

An overview of AUV algorithms research

and testbed at the University of Michigan. In:

IEEE/OES Autonomous Underwater Vehicles

Conference. 1-9. doi: 10.1109/AUV.2008.

5290531.

Goodrich, M.A., Morse, B.S., Gerhardt, D.,

Cooper, J.L., Quigley, M., Adams, J.A., &

Humphrey, C. 2008. Supporting wilderness

search and rescue using a camera-equipped

mini UAV. J Field Robot. 25:89-110.

doi: 10.1002/rob.20226.

May/J

Huang, A., Olson, E., & Moore, D. 2009.

Lightweight Communications andMarshalling

for Low-Latency Interprocess Communication.

Computer Science and Artificial Intelligence

Laboratory Technical Report. http://dspace.

mit.edu/bitstream/handle/1721.1/46708/

MIT-CSAIL-TR-2009-041.pdf.

Huang, A.S., Olson, E., & Moore, D.C.

2010. LCM: Lightweight communications

and marshalling. In: Proc. Int. Conf. on

Intelligent Robots and Systems (ROS). IEEE.

Kaess, M., & Dellaert, F. 2009. Covariance

recovery from a square root information

matrix for data association. J Robot Autonom

Syst. 57:1198-210.

Kaess, M., Ranganathan, A., & Dellaert, F.

2008. iSAM: Incremental smoothing and

mapping. IEEE Trans. Robot. 24:1365-78.

Kim, A., & Eustice, R. M. 2009. Pose-graph

visual SLAM with geometric model selection

for autonomous underwater ship hull inspection.

In: Proc IEEE/RSJ Int Conf Intell Robots Syst.

1559-65.

Kunz, C., Murphy, C., Singh, H., Willis, C.,

Sohn, R., Singh, S., … Bailey, J. 2009.

Toward extraplanetary under-ice exploration:

robotic steps in the Arctic. J Field Robot,

26(4):411-29. doi: 10.1002/rob.20288.

Larson, J., Bruch, M., & Ebken, J. 2006.

Autonomous navigation and obstacle avoid-

ance for unmanned surface vehicles. In: Pro-

ceedings of the SPIE: Unmanned Systems

Technology VIII. 6230:17-20. doi: 10.1117/

12.663798.

Leonard, J., How, J., Teller, S., Berger, M.,

Campbell, S., Fiore, G., … Williams, J. 2008.

A perception driven autonomous urban

vehicle. J Field Robot. 25(10):727-74.

Murphy, R.R., Steimle, E., Griffin, C.,

Cullins, C., Hall, M., & Pratt, K. 2008.

Cooperative use of unmanned sea surface and

micro aerial vehicles at Hurricane Wilma.

J Field Robot. 25:164-80. doi: 10.1002/

rob.20235.

Murphy, R.R., Steimle, E., Hall, M.,

Lindemuth, M., Trejo, D., Hurlebaus, S., …


Slocum, D. 2009. Robot-assisted bridge

inspection after Hurricane Ike. 2009 IEEE

International Workshop on Safety, Security

Rescue Robotics (SSRR). 1-5. doi: 10.1109/

SSRR.2009.5424144.

Newman, P. 2003. A mission oriented

operating suite. Technical Report OE2007-07.

MIT Department of Ocean Engineering.

Pacific Maritime Association. 2009. Annual

Report. Pacific Maritime Association. http://

www.pmanet.org/pubs/AnnualReports/2009/

PMA%202009%20Annual%20Report.pdf.

Pratt, K., Murphy, R., Burke, J., Craighead,

J., Griffin, C., & Stover, S. 2008. Use of

tethered small unmanned aerial system at

Berkman Plaza II collapse. IEEE International

Workshop on Safety, Security and Rescue

Robotics. 134-9. doi: 10.1109/

SSRR.2008.4745890.

Rish, J., Willcox, S., Grieve, R., Montieth, I.,

& Vaganay, J. 2001. Operational testing of

the Battlespace Preparation AUV in the

shallow water regime. In: Proc MTS/IEEE

OCEANS Conf. 1:123-9.

Schnoor, R.T. 2003. Modularized unmanned

vehicle packages for the littoral combat

ship mine countermeasures missions. In:

Proc MTS/IEEE OCEANS Conf. 3:1437-9.

doi: 10.1109/OCEANS.2003.1282587.

Sheridan, T.B. 1992. Telerobotics, Automa-

tion, and Human Supervisory Control.

Cambridge, MA: MIT Press. 1.

Showalter, S., & Manley, J. 2009. Legal

and engineering challenges to widespread

adoption of unmanned maritime vehicles.


Sohn, R., Willis, C., Humphris, S., Shank,

T., Singh, H., Edmonds, H.N., … Soule, A.

2008. Explosive volcanism on the ultraslow-

spreading Gakkel Ridge, Arctic Ocean.

Nature. 453(7199):1236-8. doi: 10.1038/

nature07075.

Steimle, E., Murphy, R., Lindemuth, M., &

Hall, M. 2009. Unmanned marine vehicle

use at Hurricanes Wilma and Ike. In:

Proc MTS/IEEE OCEANS Conf. 1-6.


Tye, C., Kinney, M., Frenzel, J., O’Rourke, M.,

& Edwards, D. 2009. A MOOS module for

autonomous underwater vehicle fleet control.


Vestgard, K., Storkersen, N., & Sortland, J.

1999. Seabed surveying with Hugin AUV.

In: Proceedings of the 11th International

Symposium on Unmanned Untethered

Submersible Technology. Durham, NH:

Autonomous Undersea Systems Institute.

von Alt, C. 2003. REMUS 100 transportable

mine countermeasure package. In: Proc MTS/

IEEE OCEANS Conf. 4:1925-30.

Wu, C. 2007. SiftGPU: A GPU implemen-

tation of Scale Invariant Feature Transform

(SIFT). Retrieved from http://cs.unc.edu/

~ccwu/siftgpu (accessed 8 April 2011).

Yoerger, D.R., Bradley, A., Jakuba, M.,

German, C., Shank, T., & Tivey, M. 2007a.

Autonomous and remotely operated vehicle

technology for hydrothermal vent discovery,

exploration and sampling. Oceanography.

20(1):152-61.

Yoerger, D.R., Jakuba, M., Bradley, A.M., &

Bingham, B. 2007b. Techniques for deep

sea near bottom survey using an autonomous

underwater vehicle. Int J Robot Res. 26(1):

41-54. doi: 10.1177/0278364907073773.

l

paper development of a flexible command and control...

Documents