establishing a-priori performance guarantees for robot ......damian lyons, ron arkin, shu jiang,...
TRANSCRIPT
Masthead LogoFordham University
DigitalResearch@Fordham
Faculty Publications Robotics and Computer Vision Laboratory
2017
Establishing A-Priori Performance Guarantees forRobot Missions that include Localization SoftwareDamian LyonsFordham University, [email protected]
Ron ArkinGeorgia Institute of Technology
Shu Jiang
Matt O'Brien
Feng Tang
See next page for additional authors
Follow this and additional works at: https://fordham.bepress.com/frcv_facultypubs
Part of the Artificial Intelligence and Robotics Commons, and the Robotics Commons
This Article is brought to you for free and open access by the Robotics and Computer Vision Laboratory at DigitalResearch@Fordham. It has beenaccepted for inclusion in Faculty Publications by an authorized administrator of DigitalResearch@Fordham. For more information, please [email protected].
Recommended CitationDamian Lyons, Ron Arkin, Shu Jiang, Matthew O'Brien, Feng Tang and Peng Tang, "Establishing A-Priori Performance Guarantees forRobot Missions that include Localization Software" International Journal of Monitoring and Surveillance Technologies Research(IJMSTR) Volume 5, Issue 1 2017.
AuthorsDamian Lyons, Ron Arkin, Shu Jiang, Matt O'Brien, Feng Tang, and Peng Tang
This article is available at DigitalResearch@Fordham: https://fordham.bepress.com/frcv_facultypubs/58
DOI: 10.4018/IJMSTR.2017010103
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
Copyright©2017,IGIGlobal.CopyingordistributinginprintorelectronicformswithoutwrittenpermissionofIGIGlobalisprohibited.
Establishing A-Priori Performance Guarantees for Robot Missions that Include Localization SoftwareDamian Lyons, Department of Computer and Information Science, Fordham University, New York City, NY, USA
Ronald C Arkin, Georgia Institute of Technology, Atlanta, GA, USA
Shu Jiang, Georgia Institute of Technology, Atlanta, GA, USA
Matthew J O’Brien, Georgia Institute of Technology, Atlanta, GA, USA
Feng Tang, Fordham University, New York City, NY, USA
Peng Tang, Fordham University, New York City, NY, USA
ABSTRACT
Oneapproachtodeterminingwhetheranautomatedsystemisperformingcorrectlyistomonitoritsperformance,signalingwhentheperformanceisnotacceptable;anotherapproachistoautomaticallyanalyzethepossiblebehaviorsofthesystema-priorianddetermineperformanceguarantees.Theaauthorshaveappliedthissecondapproachtoautomaticallyderiveperformanceguaranteesforbehavior-based, multi-robot critical mission software using an innovative approach to formal verificationforroboticsoftware.Localizationandmappingalgorithmscanallowarobottonavigatewellinanunknownenvironment.However,whethersuchalgorithmsenhanceanyspecificrobotmissioniscurrentlyamatterforempiricalvalidation.Severalapproachestoincorporatingpre-existingsoftwareintotheauthors’probabilisticverificationframeworkarepresented,andoneusedtoincludeMonte-Carlobasedlocalizationsoftware.Verificationandexperimentalvalidationresultsarediscussedforreallocalizationmissionswiththissoftware,showingthattheproposedapproachaccuratelypredictsperformance.
KEywoRdSBehavior-Based, Formal Verification, Localization, Robot Software, Uncertainty
1. INTRodUCTIoN
Forsystemsthatneedtofunctionincriticalsituations,suchasinhealthcareapplications,searchandrescuerobotics,andautomatedcounterweaponsofmassdestructions(CWMD)missions,itiscruciallyimportantthatthesystemfunctionasspecifiedortheresultmightbelossoflife,orpropertydamageorboth.Oneapproachtothisproblemistomonitorthesysteminoperation(Leucker&Schallhart,2009)andtosignalanalerttoasupervisorofthesystemwhentheperformanceisgoing,orpredictedtogo,outsidethenecessaryperformanceenvelope.Thismonitoringapproachcanbevery
49
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
50
effectiveincaseswhereasupervisorcanstepin,orthesystemcanbedisabledwithoutconsequence,whenaperformanceproblemisobserved.However,inthecasethatthesystemisautonomous,orthesupervisorcannotinteractsufficientlyquicklyandthesystemcannotsimplybedisabled,thenanalternateapproachisnecessary.InpreviousworkfortheDefenseThreatReductionAgency(Lyonsetal.,2015)wehavedevelopedanefficientapproachtotheautomatic,a-priorideterminationofperformanceguaranteesforbehavior-basedrobotmissionsoperatinginuncertainenvironments.
Thisworkisrelatedtoformalsoftwareverification(Jhala&Majumdar,2009)(DeMoura&Bjorner,2012):adesigntooltodeterminewhetherapieceofsoftwarewillfunctionproperlywithouthavingtoexecutethesoftware.Thefieldhasprogressedstronglyinrecentyearswithdevelopmentsin model-checking (Jhala & Majumdar, 2009) and satisfiability (SMT) engines (DeMoura &Bjorner,2012).However,allsuchmethodscanatbestapproximaterealrobotperformancebecauseoftheundecidabilityoftheunderlyingverificationproblem.Designingaverificationapproachformissioncriticalrobotsoftwarerequiresunderstandingwhataspectsoftherobotsoftwareareofmostimportancetotheproblem.Behavior-basedrobotprogramming(Arkin,1998)isanimportanttoolinautonomousroboticsthatyieldsrobotprogramsthatarerobusttouncertaintyaboutexactlywhatenvironmenttherobotswillfaceduringexecution.Forthisreason,inrecentwork(Lyonsetal.,2015),weaddressedtheproblemofautomaticallyverifyingbehavior-basedrobotprogramsbyleveragingthe structure of such programs. The approach employs a unique combination of static analysistechniquesandprobabilisticreasoningtoprovideperformanceguaranteesforbehavior-basedrobotprogramsoperatinginphysicalenvironmentswithuncertainknowledgeaboutobstaclestomotion.Ratherthanaddressingcomputationalverificationproblemssuchasabsenceofdeadlockorabsenceofrun-timeerrors(Trojanek&Eder,2014)(Walter,Taubig,&Luth,2010),orverifyingsoftwaregeneratedcontrolsignalswithoutconsiderationofthephysicalplatform(Kim,Kang,&Lee,2005),ourworkfocusesonestablishingperformanceguaranteesforthemissionsoftwarewithamodelofanuncertainly-knownphysicalenvironment.
A key robotics development in recent years has been the use of probabilistic mapping andlocalization algorithms that allow a robot to operate robustly in previously unseen areas byautomaticallybuildingamapandcontinuallylocalizingtherobotwithrespecttothemap(Thrun,Burgard,&Fox,2005).Manysuchalgorithmshavebeenprogrammedandtestedandmadeavailablein software libraries (Dellaert, Fox, Burgard, & Thrun, 1999). Our prior work in verifying theperformance of behavior-based robot missions involved manual and autotranslation of missions(O’BrienM.,Arkin,Harrington,Lyons,&Jiang,2014)specifiedusingGeorgiaTech’sMissionLab(MacKenzie,Arkin,&Cameron, 1997) robotmissiondevelopment toolkit.Other approaches totheverificationofbehavior-basedsystemshavealsobeenbasedonspecificprogrammingtoolkits(Kiekbusch,Armbrust,&Berns,2015).However,since therealreadyexistsoftware librariesforprobabilisticmappingandlocalization,itwouldbeadvantageoustosimplyincludeoneoftheseinarobotmissionifitprovidedthenecessaryperformance.Inthispaper,weaddresstheproblemofincorporatingexistingsoftwareintoournovelapproachtoperformanceverificationofbehavior-basedsystems.Buildingonourworkin(LyonsD.,etal.,2016),wepresentsomegeneraltheoreticalresultsrelatingtothischallenge,andthenweaddressthespecificproblemofincorporatingprobabilisticlocalization(ROSAMCL)(Dellaert,Fox,Burgard,&Thrun,1999)(Jiang&Arkin,2015)intoaC-WMDrobotmission.
Theremainderofthepaperislaidoutasfollows.InSection2,wereviewtherelevantliterature.InSection3,wepresentasbackgroundanoverviewofourapproachtoverification.InSection4,wepresentthemainchallenge:incorporationofexistingsoftwareintotheverificationprocess,andwedevelopatheoreticalbasisforaddressingthis.Section5presentsanapplicationofthetheorytothespecificexampleofaC-WMDmissionusingtheROSAMCLlocalizationmodule.Section6presentsourconclusions.
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
51
2. LITERATURE REVIEw
Approachestotheproblemofensuringthatacriticalsystemwillperformaccordingtoana-prioriformalperformanceguaranteeincludebothrun-timemonitoring(Leucker&Schallhart,2009)andformalverification(Jhala&Majumdar,2009).Run-timemonitoringhastheadvantageoftypicallybeinglightweight,however,ithasthedisadvantagethatitcanonlybeappliedifasupervisorcanbealerted,orthesystemdisabled,intheeventofarun-timemonitorperformancealert.Certaincriticalsystems, suchashealthmonitoringsystemswith life-critical interventionevents,orautonomoussystemshandlingexplosivesdisposalwithsignificantdangeroflifeandpropertydamage,donothavethisproperty.Insuchcases,itisnecessarytoestablishthatthesystemwillalwaysabidebyspecified performance guarantees in advance. Our work addresses Counter-Weapons for MassDestruction(C-WMD)robotmissionsoftware:softwareformultiple-robotteamssearchingforandlocatingWMDs.Thisfallsintothecategoryofsoftwarerequiringa-prioriperformanceguarantees.
Approachestosoftwareverificationincludemodelchecking(Jhala&Majumdar,2009),staticanalysis(Venet,2008)andtheoremproving(Shankar,2009).Themodelcheckingapproach,whichhasprovedwidelysuccessful,involvesacomprehensiveexplorationofreachablestatesoftheprogramtobeverified.Inthecaseofrobotmissions,itisveryimportanttomodeltherobotsoftware,thepropertiesof the robot actuators and sensors, and thebehaviorof the environment inwhich therobotistooperate.Theadditionofanenvironmentmodelisrarelyseeningeneralpurposesoftwareverification,butiscrucialforanysoftwarethatinteractswithacomplexphysicalsystem.Modelcheckingapproachesaresensitivetothesizeoftheproblemstatespace.Thishasbeenrecognizedasaprobleminapplyingthesetechniquestorobotics(Wongpiromsarn&Murray,2008)becauseoftherequirementtomodelsensors,actuatorsandenvironmentbehaviorinadditiontothesoftware.
Onapproachtosidesteptheadditionalstatecomplexity(Fisher,Dennis,&Webster,2013)isbyverifyingtherobot’sbeliefratherthantheactualphysicalbehavior.However,thisbegsthequestionofhowbeliefwillcorrespondtorealityonanyspecificmission.Wehavefollowedanalternateapproach(LyonsD.,etal.,2013)(LyonsD.,Arkin,Jiang,Harrington,&Liu,2014)–avoidingmakingthestate-spaceoftheproblemexplicit.Wedevelopedastaticanalysismethodtoextractasetofflow-functionsfrombehavior-basedrobotsoftware.Thefunctionsdescribehowthevariablesoftheprogram,robot,sensorandenvironmentmodelsrecurrentlyinteractduringexecution.Theperformanceguaranteeismappedtoboundaryconstraintsonthesefunctions.TheflowfunctionswhichcharacterizetherobotmissionsoftwareareusedtoconstructaDynamicBayesianNetwork(DBN),andverificationofthesoftwareconsistsoffilteringtheDBN(asopposedtoinspectingsampleexecutionpathse.g.,(Younes&Simmons,2002)).RandomvariablesarerepresentedasMixturesofGaussians(MoG)(LyonsD.,Arkin,Liu,Jiang,&Nirmal,2013),essentiallyamultiplehypothesisrepresentationforthevariablevalue.PropagationofparametricdistributionsthroughaBayesiannetworkisveryfast;manyoftheverificationstakelessthanafewminutesdespiteextensivelogginganddebuggingcommands.
There are two challenges in automatically verifying existing probabilistic robot localizationsoftware.Thefirstisthatitrequiresanenvironmentmodel,separatefrom,andinteractingwith,therobotsoftware.Themodelhastoincludethephysicallocationoftherobot,thegeometryofthemap,andtherelationshipbetweentheseandthesensormeasurements.Uncertaintyinphysicallocation(attheleast)needstobemodeled.Thesecondisthatexistingsoftwaremustbedirectlyincludedintotheverification,ratherthanrecodedandthenincluded.
Thereislittledirectlyrelatedworkinapplyingautomatedverificationtocriticalmissionsinreal-worldenvironments.Proetzschetal.(Proetzsch,Berns,Schuele,&Schneider,2007)documentverificationofabehavior-basedarchitectureofanoutdoormobilerobotRAVON.Theyrewritethesoftwareintotheformal,synchronouslanguageQuartzandcarryoutmodel-checkingusingtheAverestmodelcheckertoensurethattherobotslowsdownandstopsbasedonitsproximitysensors.Sincetheyhavenoenvironmentmodel,whattheyverifyisthatthesoftwareissuesappropriatevelocitycommands–inourworkweverifythestateoftherobotitself.Walteretal.(Walter,Taubig,&Luth,
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
52
2010)alsoverifycollisionavoidancesoftwareforamobilerobot.Theydoincludeanenvironmentmodel(theirDomainmodel)andusetheIsabelletheoremprovingsystem.TheymanuallyannotatethecodeintheirownformallanguageanduseIsabelleasanassistantforhuman-driven,ratherthanautomated,verificationoftheprogram.
Verificationofsoftwarethatincludesprobabilisticalgorithmshasbeenaddressedbytheformalverificationcommunity(Baeir&Katoen,2008)(Katoen,2010),andprobabilisticmodel-checkingtoolssuchasPRISM(Kwiatkowska,Norman,&Parker,2004)haveexistedforsometime.However,suchtoolsrequireausertocodetheiralgorithmspecificallyforverification,whereasourobjectiveis to have automatic verification, and to include existing probabilistic localization code into theverificationprocess.Therecodingapproachhasthegeneralweaknessthatare-implementationmaynotrepresenttheactualcode(itmightbeabetterorworseimplementation).Evenpublishedalgorithmdescriptionsforwidelyknownalgorithmshavebeenshowntocontainerrors(Zaks&Joshi,2008).Recodingalsoinherentlyrequiresalargeinvestmentofexpertiseandmanpower(Kim,Kang,&Lee,2005).However,themainchallengeinincludingpreexistingcodeisthatsuchcodeisdesignedtoexecuteasingleinstancewhereasverificationreasonsaboutallexecutionsthatarepossiblegiventhea-priorienvironmentmodelinformation.Includingexistingcodeispossibleinsomemodel-checkingtools–forexampleSPIN4.0andlaterallowsCcodetobeembeddedinaPromelamodelandthecodeisexecutedasanatomictransitionwithinSPIN.Ourapproachistoconsidertheembeddedcodecantransformasamplefromarandomvariable,andwedefineaframeworkforsamplingandreconstructingvariabledistributions.
3. VERIFICATIoN oF BEHAVIoR-BASEd RoBoT MISSIoNS
Asbackgroundforourapproachtoaddressingverificationofexistinglocalizationcode,wepresentanoverviewofourapproach.ThefirstsubsectionbrieflyintroducesandmotivatesourtheoreticalapproachusingPARS(ProcessAlgebraforRobotSchemas).ThesecondsubsectionintroducestheimplementationinMissionLabandVIPARS(VerificationinPARS).
3.1. Theoretical Framework for Automatic Verification of Behavior-Based SystemsAswehavediscussed,state-spaceexplosionisakeychallengeforautomaticsoftwareverification.Thisproblemisexacerbatedinourcasebytheneedtoincludenotjusttherobotsoftware,butalsoamodeloftheenvironmentinwhichtherobotprogramwilloperate.Ontopofthis,boththerobotsoftwareandtheenvironmentmodelwillneedtosupporttherepresentationofuncertainty,asine qua noninreal-worldrobotics.Toaddresstheseseriouscomputationalcomplexityproblems,weelectedto1)notfocusonstate-basedrepresentationsbutratherprocess-basedrepresentations,and2)toleveragethestructureofbehavior-basedprogramming(Arkin,1998)–asuccessful,robustrobotprogrammingparadigm.
Abehavior-basedprogramanditsenvironmentaremodeledinPARSasasetofinterconnected,recurrentprocesses,whereaprocessPiswrittenas:
Pu u i i o o v vn j k m1 1 1 1
, , , , , , , ,… …( ) …( ) … (3.1)
whereu1,…,unaretheinitialvaluesfortheprocess variables,i1,…,ijando1,…,okareinput and output port connections,andv1,…,vmare final result valuesoftheprocessvariables.Processescomputeresultvaluesfrominitialvalues,andthiscomputationmaybeinfluencedbyanycommunicationsthatoccuroverportconnections.Byfocusingonprocessesratherthanstates,weshiftthecomputationalcomplexityfromstatecombinatoricstothegeneratorofthosestates–theprocessdescription.So,ofcourse,weneedagoodlanguageforthatdescription.
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
53
Processescanbedefinedascombinationsofotherprocessesusingthefollowingcompositionoperators:parallel(‘|’),disabling(‘#’)andsequential(‘;’).Boundedrecursioniscapturedusingtail-recursiveprocessdefinitions,e.g.:
Px Qxy Py� �;= (3.2)
HereprocessPfirstactivatesprocessQwithinputvaluex.Qdeliversoutputvaluey,whichisthenusedtorecurP.AvariableflowfunctionfPrelatesthevaluesofvariablesatthestartofeachrecursivestepofPtothoseattheend.Theflow-functionforatomicprocessesarespecifieda-priori;thoseforcompositeprocess,thosedefinedascompositionsofotherprocesses,e.g.,(3.2),arecomposedfromtheflowfunctionsofthecomponentprocesses.Thiscanbeautomatedinastandardstaticanalysis(theapproachusedbycompilerstocheckoroptimizesoftware)approachtogenerateflowfunctionsgivenasetofprocesses(Lyons,Arkin,Jiang,Liu,&Nirmal,2015)withcomplexitylinearinthenumberofprocesses.SinceanyexecutionofEquation(3.2)ismodeledbyfP
n(x0)forn≥1andinitialparametervaluex0,wehaveastraight-forwardverificationmethod.However,notallprocessesaredefinedinthisform.
Thesystemtobeverifiedisexpressedastheparallel,communicatingcompositionSysofarobotcontroller,e.g.,Ctrwithvariabler1,andanenvironmentmodel,e.g.,Envwithvariabler2,written:
Sys r r Ctr r a b Env r b a1 2 1 2, ( )( ) ( )( )= (3.3)
= ( )Sys r r Sys f r rSys
' , ; ,1 2 1 2
(3.4)
f r r f r r f r rSys Sys r Sys r1 2 1 1 2 2 1 2
, , , ,, ,( ) = ( ) ( )( ) (3.5)
InEquation(3.3),theinputofCtrisconnectedtotheoutputofEnv,(a),andtheoutputofEnvisconnectedtotheinputofCtr,(b).If(3.3)wereasequentialcompositionlike(3.2)thenwecouldextractflowfunctionsforthecombinedinteractionofcontrollerandenvironmentandconductanefficientverification.Therefore, in(Lyons,Arkin,Jiang,Liu,&Nirmal,2015)wedevelopedaninterleaving theorem1 for behavior-based systems to convert Equation (3.3) to a sequential formEquation(3.4).Theintuitionisthatabehavior-basedsystemhasbehavioral‘states’eachwithanassociatedsetofsensory-triggeredresponses.AstaticanalysisalgorithmSysgenwasdevelopedtoidentifythesetofprocessesforthesestatesandrewriteparallelcompositionsoftheform(3.3)intoasequentialcomposition(3.4)whereSys’istheautomaticallyidentifiedbehavioralstateprocess,referred to as the systemperiod.OnceSysgenanalysis is complete, a system flow functioncanbeextractedfromSys’intheusualway.InthesmallexampleofEquations(3.3),(3.4)above,thefunctionextractedisshowninEquation(3.5).Thisisarecurrentfunctionthatevaluatesthevaluesforr1andr2ascomputedbytheinteractionsbetweenCtrandEnvineachexecutionofthesystemperiodSys’.Theresultofthisisthataparallelcompositionofcontrollersoftwareandenvironmentmodel,suchasinEquation(3.3),canbeautomaticallyverifiedbycheckingfSys
n(x0)forn≥1andinitialparametervalue(s)x0.
Whilethisaddressesthestatecombinatoricsproblemmentionedinthefirstparagraphof3.1,it doesnot address the challengeof representinguncertaintymentioned in that sameparagraph.Processvariables,e.g.,r1, r2inEquation(3.3),canberandomordeterministic.Randomvariablesare represented as multivariate mixtures of Gaussians, and operations on random variables areautomatically translatedintooperationsondistributions(LyonsD.,Arkin,Liu,Jiang,&Nirmal,2013);forexample,asumofrandomvariablesistranslatedtoaconvolutionoftheirdistributions.Alltheseoperationscanbewritten(asapproximationsinsomecases)asoperationsontheparametersoftheGaussianmixtures.FlowfunctionsrelatevariablevaluesatrecursionsteptofSys’tothoseatt+1,andcanbewrittenasconditionalprobabilities,e.g.:
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
54
fSys,r1(r1,t,r2,t)=P(r1,t+1|r1,t,r2,t) (3.6)
Inthefinalphaseofverification,extractedflowfunctionsareconvertedtoconditionalprobabilitiese.g.,(3.6).TheseareusedtobuildaDynamicBayesianNetwork(Russel&Norvig,2010)tocarryoutaforwardpropagationofprobabilitydistributions–aprobabilisticversionoffSys
n(x0)–todeterminewhether the combination of controller and environment will meet a performance specification.Although (Lyons, Arkin, Jiang, Liu, & Nirmal, 2015) discusses more complicated performanceguarantees,webasically restrictour attention to theguarantee that amissionwill achieve somecriteriononenvironmentvariables(usuallyaspatialaccuracyforawaypointgoaland/oratemporalrequirementforachievingthemission)withprobabilitygreaterthanathresholdbeforeatime-limithasexpired.Wehavedemonstratedthatthisapproachisfastandaccuratewhenvalidatedagainstphysicalexecutions(mostrecently(Lyonsetal.,2015)).
3.2. Implementation of Verification in MissionLab and VIPARSThe result of our research effort is a software verification tool VIPARS, which provides theperformanceguaranteeforarobotmissionbasedonhowwellthespecifiedperformancecriteriaaresatisfiedbythecontrolprogram,robot,andtheenvironmentmodelsforthemission.
VIPARS(LyonsD.,Arkin,Jiang,Liu,&Nirmal,2015)isbuiltuponMissionLab(MacKenzie,Arkin,&Cameron,1997),abehavior-based robotmissionspecificationenvironment (Figure1).MissionLabprovidesausability-testedgraphicalprogramminginterface,wheretherobot’sprogramis specified in the formofa finite stateautomaton (FSA)assembled froma libraryofprimitivebehaviors.MissionLabmissionsareautotranslated(O’Brien,Arkin,Harrington,Lyons,&Jiang,2014) to theprocess-algebranotationPARS foranalysis.Environmentmodelsarealsoprocessesinthisnotationandwehaveproposedthatastandardizedsetofenvironmentmodelscouldbeused
Figure 1. System architecture (reproduced from (Lyons et al., 2015))
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
55
tocapturedifferentclassesofenvironment(e.g.,motionuncertainty(Lyons,Arkin,Jiang,Liu,&Nirmal,2015);obstacleuncertainty(Lyonsetal.,2015)).
ThefirstphaseofVIPARSverificationisastaticanalysisoftheconcurrent,communicatingmissionsoftwareandenvironmentmodeltogenerateasetofrecurrentflow-functionscapturinghowthetwointeract.ThesecondphaseofVIPARSusesaBayesiannetworktopredicttheperformanceofthemissionfromthesefunctions.
TheoutputofVIPARSistheperformanceguarantee,currentlyquantifiedasasetofprobabilitydistributions,thatdescribesthelikelihoodofmissionsuccess.Ratherthanasingleyes/noanswerforwhethertherobotsoftwarebeinganalyzedmeetstheperformancerequirements,VIPARScanoutputrandomvariabledistributions(e.g.,thelocationoftherobotattheendofthemission)andasetofperformancegraphs(e.g.,theprobabilitythatamissionwillbesuccessfulversustime,orthespatialaccuracyofthefinalpositionasagraphofprobabilityversusspatialerror).Thisrichoutputfuelsafeedbackloopthatallowsthemissionoperatortoimprovetheperformanceofthemissionsoftware.
4. VERIFyING EXISTING LoCALIZATIoN SoFTwARE
Therearetwokeychallengestoextendingautomaticverificationtohandletheverificationofpreviouslyexistinglocalizationsoftwareincorporatedintoarobotmission.Thefirstchallengeistoprovideanenvironmentmodelforverificationthatincludesthegeometricandpositionuncertaintyneededtoverifyanyprobabilisticlocalizationapproach.Thatwillbeaddressedfirst,insubsection4.1.Thesecondchallengeistoprovideaframeworkforincludingtheautomaticanalysisofpreviouslyexistingsoftwareintoprobabilisticverificationwithoutrequiringthatthesoftwareberewritten,annotated,ormodifiedinanyway.Thatwillbeaddressedinsubsection4.2.
4.1. Environment Model for LocalizationSection3includedanintroductiontotheVIPARSverificationmoduleandtoPARS,theprocessalgebraframeworkunderlyingit.ThesoftwareforarobotmissionistranslatedfromtheMissionLabGUIEditortoasetofPARSprocessdefinitions.Werefertothetop-levelprocessasMissionandweverifythisprocessbyanalyzingitsinteractionsthroughthesensorandactuatorsignalsitgenerateswithamodeloftheenvironmentinwhichitistoexecute.ThisenvironmentmodelisalsodefinedasasetofPARSprocesses.Theconcurrent,communicatingcombinationoftherobotcontrolsoftwareMissionandtheenvironmentmodeliscalledthesystemprocess,Sys.
Inpriorwork,wehavedefinedenvironmentmodelstocapturethemotionandsensinguncertaintyoftherobot(LyonsD.,Arkin,Jiang,Liu,&Nirmal,2015),andenvironmentobstacleswithuncertainlocationandsize(LyonsD.,etal.,2015).Toverifylocalizationsoftware,theenvironmentmodelneedstocontaininformationabouttheuncertaingeometryoftheenvironment.ThesystemprocessSysforthelocalizationmissionisshowninEquation(4.1)andillustratedasaprocessnetworkinFigure2:
Sys Mission clp clh cl cv Localization D cp co ch cl cm= ( ( )( ) ( ), , , , , ,0 cclp clh
Map sysmap cm MB Laser ms mo lo cm cp ch
,
() _ , , , ,
( )( ) (( )( )( )( ))
cl
Robot P H cv cp ch co 0 0, , ,
(4.1)
RecallfromSection3,thatthenamesinparenthesisaftertheprocessnameareportconnectionnames.Theinitialvaluesfortheprocessvariablesareshowbetweenanglebrackets.TheMissionsoftware,aswe’llseeinalatersection,programsawaypointmissionandisfundamentallysimilartoallpriorwaypointmissionswehaveverifiedandvalidated.Ithasinputsclp(position),clh(heading)andcl(laserreadings);andoutputcv(velocity).Robotistheenvironmentmodel,capturingmotionandodometryerroraswellastherobotinteractionswithobstacles.Itisalsofundamentallysimilar
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
56
tothemodelsusedinourpriorwork.PO, HOaretheinitialrobotpositionandheading,andtheprocessportsareinputscv(velocity)andoutputscp, ch(odometrypositionandheading)andco(realpositiondistribution,i.e.,withoutsensingnoise–usedforperformanceguaranteeevaluationonly).
Inthebehavior-basedlocalizationapproach(Jiang&Arkin,2015),theobstacleavoidancesensor(MB_Laser)getsitsinformationfromthemap,ratherthandirectlyfrommeasuringsensoryinput.Mapmakesmapinformationavailableonitsoutputcm;MB_Laserusesthemaptogeneratemap-basedlaserdataonitsoutputcl.Localizationimplementsalocalizationmethodusingthemapcm,lasercl,androbotcp, co, chinputs,whereD0istheinitialpositionuncertainty.TheoutputofLocalization,clp,isthelocalizedposition(andheadingclh)usedbytheMissionprocess.HowtheLocalizationprocessencapsulatesanexistinglocalizationalgorithmwillbeaddressedinsubsection4.2.
Akeydifferencebetweenthislocalizationmissionandpriormissionstowhichwehaveappliedourverificationapproach(LyonsD.,Arkin,Jiang,Liu,&Nirmal,2015)(LyonsD.,etal.,2015)(O’BrienM.,Arkin,Harrington,Lyons,&Jiang,2014)isthemapandtheroleitplaysinboththeobstacle avoidance behavior and in localization. The Map process in Equation (4.1) contains amap(sysmap)whichrepresentsthegeometryofthephysicalenvironmentinwhichthemissionisexecuted.Ourassumptionisthatthismaphasbeengenerateda-prioriforanenvironmentofinterest.Ourconcernhereishowthismapcanberepresentedforthepurposesofprobabilisticverification.
RandomvariablesarerepresentedinVIPARSasMixturesofGaussiansdistributions(MG).Ifa ~ MG(CM),forCM={(μi, Σi, wi) | i ∈ 1…m}thesetofthemixtureparameters(means,variances,weights),thenaireferstomixturememberN(μi, Σi,),andw(ai)=wiarethemixtureweights,where
i
m
iw
=∑ =
1
1, � and MG(x; CM)=i
m
i i iw N x
=∑ ( )
1
; ,µ Σ . The mixture size is written | a | = m. Map
information–thelocationsandgeometryofobstacles,wallsandotherphysicalaspectsofthemissionenvironment–canbedirectlyrepresentedusingthismixturemodelasweexplainbelow.Theadvantageofthisapproachtorepresentingphysicalgeometryisthatthereisnorestrictiononthespatiallocationorextentofobstacles,andfinerprecisionofmodelingcanbeobtainedatthecostofaddingmoremixturemembers(Figure3).
Figure 2. The system process Sys for the localization mission
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
57
Definition:AnindexedmixtureofGaussiansisamixtureofGaussiansdistributiona ~ MG(CM)togetherwithanindexsetI.Themixtureisrestrictedasfollows:◦ a[x]≡aiwhereμ(ai)=x∈I,i∈1…m;◦ μ(ai)∈I,foralli∈1…m;aonly contains members indexed byI;◦ For anyx∈I,|{a[x]}|≤1;ahas at most one member for each index.
Wedefinew[x]andΣ[x]similarlytoa[x]tolabelmemberweightsandcovariances.Amapisdefinedasan indexedbivariatemixtureofGaussianswhere I=[0…X]×[0..Y]andwhereeachmemberisaGaussiankernelwithcovarianceΣ[x,y]=σm
2I,whereσmreflectsthemapresolution.Thiscorrespondssomewhatintuitivelywithanoccupancygridrepresentation,wherew[x,y]isrelatedtoprobabilityofoccupancyforthelocation(x,y).
Duringverification,thelocationrandomvariable(theconnectioncpinEquation(4.1))representsthelocationoftherobotforallpossibleexecutions.It’srelevanttocomparethiswiththerepresentationofrobotlocationinalocalizationalgorithm:therepresentationtheremaybealsobearandomvariable,buttheinterpretationisdifferent.Inanysingleexecution,therobotcanonlybeatasinglephysicallocation;thelocalizationdistributionisanestimateofthis.Inverification,theobjectiveisnottofindthesinglemostlikelylocation,buttopropagatetheeffectsofbeingatalllocations.Ratherthanusingaraytracealgorithmtodeterminehoweachlocationissupportedbysensorreadingsandrefiningthepositionestimatebasedonthat,theraytracealgorithmisusedbytheMB_Laserprocesstogatherallpossiblesensorreadingsthatcanariseduetotherobotlocationdistribution.
Webeginbydefiningarayscanonamaprepresentedasanindexedmixture:
Figure 3. Example of VIPARS Gaussian mixture model map representation
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
58
Definition:Scanline((xs,ys), θ) ⊆ R2 = { (x,y): x=xs + t cos θ, y = ys + t sin θ, t ∈ R+ }
ThesetofmapmixturemembersencounteredbyScanlineiseasilycalculatedasScanline ∩ IandforconveniencewealsodefinethemembersencounteredbyScanlineontheindexedmixtureMapas:
Definition: Scanmix((xs,ys), θ) ⊆ Map ={ (a[x,y], w[x,y]/Σw[x,y]): a[x,y]∈Map ∧ (x,y) ∈Scanline((xs,ys),θ)∩Y}
Thereturnfromasensorwillthenbeanindexedunivariatemixturecalculatedasfollows.Letbifori=1…mbethemembersofScanmix,thenwecalculateeachmembersiofthesensorreturnSensor((xs,ys), θ)fromthemembersofScanmix.EachmembermeanisthedistancefromthesensororigintothemapindexandeachmembervarianceisthemapaccuracyvarianceconvolvedwithasensorspecificnoisedistributionSN:
� , * ���� ��� , ,���s s N S where x y bi i i i N i s s i i= = ( ) = ( ) ( )µ µ σ µ µ σ == σ
m (4.2)
However,calculatingthemixtureweightsisalittlemoreinvolvedduetothepotentialocclusionofmixtures (representingobjects)byothermixtures (representingobjects) that arecloser to thesensor.TheweightsofthemembersofScanmixneedtobemodifiedforthecorrespondingmembersofSensorinorderofvisibility–thatis,inorderofμiasfollows,wherewareweightsinScanmixandw’thenew,correspondingweightsinSensor:
� [ ]�
′ =w w0 0
µ
′ = −
=∑w w w
i ij
i
jµ � ’1
0
(4.3)
Sensor((xs,ys), θ)definesthesensorreturntohaveanuncertaintyrelatedtotheaccuracyofthemap.However,itdoesnotincludeanyuncertaintyassociatedwiththerobotposition.Toincorporatetheuncertaintyinrobotpositionintothesensorreturn,weextendthedefinitionasfollows:
Definition:Sensor(N((xs,ys),Σ), θ) = {(si, w’i): i=1…m }asinEquation(4.2)butwhereΣisadiagonal
covarianceΣ =
σ
σx
y
2
2
0
0andσ σ σ θ σ θ
i m x y= + +( )� cos cos � .
However,thisdefinitionassumesadiagonalvarianceΣfortherobotmotionuncertainty.Thisisunrealisticandweneedtorelaxtheassumptionifwearetohandlerealisticrobotmotion.AnycovariancematrixcanbeinterpretedgeometricallyastheproductofrotationRθandscalingSmatrices:
Σ =
= ( )( ) =
−σ σ σ ρ
σ σ ρ σ
φφ φ θ
x x y
x y y
TR S R S where R
2
2 ��cos sinφφφ φsin cos
�
=
andSs
sx
y
0
0 (4.4)
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
59
Usingthisrelationship,theangle(ϕ)anddiagonalvariances(sx2,sy
2)ofanon-diagonalcovariancematrixcanbeidentifiedandthescananglerotatedaccordingly.
Definition: Sensor N xs ys x
y
, , ,( )
σ
σ
2
2
0
0
= ( )
� , , �Sensor N xs yss
sx
y
2
2
0
0 −
,�� φ
Havingensuredthatenvironments,mapsandsensorreadingscanallbereasonedaboutusingtherandomvariableframeworkinPARS–necessaryforbothobstacleavoidanceandlocalization–wenextturnourattentiontolocalizationitself.
4.2. Including Existing Software in Probabilistic VerificationThemaindifficultywiththeincorporationofexistinglocalizationcodedirectlyintotheVIPARSverificationalgorithmisthatlocalizationcodeisdesignedtoexecutejustasingleinstanceofarobotmission,whereasVIPARSisprobabilisticallyreasoningaboutallexecutionsthatarepossiblegiventhea-priorienvironmentmodelinformation.ConsiderthattheC++programwewanttoaddtoamissionisP.APARSprocesswrapperforPisbuilt,sothecodebehaveslikea‘blackbox’processP⟨x⟩⟨y⟩.However,whenPiscalled,itwillmaponeinputvaluextoanoutput,y;onlyonepossibleexecutionofP,whereasverificationhastocheckallpossibleexecutions.
Ourapproachtothischallengeconsidersbuildingaprocesswrapperfortheembeddedcodethatiscapableoftransformingasample(whichrepresentsasingleinstanceofamission)fromaPARSrandomvariable,andforthispurposewedefineaframeworkforsamplingandreconstructingvariabledistributions.Thisapproachhastheadvantageofusingtheactualcodethatwillgetexecutedbytherobotatrun-timeforthemission.Ithasthedisadvantageofpotentiallylengtheningverificationtimes,sincemultiplesamplesneedtobeevaluatedforarepresentativeresult.
Wedefine an extension to the flow function fP for theprocess/programP (representing theactualembeddedcode):themixtureextendedflowfunctionFPtakesarandomvariablexasinputandproducesarandomvariableyasoutput.ItsamplestheinputdistributionxandcallsfPonthesamples,andreconstructstheoutputdistributionmixturep(y | x)= FP(x)fromtheresult.Letx,y~MG(CM);ourobjectiveistoevaluatey=Π(x)(=fPabove)whereΠisthetransformationcarriedoutbytheembeddedcodeonthevariablex.Wewillpresentseveralwaysthiscanbedone.
Definition:TheMGmean-extendedrealfunction(MER)Πisdefinedasfollows:◦ IfΠisdefinedasΠ:ℜ→ℜ,wherey=Π(x),forx, y∈ℜ,then;◦ TheMGmeanextendedrealfunctionΠisdefinedasΠ:MG→MG;◦ Wherey=Π(x),forx,y∈MG(andMGisthesetofallMGs);and◦ Wherewedefiney=x;◦ Exceptμ(yi)=Π(μ(xi))forallxiinx.
So,theMERfunctionpreservesnumberofmembersandvariances.Onlythemeanistransformed,andanormaldistributionistransformedtoanothernormaldistributionwiththesamevariance.ThislimitsourmodellingofwhatthesoftwareinΠcando:
• Itcannotforexampleaddorreduceuncertaintysincevarianceisunchanged.Theentiredistributionismovedtocenteroveranewmean;
• Itcannotaddnewmemberstothemixturesincethenumberofmembersisunchanged.
Thisisclearlyverylimiting,thoughitmightbesufficientforafastestimateofperformance.ConsiderthatifwecallΠwithasinglenumberx,itreturnsanormaldistributionwiththeresultfor
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
60
xasthemeanandavariance:y=N(μ,σ)=Π(x).Now,iftheinputtocodeisaMGx,thenwecoulddefineaMGextensionofΠas:
Definition:TheMGmean extended normal function (MEN)Πisdefinedasfollows:◦ IfΠisdefinedasΠ:ℜ→ℜ×ℜ,where(μ,σ)=Π(x),forx,μ,σ∈ℜ,then;◦ TheMGnormalextendedfunctionΠisdefinedasΠ:MG→MG;◦ Wherey=Π(x),forx,y∈MG;and◦ Wherewedefiney=x;◦ Except(μ(yi),σ(yi))=Π(μ(xi))forallxiinx.
Notice however that the MEN function effectively ignores σ(xi); only the mean is used. Inprobabilisticterms,thisislikeonlyoperatingonthemostlikelyvalue.However,sinceavarianceisreturned,itmeansthecodedoesincludeanincreaseordecreaseintheuncertaintyassociatedwiththetransformedvariable.
Definition:ThemeanandvarianceofanMGisdefinedsimilarlytothemeanandvarianceofamixtureofGaussians.Forarandomvariablex~MG:
◦ μ(x)==∑i
m
i iw
1
�µ , thatis,theweightedsumofmeans;
◦ σ(x)= −( ) +
=∑i
m
i i iw
1
2 2� µ
Let’sconsidertheinputxitoactuallybeacollectionofsamples,wheretheweightofsamplesjisgivenbyN(sj;μI,σi)–thatis,memberievaluatedatsj.Thisdiscretizesandsamplesthea-prioridistribution.WecanevaluatetheMENfunctionforeachsample,andthenusetheresultstoreconstructthea-posteriori(posterior)distributionasfollow:
Definition:TheMGextended normal function(ENF)Πisdefinedasfollows:◦ IfΠisdefinedasΠ:ℜ→ℜ×ℜ,where(μ,σ)=Π(x),forx,μ,σ∈ℜ,then;◦ TheMGextendednormalfunctionΠisdefinedasΠ:MG→MG;◦ Wherey=Π(x),forx,y∈MG;and◦ Wherewedefiney=x;◦ Except(μ(yi),σ)=Π(μ(xi))forallxiinx;and◦ Whereσ(yi)is calculated as follows:
▪ (μ’j,σ’j)=Π(si)forsiasampleoftheinputxi;
▪ σ(yi)= ( ) ( )( ) −( )( ) +
=∑j
k
j i i j i jN s
1
22�; , ’x x � y ’µ
SoratherthanevaluatingtheprogramΠonasinglenumber,itisevaluatedonasetofnumbersofpredefinedsize(k)andtheresultscombinedtoformasinglenormaldistribution.
It’salsopossibletotakeasampleapproachtoarealfunctionΠ,andcalculateavarianceontheresultsample.
Definition:TheMG extended real function(ERF)Πisdefinedasfollows:◦ IfΠisdefinedasΠ:ℜ→ℜ,wherey=Π(x),forx,y∈ℜ,then;◦ TheMGmeanextendedrealfunctionΠisdefinedasΠ:MG→MG;◦ Wherey=Π(x),forx,y∈MG);and
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
61
◦ Wherewedefiney=x;◦ Exceptμ(yi)=Π(μ(xi))forallxiinx;and◦ Whereσ(yi)is calculated as follows:
▪ μ’j=Π(si)forsiasampleoftheinputxi;
▪ σ(yi)= ( ) ( )( ) −( )( )
=∑j
k
j i i j iN s
1
2; , 'x x yµ
ThissetofdefinitionsforMEN,MER,ENF,andERF,allowsforarangeofoptionsininterfacingembeddedcodewithverificationingeneral,andwithVIPARSinparticular.
5. VERIFICATIoN ANd VALIdATIoN oF A RoBoT MISSIoN wITH LoCALIZATIoN
To assess the effectiveness our approach in providing performance guarantees for probabilisticlocalizationmissions,wepresenttwowaypointmissionswheretherobotistaskedtonavigatethroughaseriesofwaypoints,avoidingobstaclesandtowardagoalwithbehaviorsthatarebasedonprobabilisticlocalization.Thegeneralassessmentprocessconsistsofthreesteps:1)verification–useVIPARStogenerateaperformanceguaranteeforthemissionwithrespecttosomespecifiedperformancecriteria,2)validation–conductexperimentaltrialsofthemissionwitharealrobot,3)evaluation–comparethepredictedperformancegeneratedbyVIPARSwiththeactualperformanceoftherobot.
ThewaypointmissionsareillustratedinFigure4.Themissionproceedswithrobotstartingat(2,2)andnavigatesbyfollowingaseriesofwaypointtothegoallocationsat(11.7,12.5)and(1.0,7.3)respectivelyforeachmission.ThebehavioroftherobotforMission-B(Figure4a)isshowninFigure5,whichwascreatedinMissionLabintheformofanFSA.TheFSAconsistsofaseriesofGoToGuardedandSpinbehaviors,whosetransitionsarepromptedbyAtGoalandHasTurnedtriggers.TheFSAforMission-AissimilartotheoneshowninFigure5butisomittedforbrevity.
Incontrasttothebehaviorswehadexaminedinourpriorwork,thebehaviorsherehaveleveragedaprobabilisticlocalizationalgorithmtoimprovemissionperformance.Specifically,theperceptualschemasofMoveToGuardedandAvoidObstacles,twooftheconstituentprimitivebehaviorsofthehigh-levelGoToGuardedbehavior,areaugmentedwithaSLAM-basedspatialmap(Jiang&Arkin,
Figure 4. Waypoint missions for verification and validation
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
62
2015).TheMoveToGuardedprimitivebehaviordrivestherobottoaspecifiedlocationwitharadiusofvelocitydropoffaroundthegoal.Insteadofusingodometryforlocalization,theperceptualschemaof MoveToGuarded is replaced with the adaptive Monte Carlo localization (AMCL) algorithm(Dellaert,Fox,Burgard,&Thrun,1999).Thisprobabilisticlocalizationalgorithmtakestherobotodometryandana-prioriacquiredmapasinputs,andoutputsanestimatedposeoftherobotalongwith a covariance matrix representing the uncertainty of the estimated pose. Furthermore, theAvoidObstacles behavior uses the spatial map, instead of using direct sensory reading from thelaserscanner,togeneraterepulsionvectors.TheperceptualschemaoftheAvoidObstaclesbehaviorismodifiedtoturnthespatialmapintopseudolaserscansoftheenvironmentthroughraytracingwithintheoccupancymap.Asaresult,theGoToGuardedbehaviorutilizesperceptualinformation(i.e.,robotposeandobstacles)generatedbyprobabilisticalgorithmstoproducemotioncommandswhilenavigatingthroughthewaypoints.
Performancecriteriaaremissionspecificationsthatmustbemet.FormissionsAandB,theseincludeconstraintson:
• Rmax:Maximumradiusofspatialdeviationallowedfromthegoal;• Tmax:Maximumallowablemissioncompletiontime.
Inthisexample,eachwaypointmissionisconsideredsuccessfulonlywhentwoconstraintsaremet:
Success=(r≤Rmax)and(t≤ Tmax) (5.1)
where r istherobot’srelativedistancetoitsgoallocationandtisthetimefortherobottofinishamission.TheobjectiveofVIPARSisthentoverifyhowwelltheseperformancecriteriaaresatisfiedbythecombinationoftherobot,itscontrolsoftware,andtheoperatingenvironment.
Figure 5. Behavioral FSA for Mission-B
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
63
5.1. VerificationThelocalizationalgorithmusedinthispaperwasAdaptiveMonteCarloSampling(AMCL)(Fox,2001)asimplementedinROS.Inthesamplingapproach,theDBNfilteringengineofVIPARSissuedrequeststoaROS-basedAMCLservertoevaluatetheERFfunctionfortheLocalizationprocess.TheinteractionisshowninFigure6:WhenevertheflowfunctionfortheLocalizationprocessneededtobeevaluatedonapositionrandomvariable,thepositionvariablewassentfromtheDBNfilteringengine(Top,Figure6)viaapipetoaconcurrentlyrunningROSIndigosystem(Bottom,Figure6).TheROSSTDRsimulatornodewasinstructedtomovetherobottotheappropriateposition,andlocalizationdatacollectedfromtheAMCLnode.Forsimplicity,theERFfunctionwasrestrictedtosinglemembermixtures,andratherthancalculatingthevariancebyevaluatingmultiplesamples,onlythemeanvaluewastransformedandthevariancecalculatedbyconvolvingthemeanwithazero-meandistributionN(0, σs).ThissimplifiedthehysteresisissuewithcallingAMCL.ThehysteresischallengeinfullyimplementingtheERFDefinitionforAMCLisdiscussedintheConclusion.
Theresultsofcarryingoutverificationonbothwaypointmissionswasasetofperformancegraphs(asdescribedSection3)showingthepredictedperformanceofthemissionswithrespecttotheperformancecriteria(5.1).
5.2. ValidationValidationexperimentsofthewaypointmissionswereconductedtoillustratethatVIPARS’predictedperformanceofthemissionisconsistentwiththerobot’sactualperformance.TherobotusedfortheexperimentaltrialsisthePioneer3-AT,afour-wheeledskid-steeredmobilerobot.Therobot
Figure 6. VIPARS-ROS architecture
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
64
isalsoequippedwithaforward-facingSICKlaserscanner.Thecompletevalidationexperimentconsistsof50trialrunsforeachwaypointmissionrespectively,whichresultedinatotalof100trialruns.SnapshotsofthewaypointmissionBareshowninFigure7.Missionsuccessisdefinedbyhowwelltheperformancecriteriain(5.1)aremet.Foreachtrial,thefollowingperformancevariablesweremeasured:
• t:Missioncompletiontime;• r:Robot’srelativedistancetoitsgoallocation.
5.3. Verification vs. Validation (V&V)Verificationandvalidationareconductedindependentlybyourtworesearchgroups,andtheresultsarenotshareduntilthefinalcomparisonstage.Figure8showstheresultsofverificationandvalidationof the waypoint missions. The performance guarantee is quantified as a probability distributionthatrepresentstherobotmission’slikelihoodforsuccess.Theseresultsalsoserveasthebasisforperformancefeedback;andhowthisinformationshouldultimatelybepresentedtothemissionoperatorwasinvestigatedinourrecenthuman-subjectsstudy(O’Brien&Arkin,2016).
Figure8comparesthevalidationandverificationresultsoftheperformanceguaranteesforthetwowaypointmissions.Figures8aand8cshowtheV&VresultsforthespatialcriterionP(r≤Rmax)of(5.1),theprobabilitythattherobotarriveswithinRmaxradiusofitsgoallocation.Figures8band8dshowthecomparisonsforthetimecriterionP t T≤( )max
,theprobabilitythatthewaypointmissionis completed under the time limit, Tmax. The results illustrate that the VIPARS verification ofperformanceguaranteesareconsistentwiththeoutcomesfromexperimentalvalidation.
TheV&Vresultscanbedividedintothreeregionsforfurtherinterpretation:HighConfidence(Unsuccessful), Uncertain, and High Confidence (Successful) regions. The High Confidence(Unsuccessful)istheregionofnearzeroverificationerrorandthemissionhasazeroprobabilityofsuccess.TheUncertainregionistheregionwhereverificationerrorissignificantlygreaterthanzero
Figure 7. Snapshots of validation for Mission-B
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
65
andtheprobabilityofmissionsuccessisbetween0and1.0.Asaresult,therobotisnotguaranteedtosucceedwiththemission.TheHighConfidence(Successful)istheregionofnearzeroverificationerrorandthemissionisguaranteedtosucceedwithprobabilityof1.0.Consequently,themissionoperator’sdecisionforrobotdeploymentcanbebasedonwhichregionthemissioncriteriafallinto.Forinstance,ifthespecifiedperformancecriterionfallswithintheUnsuccessfulregion(e.g.,Rmax=0.5m),theoperatorcaneitherabortthemissionormodifymissionparametersordesign.
Theoverallmissionsuccess(Equation5.1)isdefinedintermsofbothspatialandtimecriteria.Thus,weexaminedfurtherinFigures9and10theeffectsofvariouscombinationsofspatialandtimecriteria(RmaxandTmax)onthemissionsuccessandverificationerror.TheresultscanalsobeusedtoanswerqueriesregardingtheperformanceguaranteeforaspecificcombinationofTmaxandRmax.Figure9showstheeffectsofthetimecriterionTmaxontheV&VresultsofthespatialcriterionP(r≤Rmax)forMission A.WhiletheTmax’sinbothofitshighconfidenceregions(Figure8b)havenoeffectontheverificationerrorforP(r≤Rmax),Tmax’sthatareintheUncertainregion(e.g.,Tmax=415sec)incursignificantverificationerrors.Forinstance,forTmax=415sec,VIPARSpredictedasuccessprobabilityof0.18,whiletherobotwasactuallysuccessful76%ofthetimeinexperimentaltrials.
Figure 8. Results of VIPARS verification and experimental validation of spatial and time performance criteria for waypoint missions A and B
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
66
Figure 9. V&V of spatial criterion at various Tmax for Mission A
Figure 10. V&V of time criterion at various Rmax for Mission A
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
67
Figure10showstheeffectsofthespatialcriterionRmaxontheV&VresultsofthetimecriterionP(t≤Tmax).WhilesimilarobservationscanbemadehereasinFigure9,inthiscase,Rmax’shavemuchlessimpactontheverificationerrorofP(t≤Tmax)duetoVIPARS’saccuracyinpredictingthespatialperformanceofmissionevenintheuncertainregion(asshowninFigure8a).Nonetheless,missionswithperformancecriteriaintheUncertainregionsshouldgenerallybeavoided.
6. CoNCLUSIoN
Wehavepresentedanapproachtotheproblemofestablishinga-prioriperformanceguaranteesforrobotmissionsoftwareforC-WMDrobotmissionsinwhichfailurecouldmeansignificantlossoflifeorseverepropertydamage.Inparticular,wehaveaddressedtheissueofautomaticverificationofrobotmissionsoftwarethatincludessomepreexistingsoftwarelibrariesthathavetobeincludedinthemissionwithoutbeingrewrittenorannotated.
Twouniquetheoreticalresultswerepresented.Thefirsttheoreticalresultswasthedevelopmentofanenvironmentalmodelrepresentationforprobabilisticmaps,anindexedmixtureofGaussians,sufficient to support the reasoningnecessary for automatic verificationofmissions that includeprobabilisticlocalizationalgorithms.Thesecondtheoreticalresultpresentedacollectionoftechniquesfor ‘wrapping’ existing software in a random variable envelope so that it could be included inprobabilistic verification. A series of four wrapper functions – the mean-extended real functionMER,themean-extendednormalfunctionMEN,theextendednormalfunctionENF,andextendedrealfunctionERF–weredeveloped,eachwithdifferentconstraintsandrepresentationalpower.
Localization and mapping techniques intuitively offer advantages for robots navigating inunknownenvironments.ThispaperhasappliedMissionLab/VIPARSmissiondesignandverificationapproachtotheverificationofautonomousbehavior-basedrobotmissionsthatusetheROSAMCLlocalizationmodule.VerificationresultusingtheindexmixtureofGaussianmaprepresentationandERFwrapperfunctionforROSAMCLwerepresentedfortwodifferentlocalizationmissions.TheERFwrapperwaslimitedtoasinglememberandafixedvariancewasused.Tocompletelyimplementthemixtureextendedfunctionforthesampling-basedapproachinthispaper,thefullmotionhistoryforeachsamplerequestwouldneedtobesenttotheSTDRnodeandAMCLresetbetweensamples.Theabilitytocachethesemultiplesensoryhistorieswouldimprovecomputationtime,butatthecostofdirectlyinstrumentingAMCL–astepwewereavoidingforreasonsdiscussed.
Experimentalvalidationwasalsocarriedoutforthesetwomissions,andtheverificationandvalidationresultscomparedtodemonstratetheeffectivenessoftheapproach.Itisobservedhowever,thatVIPARSperformsmuchbetteronthespatialperformancecriterionthanonthetimecriterion.WhilethespatialrandomvariabletransformationsarecalculatedusingflowfunctionsinaBayesiannetwork,timeisrepresentedonlyasdiscreteiterationsoftheBayesiannetwork.Akeyavenueoffutureinvestigationistorepresenttimeasarandomvariableandhaveitalsocalculatedusingflow-functions,separatingitfromiterationsooftheBayesiannetwork.
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
68
REFERENCES
Arkin,R.C.(1998).Behavior-Based Robots.Cambridge,MA:MITPress.
Baeir,C.,&Katoen,J.-P.(2008).Introduction to Model Checking.Cambridge,MA:MITPress.
Bailey,T.,&Durrant-Whyte,H.(June,September2006).SimultaneousLocalizationandMapping(DLAM):PartyI,II.IEEE Robotics and Automation Magazine.
Blackman,S.(2004).MultipleHypothesisTrackingforMultipleTargetTracking.IEEE A&E Systems Magazine, 19(1).
Brahman,J.(2009).Verification and Analysis of Goal-Based Hybrid Control Systems[Thesis].P.D.CaliforniaInstituteofTechnology,Pasadena,CA.
Cowley,A.,&Taylor,C.(2011).TowardsLanguage-BasedVerificationofRobotBehaviors.Proceedings of the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
Dellaert,F.,Fox,D.,Burgard,W.,&Thrun,S.(1999).MonteCarlolocalizationformobilerobots.Proceedings of theIEEE Int. Conf. on Rob. & Aut.,Detroit.
DeMoura, L., & Bjorner, N. (2012). Satisfiability Modulo Theories: Introduction and applications.Communications of the ACM,54(9),54–67.
Fisher,M.,Dennis,L.,&Webster,M.(2013).VerifyingAutonomousSystems.Communications of the ACM,56(9),84–93.doi:10.1145/2500468.2494558
Fox,D.(2001).KLD–Sampling: Adaptive Particle Filters. Neural Information Processing Systems 14.Vancouver,Canada:NIPS.
Huang,J.,Erdogan,C.,Zhang,Y.,Moore,B.,Luo,Q.,Sundaresan,A.,&Rosu,G.(2014).ROSRV:RuntimeVerificationforRobots.Proceedings of the14th International Conference on Runtime Verification,Toronto.
Jhala, R., & Majumdar, R. (2009). Software Model Checking. ACM Computing Surveys, 41(4), 1–54.doi:10.1145/1592434.1592438
Jiang,S.,&Arkin,R.(2015).SLAM-BasedSpatialMemoryforBehavior-BasedRobots.Proceedings of the11th IFAC Symposium on Robot Control (SYROCO),Salvador,Brazil.
Katoen,J.-P.(2010).AdvancesinProbabilisticModelChecking.Proceedings of the11th International Conference VMCAI ‘10,MadridSpain.
Kiekbusch,L.,Armbrust,C.,&Berns,K.(2015).FormalVerificationofBehaviorNetworksincludingSensorFailures.Robotics and Autonomous Systems,74,331–339.doi:10.1016/j.robot.2015.08.002
Kim,M.,Kang,K.-C.,&Lee,H.(2005).FormalVerificationofRobotMovements-aCaseStudyonHomeServiceRobotSHR100.Proceedings of theIEEE Int. Conf. Robotics and Automation.
Klavins,E.(2004).ALanguageforModelingandProgrammingCooperativeControlSystems.Proceedings of the International Conference on Robotics and Automation,NewOrleans,LA.doi:10.1109/ROBOT.2004.1308780
Kress-Gazit,H.,Wongpiromsarn,T.,&Topcu,U.(2011).Correct,ReactiveRobotControlfromAbstractionandTemporalLogicSpecifications.IEEE Rob. & Aut. Mag., 18(3).
Kwiatkowska,M.,Norman,G.,&Parker,D.(2004).ProbabilisticsymbolicmodelcheckingwithPRISM:Ahybridapproach.International Journal of Software Tools and Technology Transfer,6(2),128–142.doi:10.1007/s10009-004-0140-2
Leucker,M.,&Schallhart,C.(2009).Abriefaccountofruntimeverification.Journal of Logic and Algebraic Programming,78(5),293–303.doi:10.1016/j.jlap.2008.08.004
Livingston,S.,Murray,R.,&Burdick,J.(2012).Backtrackingtemporallogicsynthesisforuncertainenvironments.Proceedings of theInternational Conference on Robotics and Automation.doi:10.1109/ICRA.2012.6225208
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
69
Lyons,D.,Arkin,R.,Jiang,S.,Harrington,D.,&Liu,T.(2014).VerifyingandValidatingMultirobotMissions.Proceedings of theIEEE/RSJ Int. Conf. on Robots and Systems,Chicago.
Lyons,D.,Arkin,R.,Jiang,S.,Harrington,D.,Tang,F.,&Tang,P.(2015).ProbabilisticVerificationofMulti-RobotMissionsinUncertainEnvironments.Proceedings of theIEEE Int. Conf. on Tools with AI.,VietrosulMare,Italy.doi:10.1109/ICTAI.2015.22
Lyons,D.,Arkin,R.,Jiang,S.,Harrington,D.,Tang,F.,&Tang,P.(2016).FormalPerformanceGuaranteesforBehavior-basedLocalizationMIssions.Proceedings of theIEEE Int. Conf. on Tools with AI,SanJoseCA.doi:10.1109/ICTAI.2016.0025
Lyons,D.,Arkin,R.,Jiang,S.,Liu,T.-L.,&Nirmal,P.(2015).PerformanceVerificationforBehavior-basedRobotMissions.IEEE Transactions on Robotics,31(3),619–636.doi:10.1109/TRO.2015.2418592
Lyons,D.,Arkin,R.,Liu,T.-L.,Jiang,S.,&Nirmal,P.(2013).VerifyingPerformanceforAutonomousRobotMissionswithUncertainty.Proceedings of the IFAC Intelligent Vehicle Symposium,GoldCoast,Australia.doi:10.3182/20130626-3-AU-2035.00034
Lyons,D.,Arkin,R.,Nirmal,P.,Jiang,S.,Liu,T.-L.,&Deeb,J.(2013).GettingitRighttheFirsttime:RobotMissionGuaranteesinthePresenceofUncertainty.Proceedings of theIEEE/RSJ Int. Conf. on Intelligent Robots and Systems,Tokyo,Japan.doi:10.1109/IROS.2013.6697122
MacKenzie,D.,Arkin,R.,&Cameron,R.(1997).MultiagentMissionSpecificationandExecution.Autonomous Robots,4(1),29–52.doi:10.1023/A:1008807102993
O’Brien,M.,&Arkin,R.(2016).AnAnalysisofDisplaysforProbabilisticRoboticMissionVerificationResults.Proceedings of the7th International Conference on Applied Human Factors and Ergonomics,LasVegasNV.
O’Brien,M.,Arkin,R.,Harrington,D.,Lyons,D.,&Jiang,S.(2014).AutomaticVerificationofAutonomousRobotMissions.Proceedings of the4th Int. Conf. on Simulation, Modelling and Prog. for Aut. Robots,Bergamo,Italy.
Piterman,N.,Pneuli,A.,&Sa’ar,Y.(2006).SynthesisofReactive(1)Designs.Proceedings of the7th International Conference VMCAI ‘06,CharlestownSC.
Proetzsch,M.,Berns,K.,Schuele,T.,&Schneider,K.(2007).FormalVerificationOfSafetyBehavioursOfTheOutdoorRobotRavon.Proceedings of the4th Int. Conf. on Informatics, Automation and Control,Dortmund,Germany.
Ropertz,T.,&Berns,R.(2014).Verificationofbehavior-basednetworks-usingsatisfiabilitymodulotheories.Proceedings of the41st International Symposium on RoboticsISR/Robotik ‘14.
Russel,S.,&Norvig,P.(2010).Artificial Intelligence.Prentice-Hall.
Shankar, N. (2009). Automated Deduction for Verification. ACM Computing Surveys, 41(4), 1–56.doi:10.1145/1592434.1592437
Thrun,S.,Burgard,W.,&Fox,D.(2005).Probabilistic Robotics.Cambridge,MA:MITPress.
Trojanek,P.,&Eder,K.(2014).Verificationandtestingofmobilerobotnavigationalgorithms.Proceedings of theIEEE/RSJ Int. Conf on Intelligent Robots and Systems (IROS),Chicago.
Venet,A.(2008).Apracticalapproachtoformalsoftwareverificationbystaticanalysis.ACM SIGAda Letters,27(1),92–95.
Walter, D., Taubig, H., & Luth, C. (2010). Experiences in Applying Formal Verification in Robotics.Proceedings of the29th International Conference on Computer Safety, Reliability and Security,ViennaAustria.doi:10.1007/978-3-642-15651-9_26
Watkins,O.,&Lygeros,J.(2003).Stochasticreachabilityfordiscretetimesystems:anapplicationtoaircraftcollision avoidance. Proceedings of the IEEE Conf. on Decision and Control, Maui, Hawaii. doi:10.1109/CDC.2003.1272482
International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017
70
Damian M. Lyons is a Professor of Computer Science at Fordham University, and Director of Fordham’s Robotics & Computer Vision Laboratory. He has degrees in Math, Engineering, and Computer Science from Trinity College, University of Dublin, Ireland, and a doctorate in Computer Science from the University of Massachusetts. His research interests include formal approaches to plan and program analysis and robot team exploration strategies. He was a senior researcher and department head at Philips Corporate Research, NY. He has served as Chair of the IEEE RAS TC on Assembly and Task Planning and is an IEEE Senior Member.
Ronald Arkin is Regents’ Professor and Associate Dean in College of Computing at Georgia Tech. He served as STINT visiting Professor at KTH Stockholm, Sabbatical Chair at Sony IDL in Tokyo, and in the Robotics/AI Group at LAAS/CNRS in Toulouse. His research interests include behavior-based control and action-oriented perception, deliberative/reactive architectures, multiagent robotics, biorobotics, human-robot interaction, and robot ethics. He served on the Board of Governors of IEEE Society on Social Implications of Technology, IEEE Robotics and Automation AdCom, and founding co-chair of IEEE RAS TC on Robot Ethics. He is Distinguished Lecturer for the IEEE SSIT and IEEE Fellow.
Shu Jiang is a Robotics Ph.D. student in the Institute for Robotics and Intelligent Machines at Georgia Institute of Technology and a member of the Mobile Robot Laboratory. He received B.S. and M.S. degrees in Electrical Engineering from the University of Florida in 2009. His research interests include formal methods, human-robot team, educational and assistive robotics.
Matthew O’Brien is a Robotics Ph.D. student in the Institute for Robotics and Intelligent Machines at Georgia Institute of Technology and a member of the Mobile Robot Laboratory.
Feng Tang is a graduate student in the Robotics and Computer Vision Laboratory, Fordham University NY.
Peng Tang is a graduate student in the Robotics and Computer Vision Laboratory, Fordham University NY.
Wongpiromsarn,T.,&Murray,R.(2008).FormalVerificationofanAutonomousVehicleSystem.Proceedings of theConference on Decision and Control.
Younes,H.,&Simmons,R.(2002).Probabilisticverificationofdiscreteeventsystemsusingacceptancesampling.Proceedings of the14th Int. Conf. on Computer Aided Verification,CopenhagenDenmark.doi:10.1007/3-540-45657-0_17
Zaks, A., & Joshi, R. (2008). Verifying Multi-threaded C programs with SPIN. Proceedings of the 15th International SPIN Workshop,LosAngelesCA.
ENdNoTES
1 Inprocessalgebra,aninterleavingtheoremrelatesthesequentialandparallelcompositionoperations.