towards continuous integration and continuous delivery in ...continuous integration pipeline...
TRANSCRIPT
TowardsContinuousIntegrationandContinuousDeliveryintheAutomotiveIndustry
Abstract
DevelopmentcyclesaregettingshorterandContinuousIntegrationandDeliveryarebeingestablishedintheautomotiveindustry.Wegiveanoverviewofthepeculiaritiesinanautomotivedeploymentpipeline,introducetechnologiesusedandanalyzeTesla’sdeliveriesasastate-of-the-artshowcase.
Introduction
Therevolutionindigitalsystemshasamassiveeffectonourdailylifeandonallbranchesofindustry.Industriestraditionallydominatedbymechanicalengineeringarenowshiftingtobesoftware-driven,ascanbeobservedinthetransitionoftheautomotiveindustry.
Customerstodayareusedtogettingthelatestupdatesautomatically,instantlyandfree.Thisdevelopmentbegunwithagilemethodsgainingpopularityinapplicationsoftwaredevelopment,thusallowingfastrelease-cycles.“FailFast”hasbecomethedrivingmotiveofinnovation-generatingSiliconValleycompanies.Today,ContinuousDeliveryisstate-of-the-artincertainapplicationdomainsandenablessoftwaredeveloperstoprovideanewreleasetoabroadbaseofcustomersatthepushofabutton.Thisallowsentirelynewconceptsofdevelopmentandacceptancetesting.Today,anewfunctioncanbeprovidedtoalimitedgroupofuserstoreceiveinstantcustomerfeedback.
Butwhatifthecustomersaredriversinsteadofmobilephoneusers?Themajordifferencehereisthatacarisasafety-criticalsystemandfaultsinsoftwaremayleadtoinjuryordeath.Soifacarmanufacturerintendsto“FailFast”,theywillhavetodosobeforetheydeliverandthusperformthoroughandautomaticchecksoftheirsoftwareiftheyintendtodelivercontinuously.
Thisisbecomingevermoreimportantwiththeindustrystartingtodeployautonomousdriving.Customersessentiallyputtheirlifefullyinthehandsofsomepieceofsoftwareiftheytrustanautonomouscartogetthemtotheirdestinationsafely.Atthesametime,theyexpectthatpieceofsoftwaretoconstantlybeattheverylatestedgeoftechnology.Theseantitheticrequirementsimposeatremendouschallengetotheautomotiveindustrymorethananyotherbranch,becauseitistheonlycyber-physicalsystemproducedinlargeseriestoday.
Thisarticleshallgiveaninsightintohowtheautomotiveindustryistryingtoovercomethesechallengestoday,whattechnologiesareusedandwhatlimitationsstillexisttoday.
DeploymentsinAutomotiveSoftware–Distributed,Embedded,Enterprise,Safety-CriticalSystemsBeforeanysoftwarecanbedeliveredtothecustomer,anumberofstepshavetobeperformedwhichincludescompilingandassemblingthefinalproduct,butalsotestingondifferentlevels.Thesestepscanbesummarizedassubsequentstagesofadeploymentpipeline[1].ThefirststagesummarizesallthestepsthatcanbeperformedautomaticallyinaContinuousIntegrationmatter,whilethesecondstageisformedofacceptanceteststhathavelongerrunningtimes.Thefinalstageisatypical
releasealongwithaUserAcceptanceTest.Eachstageistriggeredonlywhentheprecedingstagewaspassedsuccessfully.Thus,whentryingtodelivercontinuously,thegoalhastobetoperformstagestwoandthreeasquicklyasthefirststage(oratleastnearlyso).Otherwise,thecycleinwhichsoftwarechangesandnewreleasesareproducedoutpacethereleaseprocesssuchthatbuildsneedtobegroupedtoareleaseandthe
wholeideaofcontinuousdeliveriesdisappears.Indeed,technologiesandtoolsfromtheDevOpsMovements,suchasDocker(https://www.docker.com/)orPuppet(https://puppet.com/)aimatautomatizingthereleaseandenablecompaniessuchasAmazonandFacebooktocontinuouslydeploytheirlatestbuildsontheirproductionenvironments.ButwhataboutcompanieslikeTesla,whichproducehighlyembeddedanddistributedsystems?Theautomotiveindustryhasentirelydifferentstructure,processesandrequirements.Sowhatdoesa(continuous)deliverypipelinelooklikeinthisdomain?Thefirstthingthatneedstobeconsideredinordertounderstandautomotivereleaseprocessesisthehardware-andsoftware-architectureofavehicle.Moderncarshaveupto100individualECUswithdifferentpurposesthatinteracttoimplementacomplexfunctionlikeanADAS(AdvancedDriver’sAssistanceSystem).AschematicvisualizationcanbeseeninFigure1.Inordertoreducetheloadonthecommunicationchannels,certainpre-processingisalreadyperformedbythesensoricunitsthemselves:Forexample,anADASECUreceivesthedataaboutthesurroundingsintheformofalistofobjectsthatwasgeneratedfromrawdatabythecameraand
TheoriginofContinuousSoftwareEngineering
Withagilesoftwaredevelopmentbecomingstate-of-the-art,longintegrationcycleswereobsoleteandevenobstructive.In1991,theterm“ContinuousIntegration”wasfirstusedbyGradyBooch[1]todescribeaneffective,iterativewayofbuildingsoftware.ThetechniquewasquicklyadoptedintothesetoftechniquesusedinExtremeProgramminganddetailedguidelinesweresummedupbyFowlerin2006[2]:Withahighdegreeofautomation,fastintegrationtestsandasinglesource-repositoryonwhicheverysinglecommitisstored,itispossibletosetupatool-supportedpipelinethatallowstocreateastablebuildwiththepushofabutton.Withsuchcapabilities,everycommitisimmediatelyfollowedbyafullbuildinordertoquicklydetecterrorsandconstantlyhaveastablebuild.
Fowleralreadystatedthatcontinuousbuildsshouldbedeployedtoproductionenvironments.Thisisthelogicalnextstep:Ifyoucontinuouslygeneratethelatest,stablebuild,youonlyneedtotakeafewmorestepstoactuallygetthesoftwaretothecustomer.Notably,thesestepsareacceptancetestsofthefullproduct,whichcanbeautomatedtosomeextent.AdoptingthisiscalledContinuousDeliveryorContinuousDeploymentwiththeformertermexperiencinggreaterpopularityintoday’slargeDevOps-Movement.
Whilecontinuousintegrationisstandardpracticeinmanysoftwareprojectsandcontinuousdeliveryhasbeenadoptedbymanylargecompanies,mostnotablyFacebookandAmazon,moreandmoreaspectsofsoftwaredevelopmentareconsideredina“continuous”process,totheextentofincludingevenremotelyrelatedprocessessuchashumanresources[3].Thesenotionsaresummedupinthefieldof“ContinuousSoftwareEngineering”[4].
[1]Booch,Grady."Objectorienteddesignwithapplications.”RedwoodCity."(1991)
[2]Fowler,Martin,andMatthewFoemmel."Continuousintegration."Thought-Works)http://www.thoughtworks.com/ContinuousIntegration.pdf(2006).
[3]Fitzgerald,Brian,andKlaas-JanStol."Continuoussoftwareengineeringandbeyond:trendsandchallenges."Proceedingsofthe1stInternationalWorkshoponRapidContinuousSoftwareEngineering.ACM,2014
[4]Bosch,Jan,ed.ContinuousSoftwareEngineering.Springer,2014.
radarsensorsinadvance.Thus,automotivesystemstodayarenotonlyembedded,butdistributedembeddedsystems.DuetotheorganizationalstructureofOEMs,softwareteamsareequallydistributedandheterogeneous.ECUsareusuallydevelopedandproducedbyacontractor,includingallsoftware.However,hybridstructuresarealsopossible:partsoralloftheapplicationsoftwaremaybesuppliedbytheOEMorothercontractorsandthedeploymentontheECUmaybedonebyanothercontractor,asubcontractor
oreventheOEMagain.Thisimpliesthatadeliverymaybetriggeredbymanydifferentsourcesthathavealreadypassedthroughtheirowndeploymentpipeline.Finally,acarisasafety-criticalsystem,whichmeansthatfunctionalsafetycheckshavetobeperformedbeforeanydeploymenttotheproductiveenvironment.Typically,analysismethodslikeFMEA[2]orSTPA[3]areused.Theyinvolveasafetyanalystidentifyingpossiblehazardousscenariosandtestingthemagainstthereleasecandidate.
StagesoftheAutomotiveContinuousDeliveryPipeline
Theserequirementsleadtosomewhatdifferentstagesintheautomotiveindustrythatneedtobeperformedredundantlyinparallelbydifferentorganizations.AsshowninFigure2,thestartingpointisalwaysthecommitofsomesourcecodeinasingleECU.WhatfollowsimmediatelyisastandardContinuousIntegrationpipelineincludingstaticcodeanalysis,compilation,unitandintegrationtests.SinceoneECUmaycontainoneormorelibrariesofapplicationsoftwareandalwayshasaseparateoperatingsystemthatsuppliesanabstractionlayerforbasicfunctionssuchasschedulingorcommunications,furtherintegrationtestshavetobeperformed.Likeinlargeenterprisesoftware,abuildneedstobetriggeredwhenoneofthedependencieschanged[4].
Thedifferentlibrariesfirstneedtobeconfiguredandlinkedtotheoperatingsystem.ThisistypicallyamanualtaskandsupportedbyspecializedtoolssuchasVector’sDaVinciConfigurator(https://vector.com/vi_davinci_configurator_pro_en.html),which–amongstothers–offersvisualizationofinvolvedlibrariesandtheirinterfaces.Whilethissteptodayoftenrequiresexpertknowledge,thiscanbederivedfromarchitecturalinformationandcanbeautomatizedwhenacontinuousdeliverypipelineissetup.Theresultofthisstepisafullyfunctionalcontainerthatcanbeflashedonanycompatiblehardware.ThisisindeedverycomparabletothecontainerizationknownfromDevOps.
Figure1:SchematicViewofSensors,ActuatorsandProcessorsforADASinamodernVehicle
ThenextstepinvolvesIntegrationTestsperformedonasingleECU.Thesetestsaresupposedtoensurethecorrectfunctionalityoftheisolatedcontrolunitandisthuslimitedtointerfacetests.ThesetestsareeventuallycarriedoutoncustomHardware-in-the-LooporOpen-Looptestbenches.Manydifferentbuildmanagementsystems,fromthewell-knownJenkinstoproprietarycustomdevelopmentswithsophisticatedtestselectionmethods,areinusetotriggerthetests.TheconcretetestexecutionisperformedbystandardtoolslikeECU-Test(https://www.tracetronic.com/products/ecu-test/)orCanOE(http://vector.com/vi_canoe_en.html).Thesetoolsareconnectedtothebussystemsandcanmonitor,interpretandmanipulatethesignalswhengiventheaccordingarchitecturalinformation.Theyofferinterfacestospecifyadesiredbehaviour,e.g.thesimulatedinputofasignalfromacommunicatingECUalongwithpass/failcriteria,suchastheexpectationofacertainsignalbeingsentwithinagiventime.
IfallECUintegrationtestspassed,thecompiledsoftwarecontainersarecommittedtoacentralrepository.Thiscanbecomparedtoa“commit”intoasourcerepository,withthedifferencethatbinaryartifactsarecheckedinandthisisthesecondpipelinethatisbeingactivated.However,thiscommittriggersthedeploymentofsoftwarefirstonintegrationtestbenchesandlaterontestvehicles.
Thetestbenchesusedvaryincomplexityandtestinggoal.Whenitcomestofunctionalintegration,however,mosttestbenchesaredesignedsothattheycontainallECUsthatimplementacertainnumberoffunctionsalongwithacomplexHardware-in-the-Loop-SimulationfortheenvironmentandfurtherECUs.Thetestsforthesefunctionswillbeexecutedonthistypeoftestbench.
However,theeffortinthisstageisgigantic.Amodernvehiclecontainsalargenumberoffunctions.ISO26262demandstestcoverageforeachrequirementofafunction,whichresultsinanumberofseveral10.000sofintegrationtestsfortheentirevehicle.Andsincetheyareexecutedonthetargethardware,theyhavetoruninrealtimewhichleadstoanaverageexecutiontimeofseveralminutespertest.
Figure2:TheDeliveryPipelineinAutomotiveSoftware
Obviously,thisnumberneedstobereducedandatestsuitefortheexactchangehastobetailored.Inatypicalmanualfunctionalintegrationprocess,expertknowledgeisemployedtoanalyzewhichsub-functionistestedinwhichtestcase.InaContinuousScenariohowever,theexpertneedstobereplacedwithheuristics.InSoftwareEngineering,manymethodsfortestselectionhavebeenproposed,butsincethesourcecodeisnotavailable,mostofthesedonotapply.Inthiscase,testselectionmethodsthatanalyzeatestsuiteregardingthecommunicationpathsinvolvedshowgreatpotential:Testsareonlyexecutedifthesignalsmanipulatedandcheckedareactuallybeingprocessedbythechangedpieceofsoftware[6].
Thesetestsareexecutedonso-called“TestFarms”.Theseareconglomeratesofsimilartestbencheswithdifferentconfigurationstoallowparallelexecution.Insomecases,testshaverequirementsoncertainconfigurations(e.g.atestcaseisdesignedforacertainpowertrainsystem).Suchatestfarmhastobecontrolledbyacentralserversystem.Apartfromknowingthestatusofthetestbenchesandthetestsqueued,suchasystemmustcontainacomponentforload-balancingtoensuremaximumparallelizationandthateverytestisexecutedonatestbenchwiththerightconfiguration.Sometestbenchdesignsallowreconfigurationduringruntime,forexamplebyhavingredundantECU
variantsthatcanbeconnectedusingarelay.Inthiscase,thetaskoftheload-balancerbecomesacomplexschedulingproblem.Thisisataskcurrentlyonlyperformedbycustomserversystems[7].
Testingforfunctionalsafetyisabaserequirementtoallowthesoftwaretobeflashedonavehiclethatenterstheroad.Asmentionedbefore,thesystem’srequirementsandarchitectureareanalyzedusingamethodlikeFMEAorSTPA.Thisresultsintestcasesthatareexecutedonthetargethardware.ThiscanbeintegratedintoaContinuouspipelinejustlikefunctionalintegrationtests,becausethesame
Hardware-in-the-LoopandOpen-Loop
EmbeddedSystemsorCyber-PhysicalSystemsareuniqueintheirpropertythattheydirectlyinteractwiththephysicalworld.Assuchtheyrequireinputandprovideoutputwhichcannotalwaysbegeneratedorcheckedeasilywhentestingsoftware.
Toavoidthenecessitytomanipulatephysics,complexsystemslikeADASareoftentestedinsimulatedenvironments[1].Inthisapproach,sensorsarereplacedwithacomputerthatprovidestheexactsignalsthesensorwouldsendunderthedesiredphysicalcondition.Foractorsrespectivelyacomputerisconnectedthatcantranslatetheoutgoingsignalstoahypotheticalphysicalaction.
Open-LoopandHardware-in-the-Looparetwooppositionalapproaches.Thefirstconsidersysteminteractionsindividuallyandsimulateonlyoneactionandthecorrespondingreaction.Thismakesitlightweightandeasytosetup,buthaslimitationswhencomplexstagedinteractionslikeinADASneedtobetested.Hardware-in-the-Loop-systemsontheotherhandreplacetheentireenvironmentwithextremelycomplexmathematicalmodelsandcansimulateanentirecontrolloop:Itisabletocalculatetheeffectsofthesystem’sactionontheenvironmentandcanthusproperlyreactevenonconsecutivesystemactions.
Apartfromhardware,alsosoftwareandevensystemmodelscanbeputintotheloopandareusedformorelightweighttestsinearlierphases.
[1]O.Gietelink,J.Ploeg,B.DeSchutter,andM.Verhaegen,“Developmentofadvanceddriverassistancesystemswithvehiclehardware-in-the-loopsimulations”,VehicleSystemDynamics,vol.44,no.7,pp.569–590,July2006.
technologiesareused.Thedifferenceisthough,thatthesafetyanalysishastobeperformedinacontinuousmatter,too,andthereisnoattemptintheliteratureyettodothis.
Thepipelinestagesasdescribedsofarcanbeintegratedintoacontinuouspipelinewithmoreorlesseffortanddonothinderdeliverieswithinadayornightifsufficientresourcesandproperandefficienttoolingisavailable.Thisisdifferentwiththetwofollowingsteps,however:Acceptancetestsontheroadwillalwaysstayamanualaction.Manyofthesetestscanbeautomatizedandexecutedontestfarms,potentiallyreducingthisprocesstoseveraldays.Still,thiscausesadelayandconsideringseveralcommitsperday,noteverybuildwillbedeliveredina“continuous”pipelinethatincludesthisstage.
Thefinalstage,deploymentusingover-the-airtechnology,iswell-testedtodayasTeslahasshown.Withhigh-speedmobiledatastandardssuchas4G,updatinglargeamountsofsoftwareeveninremoteareasisnotaproblemanymore.Theonlyproblematthisstageisthatacarmightnotbereceivingserviceforanextendedperiodoftime.Inthatcase,deliveriescannotbemadecontinuously,butthisproblemarisesonlyinveryremoteareas.
AnOutlooktoPaloAlto
AsaSiliconValleycompany,Teslaaimstoputtheirinnovationsonthemarketasquicklyaspossible.Infact,theyaretheonlyautomobilemanufacturercreditedwithContinuousDelivery[8].Butthemechanismsbehindtheirdeliveriesareintransparent.AccordingtotheTeslaforums,usersappeartobeconfusedaboutwhetherornottheircarhasreceivedthelatestupdateyet[9].Thesameupdateversionappearstobedeployedondifferentcarsatdifferentdates.Thepubliclyavailabledatabase“TeslaFirmwareUpgradeTracker"(http://ev-fw.com/),whichenablesTeslauserstouploaddataaboutupdatestheircarreceived,trackedover1000carswithover5000singularupdatesoverthecourseoffirmwareversion7.1.Figure3showsfourrepresentativebuildsandthenumberofcarsonwhichtheyweredeployedeachday.Thefirstandmostobviousobservationisthatabuildhassomeformof“lifecycle”inwhichitisgraduallydeployeduntilitbecomesoutdatedandthattheselifecyclesoverlap,sometimestoalarge
Figure3:SelectedTeslaBuildsandthenumberofcarstheyweredeployedoneachday.
extent.Itshouldbenotedthatoverlappingbuildsareoftendeployedonthesamecaroneafteranother,soitissafetoassumethatnoteverybuildcanbe“skipped”whendeploying.The“lifecycle”canbedescribedinthreephases:
- The“releasedate”isthefirstdateonwhichabuildisavailabletoabroadpublicandmanycarswillbeupdatedonthisday.
- The“ramp-up-phase”typicallyconsistsofafewdaysfollowingthereleasedate.Thevastmajorityofcarsthatrequiretheupdatewillbeupdatedduringthistime.Thedelayismostlikelycausedbytheavailabilityofawirelessdataservice.
- Duringthe“fade-away-phase”,whichcansometimestakeseveralweekstomonths,everyotherdayafewcarsreceivetheupdate.Thereasonforthelengthofthisphaseisunclear,butcouldbecausedbyacombinationofvehiclesnotreceivingwirelessdataforanextendedtime,limitedtime-slotsforupdatesperdayanddependenciesonotherupdatesthathavetotakeplacefirst.
ThislifecyclecanbeobservedinFigure4,whichdisplaysthemostfrequentlydeployedupdatesinAugustandSeptember2016asaheatmap.Inthisdiagram,however,anotheranomalycanbeobserved.Almosteverymajorbuildisdeployedafewdayspriortothereleasedate,insomecasesuptofourweeksearlier.Itishighlyunlikelythatthisanomalyiscausedbyerrorsinthevehicles,suchasupdatetimesbeingreportedinaccurately,becauseithasbeenobservedsoregularly.15outofthe26buildsinTesla’s
Firmwareversion7.1thathavebeenreportedonmorethan50carswerereportedmorethanonedaybeforethereleasedate(notethattesladoesnotprovideofficialreleasedates).
Furthermore,thereisnoconnectionbetweenindividualcarsormodelsandtheoccurrenceofsuch“earlydeployments”.Asinglecarmayreceiveoneupdateearly,thenextcoupleofupdatesonthereleasedateorintheramp-up-phaseandsomeothersevenlater.Wecouldnotobservethatasinglecarhasreceivedanearlydeploymenttwice.Whatweobservedherecanbedescribedasaformofthecanaryreleasepattern.[1]
Whilewecanonlyspeculatewhatthereasonforthismightbe,thisisclearevidencethatTesladeploystheirsoftwareindeedcontinuously,butwithadelayofuptofourweeksormore,assumingthatabuildisadesignatedandmostimportantlyfixedrevisionofTesla’ssourcecode.
ContinuousDelivery:Waytogo!
AContinuousDeliverypipelineascomplexandcostlyasthiscouldeasilyexplainadelayof
severalweeksfromthecommitofsoftwaretothefinaldeploymentintheproductionenvironment.ThepipelinecontainsthefullContinuousDeliverypipelineasknownfromotherdomains,buthastobepassedredundantlyandforeachECUindividually.Whileonthislevelfamiliartechnologiescanbeused
Figure4:MostfrequentlydeployedbuildsofTeslaFirmware7.1in08/16to09/16.
Eachrowdesignatesoneday,thecolorindicatesthenumberofcarsthatreceivedtheupdateonthatparticularday.
andswiftprocessingiseasilyachieved,thepipelineisonlythefirststeptowardsadeploymentinacomplex,embedded,distributed,safety-criticalsystem.Allthesepropertiesofavehicleimposeadditionalrequirementsonthedeploymentpipelinethattaketheirtime.
Mostsignificantly,necessarymanualstepssuchasAcceptanceTestsandLegalApprovalcanpossiblydelaythedeploymentofanewsoftware.Errorsfoundinthesestepsrequirealengthyanalysisbeforeafixedversioncanbesentintothepipeline.
YetweobserveanenormousnumberandfrequencyofreleasesonTesla’svehicles.Thisindicatesthatfromatechnicalpointofview,mostoftheproblemswithContinuousDeliveryintheautomotiveindustrycanbeandarebeingsolved.Ifthepaceofpastyears’developmentscanbesustained,aswiftandfullContinuousIntegrationPipelinewillbeestablishedthroughouttheindustrywithinthenextyears.
References
[1]Humble,Jez,andDavidFarley.“Continuousdelivery:reliablesoftwarereleasesthroughbuild,test,anddeploymentautomation.”PearsonEducation,2010.
[2]Ishimatsu,T.,Leveson,N.G.,Thomas,J.,Katahira,M.,Miyamoto,Y.,&Nakao,H."ModelingandhazardanalysisusingSTPA."(2010).
[3]Stamatis,DeanH.Failuremodeandeffectanalysis:FMEAfromtheorytoexecution.ASQQualityPress,2003.
[4]Roberts,Mike."Enterprisecontinuousintegrationusingbinarydependencies"InternationalConferenceonExtremeProgrammingandAgileProcessesinSoftwareEngineering.SpringerBerlinHeidelberg,2004.
[6]Vöst,SebastianandWagner,Stefan."Trace-basedtestselectiontosupportcontinuousintegrationintheautomotiveindustry."ProceedingsoftheInternationalWorkshoponContinuousSoftwareEvolutionandDelivery.ACM,2016.
[7]Vöst,Sebastian."Vehiclelevelcontinuousintegrationintheautomotiveindustry."Proceedingsofthe201510thJointMeetingonFoundationsofSoftwareEngineering.ACM,2015.
[8]Claps,GerryGerard,RichardBerntsson,Svensson,andAybüke,Aurum."Onthejourneytocontinuousdeployment:Technicalandsocialchallengesalongtheway."InformationandSoftwareTechnology57(2015):21-31.
[9]TeslaMotorForums,“TrackinghighestSoftwareVersion-Isthereanythingneweroutthere?”,https://forums.tesla.com/forum/forums/tracking-highest-software-version-there-anything-newer-out-there,Accessedon27-Sept-2016
Authors
SebastianVöstisaPhDStudentatthedepartmentofSoftwareIntegrationatBMWGroup.ContacthimatSebastian.Voest@bmw.de.
StefanWagnerisaProfessoratUniversityofStuttgartandheadofthedepartmentforSoftwareEngineering.ContacthimatStefan.Wagner@informatik.uni-stuttgart.de