towards continuous integration and continuous delivery in ...continuous integration pipeline...

TowardsContinuousIntegrationandContinuousDeliveryintheAutomotiveIndustry

Abstract

DevelopmentcyclesaregettingshorterandContinuousIntegrationandDeliveryarebeingestablishedintheautomotiveindustry.Wegiveanoverviewofthepeculiaritiesinanautomotivedeploymentpipeline,introducetechnologiesusedandanalyzeTesla’sdeliveriesasastate-of-the-artshowcase.

Introduction

Therevolutionindigitalsystemshasamassiveeffectonourdailylifeandonallbranchesofindustry.Industriestraditionallydominatedbymechanicalengineeringarenowshiftingtobesoftware-driven,ascanbeobservedinthetransitionoftheautomotiveindustry.

Customerstodayareusedtogettingthelatestupdatesautomatically,instantlyandfree.Thisdevelopmentbegunwithagilemethodsgainingpopularityinapplicationsoftwaredevelopment,thusallowingfastrelease-cycles.“FailFast”hasbecomethedrivingmotiveofinnovation-generatingSiliconValleycompanies.Today,ContinuousDeliveryisstate-of-the-artincertainapplicationdomainsandenablessoftwaredeveloperstoprovideanewreleasetoabroadbaseofcustomersatthepushofabutton.Thisallowsentirelynewconceptsofdevelopmentandacceptancetesting.Today,anewfunctioncanbeprovidedtoalimitedgroupofuserstoreceiveinstantcustomerfeedback.

Butwhatifthecustomersaredriversinsteadofmobilephoneusers?Themajordifferencehereisthatacarisasafety-criticalsystemandfaultsinsoftwaremayleadtoinjuryordeath.Soifacarmanufacturerintendsto“FailFast”,theywillhavetodosobeforetheydeliverandthusperformthoroughandautomaticchecksoftheirsoftwareiftheyintendtodelivercontinuously.

Thisisbecomingevermoreimportantwiththeindustrystartingtodeployautonomousdriving.Customersessentiallyputtheirlifefullyinthehandsofsomepieceofsoftwareiftheytrustanautonomouscartogetthemtotheirdestinationsafely.Atthesametime,theyexpectthatpieceofsoftwaretoconstantlybeattheverylatestedgeoftechnology.Theseantitheticrequirementsimposeatremendouschallengetotheautomotiveindustrymorethananyotherbranch,becauseitistheonlycyber-physicalsystemproducedinlargeseriestoday.

Thisarticleshallgiveaninsightintohowtheautomotiveindustryistryingtoovercomethesechallengestoday,whattechnologiesareusedandwhatlimitationsstillexisttoday.

DeploymentsinAutomotiveSoftware–Distributed,Embedded,Enterprise,Safety-CriticalSystemsBeforeanysoftwarecanbedeliveredtothecustomer,anumberofstepshavetobeperformedwhichincludescompilingandassemblingthefinalproduct,butalsotestingondifferentlevels.Thesestepscanbesummarizedassubsequentstagesofadeploymentpipeline[1].ThefirststagesummarizesallthestepsthatcanbeperformedautomaticallyinaContinuousIntegrationmatter,whilethesecondstageisformedofacceptanceteststhathavelongerrunningtimes.Thefinalstageisatypical

releasealongwithaUserAcceptanceTest.Eachstageistriggeredonlywhentheprecedingstagewaspassedsuccessfully.Thus,whentryingtodelivercontinuously,thegoalhastobetoperformstagestwoandthreeasquicklyasthefirststage(oratleastnearlyso).Otherwise,thecycleinwhichsoftwarechangesandnewreleasesareproducedoutpacethereleaseprocesssuchthatbuildsneedtobegroupedtoareleaseandthe

wholeideaofcontinuousdeliveriesdisappears.Indeed,technologiesandtoolsfromtheDevOpsMovements,suchasDocker(https://www.docker.com/)orPuppet(https://puppet.com/)aimatautomatizingthereleaseandenablecompaniessuchasAmazonandFacebooktocontinuouslydeploytheirlatestbuildsontheirproductionenvironments.ButwhataboutcompanieslikeTesla,whichproducehighlyembeddedanddistributedsystems?Theautomotiveindustryhasentirelydifferentstructure,processesandrequirements.Sowhatdoesa(continuous)deliverypipelinelooklikeinthisdomain?Thefirstthingthatneedstobeconsideredinordertounderstandautomotivereleaseprocessesisthehardware-andsoftware-architectureofavehicle.Moderncarshaveupto100individualECUswithdifferentpurposesthatinteracttoimplementacomplexfunctionlikeanADAS(AdvancedDriver’sAssistanceSystem).AschematicvisualizationcanbeseeninFigure1.Inordertoreducetheloadonthecommunicationchannels,certainpre-processingisalreadyperformedbythesensoricunitsthemselves:Forexample,anADASECUreceivesthedataaboutthesurroundingsintheformofalistofobjectsthatwasgeneratedfromrawdatabythecameraand

TheoriginofContinuousSoftwareEngineering

Withagilesoftwaredevelopmentbecomingstate-of-the-art,longintegrationcycleswereobsoleteandevenobstructive.In1991,theterm“ContinuousIntegration”wasfirstusedbyGradyBooch[1]todescribeaneffective,iterativewayofbuildingsoftware.ThetechniquewasquicklyadoptedintothesetoftechniquesusedinExtremeProgramminganddetailedguidelinesweresummedupbyFowlerin2006[2]:Withahighdegreeofautomation,fastintegrationtestsandasinglesource-repositoryonwhicheverysinglecommitisstored,itispossibletosetupatool-supportedpipelinethatallowstocreateastablebuildwiththepushofabutton.Withsuchcapabilities,everycommitisimmediatelyfollowedbyafullbuildinordertoquicklydetecterrorsandconstantlyhaveastablebuild.

Fowleralreadystatedthatcontinuousbuildsshouldbedeployedtoproductionenvironments.Thisisthelogicalnextstep:Ifyoucontinuouslygeneratethelatest,stablebuild,youonlyneedtotakeafewmorestepstoactuallygetthesoftwaretothecustomer.Notably,thesestepsareacceptancetestsofthefullproduct,whichcanbeautomatedtosomeextent.AdoptingthisiscalledContinuousDeliveryorContinuousDeploymentwiththeformertermexperiencinggreaterpopularityintoday’slargeDevOps-Movement.

Whilecontinuousintegrationisstandardpracticeinmanysoftwareprojectsandcontinuousdeliveryhasbeenadoptedbymanylargecompanies,mostnotablyFacebookandAmazon,moreandmoreaspectsofsoftwaredevelopmentareconsideredina“continuous”process,totheextentofincludingevenremotelyrelatedprocessessuchashumanresources[3].Thesenotionsaresummedupinthefieldof“ContinuousSoftwareEngineering”[4].

[1]Booch,Grady."Objectorienteddesignwithapplications.”RedwoodCity."(1991)

[2]Fowler,Martin,andMatthewFoemmel."Continuousintegration."Thought-Works)http://www.thoughtworks.com/ContinuousIntegration.pdf(2006).

[3]Fitzgerald,Brian,andKlaas-JanStol."Continuoussoftwareengineeringandbeyond:trendsandchallenges."Proceedingsofthe1stInternationalWorkshoponRapidContinuousSoftwareEngineering.ACM,2014

[4]Bosch,Jan,ed.ContinuousSoftwareEngineering.Springer,2014.

radarsensorsinadvance.Thus,automotivesystemstodayarenotonlyembedded,butdistributedembeddedsystems.DuetotheorganizationalstructureofOEMs,softwareteamsareequallydistributedandheterogeneous.ECUsareusuallydevelopedandproducedbyacontractor,includingallsoftware.However,hybridstructuresarealsopossible:partsoralloftheapplicationsoftwaremaybesuppliedbytheOEMorothercontractorsandthedeploymentontheECUmaybedonebyanothercontractor,asubcontractor

oreventheOEMagain.Thisimpliesthatadeliverymaybetriggeredbymanydifferentsourcesthathavealreadypassedthroughtheirowndeploymentpipeline.Finally,acarisasafety-criticalsystem,whichmeansthatfunctionalsafetycheckshavetobeperformedbeforeanydeploymenttotheproductiveenvironment.Typically,analysismethodslikeFMEA[2]orSTPA[3]areused.Theyinvolveasafetyanalystidentifyingpossiblehazardousscenariosandtestingthemagainstthereleasecandidate.

StagesoftheAutomotiveContinuousDeliveryPipeline

Theserequirementsleadtosomewhatdifferentstagesintheautomotiveindustrythatneedtobeperformedredundantlyinparallelbydifferentorganizations.AsshowninFigure2,thestartingpointisalwaysthecommitofsomesourcecodeinasingleECU.WhatfollowsimmediatelyisastandardContinuousIntegrationpipelineincludingstaticcodeanalysis,compilation,unitandintegrationtests.SinceoneECUmaycontainoneormorelibrariesofapplicationsoftwareandalwayshasaseparateoperatingsystemthatsuppliesanabstractionlayerforbasicfunctionssuchasschedulingorcommunications,furtherintegrationtestshavetobeperformed.Likeinlargeenterprisesoftware,abuildneedstobetriggeredwhenoneofthedependencieschanged[4].

Thedifferentlibrariesfirstneedtobeconfiguredandlinkedtotheoperatingsystem.ThisistypicallyamanualtaskandsupportedbyspecializedtoolssuchasVector’sDaVinciConfigurator(https://vector.com/vi_davinci_configurator_pro_en.html),which–amongstothers–offersvisualizationofinvolvedlibrariesandtheirinterfaces.Whilethissteptodayoftenrequiresexpertknowledge,thiscanbederivedfromarchitecturalinformationandcanbeautomatizedwhenacontinuousdeliverypipelineissetup.Theresultofthisstepisafullyfunctionalcontainerthatcanbeflashedonanycompatiblehardware.ThisisindeedverycomparabletothecontainerizationknownfromDevOps.

Figure1:SchematicViewofSensors,ActuatorsandProcessorsforADASinamodernVehicle

ThenextstepinvolvesIntegrationTestsperformedonasingleECU.Thesetestsaresupposedtoensurethecorrectfunctionalityoftheisolatedcontrolunitandisthuslimitedtointerfacetests.ThesetestsareeventuallycarriedoutoncustomHardware-in-the-LooporOpen-Looptestbenches.Manydifferentbuildmanagementsystems,fromthewell-knownJenkinstoproprietarycustomdevelopmentswithsophisticatedtestselectionmethods,areinusetotriggerthetests.TheconcretetestexecutionisperformedbystandardtoolslikeECU-Test(https://www.tracetronic.com/products/ecu-test/)orCanOE(http://vector.com/vi_canoe_en.html).Thesetoolsareconnectedtothebussystemsandcanmonitor,interpretandmanipulatethesignalswhengiventheaccordingarchitecturalinformation.Theyofferinterfacestospecifyadesiredbehaviour,e.g.thesimulatedinputofasignalfromacommunicatingECUalongwithpass/failcriteria,suchastheexpectationofacertainsignalbeingsentwithinagiventime.

IfallECUintegrationtestspassed,thecompiledsoftwarecontainersarecommittedtoacentralrepository.Thiscanbecomparedtoa“commit”intoasourcerepository,withthedifferencethatbinaryartifactsarecheckedinandthisisthesecondpipelinethatisbeingactivated.However,thiscommittriggersthedeploymentofsoftwarefirstonintegrationtestbenchesandlaterontestvehicles.

Thetestbenchesusedvaryincomplexityandtestinggoal.Whenitcomestofunctionalintegration,however,mosttestbenchesaredesignedsothattheycontainallECUsthatimplementacertainnumberoffunctionsalongwithacomplexHardware-in-the-Loop-SimulationfortheenvironmentandfurtherECUs.Thetestsforthesefunctionswillbeexecutedonthistypeoftestbench.

However,theeffortinthisstageisgigantic.Amodernvehiclecontainsalargenumberoffunctions.ISO26262demandstestcoverageforeachrequirementofafunction,whichresultsinanumberofseveral10.000sofintegrationtestsfortheentirevehicle.Andsincetheyareexecutedonthetargethardware,theyhavetoruninrealtimewhichleadstoanaverageexecutiontimeofseveralminutespertest.

Figure2:TheDeliveryPipelineinAutomotiveSoftware

Obviously,thisnumberneedstobereducedandatestsuitefortheexactchangehastobetailored.Inatypicalmanualfunctionalintegrationprocess,expertknowledgeisemployedtoanalyzewhichsub-functionistestedinwhichtestcase.InaContinuousScenariohowever,theexpertneedstobereplacedwithheuristics.InSoftwareEngineering,manymethodsfortestselectionhavebeenproposed,butsincethesourcecodeisnotavailable,mostofthesedonotapply.Inthiscase,testselectionmethodsthatanalyzeatestsuiteregardingthecommunicationpathsinvolvedshowgreatpotential:Testsareonlyexecutedifthesignalsmanipulatedandcheckedareactuallybeingprocessedbythechangedpieceofsoftware[6].

Thesetestsareexecutedonso-called“TestFarms”.Theseareconglomeratesofsimilartestbencheswithdifferentconfigurationstoallowparallelexecution.Insomecases,testshaverequirementsoncertainconfigurations(e.g.atestcaseisdesignedforacertainpowertrainsystem).Suchatestfarmhastobecontrolledbyacentralserversystem.Apartfromknowingthestatusofthetestbenchesandthetestsqueued,suchasystemmustcontainacomponentforload-balancingtoensuremaximumparallelizationandthateverytestisexecutedonatestbenchwiththerightconfiguration.Sometestbenchdesignsallowreconfigurationduringruntime,forexamplebyhavingredundantECU

variantsthatcanbeconnectedusingarelay.Inthiscase,thetaskoftheload-balancerbecomesacomplexschedulingproblem.Thisisataskcurrentlyonlyperformedbycustomserversystems[7].

Testingforfunctionalsafetyisabaserequirementtoallowthesoftwaretobeflashedonavehiclethatenterstheroad.Asmentionedbefore,thesystem’srequirementsandarchitectureareanalyzedusingamethodlikeFMEAorSTPA.Thisresultsintestcasesthatareexecutedonthetargethardware.ThiscanbeintegratedintoaContinuouspipelinejustlikefunctionalintegrationtests,becausethesame

Hardware-in-the-LoopandOpen-Loop

EmbeddedSystemsorCyber-PhysicalSystemsareuniqueintheirpropertythattheydirectlyinteractwiththephysicalworld.Assuchtheyrequireinputandprovideoutputwhichcannotalwaysbegeneratedorcheckedeasilywhentestingsoftware.

Toavoidthenecessitytomanipulatephysics,complexsystemslikeADASareoftentestedinsimulatedenvironments[1].Inthisapproach,sensorsarereplacedwithacomputerthatprovidestheexactsignalsthesensorwouldsendunderthedesiredphysicalcondition.Foractorsrespectivelyacomputerisconnectedthatcantranslatetheoutgoingsignalstoahypotheticalphysicalaction.

Open-LoopandHardware-in-the-Looparetwooppositionalapproaches.Thefirstconsidersysteminteractionsindividuallyandsimulateonlyoneactionandthecorrespondingreaction.Thismakesitlightweightandeasytosetup,buthaslimitationswhencomplexstagedinteractionslikeinADASneedtobetested.Hardware-in-the-Loop-systemsontheotherhandreplacetheentireenvironmentwithextremelycomplexmathematicalmodelsandcansimulateanentirecontrolloop:Itisabletocalculatetheeffectsofthesystem’sactionontheenvironmentandcanthusproperlyreactevenonconsecutivesystemactions.

Apartfromhardware,alsosoftwareandevensystemmodelscanbeputintotheloopandareusedformorelightweighttestsinearlierphases.

[1]O.Gietelink,J.Ploeg,B.DeSchutter,andM.Verhaegen,“Developmentofadvanceddriverassistancesystemswithvehiclehardware-in-the-loopsimulations”,VehicleSystemDynamics,vol.44,no.7,pp.569–590,July2006.

technologiesareused.Thedifferenceisthough,thatthesafetyanalysishastobeperformedinacontinuousmatter,too,andthereisnoattemptintheliteratureyettodothis.

Thepipelinestagesasdescribedsofarcanbeintegratedintoacontinuouspipelinewithmoreorlesseffortanddonothinderdeliverieswithinadayornightifsufficientresourcesandproperandefficienttoolingisavailable.Thisisdifferentwiththetwofollowingsteps,however:Acceptancetestsontheroadwillalwaysstayamanualaction.Manyofthesetestscanbeautomatizedandexecutedontestfarms,potentiallyreducingthisprocesstoseveraldays.Still,thiscausesadelayandconsideringseveralcommitsperday,noteverybuildwillbedeliveredina“continuous”pipelinethatincludesthisstage.

Thefinalstage,deploymentusingover-the-airtechnology,iswell-testedtodayasTeslahasshown.Withhigh-speedmobiledatastandardssuchas4G,updatinglargeamountsofsoftwareeveninremoteareasisnotaproblemanymore.Theonlyproblematthisstageisthatacarmightnotbereceivingserviceforanextendedperiodoftime.Inthatcase,deliveriescannotbemadecontinuously,butthisproblemarisesonlyinveryremoteareas.

AnOutlooktoPaloAlto

AsaSiliconValleycompany,Teslaaimstoputtheirinnovationsonthemarketasquicklyaspossible.Infact,theyaretheonlyautomobilemanufacturercreditedwithContinuousDelivery[8].Butthemechanismsbehindtheirdeliveriesareintransparent.AccordingtotheTeslaforums,usersappeartobeconfusedaboutwhetherornottheircarhasreceivedthelatestupdateyet[9].Thesameupdateversionappearstobedeployedondifferentcarsatdifferentdates.Thepubliclyavailabledatabase“TeslaFirmwareUpgradeTracker"(http://ev-fw.com/),whichenablesTeslauserstouploaddataaboutupdatestheircarreceived,trackedover1000carswithover5000singularupdatesoverthecourseoffirmwareversion7.1.Figure3showsfourrepresentativebuildsandthenumberofcarsonwhichtheyweredeployedeachday.Thefirstandmostobviousobservationisthatabuildhassomeformof“lifecycle”inwhichitisgraduallydeployeduntilitbecomesoutdatedandthattheselifecyclesoverlap,sometimestoalarge

Figure3:SelectedTeslaBuildsandthenumberofcarstheyweredeployedoneachday.

extent.Itshouldbenotedthatoverlappingbuildsareoftendeployedonthesamecaroneafteranother,soitissafetoassumethatnoteverybuildcanbe“skipped”whendeploying.The“lifecycle”canbedescribedinthreephases:

- The“releasedate”isthefirstdateonwhichabuildisavailabletoabroadpublicandmanycarswillbeupdatedonthisday.

- The“ramp-up-phase”typicallyconsistsofafewdaysfollowingthereleasedate.Thevastmajorityofcarsthatrequiretheupdatewillbeupdatedduringthistime.Thedelayismostlikelycausedbytheavailabilityofawirelessdataservice.

- Duringthe“fade-away-phase”,whichcansometimestakeseveralweekstomonths,everyotherdayafewcarsreceivetheupdate.Thereasonforthelengthofthisphaseisunclear,butcouldbecausedbyacombinationofvehiclesnotreceivingwirelessdataforanextendedtime,limitedtime-slotsforupdatesperdayanddependenciesonotherupdatesthathavetotakeplacefirst.

ThislifecyclecanbeobservedinFigure4,whichdisplaysthemostfrequentlydeployedupdatesinAugustandSeptember2016asaheatmap.Inthisdiagram,however,anotheranomalycanbeobserved.Almosteverymajorbuildisdeployedafewdayspriortothereleasedate,insomecasesuptofourweeksearlier.Itishighlyunlikelythatthisanomalyiscausedbyerrorsinthevehicles,suchasupdatetimesbeingreportedinaccurately,becauseithasbeenobservedsoregularly.15outofthe26buildsinTesla’s

Firmwareversion7.1thathavebeenreportedonmorethan50carswerereportedmorethanonedaybeforethereleasedate(notethattesladoesnotprovideofficialreleasedates).

Furthermore,thereisnoconnectionbetweenindividualcarsormodelsandtheoccurrenceofsuch“earlydeployments”.Asinglecarmayreceiveoneupdateearly,thenextcoupleofupdatesonthereleasedateorintheramp-up-phaseandsomeothersevenlater.Wecouldnotobservethatasinglecarhasreceivedanearlydeploymenttwice.Whatweobservedherecanbedescribedasaformofthecanaryreleasepattern.[1]

Whilewecanonlyspeculatewhatthereasonforthismightbe,thisisclearevidencethatTesladeploystheirsoftwareindeedcontinuously,butwithadelayofuptofourweeksormore,assumingthatabuildisadesignatedandmostimportantlyfixedrevisionofTesla’ssourcecode.

ContinuousDelivery:Waytogo!

AContinuousDeliverypipelineascomplexandcostlyasthiscouldeasilyexplainadelayof

severalweeksfromthecommitofsoftwaretothefinaldeploymentintheproductionenvironment.ThepipelinecontainsthefullContinuousDeliverypipelineasknownfromotherdomains,buthastobepassedredundantlyandforeachECUindividually.Whileonthislevelfamiliartechnologiescanbeused

Figure4:MostfrequentlydeployedbuildsofTeslaFirmware7.1in08/16to09/16.

Eachrowdesignatesoneday,thecolorindicatesthenumberofcarsthatreceivedtheupdateonthatparticularday.

andswiftprocessingiseasilyachieved,thepipelineisonlythefirststeptowardsadeploymentinacomplex,embedded,distributed,safety-criticalsystem.Allthesepropertiesofavehicleimposeadditionalrequirementsonthedeploymentpipelinethattaketheirtime.

Mostsignificantly,necessarymanualstepssuchasAcceptanceTestsandLegalApprovalcanpossiblydelaythedeploymentofanewsoftware.Errorsfoundinthesestepsrequirealengthyanalysisbeforeafixedversioncanbesentintothepipeline.

YetweobserveanenormousnumberandfrequencyofreleasesonTesla’svehicles.Thisindicatesthatfromatechnicalpointofview,mostoftheproblemswithContinuousDeliveryintheautomotiveindustrycanbeandarebeingsolved.Ifthepaceofpastyears’developmentscanbesustained,aswiftandfullContinuousIntegrationPipelinewillbeestablishedthroughouttheindustrywithinthenextyears.

References

[1]Humble,Jez,andDavidFarley.“Continuousdelivery:reliablesoftwarereleasesthroughbuild,test,anddeploymentautomation.”PearsonEducation,2010.

[2]Ishimatsu,T.,Leveson,N.G.,Thomas,J.,Katahira,M.,Miyamoto,Y.,&Nakao,H."ModelingandhazardanalysisusingSTPA."(2010).

[3]Stamatis,DeanH.Failuremodeandeffectanalysis:FMEAfromtheorytoexecution.ASQQualityPress,2003.

[4]Roberts,Mike."Enterprisecontinuousintegrationusingbinarydependencies"InternationalConferenceonExtremeProgrammingandAgileProcessesinSoftwareEngineering.SpringerBerlinHeidelberg,2004.

[6]Vöst,SebastianandWagner,Stefan."Trace-basedtestselectiontosupportcontinuousintegrationintheautomotiveindustry."ProceedingsoftheInternationalWorkshoponContinuousSoftwareEvolutionandDelivery.ACM,2016.

[7]Vöst,Sebastian."Vehiclelevelcontinuousintegrationintheautomotiveindustry."Proceedingsofthe201510thJointMeetingonFoundationsofSoftwareEngineering.ACM,2015.

[8]Claps,GerryGerard,RichardBerntsson,Svensson,andAybüke,Aurum."Onthejourneytocontinuousdeployment:Technicalandsocialchallengesalongtheway."InformationandSoftwareTechnology57(2015):21-31.

[9]TeslaMotorForums,“TrackinghighestSoftwareVersion-Isthereanythingneweroutthere?”,https://forums.tesla.com/forum/forums/tracking-highest-software-version-there-anything-newer-out-there,Accessedon27-Sept-2016

Authors

SebastianVöstisaPhDStudentatthedepartmentofSoftwareIntegrationatBMWGroup.ContacthimatSebastian.Voest@bmw.de.

StefanWagnerisaProfessoratUniversityofStuttgartandheadofthedepartmentforSoftwareEngineering.ContacthimatStefan.Wagner@informatik.uni-stuttgart.de