k.bjerke-gulstuen.d.soares cruzes.system integration ...€¦ · in this experience report we...

Author'saddress:KristianBjerke-Gulstuen,Rolfsbuktveien2,1364Fornebu,Norway;email:[email protected]'saddress:DanielaSoaresCruzes,SINTEF,NO-7465Trondheim,Norway;email:[email protected](s).

SystemIntegrationTestinginLargeScaleAgile:dealingwithchallengesandpitfallsKRISTIANBJERKE-GULSTUEN,ACCENTUREDANIELASOARESCRUZES,SINTEF

Imagine driving your car towork every day.During a normal day you pass several toll gates, nothing out of the ordinary on roads inNorway.Nowimaginethatacoupleofweekslaterwhenyoucheckyourinvoicesfromthesetollgates,younoticethatyouwerechargeddoubletheusualcost.TheAutosysplatformisacomplexsystemofsystems(SoS)thatformsthefoundationforalltransactionsinvolvingmotor vehicles in Norway. The end-user experience consists of value-chain-functionality that integrates more than twenty systemsdevelopedandoperatedbydifferentgovernmentdepartmentsandagencies.Systemintegrationtesting isofhigh importance.However,theprojectexperiencedchallengesregardingunalignedSoSdevelopmentplans,uncompleteSoSintegrationdesignsandSoSintegrationtestscenariosandinsufficienttestautomationcoverage.InthisexperiencereportwedescribehowtheAutosysprojectdealtwiththesechallenges.

1. INTRODUCTION

Modernsoftwareapplicationsandsystemsarerapidlyevolving tobecomemoresophisticatedwithcomplexmulti-directional dependencies to several other integrated systems. The components, once developed, areintegrated and ultimately tested in the deployment environment. System Integration (SI) is concernedwithformingacoherentwholefromcomponentsubsystems,includinghumans,tocreateamissioncapabilitythatsatisfiestheneedsofvariousstakeholders[1].TherearedifferentformsofSI.Historically,verticalintegrationoccurswhencomponentsofasystemdevelopedbyasingleacquisitionprogramareintegratedtoproducethedesired capability. Currently, vertical integration has a broader meaning which covers both a singleorganizationandjointorganizationswhichhavemultipleacquisitionprogramsovertimethatcontributetothecreationandenhancementofamissioncapability.Horizontal integrationoccurswhensystemsdevelopedbydifferent acquisitionprograms,often fordifferent customersandpurposes, arebrought together to createanew capability. A key goal of SI is to ensure that semantic and syntactic interfaces between componentelements of the system perform as specified by “contracts” between the elements. A companion goal is toensurethattheinterfacescanbeadaptedinwell-understoodwayswithrelativelymodesteffort.AlmostallSIfailuresoccurattheinterfacelevelprimarilyduetoincomplete,inconsistent,ormisunderstoodspecifications.Invariably,therootcausesoffailuretendtobeadhocintegrationandfailuretodevelopandadheretoformallydefinedsemanticconceptsandrelationships[1].

Asystemofsystems(SoS)isacollectionofsystemsoriginallydesignedasstand-alonesystemsforspecificanddifferentpurposesbutthathavebeenbroughttogetherwithintheSoSumbrellatocreateanewcapabilityneededforaparticularmission[2].AnSoSmaybeformeddynamicallytoperformagivenmissionandthenreorganizedasneededforothermissionsincludingthosethathavenotyetbeenenvisioned.AgoodSoSdesignmighthavemodulesthatarenotasgoodastheirstand-alonecounterparts thatperformthesamefunctions.SystemofSystemsintegration(SoSI)takesonanewmeaningthatcomeswithahostofnewchallenges.Oncethecomponentsystemsaredeveloped,theyareincrementallyintegratedandtestedand,ultimately,deployedintheoperationalenvironment.

InthisexperiencereportwedescribeSoSintegrationtestingchallengesandpitfallsasexperiencedbytheAutosysproject.TheAutosysplatformisahorizontalSoSwhereend-userexperienceconsistsofvalue-chain-functionalitythatintegratesmorethantwentysystemsdevelopedandoperatedbydifferentprivateandpublicdepartmentsandagencies.AccordingtotheSystemsEngineeringGuideforSoS[3,4],anSoScanbeclassifiedaccordingtothewayitismanagedanditsopennesstochangeandnewcapabilities.TheformandrigorofSoSis directly related to SoS type. The Autosys platform can be categorized as an Acknowledged SoS: havingrecognized objectives, a designated manager and resources. Its constituent systems retain independent

SystemIntegrationTestinginLargeScaleAgile:dealingwithchallengesandpitfalls:Page-2

ownership, objectives funding, development and sustainment. Changes in the system are based oncollaboration between the SoS and the system. The Autosys project experienced delayed availability ofcomplete and testable value chains (E2E-testing), resulting in late identification of defects and unidentifieddependencies.Thiscausedagrowing lackof trust in thenewsystemandstressed theprojectas thereleasedeadlineapproached.ImportantsuccessfactorsweretosupporttheScrumteamsbyestablishingadedicatedSoSintegrationteamresponsibleforthefollowupondependenciesanddeliveryplansfromallsystemsintheSoS, early involvementof testerswith extensiveuseof the testersmindsetduring integrationdesignand toincreaseourfocusonautomatedverificationofdatausedinthedifferentvaluechains.

KristianBjerke-GulstuenisanexperiencedtestmanagerinAccenture.Hehasbeenresponsibleforplanningandexecutingtestinginseverallarge-scaleprojects,usingbothtraditionalandagiledevelopmentmethods.Inthe Autosys Project Kristian is the testmanager from Accenture, responsible for defining the project’s testprocessesandthemanagement,planningandexecutionofallqualityengineeringandtestactivitiesperformedbyAccenture.Kristianispartoftheprojectmanagementteamandhasbeenwiththeprojectsince2016.HisexperiencefromtheprojecthasbeendiscussedwithresearcherDanielaSoaresCruzesfromSINTEFwhohasafocusontesting,qualityassuranceandagiledevelopment.

The remainder of this report is organized as follows: Chapter 2 describes the project context andbackground.Chapter3describesthemainSoSintegrationtestingchallengesandhowtheprojectcollectivelydealtwith the challenges. Chapter 4 summarizes the key learning points and provides recommendations tootherprojectsdoingSoSintegrationtesting.

2. BACKGROUND

TheAutosysProjectisalarge-scaleagiledigitalizationprojectthatismanagedbyTheNorwegianPublicRoadsAdministration (NPRA). NPRA is a Norwegian government agency responsible for the construction andmaintenance of highways and county roads, including the supervision and administration of registeredvehicles and certifications. The project’s mainobjective is the digitalization of business processesrelatedtovehiclesandtoreplacethelegacyAutosysautomotive register with the new Autosys vehicleapplication platform. The Autosys platform isconsidered critical to the Norwegian society as itoperates the formal approval, by law, of vehicles,registration and change of ownership, resellersolutions, and distribution of data to other publicadministrations and selected private partners.Autosys is an SoS where end-user experienceconsists of value-chain-functionality that spansseveralsystemsthataredevelopedandoperatedbydifferent government departments and agencies(such as the police, Norwegian tax authorities,insurance companies and road toll companies). Thenewplatformismanaging11millionvehiclesandisthecoreofanimmenselycomplexecosystemwithanannualturnoverof300billionNOK.

When developing the new Autosys platform,modifications to more than twenty of the existingintegrated systems were required. These changes were necessary as the integration technology andarchitecture would change in the new platform. Figure 1 illustrates Autosys and its horizontal systemintegrationlandscape.Inordertocompletethemanyfunctionalvalue-chains,Autosysdistributestransactiondata related to vehicles to several systems. Relevant changes in data are received by several internal andexternalsystems,transactionsareprocessedand,inmanysituations,fedbacktoAutosysforthecompletionofa business process. Changes in data may be consecutively reporting on updates in the vehicle register, i.e.changeof ownership or temporaryderegistration, emissiondata andmuchmore.Other integrations canbedatalook-up(singleandbulk)ofvehiclestoprovideinformationonselectedtechnicaldataatagiventime,i.e.additionalinformationsuchashistoricaldata(ownership,numberplates,andregistrationstatuses).

Figure1.IllustrationoftheAutosysplatformasahorizontalSoS


TheAutosysprojectplanningphasestartedinthesummerof2014andtheprojectwasinitiatedtogetherwithAccentureastheselectedIT-partnerinAugust2016.Accentureisresponsiblefordevelopmentofthecorepart of the system (the inner circle in Figure 1). Approximately 120 people are daily involved in thedevelopmentprocessofthesolutions.Theprojectisscheduledtobecompletedin2021.

2.1 AutosysdeliverymodelandapproachtoagiledevelopmentandtestingTheprojectusesaPRINCE2-basedmodelandthedevelopmentphaseusesahybridagile/Scrumdevelopmentmodel(basedonPS2000SOL).ThefollowingScrumpracticesareusedinfullorpartialbytheproject:unitandintegration testing, continuous integration (CI) and continuous testing,weekly anddaily standupmeetings,incrementaldesign,dailydeployment, testdrivendevelopment, testautomation,customerside involvement,planning poker, negotiated scope contract, retrospective, epics and user stories, fixed cycles and teamcontinuity.EpicsanduserstoriesweredefinedbyNPRA.Acceptancecriteriawerespecifiedatuserstorylevelduringtheinitialandcontinuoussolutiondesignphase.Ifnecessary,newuserstoriesandacceptancecriteriacouldbeaddedduringthesprints.

A“bigbang”transitiontothenewsystemwasconsideredundesirable,hencethedeliveryplanpreparedforagiledevelopmentwitha stagedphase-outof the legacy system.Theplanned replacementprocess includedsevenmaindeliverablesandtwomaindeliverablesdeployedtoproductionperyear.Maintenancereleasesaredevelopedinparallelanddeployedwhenneeded.Thedevelopmentphaseforamainreleaseconsistedof7-12sprints,ofthreeweekseach.

Quality engineering and testinghad a highpriority in theproject, andwith extensive focus on “shift-lefttesting”. The following quality engineering practices were incorporated in the different project phases:automatedandcontinuoustesting,manualandstructuredtesting,exploratorytesting,staticanalysisandstatictesting,testdrivendevelopment,experimentation,learninganddemos.Non-functionaltestingsuchassecurity,performance and operations testingwas done both during the sprints and as part of the system- and useracceptance (UAT) testing. Our process forquality engineering and testing evolvedconstantly, however thebasics in termsof testphases, testactivitiesandtestorganizationhasbeenconsistentandasillustratedinFigure2.

The IEEE defines integration testing astesting in which software components arecombinedandtestedtoevaluatetheinteractionbetween them [5]. Integration testing was animportant part of our testing process. At theAutosysproject integrationtestingconsistedoflowlevelandhighlevelintegrationtesting.Thelow level integration testing focused at theinterface level and was performed by thedevelopers during all the sprints. High levelintegrationtestingwasdoneduringsystemtestand included SoS integration testing. All lowlevel integrationtestscoverednormalcasesandspecialcases; theyweredevelopedin jUnit, incorporated intheCIpipelineand reported inSonar.Low level integration testing reliedheavilyon stubsandmocks.Highlevel integration testing focused on SoS integration testing preferably using production candidates of allsystemsintheSoS.ThescopeofhighlevelintegrationtestingwastestingofuserscenariosthatspannedmorethanonesystemintheSoS.Thistestingwasperformedinparallelwiththesprintsandshouldstartassoonasall necessary user stories were deployed to the test environments. The testing was both manual andautomated.

Test design and test scenarios per user story and at system/SoS-level were developed as part of thecontinuoussolutiondesignphase.Unittest,integrationtest(usingstubs)andsprinttestweredoneasin-sprinttestactivitiesperuserstory.AftereachsprintNPRAwasresponsibleforexecutingacontrolpoint.Thecontrolpoint consistedof ademonstrationof the sprintdeliverable (sprintdemo)andaverificationphase tomakesurealldeliverableswerecompletedaccordingtotheDefinitionofDone.Continuousautomatedandmanualtestingwasperformedinparallelwiththesprints.Thesprintswerefollowedbyathree-weekfinalsystemtest(hardeningsprint),anda6-8-weekuseracceptancetestperformedbyNPRA.

Figure2.TestprocessattheAutosysproject


3. DEALINGWITHSYSTEMINTEGRATIONPITFALLSANDCHALLENGES

Continuousdesignanddevelopmentkeptasteadypace,howeverwhentransitioningtohigh-levelintegrationtesting,westartedtoexperienceseveralchallenges:

• UnalignedSoSdevelopmentplans• UncompleteSoSintegrationdesignsandSoSintegrationtestscenarios• Insufficienttestautomationcoverage

3.1 DealingwithunalignedSoSdevelopmentplansChallenge.Whenstartinghigh level SoS integration testingweexperienced that insufficient coordinationofSoSdeliverables causedunaligneddevelopment and test plans, hencedelays to theplanned SoS integrationtestexecution.Thescopetobeimplemented,dependenciesandtimelinesimportanttotheAutosysprojectwascommunicated and discussedwith all system owners in the SoS at the start of every release. However, theregularfollowupwasdoneinafragmentedandadhocfashion.Thissoonescalatedtounsynchronizedplansanddelayedtheavailabilityofproductioncandidatesofallintegratingsystemsinourtestenvironments.Thetestteamexperienceddifferentscenarios,asillustratedinError!Referencesourcenotfound..I.e.accordingtothedevelopmentplantheSoSintegrationtestteamcouldstartSoSintegrationtestinginsprint4,howeverseveraloftheintegratingsystemswouldinrealitynotbereadyuntilUAT.ItbecamechallengingtoaccomplishthegoalofexecutinganearlyandstructuredSoSintegrationtestofbusinessprocesses

Ourdesigners,developersandtestersfounditchallengingtohaveacompleteoverviewandcontrolofhowtransactionpatternsanddatacombinationscouldaffectallothersystemsintheSoS.Thiswasaknownriskandchallenge,andourinitialandpreferredmitigationwastoidentifydefectsthroughearlySoSintegrationtesting.However, having delays to the SoS integration test execution resulted in increased risk of identifyingintegrationdefectsandmisunderstandingslate.

Another issue was that other projects and owners of the other systems in the SoS used different JirainstancesthanAutosys,andwehadnotoolthatsupportedlinkingbetweenthedependentuserstories.Duetothesechallenges thequalityof thesolutionwasquestionedbyseveralsystemownersandadawning lackoftrustintheproject’scapabilityinmeetingthedeadlinespushedtheprojectmanagementtoact.

Figure3.DelaysresultedinlateSoSintegrationtestingwithproductioncandidatesofallsystems

Our solution. Due to the complexity and scope of the SoS integrations we learned that the design andScrumteamsneededsupportindealingwithSoSintegrations.TheprojectmanagementdecidedtostrengthentheSoSintegrationteamtogainbettercontrolwithSoSintegrationdeliverables,andwithextensivefocusonthe systems that were not within direct control of the Autosys project. Our priority was to make suredeliverables thathaddependencieswasdelivered ina timely fashion,reducingtheuseofstubs inhigh-levelintegrationtesting.TheSoSintegrationteamhadextrapersonneladdedtotheteam;andrepresentativesfromthe Scrum teams, test managers and selected system owners from the integrating systems participated inweekly stand-up meetings. The technical integration and interface specifications were rarely an issue;however, the development of corresponding modifications could have delays. These meetings thereforefocusedon sharing informationon status anddiscussingpotential actions andwork arounds if delayswerereported.


TosupportthestatusandprogressreportingwemodeledalldependenciesusingJira.TheSoSintegrationteam got access to all relevant Jira instances, anduser stories with dependencies to other systemswere tagged in Jira using a custom field. The SoSintegration team decided to use Jira-linking togroup all user stories with dependencies. Theconcept is illustrated in Figure 4. The team thenuseddifferentlinktypestosupportthetrackingofthe status of user stories necessary to becompleted before SoS integration testing withoutstubs could be performed. Due dates on the userstories were set on all relevant user stories andalignedwith the testplans.Byusing thismethodalldependencieswereautomaticallydisplayed in Jira.Theflowinglinktypeswereused:

• DEP_ON (dependent on): user stories that were required to be delivered in front of Scrum team

development.I.e.interfacecontracts.• SIT_1(systemintegrationlevel1):userstoriesthatneededtobeimplementedtoenableconnectionto

the external systems test environment, including access to stubs. These user stories had to bedeliveredtostartfirstwaveofhighlevelSoSintegrationtesting.

• SIT_2(systemintegrationlevel2):userstoriesthatneededtobedeliveredbeforecompletefunctionalbusiness process testing could be executed with production candidate versions of the externalsystems.

ThegoalwastocompleteintegrationtestingatSIT_1-levelbycompletionofthefinalsystemtest,andthat

allnecessaryuserstorieslinkedwithSIT_2werereleasedtothetestenvironmentuponUATstart.The SoS integration team lead and SoS

integrationtestmanagercollaboratedonthestatusreporting and reported weekly to the projectmanagement team. As the test team relied heavilyon risk based testing, a new status reportingtemplate combining status of the SoS integrationtesting and the test effort was established (Figure5). The report also provided information onwhether SoS integration testing using productioncandidatesorstubshadstarted.

Results. Having a dedicated team continuallydoing structured coordination and follow up ondelivery plans, dependencies and progress, theproject lowered the risk of not meeting thedeadlinesdue to SoS integration testingdelays.Bythe start of the UATwemanaged to havemost oftheSoSvalue-chaintestedatSIT_1-levelandSIT_2-level integration testing had started for several ofthesystems.TheSoScoordinationand followupcontinuedthroughallsprintsandtheteamcontinuedtheirwork during all following releases. We did not experience having an SoS integration component team asdefragmentationofresponsibilitiesorasanynegativetotheprojectsortheagilewaysofworking.

EstablishingtheintegrationdependencymodelinJirawasdoneduringtwosprintsbutneededfrequentlyreviewstoverifyifnewlyaddeduserstoriesshouldbelinked.DependenciesthatweredisplayedautomaticallyinJiragavetheteamsacompletepictureoftheSoSintegrations, includingstatuses,anditbecameeasierfortheSoSintegrationtestteamtocreatetheirtestingplansandidentifypotentialdelays.

Thenewstatusreportwasusedonaweeklybasisandwasfoundunderstandableandeasytocommunicatewhenreporting statusandprogress to theprojectmanagementandsteeringcommittee.Frequent reportinginvolvedtheprojectmanagementinthedetailsofSoSintegrationsandintegrationtestingcomplexity,resultinginbetterunderstandingofthechallenges.

Figure5.CombiningSITstatusandtesteffortreporting

Figure4.Jiralinktypesdocumentinguserstorydependencies


3.2 DealingwithuncompleteSoSintegrationdesignsandSoSintegrationtestscenariosChallenge.Eventhoughweimprovedathavinguserstoriesdeliveredinsequencealignedwiththetestplans,hencefewerdelaysinstartingthetesting,weexperiencedthatdefects,misunderstandingsandevenconflictingrequirementsblocked the SoS integration testing.Weexperienced thatwe identifiedmorenewuser storiesthan expected related to SoS integrations when doing final system test and UAT, and realized that it wasextremelydifficulttohavecompleteinsightandunderstandingofhowdatawasusedacrossallsystemsinanSoS legacy system thathadevolved trough40years.Thismade it challenging todesignSoS integration testscenarios that would cover all necessary paths through the systems, increasing the risk of not identifyingblockers and defects. This again put a high pressure on the project as the time to fix these issues becameshorter.

Our solution. The SoS integration test team analyzed the situation together with the SoS integrationdesigners and theydecided to focus on improvingboth the integrationdesigns and the SoS integration testscenarios.The two teamsapproached this taskby introducinga conceptwenamedTestDrivenDesign.ThebasicandsimpleideabehindthisconceptwastogatherresourcesfromdifferentSoSstakeholders, includingsystemowners,designers,developersandSoSintegrationtesters,and“thinkasatester”whendiscussingandexperimentingwith thedifferent functional and technical scenarios involvingSoS integrations.Wenamed itTestDrivenDesignratherthanTestDrivenDevelopmenttostressthefactthattheimprovementswereintheintegrationdesignsandtestscenarios,andnotatthedevelopment.

The activity was done having the resources collocated and it was done in iterations. The method wasstructuredwalk-throughsonpaperandbyusingboththedeployednewversionoftheAutosysplatformandthe legacysystemstill running inproductioncomparingdifferent transactionsequences.TheprioritywastogetaunifiedunderstandingonhowtheplannedmodificationswouldaffectuserscenariosspanningtheSoS,and the levelof criticalityandconsequences to theendusers if thereweredefects thatwould triggerrippleeffectssuchasfaultycalculationsinintegratedsystems.

If we identified delays to the development plans for single systems in the SoS that could not be easilyadjusted,westillstartedSoSintegrationtestingbydoingstatictestingactivities.Thestatictestingwasmostlydone as reviews andwalk-throughs using the design documentation and the learning from the test drivendesignactivitiesastestbasisinsteadofthedeployedcode.

Results. The creativemindsof the tester and their “what if…”mindsetwas experiencedas veryhelpfulwhenimprovingthecompletenessoftheintegrationdesigns.AbetterunderstandingoftransactionsequencesandhowdatawasalteredduringthestepsinabusinessprocesswasveryusefultotheSoSintegrationtesterswhendevelopingthetestscenarios.WelearnedthatevenminorvariationsinspecificdatafieldsintheAutosyscoresystemscouldhaveahugeimpactoncalculationsfurtherdownthevaluechains.

Wealsolearnedthatmost“happycases”usuallyworkedfineandasdesigned.Itwaswhenwedecidedtobroadenthetestscenarioswithcombinationsofuserscenariosnotinthe“happycase”sphere,wefoundthemore complex defects. I.e., doing a relatively straightforward change of ownership produced the expectedresultintheintegratingsystems.However,whendoingachangeofownershipandthencorrectionstocertaindatafieldsregisteredforthevehicle,thealtereddatafieldswerenotmappedcorrectlyresultingintoohighorlowcalculationsoftaxesandinsurancefees(intwooftheintegratedsystems).

DoingstaticSoS integration testingwhenwehaddelays tosinglesystemsor isolated functionality in theintegratingsystemswasfoundtobeaveryvaluableactivity.Eventhoughwecouldn’tactuallyseetheactualresultsproducedbythesystemswecouldstilldohighlevelanalysisandevaluationofthecompletenessandcorrectnessoftheuserstoriesimplemented.Byconstantlyhavingthetestteamspushandthink“shift-left”and“nothingisgoingtoblockourtesting”,alsomotivatedtheotherteamstoconstantlystriveforimprovement.

TheprojectcontinuedtousethetestdrivendesigntechniqueupfrontofSoSintegrationtestexecutionasitreally helped us to early validate if all requirements and acceptance criteria were met. The upfront effortresultedinmoreefficientSoSintegrationtestingandareducednumberofintegrationdefectsduringUAT.WeexperiencedthatmoreandmorenewuserstorieswereidentifiedandimplementedduringthesprintsratherthanduringtheUAT.AllthisresultedinahigherconfidenceamongtheSoSstakeholdersandthattheprojectwouldbeabletodelivercompleteandcorrectSoSintegrations,hencemeettheagreedgo-livedateswithahighqualitysolution.

3.3 DealingwithinsufficienttestautomationcoverageChallenge.TheScrumandtestteamhadimplementedathoughtfulapproachtotestautomation,includingtestautomationaspartofCIandautomatedvalue-chaintestingasdatadrivenautomatedtests.Ourstrategywasto


develop relatively few tests focusing on core functionality that could test larger data sets rather thanmanysmaller testes covering the entire solution. Our automated tests did a good job in testing the Autosys coresystems isolated,however theydidnotcover largervolumesofdatacombinationsanduserscenariosdoingmodifications to data important to calculations and transactions in integrated systems. Hence importantdefectsweremissed.FromtheTestDrivenDesign,wesawourautomatedtestsmostlycovering“happycases”.TheautomatedtestsdidnotuncoverthedefectsthatweidentifiedduringmanualSoSintegrationtesting,asthesetestscenariosnowcoveredmorecomplextransactionpatterns.

Oursolution.WhenphasingoutalegacysystemwehadtheluxuryofhavingablueprintforhowdatawereexpectedtolookliketotheothersystemsintheSoS.Byreplicatingvehicletransactionsfromtheproductionsystem to the test environment and comparing the result, wewere able to implement automated checkingroutinesthatcouldrunonthousandsofvehicles.Giventhatwehadaconstantlychangingnewcoresystemindevelopment, therewasaneed forquickverifications touncover ifunforeseenchangeshadoccurred inoursystems.BasedontheexperiencesfromourTestDrivenDesignweembracedthisopportunityanddevelopedautomated checking programs, in Java and Excel, that replicated vehicle changes from production in a testenvironmentandthencomparedtheresulttothevehicleinthetwoenvironments.Thisreplicate-and-comparejobwas set up to run regularly via Jenkins, ensuring thatwewerenotified quickly if anunforeseen changeoccurred.The conceptwas fairly simple; however, itwas an extremely valuable addition to our testing. Forchangeofvehicleownership,thecheckimplementedwouldlooklikethis:

1. Achangeofownershipoccurredinproductionenvironment;2. Theverificationprogrampicksupthechangeofownershiptransaction;3. Dothesamechangeofownershipinthetestenvironment;4. Lookupthevehicleintheproductionenvironmentasitisnow(afterthechangeofownership);5. Lookupthevehicleinthetestenvironment;6. Expectthesameresponseat4and5;7. Sendnotificationifanydeviations.Results.TheAutosysprojectidentifiedandcorrecteddefectsdirectlyafternewcodewasdeployedtothe

testenvironmentsbyautomaticallyand frequentlycomparingdataregardingseveral thousandvehicles.Thecheckingcoveredtransactionpatternsasdoneintheproductionenvironment,itwasdonebeforethemanualtesting and even better; before other systems in the SoS could identify the defects. Now, the automationstrategyandimplementationworkedbetterasthesafetynetitwasmeanttobe.

Thisactivitymadeusrealizethepowerofcheckingandhowimportantthisisasasupplementtothetestingofasystem.Automatingthecheckinghelpedusfreeuptimeforthetesterstodomoreexploratorytestingandto,throughexperimentationandlearning,givebetterevaluationsontheusabilityandqualityofthesolution.

Evolving our test automation strategy also resulted in a positive collaboration between testers anddevelopers.Theydesignedthecheckstogether,andweexperiencedthatthetwodisciplineswerebroughtevenclosertogether;thetestersdevelopingtheirtechnicalskillsetandthedeveloperstobebetteratunderstandingtheimportanceofdemonstratingthatthecodewillworkwhenthinkingoutsidethepre-definedanddesigneduserscenarios.

4. KEYLEARNINGPOINTS

The SoS integration testing process was critical to the delivery of the final product in the Autosys Project.However, theefficiencyandqualityof theSoS testingwasaffectedbyseveral factors.Bydealingproactivelywith the issues and shiftingquality assurance and testing activities left insteadof adding test phases to therightweregainedcontrolof theprojectprogressandsolutionquality.Allgo-livedatesweremetasplannedandweexperiencedsatisfiedend-usersandvery fewmajordefectsaftergo-live: theproject is consideredahuge success. Some SoS integration defects leaked to the production environment and the defect scenariodescribed as part of the abstract actually happened. The defect was quickly fixed; however, it was a hugereminder to all, and our first key learning point; even when implementing several actions regarding SoSintegrationsand testing, you cannot guarantee thatnodefectswill leak to theproductionenvironment.Youconstantlyneedtostayalertandactivelymakeuseofall learningandexperiencesbefore,duringandafterasolutionissetintoproduction.

The second key learning point was that an SoS project should support their teams by having anorganizational project unit, person or team, responsible for coordinating SoS integrations in scope. This


includes alignment of plans, reporting, communication, dependencies, deployment routines, test data androutines for defect fixing. In addition, the team needs to manage that user stories and defect fixing arerequestedanddeliveredaccordingtothesprintplansenablingearlytestingwithproductioncandidatesinthetest environments. The project unitmust be supportedwith a tool for tracking status and progress acrossdifferentprojects/organizations/statedepartments/stateauthorities.WhentheAutosysproject iscompletedand goes into “operations mode”, we plan to keep the component team operational, as part of portfoliomanagement,whendevelopinglargerreleasesandgraduallypasstheresponsibilitytotheScrumteamswhendevelopingfeatures.

The third key learning point was that the greater value from system integration testing arriveswhendoingtestingwithoutanyuseofstubsandmocks.ManySIfailuresoccuratinterfacelevelduetoincomplete,inconsistent or misunderstood specifications. However, to us, lack of knowledge and insight regarding theintegratedsystemsandtheiruseofAutosysdatawasanimportantadditionalsourceofSIfailures.Itwasverychallengingtohavecontrolofhowalltransactionpatternsanddatacombinationscouldaffectothersystemsinthe SoS, hence early testing with the production candidate was the best way of identifying defects andmisunderstandings. Itwas during this testingwewere able to really challenge the product and identify thelarger integration issues. Furthermore, SoS integration testing should be done by subject matter expertsrepresenting the different systems and in combinationwith skilled test experts. If dynamic testingwithoutusing stubs is not possible until late sprints, test driven design involving relevant project teams acrossorganizationsshouldbeperformedduringthedesignphase.

The fourth key learning point was to continuously (and automated) check that new code does notintroduce defects. The automated checking must run on volumes of data to demonstrate that as many aspossiblecombinationsofdataproducethesameresult intheoldandnewsolution.Theautomatedcheckingshould be performed as a supplement to the manual testing and other automated testing. The automatedchecking is important for identifying regression defects and is a good source to identify misunderstoodrequirements.Themechanismisimportantasitfreesuptimeforthetesterstofocusonexperimentationandexploratorytesting.

The fifth key learning point was that the team doing business process testing of the SoS (not onlyinterfacetesting/technicalintegration)needstobestaffedwithSoSintegrationtestingexpertsandfunctionaldomainexperts. It is importantthatall testteammembersunderstandthecompletevalue-chains intheSoS,includingdependenciesandtheconsequenceofpotentialdefectsleakingtoproductionforalltheintegratingsystems.

ThesixthkeylearningpointwasthatsystemintegrationprojectssuchasAutosysshouldabsolutelybedeliveredinanagileway,howeverthelevelofagilityneedstobediscussedandagreeduponwhenstartingtheworkonaproject release. If the levelof insecurityrelated todatausagecrosssystemsandvalue-chainsaremediumtohigh,extrameasuresintermsofadditionalplanning,followupondefinitionsofdoneetc.,shouldbeaddedtothedailyfollowupandmanagementroutines.

5. ACKNOWLEDGEMENTS

We are grateful to Piet Syhler for helpful comments and guidance on this experience report.We thank theNPRAandAccentureforallsupportduringtheprocessofwritingthisreport.ThisworkwassupportedbytheSoS-Agile(247678)project,fundedbytheResearchCouncilofNorway.REFERENCES[1]A.M.MadniandM.Sievers,Systemsintegration:Keyperspectives,experiences,andchallenges,INCOSEJSystEng16(4)2013.[2]W.H.J.Manthorpe,Jr.,Theemergingjointsystemofsystems:AsystemsengineeringchallengeandopportunityforAPL,JohnsHopkinsAPLTechDig17(1996),305–313.[3]Madni,AzadM.,andMichaelSievers."SystemofSystemsIntegration:FundamentalConcepts,ChallengesandOpportunities."AdvancesinSystemsEngineering(2014):1-34.[4]OfficeoftheUndersecretaryofDefenseforAcquisition,Technology,andLogistics(OUSDAT&L),Systemsengineeringguideforsystemsofsystems,Washington,DC,August2008.[5] InstituteofElectricalandElectronicsEngineers. IEEEStandardGlossaryofSoftwareEngineeringTerminology:ANSI/IEEEStandard610-12-1990.IEEEPress:NewYork,1990.

k.bjerke-gulstuen.d.soares cruzes.system integration ...€¦ · in this experience report we...

Documents