model-based testing for achieving maximum cd pipeline acceleration

World®’16

Model-BasedTestingforAchievingMaximumCDPipelineAccelerationAlexMartins- CTOContinuousQuality– CATechnologiesBenJohnson-Ward- ARDJediMaster– CATechnologies

DO5X29S

DEVOPS

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Abstract

MostsuccessfulDevOpsteamshavelearnedhowtoworktogetherseamlessly.Then,theygettoapointintheirjourneywheretheylookforawaytoachievemaximumsoftwaredevelopmentlifecycleaccelerationandtheythinkaboutcontinuousdelivery.Oneofthethingsmostorganizationsthatadoptcontinuousdeliveryareoverlookingistheaccelerationoftheentireapplicationlifecycle,notjustthebuild-test-deploy-operatecycle.Whataboutacceleratingtherequirementsgatheringprocess?Thesearestillcommunicatedthroughthewrittenlanguage—thesamewayit’sbeendoneacrossdifferentteamsforthepast30years:WritteninWorddocuments,Excelspreadsheetsorinrequirementsmanagementtools.Usingthiscompletelymanualprocessleadstoambiguityandintroducesapproximately50-percentofdefectsintheapplicationcode.

ThispanelofexpertsfromQAorganizationsandQAproductswillexplorethedifferentwaysthatmodel-basedtestinghasbroughtmaximumaccelerationintothecontinuousdeliverypipelineformanydifferentorganizations.

AlexMartins

CATechnologiesCTOContinuousQuality


Agenda

CHALLENGESOFACHIEVINGCONTINUOUSDELIVERY

HOWOTHERSDESIGNTHEIRPROJECTS

CLOSING

LEVERAGINGDESIGNTECHNIQUESINTOSOFTWAREDESIGN

HOWITWORKS

Q&A

1

2

3

4

5

6


Developers ReleaseTeam Operations

TESTBUILD DEPLOY

Testers

OPERATE

GapinCDPipelineAccelerationInitiatives

PerformanceTesting DeploytoProduction

CustomerExperience

Code|Integrate|Build Functional&UATesting

idea

Extremeaccelerationpossiblethroughbettertoolsandinfrastructurethatsupporttheculturalchangeontheteam

§ Wetypicallyfocusonaccelerating

code-to-deploymentactivities

§ TheIntakeprocessusuallyremainsa

bottleneckandmostlymanual

§ MostlymanualtogofromIdeato

Code(forTDD)andtoTestCasesand

TestAutomationscripts

§ HowtoautomatetheCDPipeline

Intake process?

Requirement

TestCases

TestData

VirtualServicesTest

Automation


CoreChallengestoAchievingContinuousDelivery

Requirements

Development

QA/Testing

Operations

Release

70%ofalltestingisstillmanual.5

63%oftestersadmittheycan’ttest acrossallthedifferentmobiledevicesandOSversions.6

50%oftimespentlookingfortestdata.7

79%ofteamsfaceprohibitiverestrictions,timelimits or accessfeesonneeded3rdpartyservices.3

X X X

64%oftotaldefectcostoriginateintherequirementsanalysisanddesignphase.1

50%ofdeveloperstimeisspentfixingandfindingdefects.3

?

80%ofteamsexperiencedelaysindevelopmentandQAdueto unavailabledependencies.4

?

57%aredissatisfiedwiththetimeittakestodeploynewfeatures7

!

20%ofcompaniesareperforming fullreleaseautomationtoday.8

5PerfectoMobile2014BenchmarkSurvey,WhyMobileAppsFail20146Source:BloorResearch,20147ForresterGlobalModernServiceDeliveryBenchmarkOnlineSurvey20148TheForresterWave™:ApplicationReleaseAutomation20159AnkitOberoi.5ReasonsVisitorsLeaveYourWebsite.WebsiteMagazine201410EMA™WhitePaperAPMEssentials:NavigatingtheMazeofEndUserExperienceSolutionsPreparedforCATechnologies2016

40%ofuserswillabandonyourappifittakesmorethanthreesecondstoload.9

70%of delaysinapplicationdeliveryareattributedtonewandchangingrequirements.2

50%ofcompaniesaremanagingcloudserviceswithhomegrowntoolssuchasscripts,macros,etc.Evenfewerhavemobile-specific monitoringcapabilities.10

? ?

1HyderabadBusinessSchoolGITAMUniversityQualityFlaws:IssuesandChallengesinSoftwareDevelopment20122ForresterBrief:SoftwareRequirementsPracticesAreRipeForDisruption20143StudyconductedbytheJudgeBusinessSchoolatCambridgeUniversity 20133Voke MarketSnapshotServiceVirtualization20154BloorReportAutomatedtestcasegeneration2014


ClarityandVisionThroughouttheSprint– GoalBUSINESSANALYST PROGRAMMER TESTERUSER

Thecloserthevisionmeanstheusergetsaqualityproduct.Therearefewerbugsandtheproductisdeliveredfaster.


Let’slookathowothersdesigntheirprojects


DesigningaHouse

§ Customerdescribeshisdreamhousetothearchitect:

§ WeareabouttohaveourfirstbabyandIwanttobuildabeautifulhouseformyfamilywherewecanalllivecomfortably

§ Ineedthehousetohave2bedrooms,1kitchen,1livingspace,1diningroom,1TVroomand1bathroom

§ Thehouseneedstofeelspaciousandwarm


GeneratesLOTSofQuestions

§ HowmanySq ft shouldthehousehave?

§ Howmanyfloors?

§ Howbigeachroomshouldbe?

§ Whereshouldeachroombepositioned?

§ Whatkindoflightfixturesshouldweuse?

§ Whatdoesbeautifulorcomfortablemean?

§ Endlessquestions...


APictureIsWorthaThousandWords

§ OutsideoftheITworld,Architecturecanmean:– Ageneraltermtodescribe buildings andotherphysicalstructures.[1]

– Theartandscienceofdesigning buildings and(some)non-buildingstructures. [1]

– Thestyleofdesignandmethodofconstruction ofbuildingsandotherphysicalstructures. [1]

§ ThecustomerdrawsaninitialsketchwhichisthenfurtherrefinedwiththeArchitectinmultipleiterationsuntilthecustomerissatisfiedandaccepts it[1]ShorterOxfordEnglishDictionary (1993),Oxford,ISBN0198605757


Persona1– TheHomeOwner


Persona2– TheArchitect


Persona3– TheInteriorDesigner


Persona4– TheElectricalEngineer


OtherPersonasintheValueStream

Plumbing

HomeAutomationdesign

Sounddesign

Exteriordesign

Landscaping

Andmore…

Questions

§ Couldwereallydesignahousebyjustusingwords?

§ Ifso,woulditbefaster?

§ Howtotracktheimpactacrossthelayersifthehomeownerchangeshismindthroughthedesignprocess?


Contractor1– DesigningaPlantCADSoftware


Computer-AidedDesignSoftware

§ Assistsinacceleratingdesign,butmostimportantly:

§ Enablesmultiplepersonastoworkonthesamedesign

§ Enableseachpersonatoworkonaseparatelayerofthedesign

§ Keepsalllayerstogetherwithfulltraceability

§ Automaticallyidentifiestheimpactofanychangesmadetoonelayerandpromptsthelayerownerforanaction


CivilEngineeringandSoftwareEngineeringTheBestFromBothWorlds

CivilEngineering

Design

Construction

SoftwareEngineering

Design

Construction

§ MultipleProvenPractices

§ ChangeFriendly

§ SoftwareAssisted

§ ChangeFriendly

§ HistoricallyProvenPractice

§ HardandCostlytoChange

§ ManualDesign§ HardandCostly

toChange


AligningtoAgileSoftwareDesignBuildingtheArchitecturalRunway

CivilEngineeringDesign

SoftwareEngineeringConstruction

§ MultipleProvenPractices

§ ChangeFriendly

§ SoftwareAssisted

§ ChangeFriendly

§ SWDesignPatternsoriginatedfromCivilEngineeringandArchitecturalDesign,byChristopherAlexanderin1977

§ Model-DrivenVisualizationofRequirements– alsoborrowedfromCivilEngineering

§ ServiceVirtualizationconceptsinspiredbypracticesadoptedinCivilandMilitaryEngineeringwhenworkingwith

multiplecontractors– understandtheinterfacespecsandbuildtothatspec.Flexibilitytochangeimplementation


Howdoweleveragetheseprovendesigntechniquesintosoftwaredesign?


TheFlightBookingPath

§ As apassengerIwant tosearchfordomesticflightsSothat Icanbuytheonethatmatchesmyneed

ASimpleEpic

§ AcceptanceCriteria– Icancompletethepurchasein

nomorethan6screens– Icanseealistofallflights

availablebasedonmycriteria– Icanchoosemyseats– Iwillnotseeanypromotions

§ DefinitionofDone– Userabletonavigatefrom

searchtopurchasecompletionscreen


GeneratesLOTSofQuestions

§ Doyouwanttoviewthedepartureandreturnflightsasacombinedpair,ordoyouwanttoselecteachindependently?

§ Whenshouldtheuserselectseats?Afterselectingthedepartureandreturnflightsorjustbeforepayment?

§ Doestheuserneedtobeloggedontodothesearch?Ifnot,willtheuserhavetheopportunitytologonatsomepoint?

§ Endlessquestions…


APictureIsWorthaThousandWords§ IntheITworld:

– Amodelinsystemsengineeringandsoftwareengineeringisastructuredrepresentationofthefunctions(activities,actions,processes,operations)withinthemodeledsystemorsubjectarea.[1]

– Thepurposesofthemodelaretodescribethefunctionsandprocesses,assistwithdiscoveryofinformationneeds,helpidentifyopportunities,andestablishabasisfordeterminingproductandservicecosts[2]

§ TheProductOwnerdrawsaninitialsketchwhichisthenfurtherrefinedwiththeModelerinmultipleiterationsuntiltheProductOwnerissatisfiedandaccepts it[1]FIPSPublication183releasedofIDEFØDecember1993bytheComputerSystemsLaboratoryoftheNationalInstituteofStandardsandTechnology(NIST).[2]Reader'sGuidetoIDEF0FunctionModels


Persona1– TheProductOwner


Persona2– The(CAD)Modeler

§ Streamlinesunderstandingacrossallteamsinthevaluestreambyremovingallambiguityfromthescope

§ Foundationallayerotherswillbuildontopof(singlesourceoftruth)


Persona3– TheDataEngineer


Persona4– TheServicesEngineer


Persona4– TheTestAutomationEngineer


OtherPersonasintheValueStream

UXDesigner

PerformanceEngineer

EnvironmentManager

ReleaseManager

OperationsEngineer

Andmore…

Questions

§ Couldwereallydesignanapplicationbasedonwrittenlanguage?– Yes,buthowdidthatwork

forussofar?

§ CanwetakeCDtothenextlevelofSpeedandQuality?


§ BusinessFlowsbasedonUserStories

§ AlternateScenarios§ NegativeScenarios

§ DataRequirements§ DataParameters§ TestDatamart

connection

TestData

Model

LayeredapproachtoContinuousDelivery

§ ExternalInterfacesRequirements

§ APIDefinitions§ ServiceVirtualization

Services

§ Frameworklinkage§ Keywordlinkage§ Actionsontheapp

TestAutomation

OverlayingallLayers RolesandBenefits

§ PerformanceSLAs§ LoadTesting

Scenarios§ UserProfiles

PerformanceTesting

••Drawsandmaintainsbusinessprocessflows••Baselinesunderstandingacrosstheteam••Removesallambiguities

Modeler(PO/BA/Tester)

••Findandreservedataautomatically••Matchtestdatatoeachtestcase

TestDataEngineer

••Nodependencyonexternalinterfaces••AccelerateCodeDevelopment

ServicesEngineer

••Automaticcreationofscripts••Automaticimpactanalysis

TestAutomationEngineer

••Automaticgenerationofloadscenarios••Automaticcreationofscripts

PerformanceEngineer


Let’stakealookathowitactuallyworks…


Questions?


RecommendedSessions

SESSION# TITLE DATE/TIME

DO5X47S 'Shift-Right'- RapidEvolutionwithDesignOps 11/16/2016at3:45pm

DO5X27S Whats NewinCAAgileRequirementsDesigner 11/17/2016at12:45pm

DO5X40S CaseStudy:WhyRabobankWasInspired 11/17/2016at3:45pm


MustSeeDemos

UseModelBasedTesting

CAAgileRequirementsDesigner- 515-DevOpsTheatre5

DevOps5Theatre

AchieveMaxTestCoverage

CAAgileRequirementsDesigner- 516DevOpsTheatre5Theatre

ImproveDataCompliance

CATestDataManager512- DevOpsTheatre5

DeliverTestDataFaster

CATestDataManager511- DevOpsTheatre5


Stayconnectedatcommunities.ca.com

Thankyou.