techtalk: going full circle - linking code to tests to requirements and back around again
TRANSCRIPT
World®’16
TechTalk:GoingFullCircle- LinkingCodetoTeststoRequirementsandBackAroundAgainJamesWalker– PrincipalSoftwareEngineer– CATechnologies
DO5T05
DEVOPS
2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.
Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.
ForInformationalPurposesOnlyTermsofthisPresentation
3 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Abstract
Dependencymappinglinksrequirements,codeandtests,andiscentraltogooddevelopmentpractice.
Thissessionwillshow:
§ HowCAAgileRequirementsDesignercanbeusedtocreateathreewaybindbetweentherequirements,codeandtests.
§ Howformalmodeling canbeusedtocreateafulldependencymapofeventhemostcomplexsystems,whileintroducingtraceabilitybetweenthedesign,codeandtests.
Withfulldependencymapping,whenachangeismadetoagivencomponentintherequirements,itsimpactacrosstheentiresystemcanbeidentified,informingthedevelopersofexactlywhatneedschangingandwhere.Testscanbeautomaticallygeneratedfromtheformalmodel,andcanbeupdatedautomaticallytovalidatethatthischangehasbeensuccessfullyimplementedinthecode.
Whenanotherchangeismade,rounditgoesagain…
James
WalkerCATechnologiesPrincipalSoftwareEngineer
4 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
SOFTWAREDEVELOPMENTCHALLENGES
MODELBASEDTESTING(MBT)
FEEDBACKDRIVENTESTING
CONCLUSION
1
2
3
4
Agenda
5 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
AboutMe– JamesWalker
§ BSc,MRes,PhD– SwanseaUniversity,Wales
§ ResearchinDataVisualisation/BigDataproblems
§ Grid-Tools– SoftwareEngineer(2012– 2015)
§ CATechnologies– LeadEngineeronCAARD(2015)
6 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Maybeyouwon’tquitegettoenjoyyourweekendyet!
7 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TypicalProject
Requirements
Design
Development
QA
Release
Analyst&Customer
Architect
Developer
Tester
Release
8 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Waterfall
9 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Problematic...§ Inflexible(linear)
§ Fewbusinesssystemshavestablerequirements
§ Longreleasecycle– Expensive– Faillate
10 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
WaterfallProjectRequirements
Design
Development
QA
Release
Analyst&Customer
Architect
Developer
Tester
Release
Months/Years
WinstonRoyce– demonstratedthewaterfallasacriticismforhownotto
developsoftware!
11 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ContinuousDelivery
12 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ContinuousDeliveryProjectRequirements
Design
Development
QA
Release(MVP)
Months/Years
Requirements
Design
Development
QA
Release(MVP)
Requirements
Design
Development
QA
Release(MVP)
Weeks
13 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InTheoryThisisGreat…§ Eachtimewetestthedeliverable
§ Highlyadaptivetochange
§ Deliverfaster
§ Earlierfeedback(failfast)
14 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InReality…Heroculture— Fullthrottletosavetheproject
15 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TheProblem:Alackofclarityandvisionduringeachiteration
BusinessAnalyst Programmer TesterUser
TheUserKnowswhattheywantTheAnalystspecifieswhatthatis
TheProgrammerwritesthecode
TheTesterteststheprogram
Thefurtherthevisionsdiverge
16 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Traditionaltestingmethodsaretooslow,manual,andlinear forContinuesdelivery,compromisingspeedandquality
0% 10% 20% 30% 40% 50% 60% 70%
Time/resourcesintestdatacompliance(PII)
Defectsstemmingfromambiguousrequirements
Testinginnefficienciesleadingtohighercost
Lackoftestcoveragecreatingdefects/rework
Difficultyfindingtherightdataforaparticulartest
Manualtestingleadingtoprojectdelays
What'sthebiggestproblemyouface?
17 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
1– PoorRequirements
§ Aplethoraoftechniquesexist,mostarewritteninambiguousnaturallanguage
§ Therequirementsare“static”- theyoffernowaytoderivetestsdirectlyfromthem…
§ …nowaytoupdatetestswhentherequirementschange– thishastobedonemanually
Bottlenecks
18 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
2– ManualTestCaseDesign§ Currentlymanual– atimeconsuming,error-prone process
§ Isunsystematic,adhoc,andhasnorealnotionof“coverage”
§ Over-testing andunder-testing – 10-20%coveragewith4timesover-testing
§ Poorrequirementsleadtopooroveralltesting,withtestershavingtofillinthegaps
§ Nolinkagetotestdata – processismanual,painstakingandverytime-consuming
§ Noflexibilityforchangerequests:acriticalweaknessinanagileorcontinuousdeliveryenvironment.Changestakelongerthantheoriginalrequirement.
19 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
3– PoorTesting->AutomatedtestingAutomatedtesting:Manualscriptgeneration
§ Automatedtestingframeworksareheavilyscripted
§ Scriptgenerationisusuallydonemanually
§ Aswellasthemaintenanceofscripts
§ Alternativesolutionsuse:
§ Recordplayback
§ Script-lessautomationframeworks(keyword)
Butyou’restilldoingmanualtestcasedesign!
20 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TheVisionClarityandVisionDuringDevelopment
BusinessAnalyst Programmer TesterUser
Therearelessbugsandtheproductisdeliveredfaster
Thecloserthevisionmeanstheusergetsaqualityproduct
21 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ModelBasedTesting
22 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Modelbasedtestinglet’syoudefinethebehaviorofasystemundertest– inotherwordswhatissupposedtohappen.
WhatisModelBasedTesting?
Model
System
Themodelisthenusedtotesttheactualsystemtoseeifitdoeswhatitismeantto.
23 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ModelRequirementsasan“Active”Flowchart
§ AformalmodelthatisaccessibletothebusinesswhoalreadyuseVISIO,BPM,etc.
§ Whichisalsoamathematicallyprecisemodelofasystem,sothatiteliminatesambiguityandincompleteness
§ Itcanbeusedbytestersanddevelopers– itbringstheend-user,businessandITintoclosealignment
24 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
NewDeliveryParadigm:FromIdeationtoDesign,DevelopmentandTesting
25 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TestingWiththe“Active”Flowchart
§ Testerscanoverlaytheflowchartwithallthefunctionallogicanddatainvolvedinasystem
§ Testscanthereforebeautomaticallyderivedfromit
26 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
§ Exhaustivelytestamodel– extractallpossibleroutesfromstart->end
§ Eachroute/pathbecomesatestcase
§ Numberofroutesgrowsexponentiallywitheveryaddeddecision
32nodes+62edges=1,073,741,824 possibleroutes
GeneratingTestCases
2145Paths
27 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
§ Coverage,inmostcases,isrelatedtohowmuchfunctionalityisbeingcoveredinatestcase.
§ Traversethemodelbysatisfyingcoverageconstraints:
Optimization/Coverage
19PathsEdgePairs
9PathsInOutEdges
5PathsEdges
3PathsNodes
28 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
OverlayingAutomation
§ Testautomaters overlaycodesnippetsontoprocesseswhichcorrespondtothecodewhichperformseachaction
§ Automationscriptscanthereforebeautomaticallyderivedfromit
29 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ProvideFullTraceabilityWithRequirements
§ Knowwhatneedstobere-testedandwhentheintegrityofasystemisatrisk…“IfIchangethis,whatwillIbreak?”
§ Theimpactofachangemadetoanindividualcomponentisidentifiedsystemwide
§ Theimpactontestcasesanduserstoriesupanddownasystemcanalsobeidentifiedautomatically
30 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DEMO
Visitourboothformore!
31 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
FeedbackLoops– TheFuture
32 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLDRunResults
TypicalBuildProcess
ExecutionEngine
BuildEngineDevelopment/
Code
DifferentvisualrepresentationsofPass/Failresults.
TrafficLights Dashboarding Email(Jenkins)
33 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Scenario
Testing– Grayarea
34 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
NeedleinaHaystackWithouttheRightInfrastructure
35 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TestingBermudaTriangle
Code
RequirementsTestCases
Thefurtherapartthesearethelessquality
• Modelrepresentstherequirements• Codeisimplementedusingthemodel(canaddlinkstospecificobjects/methods).
• Testcasesaregeneratedfromthemodel
Usingtheseweobtainaclosebindingbetweenallthree.
Fulltraceabilityenablessolvingthepreviousproblemsefficiently
36 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TheFutureisFeedbackLoops(NewWorldOrder)
ExecutionEngine
BuildEngine
Development/Code
Model
Runresults
37 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ScenarioRevisit– Whatdoesthepass/failrepresent?Testresultsbounddirectlytotheirroutesthrough
thesystem.
38 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ScenarioRevisit– ExploratorytestingAfterafailurethemodelcanbeusedtogeneratememoretest
casesbypinpointtesting– testcasesfocusedondiscoveringmoreknowledgeaboutthefailure.
39 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ScenarioRevisit– RiskAnalysisUsemeta-deta assignedtothemodeltoperformriskanalysis.Aretheeffectedroutesthroughthesystemaboveasetthreshold?
=63%ofUserActivity
Can’tRelease
40 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ScenarioRevisit– GettingitFixed!We’vefoundtheprocess(es)responsible– nowwecantraceback
totheresponsibleteams!
DevelopmentPinpointedFailure
Defectcreated
41 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Conclusion
42 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
WhydoIneedthecode,requirements,andtestsstronglybound?§ AshiftingITlandscapehastransformedtheroleoftesters–
testingisnowcross-functionalanditerative,withanemphasisonfeedbackthroughout
§ Newtestanddevelopmentemerged,butthe“heroculture”remained…
§ …Alongwiththechallengesofslow,manualtesting,ambiguousrequirementsand“toolshops”
43 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TheVision:ShiftLeft,ShiftRight– OverlappingFeedbackLoops
§ ShiftLeft:workiterativelywithuserstogainupfrontknowledgeoftherequirementsandusethesetodriveactivitythroughoutthedevelopmentcycle
§ ShiftRight:connect-uptoolstacks anduse(test)analyticstoharnesstheinsightsoftesting,productionandoperations
§ Feedtheseinsightsbackintothemodel,andthewholeprocessgoesbackroundagainsoyoucanenjoyyourweekend
WhydoIneedthecode,requirements,andtestsstronglybound?
44 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
RecommendedSessions
SESSION# TITLE DATE/TIME
DO5X27S What’sNewinCAAgileRequirementsDesigner 11/17/2016at12:45pm
DO5X29SModel-BasedTestingforAchievingMaximumCDPipelineAcceleration 11/17/2016at3:00pm
DO5X40S CaseStudy:WhyRabobankWasInspired 11/17/2016at3:45pm
45 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Stayconnectedatcommunities.ca.com
Thankyou.
46 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DevOps– ContinuousDelivery
FormoreinformationonDevOps– ContinuousDelivery,pleasevisit:http://cainc.to/PiTFpu