techtalk: sometimes less is more –visualization can reduce your test data while enhancing quality!

World®’16

TechTalk:SometimesLessisMore–VisualizationCanReduceyourTestDatawhileEnhancingQuality!JamesWalker– PrincipalSoftwareEngineer– CATechnologies

DO5T06T

DEVOPS

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Abstract

Effectivetestingrequireshighqualitytestdata,butmostorganizationsstillrelyonproductiondatawhichprovidesjust10-20%functionalcoverage.Thisdataisdrawnfrom“businessasusual”scenariosthathaveoccurredinthepast,andsorarelyprovidethenegativescenariosandoutliersneededtorigorouslytestsoftware.Thedatathatcomesthickandfastintoproductionisfurthertoolargeandcomplicatedforanyhumanmindtoevaluate,sothatprofilingormodellingtechnologyisneeded.Toensurethattheyhavethequalitydataneededfortesting,organizationsneedtobeabletoevaluatewhichattributesexistinexistingdata,aswellashowtheycombine.Onlythencantheyevaluatewhichmissingattributesareneededtoexecutethetestsneededtodeliverqualitysoftware.

ThisTechTalkwillshowhowdatavisualizationprovidesaquickandreliablemethodtomeasurethetestcoverageprovidedbyexistingtestdata,spottinganymissingorinvaliddataataglance.Presentingdataattributesanddimensionsinpictorialformallowsuserstounderstandwhatdatatheyhave,howitsattributesrelate,andwhatdataismissing.TheaccuratemodelcreatedinCA’sDataVisualizationcanfurtherthenbefedintoCAAgileRequirementsDesignerandCATestDataManager,creatingthesmallestsetofdataneededtosatisfyeverypossibletestautomatically.

JamesWalker

CATechnologiesPrincipalSoftwareEngineer,CAAgileRequirementsDesigner


AboutMe

§ BSc,MRes,PhD– SwanseaUniversity,Wales

§ ResearchinDataVisualisation/BigDataproblems

§ Grid-Tools– SoftwareEngineer(2012– 2015)

§ CA– LeadSoftwareEngineerARD


MOTIVATION

INTRODUCTIONTODATAVISUALIZATION

DATAVISUALIZATIONFORTESTDATA

DEMO

1

2

3

4

Agenda

CONCLUDINGTHOUGHTS5


TestDataChallenges

§ Productionisoftenviewedasthetruesourceofgoodtestdata(defaulttakeacopyofproductiondata-masking)

§ Productiondatahashighvolumesandlowvariance(edgecases)

§ Betterdataindevelopmentshouldbethegoal(subjective–whatisbetter?)

Todothisyouneedtobeabletoanswerawholebunchofquestions


TestDataChallenges

“WhatdatadoIhave?”

“Whatdatadon’tIhave?”“DoIhave thedataIneed

formytests?”

“WhereamIundertesting?”

“WhereamIovertesting?”

“Howeffectiveismytestdata?”

“Whatismydatacoverage?”


CustomSQL

Data Views/Cubes Off-the-shelf visualisationtool

OldWorldOrder…1.

2. 3.


NewWorldOrder…TestingtotheBigDataFieldTechnologicaladvancementsoverthepastdecadehaveincreasedourabilitytocollectdatatopreviouslyunimaginablevolumes

Estimatedthatpeoplewillgenerate4.3exabytes ofdataintheirlifetime(1).

Datacontainshugeamountsofvalueforgaininginsight,understanding,decisionmaking,andprediction.

Virtuallyeveryfieldofscienceandindustryistakingadvantageofanalytics(medicine,sports,weather,finances,etc).

Testingislatetotheparty – Hugeopportunitiesforbigdatatechniquestohelpustestoursoftware,understandtheresults,gaininsightintoquality,makedecisions(shouldwerelease?),andeventuallypredictresultsbeforewe’veevenranasingletestcase...


IntroductiontoDataVisualization


“Thepurposeofcomputingisinsight,notnumbers”

Visualization:

§ Atoolthatallowstheusertogaininsightintodata

§ Toformamentalvision,image,orpictureof(somethingnotvisibleorpresenttothesightoranabstraction);tomakevisibletothemindorimagination[OxfordEnglishDictionary,1989]

RichardW.Hamming,1962


http://www.comm-dev.org/


VisualizationisVeryOld

§ Oftenanintuitivesteptomakephenomenaclearere.g.agraph

§ Classical(easy)approachesknownfrombusinessgraphics(excel,etc)

§ Onlynowinthepastdecadeisthevaluestartingtobecomeprevalent

https://utah.com/parowan-gap


DataSetsAreEver-increasinginSize– AGraphicalApproachIsNecessaryBefore – Simpletabulardata(verylownumberofdataanddimensions

Now – Distributedsystemscreatingmillionsofrowsasecond


VisualizationisGoodfor:

§ Exploration– Findtheunknown,unexpected– Hypothesisgeneration

§ Analysis– Confirmorrejecthypotheses– Informationdrill-down

§ Presentation– Communicate/disseminateresults


https://fluidi.wordpress.com

World®’16©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD15


SoWhatisDataVisualization?

§ Datavisualizationistheprocessofcreatinggraphicalabstractionsofdata

§ Usevisualisationonthedailybasis(i.e.Tubemap,weatherreport,stockmarket,webtraffic…)

§ Techniqueshaveenormousvaluetoallaspectsoftheworldwelivein– todaywefocusontesting&testdata!


CATestDataVisualizer


LargeRelationalDatabases

Protein Data Bank – pdb.org


DataCombinations

§ Itisimpossibletoconsider“AllCombinations”ofdata

n timesn timesn timesn =verylarge

Eachspinofthelockisadataattribute40possiblepositions4inputsrequired

40x40x40x40=2,560,00040x40x40x40x40=102,400,000

40x40x40x40x40x40=4,096,000,000


DataConcepts

§ Dataconcepts– onlysomecombinationsarerelevantforatestcase(testrequirements).

§ Howdotestsrelatetothedata?

§ Notallcolumnsmatter,buttheircombinedeffectdoes

§ Wecreateameta-layeroftestdataattributes


DataView– FlattentheDataandPickRelevantAttributes


Demo


Conclusion– TestDataVisualizer

§ Avisualizationtool– designedtoanalyse &andassistinbuilding‘better’testdata

§ Useadvancedspotdiagramsandparallelcoordinates– Comparedataforvalidandinvalidsetsofcombinations– Identifymissingcombinationsofdata– Indentify overandunder-testing– Compareenvironmentsforcoverage(QA1,QA2)– Measuredatacoverageaccurately– Reservedataamongstteammembers


TestDataVisualizer

“WhatdatadoIhave?”

“Whatdatadon’tIhave?”“DoIhave thedataIneed

formytests?”

“WhereamIundertesting?”

“WhereamIovertesting?”

“Howeffectiveismytestdata?”

“Whatismydatacoverage?”


RecommendedSessions

SESSION# TITLE DATE/TIME

DO5T17SCaseStudy:Nationwide'sCATestDataManagerSuccessStory 11/17/2016at1:45PM

DO5T07TTechTalk:WhatHappenedintheBackend?ThePowerofDBCompare 11/17/2016at3:00PM

DO5X42STechVision:TestDataonDemand:DeliveringtheRightData,totheRightPlace,attheRightTime 11/17/2016at4:30PM


Stayconnectedatcommunities.ca.com

Thankyou.


DevOps– ContinuousDelivery

FormoreinformationonDevOps– ContinuousDelivery,pleasevisit:http://cainc.to/PiTFpu

techtalk: sometimes less is more –visualization can reduce your test data while enhancing quality!

Technology