software anomalies test planning - department of · pdf fileagenda • iso/iec/ieee 29119...
TRANSCRIPT
TDDD04Defect taxonomies and test plans Ola Leifler, [email protected]
Agenda• ISO/IEC/IEEE 29119 - Test planning • ISO/IEC/IEEE 1044 - Software anomalies
2
3
Organizationaltestprocess
ISO/IEC/IEEE, “ISO/IEC/IEEE international standard for software and systems engineering – software testing part 2: Test processes,” ISO/IEC/IEEE 29119-2:2013(E), pp. 1–68, Sept 2013.
Testmanagementprocesses
Testplanningprocess
Testmonitoringandcontrolprocess
Testcompletionprocess
Dynamictestprocesses
Testdesign&implementationprocess
Testenvironmentsetup&maintenanceprocess
Testexecutionprocess
Testincidentreportingprocess
4
Test planningUnderstandcontext
Organisetestplandevelopment
Iden5fyandanalyserisks
Iden5fyriskmi5ga5onapproaches
Designteststrategy
Determinestaffingandscheduling
Recordtestplan
Gainconsensusontestplan
Communicatetestplanandmake
available
Start
Testplan
scope
Testplandevelopmentschedule
Analyzedrisks
Mi5ga5onapproaches
Teststrategy
Scheduling,staffingprofile
DraHtestplan
Approvedtestplan
5
Organizationaltestprocess
ISO/IEC/IEEE, “ISO/IEC/IEEE international standard for software and systems engineering – software testing part 2: Test processes,” ISO/IEC/IEEE 29119-2:2013(E), pp. 1–68, Sept 2013.
Testmanagementprocesses
Testplanningprocess
Testmonitoringandcontrolprocess
Testcompletionprocess
Dynamictestprocesses
Testdesign&implementationprocess
Testenvironmentsetup&maintenanceprocess
Testexecutionprocess
Testincidentreportingprocess
Test design and implementation process
6
Iden%fyfeaturesets
Derivetestcondi%ons
Derivetestcoverageitems
Derivetestcases
Assembletestsets
Derivetestprocedures
Start
Featuresets
Testcondi%ons
Testcoverageitems
Testcases
Testsets
Testdesignspecifica%on
Testcasespecifica%on
Testcasespecifica%on
Testprocedures
andtestscripts
Example“The system shall accept insurance applicants over the age of 18 and under the age of 80 years on the day of application based on their input age in whole years; all others shall be rejected. Accepted applicants of 70 and over shall receive a warning that in the event of a claim they shall pay an excess of $1000.”
7
Derive test conditions• Completion criterion? • Valid input? • Invalid input? • What if we note the following: 40 <= Age <= 55 results in a discount
message (unspecified in the description). How do we handle that?
8
Derive test coverage items• Equivalence class items to cover
9
Derive test cases• Select representatives from each class to achieve 100% equivalence class
coverage
10
Assemble test sets• What can be automated? • What must be manually tested?
11
Derive test procedures• Ordering of test cases based on exposure/dependencies • Traceability
12
Software anomaliesHow to classify anomalies according to ISO/IEC/IEEE 1044
ISO/IEC/IEEE,“IEEEstandardclassificationforsoftwareanomalies,”IEEEStd1044-2009(RevisionofIEEEStd1044-1993)
Error(Mistake)
Fault(Defect,Bug)
Failure
maycause
Incident(Symptom)
maybeobservedas
Test
Testcaseexercises
mayinduce
Remember this?
Defect - FaultFaults are executed defects. Some defects are detected before they become faults
Attribute Definition
DefectID
Description Whatismissing,wrong,orunnecessary?
Status Currentstatewithindefectreportlifecycle.
Asset Thesoftwareasset(product,component,module,etc.)containingthedefect.
Artifact Thespecificsoftwareworkproductcontainingthedefect(specification,design,code).
Versiondetected
Versioncorrected
Priority Rankingforprocessingassignedbytheorganization
Severity Thehighestfailureimpactthatthedefectcould(ordid)cause,asdeterminedby(fromtheperspectiveof)theorganizationresponsibleforsoftwareengineering.
Probability
Effect Theclassofrequirementthatisimpactedbyafailurecausedbyadefect.
Type Isthedefectduetocode,data,configuration?
Mode Omissionoffeature?Misrepresentationofspec?Unnecessaryfeature?
Defect attributes
…
Example 1 Attribute Value Definition
Effect Functionality Actualorpotentialcauseoffailuretocorrectlyperformarequiredfunction(orimplementationofafunctionthatisnotrequired),includinganydefectaffectingdataintegrity.
Effect Usability Actualorpotentialcauseoffailuretomeetusability(easeofuse)requirements.
Effect Security Actualorpotentialcauseoffailuretomeetsecurityrequirements,suchasthoseforauthentication,authorization,privacy/confidentiality,accountability(e.g.,audittrailoreventlogging),andsoon.
Effect Serviceability Actualorpotentialcauseoffailuretomeetrequirementsforreliability,maintainability,orsupportability(e.g.,complexdesign,undocumentedcode,ambiguousorincompleteerrorlogging,etc.).
Example 1
Attribute Value Definition
Type Data Defectindatadefinition,initialization,mapping,access,oruse,asfoundinamodel,specification,orimplementation.Examples:Variablenotassigned,initialvalueorflagnotset,incorrectdatatype…
Type Interface Defectinspecificationorimplementationofaninterface(e.g.,betweenuserandmachine,betweentwointernalsoftwaremodules,betweensoftwaremoduleanddatabase,betweeninternalandexternalsoftwarecomponents,betweensoftwareandhardware,etc.).Examples:Incorrectmoduleinterfacedesignorimplementation,incorrectreportlayout(designorimplementation),incorrectorinsufficientparameterspassed
Attribute DefinitionFailureID Uniqueidentifierforthefailure.Status Currentstatewithinfailurereportlifecycle.Title Briefdescriptionofthefailureforsummaryreportingpurposes.Description Fulldescriptionoftheanomalousbehaviorandtheconditionsunderwhichitoccurred,
includingthesequenceofeventsand/oruseractionsthatprecededthefailure.
Analysis Finalresultsofcausalanalysisonconclusionoffailureinvestigation.Environment Identificationoftheoperatingenvironmentinwhichthefailurewasobserved.
Configuration Configurationdetailsincludingrelevantproductandversionidentifiers.
Testreference Identificationofthespecifictestbeingconducted(ifany)whenthefailureoccurred.
Incidentreference
Identificationoftheassociatedincidentifthefailurereportwasprecipitatedbyaservicedeskorhelpdeskcall/contact.
Defectreference
Identificationofthedefectassertedtobethecauseofthefailure.
Failure attributes
…
Case 1
“Sue calls service desk and reports she cannot log in to timesheet system because the password field is missing from the login screen.” — In this example, Sue has a problem in that she cannot log in, caused by a failure wherein the password field did not appear on the login screen, which was in turn caused by a defect inserted during coding of the Login.asp artifact.
Failure/Defect Attribute ValueFailure FailureID 1Failure Status OpenFailure Title MissingpasswordfieldFailure Description
Failure AnalysisFailure Environment
Defect Asset
Defect Artifact
Defect Severity
Defect Probability
Defect Effect
Defect Type
Defect Mode
Case 1
Case 2
“Joe calls service desk and reports he cannot log in to timesheet system because the password field is missing from the login screen.”
Duplicate: several failures can indicate the same fault
Case 3
“During a peer review for software requirements for a new financial management system, Alice discovers that values are in the requirements as thousands of dollars instead of as millions of dollars.”
Failure/Defect Attribute ValueDefect DefectID 2Defect Status ClosedDefect Description
Defect Asset
Defect Artifact
Defect Severity
Defect Probability
Defect Effect
Defect Type
Defect Mode
Case 3
Case 4
“Company A’s battery ran out of power because there was no low-power warning. The design of a security system monitoring system did not include a warning for low battery power, despite the fact that this feature was specified in the requirements.”
Failure/Defect Attribute ValueFailure FailureID 3Failure Status OpenFailure Title MissinglowpowerbatteryalertFailure Description
Failure AnalysisFailure Environment
Defect Asset
Defect Artifact
Defect Severity
Defect Probability
Defect Effect
Defect Type
Defect Mode
Case 4
SEI Risk Identification TaxonomyClass Element Attribute
ProductEngineering
Requirements
Stability
Completeness
Clarity
Validity
Feasibility
Precedent
Scale
DesignFunctionality
Difficulty
So, how does that relate to me as a tester?
IfIamconcernedabout… ThenIwanttoemphasize
Requirementstability Traceability(testplanning)
Incompleterequirements Exploratorytesting
Unclearrequirements Decisiontables/statetransitiontesting
Designdifficulty Control-flowtesting
… …