copyright © 2003 software quality research laboratory software production essentials seeing past...
Post on 19-Dec-2015
214 views
TRANSCRIPT
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Software Production Software Production EssentialsEssentials
Seeing Past the Buzz WordsSeeing Past the Buzz Words
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Why Process?Why Process? QualityQuality
Maximize Customer SatisfactionMaximize Customer Satisfaction Minimize Rework and RepairMinimize Rework and Repair
ProductivityProductivity Optimize Production CostOptimize Production Cost Shorten Time to MarketShorten Time to Market
This is not a tradeoff. Quality is This is not a tradeoff. Quality is Free.Free.
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Cost of Correcting an ErrorCost of Correcting an Error
1X in Requirements Analysis1X in Requirements Analysis 5X in Specification and Design5X in Specification and Design
10X in Coding10X in Coding 30X in Integration30X in Integration
70X in Acceptance Test70X in Acceptance Test 300X in Field300X in Field
Typical Range: $50 - $14,000Typical Range: $50 - $14,000
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Basic Process ElementsBasic Process Elements
Requirements DefinitionRequirements Definition
CodingCoding
ReleaseRelease
Everybody does it, but the order may vary.
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
How Do We Fill the Gaps?How Do We Fill the Gaps?
Requirements DefinitionRequirements Definition Specification?/Design?/Verification?/?Specification?/Design?/Verification?/? CodingCoding Test?/ Inspection?/?Test?/ Inspection?/? ReleaseRelease
No shortage of lifecycle/process candidates.
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Software Production is Software Production is IncrementalIncremental
An efficient, repeatable process is necessary for expanding, changing requirements.
?
?
?
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Software Production KeysSoftware Production Keys
Architecture
ArchitectureDefinition
Requirements
Code
Implementation
Specification
Test Planning
Test Cases
Certification
ReleaseDecision
Code Inspection
Peer Review
Peer Review
Peer Review
(1) A seamless, traceable transformation from requirements to code specification.
(2) Mapping specified code to a robust, maintainable architecture.
(4) Quantitative reliability assessment via model-based statistical testing.
(3) Peer review of key work products including 100% code inspection.
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Specification Process OverviewSpecification Process Overview
SpecificationFunctions
CanonicalSequences
Black BoxSpec
State DataTables
State BoxSpec
StimulusGathering
ResponseGeneration
State DataState
Transitions
ResponsesStimuli
Enumeration
RequirementsInitial Requirements
Capture
Systematic BehaviorDiscovery byEnumeration
Black Box and StateBox Transformations
Code Definition
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Rigorous Specification ProcessRigorous Specification Process
Establish Establish system boundarysystem boundary.. Define human/software/hardware Define human/software/hardware interfacesinterfaces.. Itemize Itemize stimulistimuli.. Itemize Itemize responsesresponses.. Perform Perform enumerationenumeration.. Perform Perform canonical sequence analysiscanonical sequence analysis.. Generate Generate state machine specificationstate machine specification.. Transform to Transform to formal notationformal notation (e.g., ACL2). (e.g., ACL2).
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Map Specified Code to Map Specified Code to ArchitectureArchitecture
Architecture
+Method1()+Method2()+Method3()
-data1-data2-data3
Exec
+set1()+set2()
-Resp1-Resp2
Display
+set1()+set2()
-Resp1-Resp2
Control
+set1()+set2()
-Resp1-Resp2
Console
+get1()+get2()
-Stim1-Stim2
Sensor
-End1
1
-End2
1
-End3
1..*
-End4
1
-End5
1
-End6
1-End7
1 -End8
1..*
StimulusGathering
ResponseGeneration
State MachineData/Transitions
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Independent ReviewIndependent Review Domain Expert ReviewDomain Expert Review
Initial RequirementsInitial Requirements Black Box SpecificationBlack Box Specification Usage ModelUsage Model
Development Team Peer ReviewDevelopment Team Peer Review Architecture SpecificationArchitecture Specification Test PlanTest Plan
Code InspectionCode Inspection Automated Enforcement of Coding StandardsAutomated Enforcement of Coding Standards Manual Verification of Functional CorrectnessManual Verification of Functional Correctness
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Testing is Always SamplingTesting is Always Sampling
Population(All Uses)
Sample(Tests)
What to test: a statistically appropriate
sample
How much to test: to an acceptable
reliability estimate
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Statistical Usage TestingStatistical Usage Testing
Statistical Usage TestingStatistical Usage Testing involvesinvolves Modeling the population of usesModeling the population of uses Selecting a sample for testingSelecting a sample for testing Estimating field performance from Estimating field performance from
test sample resultstest sample results Making a release decisionMaking a release decision
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Model-Based Statistical TestingModel-Based Statistical TestingUsage Modeling
andTest Planning
Test Case Generationand Automation
Support
Results Managementand Quantitative
Analysis
Build Model
Analyze/ReviseModel
Generate TestCases
Analyze/ReviseTest Suite
Execute Tests
Release Decision
ReliabilityAnalysis
Retest
Release
TML Files
HTMLReport
Test Cases/Scripts
HTMLReport
TestResults
ReleaseCriteria
HTMLReport
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Usage ModelUsage Model: All Possible : All Possible Uses and their LikelihoodUses and their Likelihood
Begin
Beta
EndStart
A
B
Quit
Quit
STATE MACHINENodes are states-of-useArcs are possible stimuliProbabilities define expected usageTest case is path from initial to terminal state
AB
B
A
p=1p=.9
p=.1
p=.1p=.1
p=.01
p=.05
p=.85
Idle
Alpha
p=.89
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
A Usage Model is A Usage Model is a Finite-State a Finite-State Markov ChainMarkov Chain
Well-understood formalismWell-understood formalism Rich body of analytical resultsRich body of analytical results Engineering basis for testingEngineering basis for testing Objectivity in test planning and Objectivity in test planning and
managementmanagement Simulates “use” of product and not Simulates “use” of product and not
the product itselfthe product itself
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Statistical Testing based on a Statistical Testing based on a Usage Model affords optionsUsage Model affords options
Coverage testing…all states and arcsCoverage testing…all states and arcs Importance testing…most probable pathsImportance testing…most probable paths Random testing…representative usageRandom testing…representative usage Testing to reliability goal…0.99, 0.999, …Testing to reliability goal…0.99, 0.999, … Testing to level of confidence…95%, 99%…Testing to level of confidence…95%, 99%… Testing to budget with measured reliabilityTesting to budget with measured reliability
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Test AutomationTest Automation
Partition andSample Input Space
UsageModel
GenerateTest
Cases
TestExecution
CompareResults to
Oracle
Test Record
Export TestScripts
Record Pass/Fail
AnalyzeResults
Result Checkingand Comparison
Parametric Data Generation forAbstract Stimuli
Test Case Generation
Release?
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Results Analysis / Release DecisionResults Analysis / Release Decision
Record all failures by test case number Record all failures by test case number and transition where failure occurs, then:and transition where failure occurs, then:
Compute Compute estimated reliabilityestimated reliability based on based on testing experiencetesting experience
Evaluate Evaluate stopping criteriastopping criteria ReleaseRelease or or fix and retestfix and retest based on based on
semi-quantitative risk assessment semi-quantitative risk assessment
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Benefits of Benefits of Model Based TestingModel Based Testing
Better ProductBetter Product Clearer requirements, improved specificationClearer requirements, improved specification
Better Use of ResourcesBetter Use of Resources Quantitative optimization of testing strategyQuantitative optimization of testing strategy Reusable assets: model, test framework, oracleReusable assets: model, test framework, oracle
Shorter Life CycleShorter Life Cycle Test planning in parallel with developmentTest planning in parallel with development Easier test automationEasier test automation
Better ManagementBetter Management Quantitative support for management decisionsQuantitative support for management decisions Quantification of expected field reliabilityQuantification of expected field reliability Clear path to continuous process improvementClear path to continuous process improvement
Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory
Essential Process ElementsEssential Process Elements Requirements DefinitionRequirements Definition Robust Architecture DefinitionRobust Architecture Definition Sequence-Based SpecificationSequence-Based Specification Domain Expert and Peer ReviewDomain Expert and Peer Review CodingCoding 100% Code Inspection100% Code Inspection Model-Based Testing/CertificationModel-Based Testing/Certification IncrementalIncremental Release Release
Fill the gaps with rigorous engineering Fill the gaps with rigorous engineering practices.practices.