place your school logo here verification (cs340 john knight 2004) 1 verification

Verification (CS340 John Knight 2004) 1 Place your School Logo Here

Verification


Validation vs. Verification Validation:

“Did We Build The Right Thing?” Does The Software Do What User Wants?

Verification: “Did We Build The Thing Right?” Does The System Implement The Specification?

Actual SoftwareIdea

Development

Validation Verification

Validation

SoftwareSpecification


Verification Techniques Inspection:

Examine The Work Product By Eye Detailed Discussion—Laboratory Topic

Static Analysis: Have A Program Examine The Work Product Summary

Testing: For An Executable Work Product, Try It Out Summary

Formal Proof: Use Mathematics To Demonstrate The Desired Property Summary

Inspections are the best return on investment.

Always include inspection in your software processes.


Reviews, Walkthroughs, & Inspections

A Variety Of Techniques Tried: Reviews Walkthroughs Inspections

All Based On Same Premise:Humans looking at work products

find defects quite well.

Very Effective In PracticeApplicable To All Work productsImportant Observation:

You can inspect all of aprogram but you cannot

test all of a program.


Fagan Inspections Developed By Michael Fagan At IBM Extremely Successful, Very Popular Hard To Do Well, Easy To Botch Steps:

Planning - Fairly Brief Overview - About One Hour Preparation - Several Hours Over Several Days Inspection - About A Two-Hour Meeting Rework - Depends On What Is Found Follow-Up - Check That Necessary Changes Made

People: Moderator - Manages Process Author - Developer Of Artifact Reader - Provides Inspection Focus Inspectors - General Participants Specialists - Specific Technical Areas


Fagan Inspections

Specifications Designs Implementations Test Plans Process Concepts Etc.

Author’sOverview

Inspector’sReading



Insp

ectio

n

Author’sRework


Inspection Activities Goal:

Detect Defects In Work Product Defects Might Be Any Deviation From Desired Quality Not Just Source Code "Bugs"

Checklist-driven: Usually A Corporate Standard Set Of Checks Determines Specialists Who Have To Inspect

Inspectors Assumed To Have Read/Inspected Work Product

Moderator: Schedules Inspection Determines Length Monitors/Manages Meeting Determines Which Part Of Artifact To Inspect


Inspection Activities Reader:

“Guides” Inspection By Reading Aloud, Line-by-line Author:

Provides Overview Answers Questions Undertakes Rework

Inspectors: Apply Product Knowledge Language/Tools Knowledge Technical Experience

Specialists—Attention To: Portability Maintenance Real-time Use Of Language Etc.


Review—Issues With Fagan Inspections Why The Overview? Documents? No Enforced Preparation—People Tend Not To Bother... Inspection Based On Paper Documents Not Rigorous:

Driven By The Clock Dominated By Strong-Willed Individuals No Precise Format No Precise Schedule

No Matching Of Inspector Skill Levels: Senior Engineers Argue About Comments And Spelling

No Conclusions EnsuredRemains A Controversial Technology

Done Well, Inspection Is The Best Value For Money Available


Active Reviews Developed By Parnas And Weiss At NRL Attempt To Deal With Problems Of Fagan Inspections Sophisticated Technique Some Key Points:

Inspectors Work On A Single Technical Area—Their Area Of Expertise

Inspectors Work Alone Authors Supply Questions To Inspectors To Check

Comprehension Results Collected And Product Reworked As Necessary

Found To Be Very Effective At NRL On A7


Phased Inspections Developed By Knight And Myers At UVa Attempt To Deal With All The Problems Of Fagan Inspections Goal—A Very Rigorous, Repeatable Software Inspection Process Premise:

There is far too much to check in a single inspection.Inspections should be in tightly focused, separate steps.

Steps Are Called Phases—Each Phase Building On Prior Phases Two Types Of Phase—Single- And Multi-Inspector Use Questions And Isolated Inspection From Active Reviews Multi-inspector Phase:

Several Inspectors Work In Parallel—Try To Find All Defects In Certain Class

Inspectors Meet To Reconcile Results Experiments At UVa Indicate Considerable Benefits In Approach


Phased Inspections

Inspector(s) A---

CheckCompliance

WithSpecificQualityGoal

Inspector(s) B---

CheckCompliance


Assume AllPrior Checks

Inspector(s) I---

CheckCompliance


Assume AllPrior Checks

Computer Support

Signature Signatures

Phase 1 Phase 2 Phase N


Types Of Phases

SpecificQualityCheck Specific

QualityCheck

SpecificQualityCheck

SpecificQualityCheck

Reconciliation

(a) Individual (b) Group


Phased Inspection—Example

DefectInspection

By Individual

DefectInspection

By Individual

DefectInspection

By Individual

ReconciliationDesign

DocumentationInspection

GlobalProgramming

PracticesInspection

InternalDocumentInspection

Source CodeFormat

Inspaction

LocalProgramming

PracticesInspection


Cleanroom Development Developed By Mills At IBM Premise:

Provide a software development environment, a “cleanroom”,that would not permit defects to enter the software.

Draws On Analogy With Hardware Engineering Sophisticated Technique—Some Key Points:

Software Developed Without Execution Authors Not Permitted To Execute Software Some Variants Do Not Permit Compilation Software Tested By Independent Group Test Results Returned To Authors For Software Repair Quality Assessment By Statistical Estimation

Results Include: Better Productivity (Actually, Much Better) Better Quality (Actually, Much Better)


(Insert Course #) Inspections Goal Is To Try Inspection Exercise Three-phase Phased Inspection:

Source-code Layout Programming Practices Functional Correctness

Author-supplied Questions For Comprehension Serious Measurement Of Activities Laboratory Activity To Be Presentation Of Results Post-lab Is To Be Inspection Report

Read the lab manual for lab 8.


Static Analysis

Examine Work Product Looking For Legal But Probably Erroneous Things

Typical Uses: Set/Use Anomalies Coding Standard Checking Design-rule Checking

Static Analyzer Report

Work Product

(Any Type, NotJust Source Code

Parse InputCreate Internal RepresentationConduct Check(s)Produce Report


Difficulties With Static Analysis Not A Panacea—Many Things Cannot Be Checked:

if a > b thenc := 1;...if d < e then

f := g / c;

-- Is c initialized on every execution? Program Complexity:

...very_large_function(a, b, c, d, e, f, g, h);

-- What is the effect of the function on the prgrm variables? Language Specific, Difficult To Construct—Hence Not As Common As They

Should Be Often Supplement Compilers (Finish The Compiler’s Job), E.g. Lint. Very High Performance System Developed By Dave Evans—Splint


Testing Summary Testing As Verification Technique Testing And Validation Testing Compared To Debugging Testing Phases:

Unit - Integration - System Acceptance - Regression

Black Box vs. White (Clear) Box Testing Coverage:

Functional Statement Branch Path

Automation Of Test Execution Checking Correctness Of Output


Testing In Engineering

Hydraulic Rams

AEROSPACE ENGINEERING SOFTWARE ENGINEERING

Structural Analysis By Finite Element Technique

Testing Confirms Analysis

Almost No Comprehensive Analysis Possible

Testing Establishes Conclusion


Some Testing Terminology Confusing Terms—Use Them Properly:

Error And Erroneous State Fault And Latent Fault Failure Commission vs. Omission

Test Oracle—Hardest Part Of Testing: Beethoven Exxon

Integration Strategies: Bottom Up Top Down

Test “Case”? Software Reliability?


Functional Testing Black Box Approach Function/Test Case Matrix Coverage Metric:

All Functions Covered Singly Function Covered In Combinations

Equivalence Partitioning Boundary Value Analysis Cause-Effect Graphing Special Cases:

Derive Cases From Conditions In Software Base Cases On Intuition


Structural Testing White/Glass Box Approach Base Test Case Selection On Software Structure Control Flow Based Testing—Try To Meet Coverage

Metrics: Statement Coverage Branch Coverage (All Flavors) Path Coverage

How Can These Be Measured? Mutation Testing:

Test The Test Cases Benign Mutants Killing Mutants Other Defects?


Testing Special Types Of Program Object-Oriented Programs:

Tricky... How Do You Test A Template? Programs Manipulate Objects, Not Classes (Or Templates) What Can You “Trust” When Inheriting A “Tested” Class?

User Interfaces: How Should The Input Be Generated?

Concurrent Programs: Order Of Events Test Harnesses

Real-Time Programs: Correctness Includes Timing Timing Affected By Measurement....