tom gilchrist [email protected] the tools and techniques of sqa sasqag, february 17, 2000

Tom [email protected]

The Tools and The Tools and Techniques of SQATechniques of SQA

SASQAG, February 17, 2000SASQAG, February 17, 2000

[email protected]

SASQAG 2/17/00

Disclaimers

• The views and opinions expressed in this presentation are not necessarily those of my employer.

• Your results may vary.• You are free to use any

slide in your work, but if you do, please include my name and email.

[email protected]

SASQAG 2/17/00

Goals and Objectives

• Some Terms• Peer Reviews• Kickoff Meetings• Audits• Checklist and

Worklists

Sharpen YourSQA Skills

[email protected]

SASQAG 2/17/00

Software Quality Assurance

• Check software products and processes to verify that they comply with the applicable procedures and standards. (Process Checks/Audits)

• Review and measure the quality of software products and processes throughout development. (Peer Reviews)

• Provide software project management (and other appropriate parties) with the results of reviews and process checks.

• Work with the software project during early stages to establish plans, standards, and procedures to keep errors from occurring in the first place.

[email protected]

SASQAG 2/17/00

Validation and Verification

• Validation - Are we building the right product?

• Verification - Are we building the product right?

• Static Verification (of software) - Verification of a program by

examination and analysis of its program elements.

[email protected]

SASQAG 2/17/00

QualityIn

Perspective

[email protected]

SASQAG 2/17/00

Customer vs. End User

• End User:

- The person who uses the software product in the value stream

• Customer:

- Anyone who is affected by the end product or intermediate software engineering deliverables being produced (reviewed).

[email protected]

SASQAG 2/17/00

Healthy Customer/Supplier Relationship

• Implies product use by customer!

• Implies choice

• Communications centers on expectations.

- How we identify expectations

- How we document and communicate them

- How we measure/improve them

[email protected]

SASQAG 2/17/00

Value vs Quality

Customers are concerned with value not just quality.

• "Value is not overpaying for quality."

• Value is..."the right combination of product quality, fair price, and good service."

* From Business Week, Nov 11, 1991

[email protected]

SASQAG 2/17/00

Low Quality in Software

• Unneeded ComplexityUnneeded Complexity• ReworkRework

-Wrong-Unclear-Missing-Extra

[email protected]

SASQAG 2/17/00

The Project

ENTRY

EXIT

Program ElementCreation

ENTRY

EXIT


ENTRY

EXIT


ENTRY

EXIT


Requirements

FinishedProduct

Specs Design Code Test

Time

ProjectPlan

Standards,templates, rulesand checklists




[email protected]

SASQAG 2/17/00

Where is rework introduced?

ENTRY

EXIT


ENTRY

EXIT


ENTRY

EXIT


ENTRY

EXIT


Requirements

FinishedProduct


Time

ProjectPlan





[email protected]

SASQAG 2/17/00

Where is rework found?

ENTRY

EXIT


ENTRY

EXIT


ENTRY

EXIT


ENTRY

EXIT


Requirements

FinishedProduct


Time

ProjectPlan





[email protected]

SASQAG 2/17/00

Where is rework fixed?

ENTRY

EXIT


ENTRY

EXIT


ENTRY

EXIT


ENTRY

EXIT


Requirements

FinishedProduct


Time

ProjectPlan





[email protected]

SASQAG 2/17/00

The Quality Lever

Requirements

Design

CodingDevelopment

TestingAcceptance

Testing

ImproveProduct

Production40-100X

30-70X15-40X

10X3-6X

1X

Data from Boehm: Software Engineering Economics, 1981

[email protected]

SASQAG 2/17/00

The Quality Lever

Requirements

Design

CodingDevelopment

TestingAcceptance

Testing

ImproveProduct

Production40-100X

30-70X15-40X

10X3-6X

1X


HighVisibility/Reward

[email protected]

SASQAG 2/17/00

The Quality Lever

Requirements

Design

CodingDevelopment

TestingAcceptance

Testing

ImproveProduct

Production40-100X

30-70X15-40X

10X3-6X

1X


HighVisibility/Reward

LowVisibility/Reward

[email protected]

SASQAG 2/17/00

Error/Defect Classification

Number of Errors

Minor Major "ShowStoppers"

ErrorSeverity

[email protected]

SASQAG 2/17/00

Error/Defect Classification

Number of Errors

Minor Major "ShowStoppers"

Find and remove as many Major and Show Stopper errors as early as possible.

ErrorSeverity

[email protected]

SASQAG 2/17/00

Terminology

• Major ErrorMajor Error- Something that is wrong

- High degree of probability that it will cause rework

- Rework later will cost much more if left allowed to migrate downstream.

• Minor ErrorMinor Error An issue or defect that is not major.

The costs of fixing later is not significantly higher than fixing now.

[email protected]

SASQAG 2/17/00

Quality Metric…Rework

• Number of Major Errors / Unit of Number of Major Errors / Unit of WorkWork

- ME/Page

- ME/Drawing Zone

- ME/KLOC

- ME/?

[email protected]

SASQAG 2/17/00

Peer Reviews

[email protected]

SASQAG 2/17/00

Initial Level

InformalChecking

DoWork

Down-stream

Customers

InputSource

Materials

ErrorsFound

ProcessProduct

EntryCriteriaLooselyDefined

ExitCriteriaForQualityNotDefined

Work Process

ENTRY

EXIT

[email protected]

SASQAG 2/17/00

Initial Level

InformalChecking

DoWork

Down-stream

Customers Re-Work

InputSource

Materials

ErrorsFound

ProcessProduct


ExitCriteriaForQualityNotDefined

Work Process

ENTRY

EXIT

[email protected]

SASQAG 2/17/00

Measure and Improve Product Quality

DoWork

Down-stream

CustomersRe-

Work

ErrorsFound

ProcessProduct

CrossfunctionalChecklists

SW Inspection

InputSource

Materials


ExitCriteriaForQualityDefined

ENTRY

EXIT

Work Process

[email protected]

SASQAG 2/17/00

Improve Product Quality and Reduce Variation

DoWork

Down-stream

CustomersRe-

Work

ErrorsFound

ProcessProduct


InputSource

Materials

EntryCriteriaDefined

ExitCriteriaForQualityDefined

ENTRY

EXIT

Work Process

SW Inspection

[email protected]

SASQAG 2/17/00

To Improve Product AND Process

(EntryCriteria

Met)

DoWork

Down-stream

CustomersRe-

Work

InputSource

Materials

ErrorsFound

ProcessProduct


ExitCriteriaMet

Statistical DataAnalysis and

Causal Analysis

PD

CA

DemingCycle

(Defect Prevention Process)

Repeatable Work Process

SW Inspection

[email protected]

SASQAG 2/17/00

SW Inspection Process Flow

Planning

Collect andStore Data

AnotherChunk?

Verify andApprove

Authors Moderator Inspection Team Members

OK

Re-Work

YES

NO

Other Inspection andCI Processes

ProduceProduct

Rework

ProductRelease

IndividualPreparation

OverviewMeeting

ErrorLoggingMeeting

7

[email protected]

SASQAG 2/17/00

SDLC

ENTRY

EXIT

Output...ENTRY

EXIT

Output

InputInput





Time

[email protected]

SASQAG 2/17/00

SW Inspections

ENTRY

EXIT

Output...ENTRY

EXIT

Output

InputInput





• SW Inspection• SW Inspection • SW Inspection• SW Inspection

[email protected]

SASQAG 2/17/00

Kickoff Meeting

ENTRY

EXIT

Output...ENTRY

EXIT

Output

InputInput





TaskKickoffMeeting

TaskKickoffMeeting

TaskKickoffMeeting

TaskKickoffMeeting


[email protected]

SASQAG 2/17/00

Task Kickoff MeetingTask Kickoff Meeting

• Identify people/organizations affected by errors in the activity

• Identify and document potential major errors (build/modify checklists)

• Confirm task/process entry conditions

• Identify resources

• Forum for learning

Reduce the amount of time needed by the author(s) to produce a deliverable by giving

technical and resource information to do the job.

[email protected]

SASQAG 2/17/00

Kickoff Meeting

ENTRY

EXIT

Output...ENTRY

EXIT

Output

InputInput



TaskKickoffMeeting

TaskKickoffMeeting

TaskKickoffMeeting

TaskKickoffMeeting




[email protected]

SASQAG 2/17/00

Walkthrough

ENTRY

EXIT

Output...ENTRY

EXIT

Output

InputInput



• Walkthrough• Walkthrough



• Walkthrough• Walkthrough

21

TaskKickoffMeeting

TaskKickoffMeeting

TaskKickoffMeeting

TaskKickoffMeeting


[email protected]

SASQAG 2/17/00

Walkthrough ProcessWalkthrough ProcessReduce the amount of time needed for re-work

by reviewing work completed to date and address questions and concerns.

• Led by author

• Discussion of deliverable style and format

• Examine technical alternatives

• Evaluate checklists, standards, and rules.

• Capture undocumented issues

• Forum for learning

[email protected]

SASQAG 2/17/00

Peer Review Methods

WalkthroughWalkthrough

Methods Typical Goals Typical Attributes

• Minimal overhead• Developer training• Quick turnaround

•Little/no preparation•Informal process•Meetings•No measurement•Led by the author

SW SW InspectionInspection

• Detect and report all defects efficiently and effectively.

• Usage of deliverable

•Formal process•Known Coverage Rate•Moderator•Checklists•Customer Reviewers•Measurement

Desk Desk CheckCheck

• Minimal overhead• Quick turnaround

•Little/no preparation•Informal process•No measurement•No Meetings•Done by the author

[email protected]

SASQAG 2/17/00

SQA Interventions

OutputENTRY

EXIT

Input Program ElementCreation

Start During End

• Kickoff Meeting

[email protected]

SASQAG 2/17/00

SQA Interventions

OutputENTRY

EXIT


Start During End

• Kickoff Meeting

• Walkthrough

• Desk Check

[email protected]

SASQAG 2/17/00

SQA Interventions

OutputENTRY

EXIT


Start During End

• Kickoff Meeting

• Walkthrough

• Desk Check

• SW Inspection

[email protected]

SASQAG 2/17/00

SQA Interventions

OutputENTRY

EXIT


Start During End

• Kickoff Meeting

• Walkthrough

• Desk Check

• SW Inspection• Formal

Walkthrough• Formal Desk

Check

[email protected]

SASQAG 2/17/00

Peer Reviews

ENTRY

EXIT

Output...ENTRY

EXIT

Output

InputInput



TaskKickoffMeeting

TaskKickoffMeeting

TaskKickoffMeeting

TaskKickoffMeeting

• Walkthrough• Desk Check




• SW Inspection• Formal Walkthrough• Formal Desk Check






[email protected]

SASQAG 2/17/00

Audits

[email protected]

SASQAG 2/17/00

Audit ProcessAudit ProcessSoftware Quality Assurance is to provide management with appropriate visibility into the process being used by the software developers and of the products being built.

• Determine:

• Conformance to policies, procedures, and standards• Adequacy of policies, procedures, and standards• Effectiveness and efficiency of policies, procedures, and

standards• Assess personnel familiarity to requirements and

documentation• Assure availability, use and adherence to software

standards

[email protected]

SASQAG 2/17/00

Audit Process

Un-satisfactory

Report?

OK

NO

YES

ProduceProduct

CorrectiveActions

AuditKickoffMeeting

DevelopAudit

Checklist

ConductAudit

Developers Auditor & Project Manager

Write-upReport &Findings

Auditor

IdentifyRequirements

Follow-upAudit

Re-Work

CloseAudit &

File END

Start

Reviewwith

Manager

[email protected]

SASQAG 2/17/00

Checklists & Worklists

[email protected]

SASQAG 2/17/00

Checklists

• Used to identify Major Errors• Usually in the form of questions• When used….

- By the authors to keep from inserting rework into the work stream

- By reviewers to find and report as many Major Errors as possible

- By future authors and reviewers

[email protected]

SASQAG 2/17/00

Generic Document Checklist

• Purpose Has the document satisfied the stated purpose?

• Redundant Is the same information presented elsewhere within the document?

• Acronyms Have all acronyms been defined prior to use?

• Accuracy Is the information wrong?

• Extraneous Is there information outside the stated scope of the document?

• Completeness Is necessary information missing?

• Standard Does the document follow the xyz documentation standard?

[email protected]

SASQAG 2/17/00

Generic Document Worklist• Before creating the deliverable...

- Hold Kickoff Meeting• Invite the right participants

• Plan for a successful document

• Build/Refine Checklists

• While working on the deliverable…- Use the checklists

- Use Walkthroughs and Desk Checks as needed to reduce rework (major errors).

• When finished with the deliverable…- As per SQA plan, perform the appropriate Peer review

- Send peer review data summary to SQA focal

[email protected]

SASQAG 2/17/00

For More Information

[email protected]

SASQAG 2/17/00

Resources...• SASQAG WEB Site

(sasqag.org)-Past Meetings-Links

• UW Certificate Course• Commercial Courses• SASQAG Training Courses

[email protected]

SASQAG 2/17/00

Goals and Objectives

• Some Terms• Peer Reviews• Kickoff Meetings• Audits• Checklist and

Worklists

Sharpen YourSQA Skills

tom gilchrist [email protected] the tools and techniques of sqa sasqag, february 17, 2000

Documents