by tom gilchrist, csqa, csqe, software process reviews/audits process overview

by

Tom Gilchrist, CSQA, CSQE,

Software Process Reviews/AuditsSoftware Process Reviews/Audits

Process OverviewProcess Overview

SASQAG 10/17/2002 tomg@tomgtomg.com 2

Before we start…

• SQA Context

• Overview of SW Audit Process

• SW Audit Examples

Information in this presentation are my opinions and not necessary those of my employer.

SASQAG 10/17/2002 tomg@tomgtomg.com 3

Some Terms/Ideas

• Process

• Deterministic vs. Non Deterministic

• Quality vs. Value

SASQAG 10/17/2002 tomg@tomgtomg.com 4

Software Quality Assurance

• Check software products and processes to verify that they comply with the applicable procedures and standards. (Process Reviews or Audits)

• Review and measure the quality of software products and processes throughout development. (Dynamic & Static Testing)

• Provide software project management (and other appropriate parties) with the results of reviews and process checks.

• Work with the software project during early stages to establish plans, standards, and procedures to keep errors from occurring in the first place.

SASQAG 10/17/2002 tomg@tomgtomg.com 5

Formal Definition

Audits provide an independent evaluation of software products or processes to ascertain compliance to standards, specifications, and procedures based on objective criteria that included documents that specify:

– The form or content of the product to be produced

– The process by which the products shall be produced

– How compliance to standards or guidelines shall be measured.

IEEE STD 1028, (1988)

SASQAG 10/17/2002 tomg@tomgtomg.com 6

Audit Types

• First Party AuditFirst Party Audit– Within you company or organization

• Second Party AuditSecond Party Audit– Sometimes called “external audits”– By a Customer on his Supplier– By a Supplier on you.

• Third Party AuditThird Party Audit– Outside third party is contracted to do

the audit.

SASQAG 10/17/2002 tomg@tomgtomg.com 7

Audit/Process Review Principles

• Conducted by individuals who are organizationally independent of the developers.

• Begin early in the requirements phase and continue throughout the development process.

• Professionally planned, conducted and documented.

• Follow-up on corrective action.• Project Management is involved in the Audit

process and is responsible for rework and process improvements.

SASQAG 10/17/2002 tomg@tomgtomg.com 8

What Software Audit Should Do

• Determine:• Compliance to requirements• Conformance to plans, policies, procedures, and

standards• Drive process improvement based on:

• Adequacy of plans, policies, procedures, and standards

• Effectiveness and efficiency of plans, policies, procedures, and standards

• Assess personnel familiarity to requirements and documentation

• Assure availability, use and adherence to software standards

SASQAG 10/17/2002 tomg@tomgtomg.com 9

What Triggers an Audit?

• Quality Assurance Plan• Event• Date

• Requests from management• Requests from developers• Requests from customers• Integration with process improvement activities• Outside requirements — regulatory• Gut feel

SASQAG 10/17/2002 tomg@tomgtomg.com 10

Scope: Requirements, Time, and Target

Audit

Target

External

Standards

Organizational

Procedures and

Methods

• Spread around organization

• Cover all functions and activities

• Try to hit things early

• Move towards process audits

SASQAG 10/17/2002 tomg@tomgtomg.com 11

Process Review/Audit Process

OK

PrepareAudit

Developers Project ManagerAuditor

ConductAudit

Write-upReport &Findings

Follow-upAudit

Re-Work

Findings?

NO

YES

CloseoutAudit &

File END

Reviewwith

Manager

Plan(Requirements,

Scope, & Checklist)Start

CorrectiveActions

SASQAG 10/17/2002 tomg@tomgtomg.com 12

Identify Requirements

• Policies/Standards Corporate, Group, IEEE• Processes/Plans SCMP, SQAP, SDP, Project Plan• Procedures Change Management, Design

Reviews, Document Standards,

Testing • Task Instructions Library updates, unit testing, peer

reviews

• Success of an audit is directly proportional to preparation, research and analysis conducted before the audit is performed.

SASQAG 10/17/2002 tomg@tomgtomg.com 13

Requirement Types

• Functional (ascertainably true or false)• Quality (range of acceptable values)

SASQAG 10/17/2002 tomg@tomgtomg.com 14

Types of Audits (Internal)

• Quality System Audits

• Product Audit

• Process Audit

• Project Audit

• CM Audit

SASQAG 10/17/2002 tomg@tomgtomg.com 15

Evidence Collection

• Collect Factual Information• Analyze and Evaluate the Evidence• Draw Conclusions• Generate Findings

SASQAG 10/17/2002 tomg@tomgtomg.com 16

Corrective Action of Findings

• Determine Action– Immediate Remedial Action– Process Improvement/Fix– Acceptable Risk

• Identify Root Cause• Corrective Actions Plan • Manage CA Plan to completion• Analyze Effects of CA

SASQAG 10/17/2002 tomg@tomgtomg.com 17

Develop Audit Checklist

• Focus on clear requirements (or unclear to fix)

• Select subset of requirements• Focus on important steps/products• Write clear concise questions• Canned checklist vs. straw horse

SASQAG 10/17/2002 tomg@tomgtomg.com 18

Checklist Sample

Requirement Checklist Item Details Observations Results (P/F)

Company Standard ABC-234, page 7

Does project QA plan will have a list of deliverables subject to Peer Reviews?

Check SQA document for a list of approved peer reviews and which documents are to be reviewed. (if no documents are found, then fail. If no peer review procedures are referenced, then fail)

Project SQA Plan

Were the number of audits completed equal to the number planned?

Check to see which audits were planned for the last 60 days. Check for evidence that the audit was completed and if there were findings, that a CA plan was signed.

Project SQA Plan

Were the number of peer reviews completed equal to the number planned?

For each peer review type, check the CM records for the past 60 days to see if the document type specified in the QA plan was checked into CM for the first time. If so, check for records of the peer review being completed as per peer review process cited in SQA plan.

SASQAG 10/17/2002 tomg@tomgtomg.com 19

Interviewing

• Ask open-ended questions• Know the types of answers expected• Focus on Process and not People• Seek Corroboration and Evidence

SASQAG 10/17/2002 tomg@tomgtomg.com 20

Sample Interview Questions

• How do you track your progress?• Do you have a CM Plan?• Tracing

– What are you working on?– Is it a configured item?– Do you have an approved CR or PR?– Is the version you are working on

checked out of CM?

SASQAG 10/17/2002 tomg@tomgtomg.com 21

Desirable Auditor Characteristics

• EmotionalEmotional• Interviews• Group

dynamics• Oral reports• Empathy• Don’t take

things personally

• MechanicalMechanical• Sampling• Root Cause

Analysis• IntellectualIntellectual

• Writing• Planning• Speaking• Detail

Oriented• Concise

SASQAG 10/17/2002 tomg@tomgtomg.com 22

Desirable Auditor Characteristics(Cont.)

• Knowledge of Audit process• Knowledge of target (SW) processes• Knowledge of techniques • Professional attitude• Good listener• Inquisitive/analytical• Communicates at all levels• Detailed Notes and Observations• Diplomatic