andy moyer. cleanroom software engineering what is it? goals properties of cleanroom cleanroom...

CLEANROOM SOFTWARE

ENGINEERING

Andy Moyer

Cleanroom Software Engineering What is it? Goals Properties of Cleanroom Cleanroom Technologies Case Studies Critiques

Cleanroom Software Engineering: What is it? Set of principles and practices for the

specification, development, and certification of software-intensive systems.

Based on the work of Harlan D. Mills who was an employee of IBM in early 1980s.Influenced by Dijkstras structured programming,

Nicholas Wirth on stepwise refinement, and David Parnas on modular design

Methods based on two principles:Programs are rules for mathematical functionsSoftware testing is sampling

Where the Idea Came From

The cleanroom model follows the ideas of manufacturing semiconductors in a “cleanroom” environment.

Goals of the Cleanroom Process

Main goal: Achieve or approach zero defects

Other goals:Prevent defectsCreate correct, verifiable softwareIncremental development

Properties of Cleanroom

Spend a lot of time and money "up-front" preventing defects

Use statistical methods to ensure quality Formally state and “prove” requirement

needs Specification, development, and

certification may be done by separate teams depending on the size of the project

Cleanroom Technologies

1. Incremental development under Statistical Process Control

2. Function-Based Specification, Design, and Verification

3. Correctness Verification

4. Statistical Testing and Software Verification

Incremental Development Under Statistical Process Control Used to reduce the risk associated with

managing large systems by focusing on smaller, manageable subsystems of increments.

Each increment is developed and tested separate from the whole system before integrating into the whole system.

As increments are added, the whole system is tested.

Some amount of end-user function used to obtain customer confirmation or clarification

Incremental Development Under Statistical Process Control

Increment should have these properties:Externally executableContain all functionality of prior increments

Benefits:Specification, design, and certification

activities may be performed oftenTimely feedback can be obtained from

customerIntellectual control over a system

Cleanroom Technologies

1. Incremental development under Statistical Process Control

2. Function-Based Specification, Design, and Verification

3. Correctness verification

4. Statistical Testing and Software Verification

Function-Based Specification, Design, and Verification

Objectives:Expand the specification into

implementation via small stepsMaintain intellectual control by designing

data and control structures at appropriate levels of abstraction

The Three Boxes:Black BoxState BoxClear Box

Function-Based Specification, Design, and Verification

The Box StructuresBegins with an external view – Black BoxTransformed into a state machine – State

BoxFully developed into a procedure – Clear

Box

The Black Box

A view of an object that hides data implementation and process implementation. It describes how a system responds to stimuli, usually in a formal specification language.

No internal state is looked at Our good friend – Z (Zed)

The State Box

A view of an object that shows data implementation but hides process implementation. It describes how "state" information is transformed.

Derived from the black box – first step to implementation

This is represented as a finite state machine

The Clear Box

A view of an object that shows both data implementation and process implementation. The goal is to stepwise refine procedures and to prove them correct.

Derived from the state box May introduce new black boxes defining

major operations Creates the required responses (outputs)

to stimuli (inputs)

Stepwise Refinement of Specifications to Code Steps:

1. Expand clear box into lower level designs

2. Verify the correctness of the expanded designs

3. Expand low level designs into code

Cleanroom Technologies

1. Incremental development under Statistical Process Control

2. Function-Based Specification, Design, and Verification

3. Correctness verification

4. Statistical Testing and Software Verification

Correctness Verification The primary debugging process for

Cleanroom Each box structure subject to correctness

verification Objectives:

Determine that the box structures correctly implements design

Remove any errors that were introduced during development

Completely review the code for completeness, consistency, and correctness

Correctness Verification

So what is verified?If the box structure behaves as definedCorrectness of each refinementResponse mapping defined in one step is

preserved

Cleanroom Technologies

1. Incremental development process model

2. Stepwise refinement of specifications to code

3. Correctness verification of developed code

4. Statistical Testing and Software Verification

Statistical Testing and Software Verification

Each increment of compilable functionality is statistically tested to:Ensure that it meets the quality standards

defined by the development organizationCertify a level of reliability that the product

will deliver in the fieldProvide feedback for quality control and

process improvement

Statistical Testing and Software Verification

Steps taken:1. Plan the test

○ Stratification plan which describes each layer to be developed

○ The sampling plan○ A description of which physical testing

resources are to be devoted to each increment and each stratum

2. Develop usage models required by the test plan

3. Develop and validate the usage distribution

Example Usage Model

NASA Case Study

In March of 1990 NASA preformed a case study of SEL Cleanroom versus their typical process

Each group applied the process to the same project – a Coarse/Fine Attitude Determination Subsystem of the Upper Atmosphere Research Satellite.

Completed system contained about 34,000 Source Lines of Code (SLOC) of primarily FORTRAN

NASA Case Study-SEL Cleanroom Lifecycle

NASA Case Study-Experience Comparison

NASA Case Study

Testers and developers are on completely separate teams

Developers have no access to the mainframe computer for compilation and testing

Developers rely on code reading instead of unit testing to verify correctness

Testers use a statistical testing approach

NASA Case Study - Results

NASA Case Study - Results Failure Rate:

Standard SEL process – 6 errors/KSLOCCleanroom SEL procees – 3.3

errors/KSLOC The inexperience of the cleanroom team

had no effect on the outcome Impact of unstable requirements

lessened by concentrated effort in team reviews

DoD Case Study

The STARS program is a US Department of Defense (DoD) reasearch and development program

Emphasizes:Process drivenRe-use basedIntegrated software engineering

environment

DoD Case Study Typical Process

Productivity measured at 121 loc/person month

Failure rate: 23~27 failures/KLOC

Cleanroom ProcessProductivity

measured at 559 loc/person month

Failure rate: 1 failure/KLOC

DoD Case Study

Important benefits:Improved ProductivityImproved qualityHigh staff morale

Critiques

In 1997, Beizer argued the elimination of unit testing is contradicting “known testing theory and common sense”

Also, is it possible to find bugs without compiling/running code?

References 1.  Cleanroom Software Engineering. Retrieved

from University of Texas at Arlington website: http://www.uta.edu/cse/levine/fall99/cse5324/cr/clean/page1.html

2.  DACS, The Data and Analysis Center for Software. Retrieved January 5, 2009  Found at: www.dacs.dtic.mil/databases/url/key.php? keycode=64

References 3.  Green, Scott et al. (1990). "The

Cleanroom Case Study in the Software Engineering Laboratory: Project Description and Early Analysis". NASA.  Retrieved from http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19910008271_1991008271.pdf

4.  Prowell, Stacy et al. (1993). "Cleanroom Software Engineering: Technology and Process". Reading, MA: Addison Wesley Longman, Inc.