software process assessment using sei’s software...

ment apability

ineering

Slide -1

Software Process AssessUsing SEI’s Software CMaturity Model

Neal S. CoulterCollege of Computing Sciences and EngUniversity of North Florida

ractices

om 1987-1990 vings.

ngineering pared with 89% , 25% of total y the customer; rted.

Slide -2

Can Software Development PImprove Dramatically? - 1

•Hughes Aircraft spent $445,000 frand realized $2,000,000 annual sa

•Schlumberger completed 94% of eprojects on schedule in 1992, comin 1991 and 51% in 1990. In 1989product defects were discovered bin 1991, 10% were customer repo

ractices

dollar invested ivity and hired eet the demand

e in SLOC per fect density livered code to costs dropped ost overruns for

8 to 3% in 1991.

Slide -3

Can Software Development PImprove Dramatically? - 2

•Raytheon realized $7.70 for everyand a two-fold increase in product100 additional staff members to mfor new business. •Raytheon realized a 190% increasperson-month from 1988-1996; dedropped from 17.2 per KLOC of de4.0 per KLOC. In two years reworkfrom 41% of project cost to 20%. Cprojects dropped form 40% in 198

ments?

rger and oth- and modify-

ing the I) Software

the improve-

Slide -4

What Caused these Improve

Hughes, Raytheon, Motorola, Schlumbeers reported such changes after assessinging its software development process usSoftware Engineering Institute’s (SECapability Maturity Model (CMM).

Hence, CMM could be the reason for ments!

?

ue of Level 2 are

3”

Slide -5

What’s In It for the Developers

Less stress“The only ones questioning the valthose who have not achieved it”“A coherent culture exists at Level

Better morale

Less turnover

Less overtime

Job security

stitute

f Defense (DoD)

iversity advancing the ering to improve on software

Slide -6

The Software Engineering In

•Operated by the US Department o•Opened in 1984 in Pittsburgh•Affiliated with Carnegie Mellon Un•Mission is to provide leadership instate of practice of software enginethe quality of systems that depend

re Pro-

tracts found missed by 20 billion A12 e to software

urse.

Slide -7

Motivation for an SEI Softwacess Model - 1

A review of 17 major DoD software conthat the average 28-month schedule wasmonths; no project was on time. The $58aircraft program was canceled mainly duproblems.

Many similar stories are common, of co

re Pro-

rge a gap verage current

h military soft-problems but tanding of soft-lopment as a pro-

g complexity and

Slide -8

Motivation for an SEI Softwacess Model - 2

DoD concluded “few fields have so labetween current best practices and apractice...today’s major problems witware development are not technical management problems... the undersware as a product and software devecess is not keeping pace with growinsoftware complexity...”

developing a brief descrip-mary archi-

Slide -9

History of CMM - 1

In November 1986, SEI and Mitre beganprocess maturity framework. In 1987, a tion of the model was released by its pritect, Watts Humphrey.

Methods were developed forSoftware process assessments (SPA)Software capability evaluation (SCE)

s improve-

alified to per-

in evaluating ftware.

Slide -10

History of CMM - 2

SPAs are intended for an internal procesment program

SCEs are used to appraise contractors quform work

A questionnaire was developed to assistorganizations’ abilities to develop so

work into the CMM, now

ce

.cmu.edu)

Slide -11

History of CMM - 3

In 1991, SEI evolved the maturity frameCapability Maturity Model for SoftwareVersion 1.1.

The CMM• Is based on actual practices• Reflects the best state of the practi• Reflects the needs of individuals• Is documented• Is publicly available (http://www.sei

-Based

eds of profes-

total agreement

Slide -12

Benefits and Risks of ModelImprovement - 1

Benefits•Provides a common language•Supports measurement•Based on collaboration with hundrsionals•Represents a consensus, but not

-Based

ls are useful” -

hes on some le and technol-

ains, advocate ow to hire,

Slide -13

Benefits and Risks of ModelImprovement - 1

Risks•All models are wrong; some modeGeorge Box•Not comprehensive; it barely toucnonprocess factors, such as peopogy•Does not address application domspecific technologies, or suggest htrain, and retain people.

ent (TQM) e goals of

M does not delighted) at the soft-satisfy the the requirements of the total sys-re development nship.

Slide -14

CMM and TQM

CMM is built on Total Quality Managemprinciples. The goal of TQM is to meet ththe customer, now and in the future. CMstate the customer should be satisfied (orwith the software product. It does state thware supplier should build products that customer’s needs as documented in allocated to the software component tem or product being supplied. Softwais usually on part of a business relatio

ING

MATURITY

LEVELS

Slide -15

The Basic CMM ModelOPTIMIZ

5

MANAGED4

DEFINED3

REPEATABLE2

INITIAL1

Continuouslyimproving

process

PredictableProcess

Standard,consistent

process

Disciplinedprocess

for a given

ethods, prac-ploy to .

xpected rocess.

ual results s.

Slide -16

Some Basic Definitions - 1

Process - A sequence of steps performedpurpose. What you do.

Software process - A set of activities, mtices and transformations that people emdevelop and maintain software products

Software process capability - Range of eresults by following a certain software p

Software process performance - The actachieved by following a software proces

o which a ntrolled and r growth in s of an orga-cy of its applica-

Slide -17


Software process maturity - The extent tspecific process is defined, measured, coeffective. Maturity implies a potential focapability and indicates both the richnesnization’s process and the consistention across projects.

nary plateau cess. Each goals that, rt of the soft-e maturity nent in the

e in the pro-

Slide -18


Maturity level - A well-defined evolutiotoward achieving a mature software promaturity level comprises a set of processwhen satisfied, stabilize an important paware process. Achieving each level of thframework establishes a different composoftware process, resulting in an increascess capability of the organization.

nd Risk RESULT

ty

Risk

Slide -19

CMM, Productivity, Quality, a

5Optimizing

4Managed

3Defined

2Repeatable

1Initial

LEVEL

Productiviand

Quality

cterized as ad processes are al efforts and

nt processes nd functional-n place to similar appli-

Slide -20

CMM Level Summaries - 1

1. Initial - The software process is charahoc, and occasionally even chaotic. Fewdefined and success depends on individuheroics.

2. Repeatable - Basic project managemeare established to track cost, schedule, aity. The necessary process discipline is irepeat earlier successes on projects withcations.

h manage-ented stan-oftware e organization’s

Slide -21


3. Defined - The software process for botment and engineering activities is documdardized, and integrated into a standard sprocess for the organization. Called thstandard software process.

oftware pro-d used for

rovement is he process echnologies.

Slide -22


4. Managed - Detailed measures of the scess and product quality are collected ananalysis and control.

5. Optimizing - Continuous process impenabled by quantitative feedback from tand from piloting innovative ideas and t

l 5,

f the 379 organi-anced enough to

s in place and ments, 73% don’t

Slide -23

So, Everyone but Us is LeveRight?

According to SEI’s reportable data, ozations at 99 companies that are advhave process improvement programhave conducted SEI maturity assessrate higher than Level 1.

The others have not applied.

vels - 1EL 4

sses are ta-under-and zed

es of ual ms are stood imi-

LEVEL 5

Processes are continuously and systemat-ically improved

Common sources of problems are understood and elimi-nated

Slide -24

Advancing Through CMM Le

PROCESS

LEVEL 1

Few stable processes exist or are used

“Just do it”

LEVEL 2

Documented and stable estimating, planning, and commitment processes at the project level

Problems are recognized and corrected as they occur

LEVEL 3

Integrated management and engineer-ing processes are used across the organization

Problems are anticipated and pre-vented, or their impacts are mini-mized

LEV

Procequantitively stood stabili

Sourcindividprobleunderand elnated

vels - 2EL 4

sense mwork within roject

LEVEL 5

Strong sense of teamwork exits across the organiza-tion

Everyone is involved in process improvement

Slide -25


PEOPLE

LEVEL 1

Success depends on individual heroics

“Fire fighting” is a way of life

Relationships between disci-plines are uncoordi-nated, perhaps even adversary

LEVEL 2

Success depends on individuals; management system sup-ports

Commitments are understood and managed

People are trained

LEVEL 3

Project groups work together, per-haps as an integrated product team

Training is planned and provided according to roles

LEV

Strongof teaexistseach p

vels - 3EL 4

chnol-re ed on itative

LEVEL 5

New technol-ogies are pro-actively pursued and deployed

Slide -26


TECHNOLOGY

LEVEL 1

Introduction of new tech-nology is risky

LEVEL 2

Technology supports established, stable pro-cesses

LEVEL 3

New technol-ogies are evaluated on a qualitative basis

LEV

New teogies aevaluata quantbasis

vels - 4L 4

fini- col-

are dized he ation

e used rstand ess -

nd sta-

LEVEL 5

Data are used to evaluate and select pro-cess improve-ments

Slide -27


MEASUREMENT

LEVEL 1

Data collec-tion and anal-ysis is ad hoc

LEVEL 2

Planning and management data used by individual projects

LEVEL 3

Data are col-lected and used in all defined pro-cesses

Data are sys-tematically shared across projects

LEVE

Data detion andlection standaracross torganiz

Data arto undethe procquantitatively abilize it

productive tion for the undation fail

under stress--ent.

t is character- they can not

er-level orga-ul in moving

Slide -28

Skipping Levels

Trying to skip maturity levels is counterbecause each level is a necessary foundanext level. Processes without a proper foat the same time they are needed most--and provide no basis for future improvem

Many developers have collected data thaistic of Level 4 organizations only to finduse it. However, some attributes of highnizations, such as an SEPG, can be helpfup levels.

ity Level

en repeating roduce

Target

Time/$/...

evel 3

Slide -29

Process Capability by Matur

A process is under statistical control whthe work in roughly the same way will proughly the same results.

Level 1 Target

Time/$/...

Probability

Level 5Target

Time/$/...

Probability

Probability

L

Levels

.

y practices

e

Slide -30

Internal Structure of Maturity

Each maturity level has constituent parts

Maturity levels

Key process areas

Common features

Ke

Processcapability

Goals

Implementation orinstitutionalization

Activities orinfrastructure

Indicate

Achieve

Address

Describ

ents - 1

elated activi-ieve a set of

process capa- processes for atisfied and

lture and ctices, poli-ngoing way e underlying

Slide -31

Some Maturity Level Compon

A key process area (KPA) is a cluster of rties that, when performed collectively, achgoals considered important for enhancingbility. To achieve a maturity level, the keythat level (and the lower levels) must be sthe processes must be institutionalized.

Institutionalization is the building of a cuinfrastructure to support the methods, pracies, and procedures so that they are the oof doing business. The infrastructure is thframework of an organization or system.

ents - 2

practices and ganization has als signify the A.

actices. Key structure that ntation and practices not mandate nted.

Slide -32

Some Maturity Level Compon

The goals of each KPA summarize its keycan be used in determining whether an oreffectively implemented the KPA. The goscope, boundaries, and intent of each KP

Each KPA is described in terms of key prpractices describe the activities and infracontribute most to the effective implemeinstitutionalization of the KPAs. The keydescribe “what” is to be done, but do “how” the process should be impleme

s

organized by lementation

ective, repeat-res are:

, sponsorship...ining...dures, track-

ple measure-

s, audits...

Slide -33

Components of Key Practice

The practices that describe the KPAs arecommon features that indicate if the impand institutionalization of a KPA are effable, and lasting. The five common featu

•Commitment to perform - Policies•Ability to perform - Resources, tra•Activities performed - Plans, proce

ing...•Measurement and analysis - Sam

ments...•Verifying implementation - Review

nge managementanagement

Slide -34

KPAs by Maturity Level - 1

Initial

Organization process focusOrganization process definitionTraining programIntegrated software managementSoftware product engineeringIntergroup communicationPeer reviews

Quantitative process managementSoftware quality management

Managed

Defined

Repeatable

Requirements managementSoftware project planningSoftware tracking and oversightSoftware subcontract managementSoftware quality assuranceSoftware configuration management

Defect preventioTechnology chanProcess change m

Optimizing

Sl

K

L

L project-man-a

L anizational is infrastructure th ngineering a ojects.

ide -35

PAs by Maturity Level - 2

evel 1 - No KPAs

evel 2 - Focuses on establishing basic gement controls.

evel 3 - Addresses both project and orgsues, as an organization establishes an at institutionalizes effective software e

nd management processes across all pr

titative ess and the

anizations continuous

Slide -36

KPAs by Maturity Level - 3

Level 4 - Focuses on establishing a quanunderstanding of both the software procsoftware work products being built.

Level 5 - Focuses on issues that both organd projects must address to implement and measurable process improvement.

me>

X> is

mmary>cess X as

Slide -37

KPA Template - 1

<Key Process Area X>

a key process for level n: <Level na

The purpose of <Key Process Area <statement>

<Key Process Area X> involves <su<Additional elaboration on Key Pro appropriate>

as goal>

written for <X>

Slide -38

KPA Template - 2

Goals

Goal 1 <Process summary statement...

Commitment to Perform

Commitment 1 The project follows aorganizational policy

...

nsible for

in Key Pro-ng to a docu-

Slide -39

KPA Template - 3

Ability to Perform

Ability 1 A group that is respo<X> exists

...

Activities PerformedActivity 1 <Activity performed

cess Area X> accordimented procedure

...

ade to deter- activities

> are manage-asis

Slide -40

KPA Template - 4

Measurement and Analysis

Measurement 1 Measurements are mmine the status of thefor <X>

...Verifying Implementation

Verification 1 The activities for <Xreviewed with seniorment on a periodic b

ble

Management of products e project’s soft-

gement

Slide -41

...SCM Template - 1

A key process area for Level 2: Repeata

The purpose of Software Configuration is to establish and maintain the integrityof the software project throughout thware life cycle....

GoalsGoal 1 Software configuration mana

activities are planned

written for imple-

ity for man-e exists or is

Slide -42

...SCM Template - 2

Commitment to PerformCommitment 1 The project follows a

organizational policymenting SCM

...

Ability to PerformAbility 1 A board having the author

aging the project’s baselinestablished

...

r each soft-a docu-

ade to deter- SCM activi-

Slide -43

SCM Template - 3

Activities PerformedActivity 1 A SCM plan is prepared fo

ware project according to mented procedure

...

Measure and AnalysisMeasurement 1 Measurements are m

mine the status of theties

...

re reviewed ent on a

Slide -44

SCM Template - 4

Verifying ImplementationVerification 1 The SCM activities a

with senior managemperiodic basis

...

1

hat requires sts of the

re

hs and weak-

of the organiza-

Slide -45

CMM-Based Assessments -

Complete assessment is a detailed task torganization wide participation. It consiphases:•Select a team•Complete the maturity questionnai•Analyze the responses•Conduct a detailed site visit•Produce a list of findings of strengtnesses•Prepare a key process area profiletion

2

3

ResponseAnalysis

6

KPAProfile

Slide -46

CMM-Based Assessments -

1

TeamSelection

2

MaturityQuestionnaire

samplesthe

CMM

5

Findings

based on the CMM

4

On-SiteVisit

Interviews and documentreviews

all key practices

ther

n’t Know

Slide -47

Maturity Questionnaire

•Based on CMM Version 1.1•Organized by CMM KPAs•Covers all 18 KPAs•Addresses each KPA goal but not •6 to 8 questions per KPA•Can be completed in one hour•No standard scoring scheme•Identifies issues to be explored fur

Each question can be answered: Yes, No, Does not Apply, Do

Also place for comments

oject?

and made ough the use

nizational anagement

Slide -48

SCM Maturity Questionnaire

Has eight questions:

1. Are SCM activities planned for the pr

2. Has the project identified, controlled,available the software work products throf configuration management?...5. Does the project follow a written orgapolicy for implementing configuration mactivities?...

software process assessment using sei’s software...

Documents