sixsigma and cmmi everything you wanted to know, but didn’t know who to ask

Strive to be

Level 5

1

SixSigma and CMMIEverything You Wanted to Know,

But Didn’t Know Who to Ask

Tom Lienhard

Six Sigma BlackBelt

Strive to be

Level 5

2

I have 90 Minutes

Today’s Mission

• QUICK Introduction to Six Sigma

• Relationship of Six Sigma to the CMMI

• Applying Six Sigma

• Examples of Six Sigma Tool Usage to Improve

BlackBelt training is 160 hrs Intro to CMMI is 24 hrs

Strive to be

Level 5

3

What Do You Think Six Sigma Is All About?

Benchmark?

Method?

Metric?

Vision?Philosophy?

Stretch Goal?

Pipe Dream? Disease?

Strive to be

Level 5

4

What Is Six Sigma?

A customer-driven management methodology to significantly

increase customer satisfaction through reducing and eliminatingvariation and thus defects.

A concept developed by Motorola to achieve a desired level of quality, virtually defect-free products and processes

With this definition, six sigma is applicable to any development or manufacturing process

Strive to be

Level 5

5

The Many Facets of Sigma

Sigma is a letter inthe Greek alphabet “Sigma” is used to designate the

distribution about the mean of any process or product characteristic(standard deviation)

“Sigma value” indicates how often defects are

likely to occur

“Sigma level” is the number of standard deviations between the process center and the CLOSEST spec limit

Strive to be

Level 5

6

Premise for Six Sigma

• Sources of variation can be• Identified• Quantified• Mitigated by control or prevention

• The elements of a process responsible for variation are called the “6 M’s”• Man (people)• Machine• Material• Method• Measurement• Mother Nature

Strive to be

Level 5

7

Variation is the Enemy

• Variation exists in everything

• If it is assumed that everything is a result of some process, then it is safe to conclude that the process introduces product variation

• Variation in the product is due to variation in the process

• Variation comes in two forms - common and special cause

We Must Slay the Enemy

Strive to be

Level 5

8

Common and Special CausesCommon Causes• Continuously active in the process• Predictable

Special Cause• Extraordinary events• Can assign a reason• Can do something about it

87654321

11

10

9

8

7

6

5

4

3

2

X=7.4

3.0SL=9.2.

-3.0SL=5..6

Special causes

Common causes

Control limits reflect the built in or natural process variationControl limits are not related to standards or specificationThe control limits allow predictions about the future to be made

Strive to be

Level 5

9

0

2000

4000

6000

8000

10000

12000

14000

6/1

4/9

9

6/2

8/9

9

7/1

2/9

9

7/2

6/9

9

8/9

/99

8/2

3/9

9

9/6

/99

9/2

0/9

9

10

/4/9

9

10

/18

/99

11

/1/9

9

11

/15

/99

11

/29

/99

12

/13

/99

12

/27

/99

1/1

0/0

0

1/2

4/0

0

2/7

/00

2/2

1/0

0

3/6

/00

3/2

0/0

0

We

ekl

y S

tan

dar

d H

ou

rs

Demonstrated

UCL

2 SIGMA

1 SIGMA

MEAN

1 SIGMA

2 SIGMA

LCL

Planned

Historical output to stores in standard hours

99.7% probability that output of an “in-control”process will fall between the upper control limit(UCL) and the lower control limit (LCL).

68% probability that output of an “in-control”process will fall between the 1 sigma limits.

95% probability that output of an “in-control”process will fall between the 2 sigma limits.

Planned requirements in standard hours

Average historical output

1

2

3

4

5

8

6

7

1 One point more than three sigmas from the mean

Tests for out-of-control conditions in a control chart

2 Nine points in a row on the same side of the mean

3 Six points in a row all increasing or decreasing

4 Fourteen points in a row, alternating up and down

5

6 Four out of five points more than 1 sigma from the mean

7 Fifteen points in a row within 1 sigma of the mean

8 Eight points in a row more than 1 sigma from the mean

Two out of three points more than 2 sigmas from the mean

Not shownControl limits are calculated based on the week-to-week variation of the demonstrated output

The control limits are used as a predictor of future output if the historical output is in control (i.e. does not fail any of the 8 tests below) and the process has not changed significantly

Violation of any of the 8 tests below by the plannedrequirements indicates a nearly zero probability thatthe actual output can meet the plan. A majorprocess change or re-plan is required.

Shows the difference between planned and actual for a given week

Reading a Control Chart

Strive to be

Level 5

10

largest source of variation (components of variation, analysis of variance)

factors which have the greatest influence on variation (Process Map, FMEA, Screening Design of Experiments)

where to set factors to reduce variation and maximize the output (Optimizing Design of Experiments)

Six Sigma is All About Variation

Six Sigma tools help identify

types of variation (control chart) 6s

87654321

110

100

90

80

70

60

50

40

30

20

1

1

1

X=74.46

3.0SL=92.36

-3.0SL=56.55

Special causes

Common causes

Strive to be

Level 5

11

Six Sigma Tool Box – Example Usage

• Thought Process Map• Process Map• Failure Modes and Effect Analysis• Pareto Chart• Control Chart• Confidence Intervals• Product Scorecard• Postmortem• Measurement System Evaluation• Regression Analysis• Design of Experiments• Main Effects Plot• Interaction Plot

6s

Strive to be

Level 5

12

Identify variation

Reduce or eliminate variation

Maximize output and deliver less defects

Satisfied Customer

Customer Satisfaction Through Reducing Defects

Characterize variation

Strive to be

Level 5

13

Six Sigma is Based on Statistics

Mean (Process Center)

1 Standard Deviation

Strive to be

Level 5

14

Process CenterThe distribution of Our Process output

REWORKREWORK GOOD GOOD

Upper SpecificationLimit

Lower SpecificationLimit

Determined by the customer


Incapable Process

• We have REWORK because some of what we make is waste

• Process is WIDER than the specification limits

Capable Process

• We make as much as the customer needs and have very little waste

• Process FITS within the specification limits

States of a Process

Process Center Upper SpecificationLimit

Lower SpecificationLimit



GOOD GOOD

Strive to be

Level 5

15

PCB

PCB : Process Capability Baseline = “normal” variation for the sample

If you take the bell shaped curve and turn it on its side, you have a control chart

X

X : Sample Mean = arithmetic average of a set of values (process center)

s

: s Standard Deviation = the square root of the variance

UCL

UCL : Upper Control Limit = +3s from the Sample Mean

LCL

LCL : Lower Control Limit = -3s from the Sample Mean

X Bar What? Standard Who?

Strive to be

Level 5

16

Four Possible States of a Process

Spec. limit

Control LimitCapable but

Unstable

Brink of Chaos

Unstable andnot Capable

Chaos

Stable butnot Capable

Threshold

Stable andCapable

Ideal

Strive to be

Level 5

17

So Long As We Satisfy the Requirements, Right?

Pilot A Pilot B

Which pilot would you want to fly with?

Both within spec, same average landing

Why?

Strive to be

Level 5

18

LowerSpec Upper

Spec

LowerSpec

UpperSpec

Target Target

ProcessCenter

ProcessCenter

4s

s s ss

6s

s s s ss

1 StandardDeviation

s

1 StandardDeviation

Pilot A is at 4 sigmaIn order to be six sigma, like Pilot B, the

variation in Pilot A’s landings must be reduced

Moving Towards Six Sigma

Pilot A’s Distribution Pilot B’s Distribution

Strive to be

Level 5

19

Pilot A and B at Sky Harbor Airport

750

463

100

9

0.3

0.005

0 100 200 300 400 500 600 700 800

1

2

3

4

5

6

Sigma Level

Bad Landings / Day

0.005

Pilot A

Pilot B

Wouldn’t stay in business too long...

Strive to be

Level 5

20

Higher Sigma Equates toMajor Reduction in Defects

500,000

308,500

66,810

6,210 233 3.40

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

450,000

500,000

1 2 3 4 5 6

Sigma Level

Def

ects

/ M

illi

on

Op

po

rtu

nit

ies

Strive to be

Level 5

21

Six Sigma -The Other Story

1s - 170 misspelled words per page in a book

2s - 25 misspelled words per page

3s - 1.5 misspelled words per page

4s - 1 misspelled word per 30 pages

5s - 1 misspelled word in a set of encyclopedias

6s - 1 misspelled word in all the books in a library

Understand you customers’ needs And balance with cost

Strive to be

Level 5

22

CMMI Meets Six Sigma

Strive to be

Level 5

23

CMMI Reminder

Optimizing

Managed

Defined

QuantitativelyManaged

Level Predicted Performance

Time/$/Quality/...

Time/$/Quality/...

Time/$/Quality/...

Time/$/Quality/...

the best practices reside in the projects

the organization has a defined common process which includes the collection of product and process metrics

all the process assets accumulated from Level 2 and 3 are used to quantitatively understand the project

the organization concentrates on improving the process to gain an advantage in quality, productivity, and timeliness

Strive to be

Level 5

24

Level 2 - Measure the Project

The initial focus for measurement activities is at the project level. As the organization moves toward Level 3, the measures prove useful for addressing organization and/or enterprise-wide information needs.

Measurement and Analysis (and GP 2.8) Monitor and control the [PA Name] process against the plan for performing the process and take corrective action

Strive to be

Level 5

25

Level 3 – Understand the Organizational Practices

At Level 3, project data is collected and analyzed by the organization. Thresholds are set based on good engineering judgment which trigger corrective action

Level 2/3 set the stage for quantitative management (statistical process control).• Repeatable, Consistent Process• Adequate Resources• Defined, Consistent Process Measures• Data Collected From Above Process• Assurance of Process Compliance

Strive to be

Level 5

26

Level 4 - Stabilizing the ProcessAt maturity level 4, an organization applies quantitative and statistical techniques to manage process performance and product quality. Quality and process performance is understood in statistical terms. Special causes of process variation are identified and, where appropriate, the sources of special causes are corrected to prevent future occurrences.

87654321

110

100

90

80

70

60

50

40

30

20

1

1

1

X=74.46

3.0SL=92.36

-3.0SL=56.55

Special causes

Design Peer Review

CodePeer Review

TestPeer Review

Formal TestPeer Review

Tota

l # o

f D

efec

ts

Life Cycle Phase

Customer Before Delivery

Req’ts Peer Review

Strive to be

Level 5

27

Level 4 - Stabilizing the Process

When performance falls outside normal range of process performance• Identify the reason

• Take corrective action when appropriate

A critical distinction between maturity level 3 and 4 is the predictability of process performance. At maturity level 4, the

performance of processes is controlled using statistical and other quantitative techniques, and is quantitatively predictable.

Strive to be

Level 5

28

Level 5 - Optimizing the Process

At maturity level 5, an organization continually improves its processes based on a quantitative understanding of the common causes of variation inherent in processes.

When performance is within normal range of process performance but unable to meet the established objectives• Narrow the performance limits• Shift the performance limits

87654321

150

100

50

0

X=74.46

3.0SL=136.0

-3.0SL=12.91

Common causes

Strive to be

Level 5

29

Am I Acting 4 or 5?

A critical distinction between levels 4 and 5 is the type of process variation addressed.

• At maturity level 4, the organization is concerned with addressing special causes of process variation and providing statistical predictability of the results. Although processes may produce predictable results, the results may be insufficient to achieve the established objectives.

• At maturity level 5, the organization is concerned with

addressing common causes of process variation and changing the process (to shift the mean of the process performance or reduce the inherent process variation experienced) to improve process performance and to achieve the established quantitative process-improvement objectives

Strive to be

Level 5

30

Six Sigma allows the organization to statistically analyze whether a process change was actually an improvement to the normal range of process performance

Level 5-Statistical Process Improvement

Level 5 involves continuous improvement to the processes with the goal of improving quality, productivity and decreasing cycle time

Old Process New Process

Cyc

le T

ime

Strive to be

Level 5

31

Six Sigma Tools Enable Levels 4&5

The Six Sigma methodology provides the tools that implement

• Quantitative Project Management,• Organization Process Performance• Causal Analysis and Resolution • Organizational Performance Management

A Level 5 organization is one that has implemented, among other things, statistical process control (SPC)

6s

Strive to be

Level 5

32

Applying Six Sigma

Develop Infrastructure

Quantitatively UnderstandProcess Performance

ContinuousMeasurable

Improvement

PROCESS

Strive to be

Level 5

33

Develop Infrastructure

• Commitment to Process Improvement• Repeatable, Consistent Process• Adequate Resources• Defined, Consistent Process Measures• Data Collected From Above Process• Assurance of Process Compliance

Only required for the window you are looking through

Strive to be

Level 5

34

Quantitatively Understand Process Performance

Use Six Sigma tools to identify sources of variation (quantitatively understand process performance) and then act to eliminate or reduce those sources

Measurements of processperformance on project

Quantitatively understand the process performance

Discover sources of variation

Determine the magnitude of the effects due to process changes

Implement improvements

Monitor the impact

Strive to be

Level 5

35

Six Sigma – Real Life Example

1-11-11-11-11-1

80

60

4080

60

4080

60

4080

60

4080

60

40

Program

Experience

Training

Criteria

Num People

1

-1

1

-1

1

-1

1

-1

1

-1

Interaction Plot (data means) for % Match

Determine the magnitude

8765432Subgroup 1

1101009080706050403020

Sam

ple

Mea

n

1

1

1

X=74.46

3.0SL=92.36

-3.0SL=56.55

Monitor the impact

PlanningCustomerRqmts. Analysis

DesignImplementation

Test Formal Test

Customer BeforeTOTALLeaked

Planning2.03 0 0 0 0 0 0 0 2.03 0Customer0 1.8 1.4 0 4.06 0 16.15 0 23.41 21.61Rqmts. Analysis0 0 7.32 12.04 32.77 41.6 56.09 0.79 150.61143.29Design 0 0 0.13 41.99 8.2 23.2 118.94 5.28 197.74155.75

Implementation 0 0 0.17 0.5 154 90.3 88.88 23.3 357.15203.15Test 0 0 0 0.16 0.03 19.92 4.5 0 24.61 4.69

Formal Test 0 0 0 0 0 2.34 149.25 0 151.59 2.34

Customer Before 0 0 0 0 0 0 2.7 13.6 16.3 13.6

TOTAL2.03 1.8 9.02 54.69 199.06177.36436.5142.97923.44544.43

Measurements

8765432Subgroup 1

1101009080706050403020

Sam

ple

Mea

n

1

1

1

X=74.46

3.0SL=92.36

-3.0SL=56.55

Statistically understand

Strive to be

Level 5

36

Continuous Measurable Improvement

Measurements of processperformance of project

Statistically understand the process capability


Determine the magnitude of theeffects due to process changes

Implement improvementsMonitor the impact

Measurements of processperformance of project

Statistically understand the process capability


Determine the magnitude of theeffects due to process changes

Implement improvementsMonitor the impact

• Can be next biggest hitter• Can be next quickest• Can be next logical problem• Can be anything the org. determines

Strive to be

Level 5

37

Six Sigma Tool Box – Example Usage

• Thought Process Map• Process Map• Failure Modes and Effect Analysis• Pareto Chart• Control Chart• Confidence Intervals• Product Scorecard• Postmortem• Measurement System Evaluation• Regression Analysis• Design of Experiments• Main Effects Plot• Interaction Plot

6s

Strive to be

Level 5

38

Thought Process Map

Problem to be solved

What is theprocess?

Process Mapcurrent process

What are thepossible

weaknesses?

FMEA/CEPostmortems

What is the currentdata ?

Gather data What data?

What about older projects?No, data not available

Data from

New projectsOrganize data to

effectively analyze

What data?What want to know?Estimate rework

by phase

How do wemeasure theprocess?

Look for differentrequirementsDifferent customersMultiple perspective

Involve multiple projectsMultiple disciplines

BARRIER-Lack of data?

Red - Question or expected result Blue - Answer or actual result

BARRIER-Lack of data?

Is there one?How Determine?

What arex's, y's?

Howtell?

Data forFMEA

Underlined - Barrier

How well is itworking now?

What is datashowing us?

Refine estimate as site databecomes available

Drill down into thedata/NEM/Control

Charts

Is the data anygood?

Validate MeasurementSystem MSE(KAPPA/ICC

or Nested Design)

DOE

Update process toensure data has

higher confidencerate/ train

NotAdequate

What areimportant factors?

BARRIER-People's time

Determine actionBased on data Make and

communicateimprovements

Did change causeimprovement?

Set up controlplan and Use

Control Charts toMonitor

Effectiveness ofImprovements

Adequate

NewslettersLiaison Meetings

BARRIER-Projects not

required to follow?



Choice of factors

Question to answer

Stable process

Yes

Identify andRemove Special

Causes

No

CharacterizeOptimizeProcess

Strive to be

Level 5

39

Process Map

Activity/Sub-process

C - InputN - InputS - InputX - Input

OutputOutputOutput

Rework

Defects?

Rework

X - Critical (statistical proven critical)N - Noise (can't or choose not to control)S - SOP (the standard way to do it)C - Controllable (can be changed to see effect)

Activity/ProcessInputs Outputs

Rework

50,000 ft level

500 ft level





Rework

Defects?

No No

C - InputN - InputS - InputC - Input

X - InputN - InputS - InputX - Input

C - InputS - InputS - InputS - Input

C - InputN - InputS - InputX - Input

OutputOutput

Output

OutputOutput

OutputOutputOutput

Strive to be

Level 5

40

Process or Product Name:

In-Phase Peer ReviewPrepared by: Tom Lienhard & Team

Page ____ of ____

Responsible:Tucson Software Engineering Process Group

FMEA Date (Orig) 23 Sept 99 (Rev) 29 Sept 99

Process Step/Input

Potential Failure Mode Potential Failure EffectsSEV

Potential CausesOCC

Current ControlsDET

RPN

Actions Recommended

What is the process step/ Input under

investigation?

In what ways does the process step go wrong?

What is the impact on the Key Output Variables (Customer Requirements) or internal requirements?

What causes the process step to go wrong?

What are the existing controls and procedures (inspection and test) that prevent eith the cause or the Failure Mode? Should include an SOP number.

What are the actions for reducing the

occurrance of the Cause, or improving detection? Should

have actions only on high RPN's or easy

fixes.

Peer Review Planning

No Review Team identified upfront w/review package(roles & responsibilities)

Product not reviewed by appropriate disciplines

6

SW plans do not require this

1

SEPG/SQA and peer review of plans

1 6

None

6

Lack of process awareness

6

Moderator to ensure review package complete (contains reviewers)

6 216

Re-train moderator and conduct process evaluations

No product evaluation criteria identified with review package

Product not reviewed to customer and/or process requirements

9

SW plans do not require this

1

SEPG/SQA and peer review of plans 1 9

None

9


6

Moderator to ensure review package complete (contains review criteria)

6 324


No notice or agenda with review package

Team not able to give adequate review time 6

SW plans do not require this1


None

6


6

Moderator to ensure review package complete (contains notice and agenda)

6 216


Product Review Product not reviewed Defects not found

9

Adequate time not given to review

6

Moderator and SQD ensure adequate time was given to review product 3 162

None - cultural thing

9

No or inadequate evaluation criteria

6

Moderator to ensure review package complete (contains evaluation criteria)

9 486

Create generic checklists for site (to highest level) and conduct process evaluations

9

Inappropriate reviewers

6

Moderator to ensure appropriate reviewers

9 486

Update command media to identify required participants on notice & agenda and conduct process evluations

Metrics not captured Organization quantitative data incorrect/incomplete 3

SW plans do not require this1


None

3


9

Moderator to ensure review process followed (metrics captured)

6 162


Process/Product Failure Modes and Effects Analysis

(FMEA)

1 3

Severity9 - Defects go to customer6 - Defects cause rework3 - Data not collected1 - No harm/no foul

Occurrence:9 - Regular occurrence6 - Occurs more than occasionally3 - Occurs occasionally1 - Rare occurrence

Detection:9 - Nothing in place6 - Based on individuals3 - Check in place, usually works1 - Check in place & working

FMEA

Strive to be

Level 5

41

Potential causes (factors) which kept showing up over and over on the FMEA:• Inappropriate review team (“wrong” moderator, dominant, inexperienced, or

yes-people made up the team)• Lack of process awareness (both unintentional and deliberate)• No or inadequate review criteria (review what is there not what is missing,

biased review based on experience with phase)

Plan to minimize the occurrence and increase the detection :• Update the process to highlight required participants, their roles and

responsibilities on the Notice and Agenda• Roll-out Peer Review training• Perform peer review process evaluations• Generate common evaluation criteria for all work products that can be used

across the entire organization

Use what was learned about factors as an input into DOE:

What Was Learned

Strive to be

Level 5

42

Planning Customer Rqmts.

Analysis

Design Implement

ation

Test Formal

Test

Customer

Before TOTAL LeakedPlanning

2.03 0 0 0 0 0 0 0 2.03 0Customer

0 1.8 1.4 0 4.06 0 16.15 0 23.41 21.61Rqmts.

Analysis 0 0 7.32 12.04 32.77 41.6 56.09 0.79 150.61 143.29Design

0 0 0.13 41.99 8.2 23.2 118.94 5.28 197.74 155.75Implement

ation 0 0 0.17 0.5 154 90.3 88.88 23.3 357.15 203.15Test

0 0 0 0.16 0.03 19.92 4.5 0 24.61 4.69Formal

Test 0 0 0 0 0 2.34 149.25 0 151.59 2.34Customer

Before 0 0 0 0 0 0 2.7 13.6 16.3 13.6

TOTAL 2.03 1.8 9.02 54.69 199.06 177.36 436.51 42.97 923.44 544.43

Phase Detected

Ph

ase

Intr

odu

ced

Data has been changed to protect the innocent

Cost of Rework due to defects-in days

Strive to be

Level 5

43

$0

$50,000

$100,000

$150,000

$200,000

$250,000

$300,000

$350,000

$400,000

Plannin

g

Custom

er

Reqs A

nalys

is

Design

Implem

enta

tion

Test

Formal T

est

Custom

er B

efor

e

Phase Detected

Cu

mu

lati

ve C

ost

s

Existing Detection

Improved Detection

Existing defect detection profile

Goal defect detection profile

Same number of total defects introduced in the same phases

IMPACT ONBOTTOM LINE

Cost Savings Goal

Strive to be

Level 5

44

100

90

80

70

60

50

40

30

20

Per

cent

Def

e cts

Fou

nd I

n P

hase

Goal

Plan

ning

Cu s

t . Pl

an

Req

’ts

Des

i gn

Imp l

em.

Test

Form

al T

est

Cu s

t om

er

These 3 phases accountfor > 92% of rework

Distribution Characteristics of Data

Strive to be

Level 5

45

2520151050

1.0

0.5

0.0

Sample Number

Pro

portio

n

P Chart for In Phase

P=0.7678

3.0SL=1.000

-3.0SL=0.00E+00

P Chart – Determines Stability

But we sampled only 3 projects - what does the population look like?

Strive to be

Level 5

46

25 26 27 28 29 30 31

95% Confidence Interval for Mu

20 25 30 35

95% Confidence Interval for Median

Variable: Req

A-Squared:P-Value:

MeanStDevVarianceSkewnessKurtosisN

Minimum1st QuartileMedian3rd QuartileMaximum

20.6813

1.6737

25.0000

0.3380.200

28.6667 3.214610.3333

-1.54539

3

25.000025.000030.000031.000031.0000

36.6521

20.2026

31.0000

Anderson-Darling Normality Test


95% Confidence Interval for Sigma


Descriptive Statistics

60 70 80 90 100


0 20 40 60 80 100 120 140


Variable: Cust1

A-Squared:P-Value:



7.512

13.237

55.000

0.4460.081

70.666725.4231646.3331.72000

3

55.000 55.000 57.000100.000100.000

133.821

159.777

100.000






55 60 65 70 75


40 50 60 70 80 90


Variable: Design

A-Squared:P-Value:



42.2518

4.8377

55.0000

0.2590.384

65.3333 9.291686.3333

-1.18512

3

55.000055.000068.000073.000073.0000

88.4149

58.3951

73.0000






65 67 69 71 73


60 70 80


Variable: Implem

A-Squared:P-Value:



58.6271

2.1042

65.0000

0.2120.536

68.6667 4.041516.3333

0.722109

3

65.000065.000068.000073.000073.0000

78.7062

25.3995

73.0000






82 84 86 88 90 92


72 82 92 102


Variable: test

A-Squared:P-Value:



73.985

2.868

82.000

0.1930.616

87.6667 5.507630.3333-2.7E-01

3

82.000 82.000 88.000 93.000 93.000

101.348

34.614

93.000






97 98 99


96 97 98 99 100 101


Variable: Form Tst

A-Squared:P-Value:



95.516

0.521

97.000

0.1890.631

98 110

3

97.000 97.000 98.000 99.000 99.000

100.484

6.285

99.000






Confidence Intervals

Strive to be

Level 5

47

phase

pe

rce

nt

Planning Cust. Reqs Design Implem Test FormalTst

Cust.

40

90

140

0

If I were a betting man…. the true population means are within these intervals

7

134

21

3742

88

59

78

74

101

96

100

14

139

100

Goal

…Gives Us Population Range

Strive to be

Level 5

48

Analysis of Variance for percent

Source DF SS MS F Pphase 7 10841.9583 1548.8512 8.701 0.000project 16 2848.0000 178.0000Total 23 13689.9583

Variance Components

Source Var Comp. % of Total StDevphase 456.950 71.97 21.376project 178.000 28.03 13.342Total 634.950 25.198

Phase variationProject variation

ANOVA Tells Us the Variation is Between Subgroups

Strive to be

Level 5

49

If no action is taken 95% confident that• the Requirements Phase will find between 21% - 37% of defects in phase• the Design Phase will find between 42% - 88% of defects in phase• the Implementation Phase will find 59% - 78% of defects in phase

Variation between the phases (72%) is greater than variation between projects (28%)• need to work largest source of variation - what changed between, what didn’t, etc.

80 %World Class

100 %

SW Phase

Green -customer processRed - < 92 5 of reworkBlue - testing process

What Was Learned

Strive to be

Level 5

50

StdOrder RunOrder Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 % Match1 1 -1 1 -1 -1 -1 705 2 -1 -1 -1 -1 1 553 3 -1 -1 -1 1 -1 307 4 -1 1 -1 1 1 602 5 -1 -1 1 -1 -1 706 6 -1 1 1 -1 1 804 7 -1 1 1 1 -1 658 8 -1 -1 1 1 1 609 9 1 -1 -1 -1 -1 4513 10 1 1 -1 -1 1 5515 11 1 -1 -1 1 1 3511 12 1 1 -1 1 -1 5514 13 1 -1 1 -1 1 8010 14 1 1 1 -1 -1 8012 15 1 -1 1 1 -1 5516 16 1 1 1 1 1 70

Fractional Factorial DesignFactors: 5 Base Design: 5, 16 Resolution: VRuns: 16 Replicates: 1 Fraction: 1/2Blocks: none Center pts (total): 0

Design of Experiment (DOE)

Strive to be

Level 5

51

20100-10

1

0

-1

Effect

No

rmal

Sco

re

C

A

B

Normal Probability Plot of the Effects(response is % Match, Alpha = .10)

A: ExperienB: TrainingC: CriteriaD: Num Peop

151050

B

C

A

AB

AD

ACD

AC

D

CD

BD

BCD

BC

ABD

ABC

Pareto Chart of the Effects(response is % Match, Alpha = .10)

A: ExperienB: TrainingC: CriteriaD: Num Peop

Shows Training, Criteria, Experienceas the influential factors

Data looks pretty normal

Strive to be

Level 5

52

Fractional Factorial FitEstimated Effects and Coefficients for % (coded units)

Term Effect CoefConstant 60.312 Program -1.875 -0.938 Experien 13.125 6.562 Training 19.375 9.687 Criteria -13.125 -6.562 Num Peop 3.125 1.562 Program*Experien -1.875 -0.937 Program*Training 4.375 2.187 Program*Criteria 1.875 0.937 Program*Num Peop -1.875 -0.937 Experien*Training -5.625 -2.812 Experien*Criteria 4.375 2.188 Experien*Num Peop -4.375 -2.187 Training*Criteria -1.875 -0.938 Training*Num Peop 1.875 0.937 Criteria*Num Peop 1.875 0.937

Analysis of Variance for % (coded units)

Source DF Seq SS Adj SS Adj MS F PMain Effects 5 2932.8 2932.8 586.56 * *2-Way Interactions 10 440.6 440.6 44.06 * *Residual Error 0 0.0 0.0 0.00Total 15 3373.4

Shows same thingTraining, Criteria, Experience

as the influential factors

Strive to be

Level 5

53

1-1 1-1 1-1 1- 1 1- 1

70

65

60

55

50

% M

atc h

Main Effects Plot (data means) for % Match

Process pretty robust tothese factors

Factor 1 Factor 2 Factor 3 Factor 4 Factor 5

Want to set these factors “high”

DOE Main Effects Plot

Strive to be

Level 5

54

1-11-11-11-11-1

80

60

4080

60

4080

60

4080

60

4080

60

40

1

-1

1

-1

1

-1

1

-1

1

-1Factor 1

Factor 2

Factor 3

Factor 4

Factor 5

DOE Interaction Plot

Strive to be

Level 5

55

Experience is a no brainer - more is better

Training is no brainier - have to have it!

Number of people was eye opening - didn’t make too much difference Since more people cost more money, keep it at 2

Criteria was a shock

Need to do follow-up since I was not present to take copious notes

Follow-up revealed that having criteria limited the scope of the review,reviewed only what was there, did not use criteria as intended

What Was Learned

Strive to be

Level 5

56

Strive to be

Level 5

57

AFTER IMPROVEMENTS

PHASE DETECTED

PHASEPlanning Customer Rqmts.

Analysis

Design Implementa

tion

Test Formal

Test

Customer

Before TOTAL

IPlanning

235 2 0 3 0 0 0 0 240

NCustomer

0 22 22 7 4 0 0 0 55

TRqmts.

Analysis 0 1 161 13 7 2 0 0 184

RDesign

0 0 3 173 111 2 0 0 289

OImplementa

tion 0 0 0 2 342 20 0 1 365

DTest

0 0 0 2 2 22 1 0 27

UFormal

Test 0 0 0 0 0 0 2 1 3

CCustomer

Before 0 0 0 0 0 0 0 3 3

D TOTAL 235 25 186 200 466 46 3 5 1166

Planning Customer Rqmts. Analysis

Design Implementation

Test Formal Test

Customer Before TOTAL

Planning29 0 0 0 0 0 0 0 29

Customer0 12 2 0 2 0 5 0 21

Rqmts. Analysis 0 0 61 14 29 26 71 1 202Design

0 0 1 323 82 29 38 2 475Implement

ation 0 0 1 5 220 43 44 10 323Test

0 0 0 2 1 249 30 0 282Formal

Test 0 0 0 0 0 13 597 0 610Customer Before 0 0 0 0 0 0 1 4 5

TOTAL 29 12 65 344 334 360 786 17 1947

Number of defects identified by phase introduced/phase detected (from Product Scorecard)

Number of defects identified by phase introduced/phase detected (from Product Scorecard)

Strive to be

Level 5

58

AFTER IMPROVEMENTS

PHASE DETECTED

PHASEPlanning Customer Rqmts.

Analysis

Design Implementa

tion

Test Formal

Test

Customer

Before TOTAL

IPlanning

98% 1% 0% 1% 0% 0% 0% 0% 21%

NCustomer

0% 40% 40% 13% 7% 0% 0% 0% 5%

TRqmts.

Analysis 0% 1% 88% 7% 4% 1% 0% 0% 16%

RDesign

0% 0% 1% 60% 38% 1% 0% 0% 25%

OImplementa

tion 0% 0% 0% 1% 94% 5% 0% 0% 31%

DTest

0% 0% 0% 7% 7% 81% 4% 0% 2%

UFormal

Test 0% 0% 0% 0% 0% 0% 67% 33% 0%

CCustomer

Before 0% 0% 0% 0% 0% 0% 0% 100% 0%

D TOTAL 20% 2% 16% 17% 40% 4% 0% 0% 100%

0%

0%10%

Planning Customer Rqmts.

Analysis

Design Implement

ation

Test Formal

Test

Customer

Before TOTALPlanning

100% 0% 0% 0% 0% 0% 0% 1%Customer

0% 57% 0% 10% 0% 24% 1%Rqmts.

Analysis 0% 0% 30% 7% 14% 13% 35% 0% 10%Design

0% 0% 0% 68% 17% 6% 8% 0% 24%Implement

ation 0% 0% 0% 2% 68% 13% 14% 3% 17%Test

0% 0% 0% 1% 0% 88% 11% 0% 14%Formal

Test 0% 0% 0% 0% 0% 2% 98% 0% 31%Customer

Before 0% 0% 0% 0% 0% 0% 20% 80% 0%

TOTAL 1% 1% 3% 18% 17% 18% 40% 1% 100%

Percentage of defects identified by phase introduced/phase detected (from Product Scorecard)

Percentage of defects identified by phase introduced/phase detected (from Product Scorecard)

Strive to be

Level 5

59

PHASE DETECTED

PHASEPlanningCustomerRqmts.

AnalysisDesignImplementa

tionTest Formal

TestCustomer

Before TOTAL Leaked

IPlanning

16.45 2 0 0.51 0 0 0 0 18.96 2.51

NCustomer

0 3.3 15.4 7 8.12 0 0 0 33.82 30.52

TRqmts. Analysis 0 1 19.32 11.18 7.91 3.2 0 0 42.61 23.29

RDesign

0 0 0.39 22.49 11.1 1.6 0 0 35.58 13.09

OImplementa

tion 0 0 0 0.2 239.4 42 0 2.33 283.93 44.53

DTest

0 0 0 0.16 0.06 1.76 0.15 0 2.13 0.37

UFormal Test 0 0 0 0 0 0 0.5 0.58 1.08 0.58

CCustomer

Before 0 0 0 0 0 0 0 10.2 10.2 10.2

D TOTAL 16.45 6.3 35.11 41.54 266.59 48.56 0.65 13.11 428.31 125.09

Planning Customer Rqmts. Analysis

Design Implementation

Test Formal Test

Customer Before TOTAL Leaked

Planning2.03 0 0 0 0 0 0 0 2.03 0

Customer0 1.8 1.4 0 4.06 0 16.15 0 23.41 21.61

Rqmts. Analysis 0 0 7.32 12.04 32.77 41.6 56.09 0.79 150.61 143.29Design

0 0 0.13 41.99 8.2 23.2 118.94 5.28 197.74 155.75Implement

ation 0 0 0.17 0.5 154 90.3 88.88 23.3 357.15 203.15Test

0 0 0 0.16 0.03 19.92 4.5 0 24.61 4.69Formal

Test 0 0 0 0 0 2.34 149.25 0 151.59 2.34Customer

Before 0 0 0 0 0 0 2.7 13.6 16.3 13.6

TOTAL 2.03 1.8 9.02 54.69 199.06177.36436.51 42.97 923.44 544.43

AFTER IMPROVEMENTS

Cost of Rework due to Defects in Days (from Product Scorecard)

Cost of Rework due to Defects in Days (from Product Scorecard)

Strive to be

Level 5

60

AFTER IMPROVEMENTS

87654321

1 0 0

9 0

8 0

7 0

6 0

5 0

4 0

3 0

2 0

C 4 4

Ba

se

line

Three Phases

87654321

1 0 0

9 0

8 0

7 0

6 0

5 0

4 0

3 0

2 0

C 5 1

Imp

ro

ve

d Three Phases

95% Confident That Two of he ThreePhase Will be World-Class. The ThirdHad Drastic Improvement

95% Confident That Two of he ThreePhase Will be World-Class. The ThirdHad Drastic Improvement

What happened here? Remember weare measuring % not #. We went from282 to 27, a huge improvement in rework $ and effort, but % wise it was a decline

Strive to be

Level 5

61

AFTER IMPROVEMENTS

Rework Effort in Dollars by Phase

$0

$50,000

$100,000

$150,000

$200,000

$250,000

Plannin

g

Custom

er

Rqmts. A

nalys

is

Design

Implem

enta

tion

Test

Formal T

est

Custom

er B

efor

e

TOTAL

Software Development Phase

Do

llar

s

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Leaked Rework

In-Phase Rework

% Leaked

Cost of Rework due to DefectsCost of Rework due to Defects

Rework Effort in Dollars by Phase

$0

$50,000

$100,000

$150,000

$200,000

$250,000

Software Development Phase

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Leaked Rework

In-Phase Rework

% Leaked

Strive to be

Level 5

62

$0

$50,000

$100,000

$150,000

$200,000

$250,000

$300,000

$350,000

$400,000

Planning Customer Reqs Analysis Design Implementation Test Formal Test CustomerBefore

Phase Detected

Existing Detection

Goal

Actual

Baseline defect cost profile

Improved defect cost profile

Goal defect cost profile

Bottom Line SavingsBottom Line Savings

$120,000 in six months on just 3 software projects

(Notice the increased up front spending)

Strive to be

Level 5

63

AFTER IMPROVEMENTS

Three Phase Where Project Concentrated(Three Phases With > 92% of Rework)Three Phase Where Project Concentrated(Three Phases With > 92% of Rework)

Recalculating the control limitsto see if there is significance….

654321

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

Sample Number

Pro

port

ion


P=0.5751

3.0SL=0.6845

-3.0SL=0.4658

654321

1.0

0.9

0.8

0.7

0.6

0.5

0.4

Sample Number

Pro

port

ion


P=0.6582

3.0SL=0.7419

-3.0SL=0.5745

654321

1.0

0.9

0.8

0.7

0.6

0.5

Sample Number

Pro

port

ion


P=0.8145

3.0SL=0.8756

-3.0SL=0.7535

Strive to be

Level 5

64

654321

1.0

0.5

0.0

Sample Number

Pro

port

ion

P Chart for In Phase by Group

P=0.87503.0SL=0.9481

-3.0SL=0.8019

2

Control Limits for Requirements AnalysisPhase has no Overlap Whatsoever

Control Limits for Requirements AnalysisPhase has no Overlap Whatsoever

654321

1.0

0.9

0.8

0.7

0.6

0.5

0.4

Sample Number

Pro

port

ion


P=0.6252

3.0SL=0.7106

-3.0SL=0.5398

2

Design Phase is More Interesting

Design Phase is More Interesting

654321

1.0

0.9

0.8

0.7

0.6

0.5

Sample Number

Pro

port

ion


P=0.93733.0SL=0.9754

-3.0SL=0.8993

2

Control Limits for ImplementationPhase Have Significantly Changed

With Little Overlap

Control Limits for ImplementationPhase Have Significantly Changed

With Little Overlap

Range is much larger which widens the control limits.

However, looking at the scorecard, • only 1% of defects made it to test,

whereas before it was 14%. • Rework due to leakage was a

mere 13 days compared to 156 days!

Strive to be

Level 5

65

Make decisions based on data. Experience is only one input parameter and can steer you wrong

Let the Six Sigma tools work for you to narrow the scope and identify where to apply resources

A few simple process changes resulted in BIG impact to bottom line!

This project spun off others team which helped meet the customers’ needs

Quick look at charts show that the process is now in control

Greatest source of variation is now within the subgroups, not between them• Project to project variation is larger than phase to phase variation

The three phases that were concentrated upon improved greatly• Two of the three are world-class

Just measuring one aspect of the process does not tell the whole story• Finding a larger percentage of defects in phase may not be

cost effective

Conclusion

Strive to be

Level 5

66

New Challenge Was Presented

Strive to be

Level 5

67

478

239

14

280

0

100

200

300

400

500

600

Jul Aug Sep Oct Nov Dec

Requirements to review Requirements reviewed

Design reviewed Code reviewed

Item 17

Peer Review Tracking

Strive to be

Level 5

68

AS900 Peer Review Effort

0

10

20

30

40

50

60

70

80

90

100

1 5 9 13

17

21

25

29

33

37

41

45

49

53

57

61

65

69

73

77

81

85

89

93

97

101

105

Review Number

Effort per Review

Avereage Effort

Overall Peer Review Effort

12.6

Strive to be

Level 5

69

10050Subgroup 0

200

100

0Ind

ivid

ual V

alue

1

1 1 1 1 1

X=12.633.0SL=42.63

-3.0SL=-17.37

Plans# defects

Mod Hrs Prep Hrs

Peer Review Effort

Strive to be

Level 5

70

Peer Review Effort by Product Type

10050Subgroup 0

200

100

0

Ind

ivid

ual V

alue

1

1 111 1

1

X=8.7353.0SL=37.96

-3.0SL=-20.49

2 3 4 5 6Docs Reqs Design Code Test Change

Strive to be

Level 5

71

15105Subgroup 0

200

100

0

I nd

i vid

ua

l V

al u

e

1

X=24.52

3.0SL=93.70

-3.0S L=-44.66

Peer Review Effort by Product Type

2010Subgroup 0

40

30

20

10

0

-10

-20

I nd

i vid

ua

l V

al u

e

1

X=10.99

3.0SL=37.56

-3.0S L=-15.57

Docs Reqs

Design Code

Test Change

2010Subgroup 0

20

10

0

I nd

i vid

ua

l V

al u

e

X=6.904

3.0SL=16.81

-3.0S L=-2.997

252015105Subgroup 0

70

60

50

40

30

20

10

0

-10

-20

I nd

i vid

ua

l V

al u

e

1

11

X=13.16

3.0SL=43.83

-3.0S L=-17.51

32Subgroup 1

40

30

20

10

0

-10

-20

Ind

ivid

ua

l V

alu

e

X=11.69

3.0S L=35.41

-3.0S L=-12.02

252015105Subgroup 0

70

60

50

40

30

20

10

0

-10

-20

-30

I nd

i vid

ua

l V

al u

e

1

1

X=8.735

3.0SL=37.96

-3.0S L=-20.49

Strive to be

Level 5

72

10050Subgroup 0

300

200

100

0Ind

ivid

ual V

alue

1

1

X=18.323.0SL=60.85

-3.0SL=-24.20

Peer Review Number of Defects

Strive to be

Level 5

73

Peer Review Number of Defects by Product Type

10050Subgroup 0

300

200

100

0

Ind

ivid

ua

l Va

lue 1

1

X=15.103.0SL=43.65

-3.0SL=-13.45

2 3 4 5 6

Docs Reqs Design Code Test Change

Strive to be

Level 5

74

15105Subgroup 0

300

200

100

0

Ind

ivi d

ua

l V

alu

e

1

X= 27.00

3.0SL=107.1

-3.0SL= -53.14

2010Subgroup 0

60

50

40

30

20

10

0

-10

-20

Ind

i vi d

ua

l V

alu

e

X= 17.63

3.0SL=51.91

-3.0SL= -16.65

2010Subgroup 0

40

30

20

10

0

-10

Ind

ivi d

ua

l V

alu

e

X= 15.14

3.0SL=38.06

-3.0SL= -7.787

2010Subgroup 0

200

100

0

Ind

i vi d

ua

l V

alu

e

1

X= 23.59

3.0SL=83.87

-3.0SL= -36.69

32Subgroup 1

20

10

0

Ind

ivi d

ua

l V

alu

e

X= 8.000

3.0SL=19.97

-3.0S L= -3.968

2010Subgroup 0

70

60

50

40

30

20

10

0

-10

-20

Ind

i vi d

ua

l V

alu

e1

1

X= 12.22

3.0SL=39.30

-3.0SL= -14.86

Peer Review Number of Defects by Product TypeDocs Reqs

Design Code

Test Change

Strive to be

Level 5

75

198 Requirements* to peer review(predict 11 hrs and 18 defects/requirement review)

239 Design elements* to peer review(predict 7 hrs and 15 defects/design review)

464 Code elements* to peer review(predict 13 hrs and 24 defects/code review)

Reviews Remaining

* multiple elements may be reviewed in a single review

Strive to be

Level 5

76

478 481 492 495 502 502480 480 490

239 241

337

420

477502

1446

132

256

398

478

280

502 502

0

100

200

300

400

500

600

Jul Aug Sep Oct Nov Dec

Requirements to review Requirements reviewed

Design reviewed Code reviewed

Mission Accomplished!

Strive to be

Level 5

77

Tom [email protected]

That's all folks

mailto:[email protected]

sixsigma and cmmi everything you wanted to know, but didn’t know who to ask

Documents

sigma limits

sigma relationship

sigma examples

sigma sources of variation

product variation variation

occursigma level

sigma tool usage

process center