quantitatively measured process improvements at …… · ppt file · web view ·...
TRANSCRIPT
Quantitatively Measured Process Improvements at Northrop Grumman IT
Craig HollenbachNorthrop Grumman IT
Agenda
Northrop Grumman IT Overview 2002 SCAMPI AppraisalSample Project Data
• Inventory Tracking System (ITS)• AIT• JPATS• SIGS
Conclusions
Northrop Grumman Overview
Northrop Grumman• $26 B in revenue; 120,000 employees; 50 states; 25 countries
Information Technology (IT) Sector• $4 B in sales; 22,000 employees; 48 states; 15 countries
Defense Enterprise Solutions (DES) Business Unit• $548 M in sales; 2,900 employees, 23 states, 3 countries
DES provides enterprise-wide technology solutions to the Defense marketplace
Major Applications:
Logicon LIS
Litton TASC
Logicon LTS
L5 Litton PRC
(to other units)
DES
2001 20021999 2000
Logicon LATL3
LDES L5
(to other units)
L3
CMMI
(to other units)
Logicon LISS L3
ENABLER
LIEB
SPII
DES Maturity Pedigree
2002 CMMI Approach
Background• Kent’s quote about problems at beginning of 2002
Personnel & Teams• PA Process Owners • DES Organizational Units (e.g., EPG, training, procurement)• High Maturity Process Area Teams, composed of project
representatives (L4WG, L5WG, MO, DPWG, TCMSIG)Approach
• DES Organizational Improvements– CMMI Process Gap Analysis– Built Umbrella processes for legacy orgs
• DES Project Improvements– Assigned support reps to assist project personnel– Project representatives participated on high maturity process area teams
2002 SCAMPI Appraisal
SCAMPI appraisal led by independent SEI-certified appraisers in December 2002 determined that DES achieved• CMMI-SE/SW maturity level 5• CMMI-SE/SW capability level 5 in PMC, IPM, TS, and VER• SW-CMM maturity level 5
DES works with other IT Business Units to transfer our process improvement experience throughout the sector
Inventory Tracking System (ITS)
Inventory Tracking System
Project Description:USAF/AFMC/MSG Inventory Tracking System (ITS)
ModernizationA 3.5-year, $11M Firm-Fixed Price project with a development
staff of approximately 15 membersDevelopment Team uses SEI Personal Software Process (PSP)Implemented CMMI Level 5 quantitative management
processes to dramatically improve the cost, schedule, and delivered quality of the software
Currently in preparation for 1st contractual customer driven test cycle
Contractual Quality Goal is to deliver no known severity 1-3 defects (1-Critical, 2-Urgent, 3-Routine).
ITS Critical + Urgent Defect Density
Quantitative Management Plan Goal: 5/KLOC for Critical + Urgent
-0.015
-0.010
-0.005
0.000
0.005
0.010
0.015
0.020
0.025
1/9/
02
1/22
/02
2/5/
02
2/13
/02
3/5/
02
3/28
/02
5/10
/02
6/6/
02
6/17
/02
6/25
/02
3/29
/02
6/3/
02
7/11
/02
7/26
/02
8/5/
02
8/8/
02
8/14
/02
7/2/
02
8/20
/02
9/6/
02
9/16
/02
9/25
/02
9/27
/02
10/4
/02
10/2
2/02
10/2
4/02
10/2
8/02
10/3
1/02
11/8
/02
8/29
/02
10/2
8/02
10/3
1/02
11/1
2/02
11/1
4/02
11/1
5/02
X
DDr DDr Mean UNPLX LNPLX U2S L2S U1.5S L1.5S
5.3 4.9 3.2
1.31.4
Peer Review - Builds 1 - 5
KLOC = Thousand Lines of Code
Peer Review Defect Density (Critical + Urgent cont.)
DP 2 DP 3
2.1
3.93.5
6.16.6
0
1
2
3
4
5
6
7
1 2 3 4 5
Build
Defe
ct D
ensi
ty (D
efec
ts/K
LOC)
DP 1
Cost Variance by Build
-46%
11%21%
41%
-15%
-60%
-40%
-20%
0%
20%
40%
60%
1 2 3 4 5
Build
Cost
Var
ianc
eDP 2 DP 3DP 1
Schedule Variance by Build
36%
49%
38%
-1%1%
-10%
0%
10%
20%
30%
40%
50%
60%
1 2 3 4 5
Build
Sche
dule
Var
ianc
e
DP 2 DP 3DP 1
Return on Investment – Construction Phase
Hours invested: 124• Team training: 48• Conducting DP Cycles: 76
Defects avoided: If the Defect Density had remained at 6.6 (Build 1), we would have injected 110 more defects.
Hours saved: At an estimated cost of 15 hours per defect this equals 1650 hours.
Return: • Hours:1650/124 = 1330%• Customer satisfaction: Priceless! – “The contractor has
always provided products and services with less defects that industry standards. Most have been provided with no defects. Personnel have been used that show a complete understanding of their subject area and are able to convey this information in a highly professional manor.”
Inventory Tracking System – Test Phase
12
Need to understand Total Defect Density in Peer Review in Construction Phase to relate to Defect Density in Test Phase• DP Cycles had an effect on Total defect density also
– Build 1 Total defect density = 21.6 defects/KLOC– Build 5 Total defect density = 13 defects/KLOC
• Total Defect Density for Construction Phase = 19 defects/KLOC• Total Defect Density for Testing to Date = 4.5 defects/KLOC
0
5
10
15
20
Construction Test
Defect Density
400% Reduction
Inventory Tracking System – Test Phase
Management Goals in Test are being exceeded! Critical/Urgent Defect Total Defects
DDt Unit .5/KLOC 2/KLOCDDt .25/KLOC 1/KLOC(all internal integration cycles)
ITS Test Defects By Test CycleActual Defect Density by KLOC
12
0
20
40
60
80
100
120
140
160
180
Unit Test Build .01 -0.6
Build 0.7 Build 0.75 Build 0.80(Informal)
Critical + Urgent Total
.46
.2
.056
.33
1.4
.18.065
.233.32
.123
DP Cycle
0100
200300400
500600700
800900
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
Project Month
Defe
cts
Disc
over
ed
Theoretical Curve Actual Values
Defect Discovery Rayleigh CurveAll Test Defects
Feb 03 Prediction = 14.55 defectsFeb 03 Actual = 15 defects .14 defects per K LOC/month
Inventory Tracking System – Test Phase
Quantitative Management Plan Goals: DDs – Defect Discovery 1/KLOC
.21
.065
When the Total Defect Discovery rate falls under 1 defect per KLOC per month the project manager and test lead have enough confidence to stop test cycle.
0100
200300
400500
600700
800900
0 10 20 30
Project Month
Defe
cts
Disc
over
ed
Theoretical Curve Actual Values
Prediction = 1.44 defectsActual = 1 defect (last week of May)
Defect Discovery Rayleigh CurveBuild .75 Test Defects
Results
CMMI Quantitative Management & Defect Prevention Cycles have a huge return on Investment in the Construction Phase.• Specific results from the first coding cycle to the fifth are:
Critical / Urgent defect density reduced by 68%, Cost Variance improved from -59% to +39%, and Schedule Variance improved from +26% to +49%.
This return has a significant effect on the Test Phase• Where most projects have the highest defect detection rate in
Test, ITS has its lowest defect detection rate. Latent defect analysis estimates delivering a defect density of between 0.35 and 0.7; a total of 20 to 40.
Understanding the quality of the product allows for better management decisions and with highly satisfied customers.
Automated Identification Technology (AIT)
AIT Document Defect Data QM
Began collecting data in Feb ‘02 as part of DP Cycle • Process improvement techniques per DP Cycle identified
– Use of CM controlled Templates for documents enforced for authors– Type classification identified for defects: technical/non-technical– Documentation Input Defect Report checklist completed
Identify number of pages each document Identify document types Identify defects as technical/non-technical
– Management and personnel awareness of data collection and purposes• Six months data had defect rate vary from 4.9% to 11.6%, with one
outlier higherProcess Improvement implementation
• Resulted in re-evaluation of upper and lower limits• Increased personnel and management focus on data• Data for last 12 months has defect rate vary from 0.4% to 5.9%
Document Input Defect X Control Chart
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
55%
60%
Feb-
02
Mar
-02
Apr-
02
May
-02
Jun-
02
Jul-0
2
Aug-
02
Sep-
02
Oct
-02
Nov
-02
Dec
-02
Jan-
03
Feb-
03
Mar
-03
Apr-
03
May
-03
Jun-
03
Jul-0
3
X
% Defects % Defects Mean UNPLX LNPLX U2S
L2S U1.5S L1.5S Upper Spec LimitNorthrop GrummanInformation Technology Company Sensitive / Company Proprietary -- Subject to Nondisclosure
Document defect data Feb ’02 – Jul ‘03
JPATS TIMS
JPATS Challenge
JPATS Build 1.05 (June 2002)• The first build to be released after progressing from
“Development” to the “Maintenance” phase of the program.– Builds 1.01 – 1.04 were internal builds, not released for customer
verification
• Used “development”–style processes for fixing STRs • Failed 9/30 (30%) of the on-site STR verification tests with
customer witnesses. – NOTE: STRs Fixed vs. STRs Accepted is a measure that is quantitatively
measured by the JPATS Program
As a Result … Kicked off JPATS DP Cycle #1• GOAL: Reduce the STR verification failures to < 5% for JPATS
builds 1.0.6 and 1.07
DP Cycle Findings
Root causes included• Lack of “maintenance”-style processes (e.g. streamlined for
dealing with many (~30-130) STRs/build)• Lack of “maintenance”-style build planning & tracking
~40 Countermeasures identified• Many top-tier countermeasures focused on
improving/updating our STR build processes– Most of these were approved for action by the sponsor
DP Cycle Improvements
Actions• JPATS updated/developed the following processes
specifically for the Contractor Logistics Support (CLS = maintenance) phase
– BP 100, CLS Software Build Process– BP 200, Define a Build – BP 300, Plan and Track a Build– BP 400, Develop a Build
UT 100, Unit Test Procedure PR 100, Peer Review Procedure (Code) RT 100, Regression Test Procedure
– BP 500, Deploy a Build
• JPATS developed a build planning and tracking matrix called the “STR Big Board” to track all the elements required by the process per STR across all STRs
DP Cycle Effectiveness
Build 1.06 (July 2002)• 0% STR verification failure rate (0/11)• Goal of verification failure rate < 5% was met
Build 1.07 (Oct 2002)• 2.7% STR verification failure rate (3/113)• Goal of < 5% verification failure rate was still met
Subsequent builds have continued to perform well• See next slide showing current JPATS QM measure for STR
Verification
DP Cycle Effectiveness
STRs Projected Fixed vs Actual Fixed: (Actual - Projected) / Projected
-60%
-40%
-20%
0%
20%
40%
1.05 1.06 1.07 1.08 1.1 1.2
Build #
% D
evia
tion
DDrc Goal X-bar UCL LCLNorthrop GrummanInformation Technology Company Sensitive / Company Proprietary -- Subject to Nondisclosure
Problem Occurred
DP Cycle Subsequent Results
Synthetic Imagery Graphical System (SIGS)
SIGS Schedule Performance
Goal: SPI (X bar) of 85% in the 1st third of each PoP, 90% in the 2nd third, and 95% in the last third
Actual: 92.1% over multiple PoPs; 88.3% at the end of the 1st PoP; 88.7% at the end of the 2nd; and 96.8% at the end of the last PoP
Highlights: Cost Performance (CPI) was 96.8% over the same period; Award Fee average was 99% over the same period
SPIm X Chart for SIGS
50%
60%
70%
80%
90%
100%
110%
120%
130%
Date
Perc
ent (
%)
SPIm X bar UNPL LNPL Wt. Ave.
SIGS Schedule Performance (Cont’d)
Situation: O&M project undertaking a major redesign of the system over multiple years using new technology
At the beginning: unfamiliar technology meant that schedule estimates had large uncertainty since there was no available historical data to support the basis of estimate
Process changes: introduced Earned Value (EV) tracking combined with statistical process control (SPC) techniques allowed better monitoring of progress against the plans and identifying when there are special causes of variation
Improvement: by closely tracking the actual effort required to complete the earlier activities, we were able to feed that back into the estimates for the later activities and thus able to produce schedules with less uncertainty
Conclusions