lockheed martin benefits continue under cmmi · lockheed martin’s integrated engineering process...

Joan WeszkaLockheed Martin

Corporate Engineering & TechnologySystems & Software Resource Center

© Copyright Lockheed Martin Corporation 2004

CMMI® Technology Conference 2004November 17, 2004

.

Lockheed Martin Benefits Continue UnderCMMI

® CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.

11/17/04 © Copyright Lockheed Martin Corporation 2004 Joan Weszka - 2

Agenda

� Context� Background on Lockheed Martin’s (LM) CMMI®

transition approach� Benefits in the Software CMM® and CMMI® eras

�LM Systems Integration (Owego, NY)�LM Maritime Systems & Sensors – Tactical Systems

(Eagan, MN)�LM Maritime Systems & Sensors – Radar Systems

(Syracuse, NY)�LM Maritime Systems & Sensors – Undersea Systems

(Manassas, VA)


Context

� Lockheed Martin organizations that trackedquantitative process improvement benefits duringtheir SW-CMM high maturity journey have nowtransitioned to CMMI

� Previously reported data showed these benefitscontinued with CMMI implementation�This trend continues in 2004

� However, benefits derived are not attributable onlyto CMMI�Many initiatives are underway concurrently with

CMMI deployment (and were underway when the SW-CMM was in use)

CMM is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.


Key Tenets of Lockheed Martin’sCMMI® Transition

� Address CMMI® in the context of yourorganization’s business requirements�Lockheed Martin’s Integrated Engineering Process

(LM-IEP) standard includes CMMI®, in addition toother standards and requirements (e.g., ISO/IEC 15288,ISO 9001:2000)

� Use SCAMPISM A for benchmarking� Adopt an incremental appraisal approach for

process improvement and SCAMPISM readiness�Lockheed Martin Continuous Appraisal Method (CAM)

has been successfully deployed with CMMI®

SM SCAMPI is a service mark of Carnegie Mellon University.


LM-IEP Context Diagram

OrganizationalOrganizationalStandardStandard

Process(Process(eses))LMLM--IEPIEPStandardStandard

Industry StandardIndustry Standard

Gov’tGov’t StandardStandard

ANSI/EIAANSI/EIA--632632

ISO 9001:2000ISO 9001:2000

IEEE 1220IEEE 1220

LM HardwareLM HardwareProcess StandardProcess Standard

Industry StandardIndustry Standard

Gov’tGov’t StandardStandard

Project Specific Standards

ProjectProjectDefinedDefinedProcessProcess

ISO/IECISO/IEC--1220712207

CMMICMMI®® V1.1V1.1

ISO/IECISO/IEC--1528815288

LMLM CPSsCPSs

AS9100AS9100

IEP – Integrated Engineering Process

LM – Lockheed Martin

CPS – Corporate Policy Statement

Domain Specific Standards


Continuous Appraisal Method (CAM)Design Goals

� Minimize appraisal preparation and reduce cost� Integrate process improvement with process

appraisal activities� Facilitate appraisal scheduling and minimize

disruption for participants� Provide an appraisal environment conducive to

process improvement� Promote institutionalization

CAM was originally developed for use with EIA/IS 731 and is beingused with CMMI®


ProjectAppraisals

Overview of Incremental AppraisalUsing CAM

MaintenanceReview

Institutionalization focus with minimal project disruption

OSP = Organizational Standard ProcessPA = Process Area

OSPAppraisal

Appraise OSP Fix Weaknesses Re-appraise

Fix Weaknesses Re-appraiseAppraise PA 1

...

Planning




Feedback on CAM usage

� 14 CAMs have been completed or are underway atLockheed Martin operating units using CMMI®

� 6 prior CAMs have been completed using EIA/IS 731� Experience with CAM has been positive:

�More focus on process improvementFacilitated by the incremental approach

�Less invasive to programs�Less stressful to the organization�More value-add, in-depth findings�More cost effective�Effective method to prepare for SCAMPISM

SM SCAMPI is a service mark of Carnegie Mellon University.


Lockheed Martin Systems IntegrationLockheed Martin Systems Integration -- Owego Process CredentialsOwego Process Credentials

Q-100 Award

19901990

19931993 IBM Baldrige Gold Award

New York StateExcelsior Award

19941994

19951995 RIT / USA TODAYQuality Cup Finalist

19961996 ISO 9001:1994 /SW CMM Level 3

19971997 SW CMM Level 5

19981998 LM21 Best Practices

20012001 ISO 9001:2000 /SEI CMMI Pilot Site

20032003 CMMI Level 5 /AS 9100A

20002000 Lean / Six Sigma

TQM

20022002

20032003

20022002

20012001

20002000

19981998

19971997

19961996

19951995

19941994

19931993

19901990


0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

120.0%

140.0%

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003

CMM

ILe

vel 5

SWCM

MLe

vel 3

SWCM

MLe

vel 5

Improvements Since 1993:12.7% Average per Year

ISD

Proc

esse

s

SWCM

MLe

vel 3

Capa

ble

Lockheed Martin Systems Integration OwegoLockheed Martin Systems Integration Owego -- Software ProductivitySoftware Productivity

Software Productivity (All Software including Reuse)

CMM

ILe

vel 3

Capa

ble

Contributors:•Increased Reuse (Domain Specific)•Process Maturity and Compliance•Process Consistency•Increased use of High Order Language/ 4th Generation / Object Oriented•Use of development and test tools


Lockheed Martin Systems Integration OwegoLockheed Martin Systems Integration Owego -- QualityQuality

Software Defects per Million Delivered Source Lines of Code

-80%

-60%

-40%

-20%

0%

1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

%R

educ

tion

STR

s / M

SLO

C Improvements Since ‘92

80.5% Overall

Inte

grat

edPr

oces

ses

SWCM

MLe

vel 5

CMM

ILe

vel 3

Capa

ble

CMM

ILe

vel 5

* STR / MSLOC = Software Trouble Report/Million Source Lines of Code

SWCM

MLe

vel 3

2003

®® CMMI and CMM are registered in the U.S. Patent and Trademark OffCMMI and CMM are registered in the U.S. Patent and Trademark Office by Carnegie Mellon Universityice by Carnegie Mellon University..

Note: Started counting defects differently in 2004. Redefined from “Latent” to “Openat Delivery”.


LM Maritime Systems & Sensors Tactical SystemsProcess Improvement Credentials

� Oct. 1999 – Attained SW-CMM® level 4� Dec. 2000 – Attained Systems Engineering Capability

Model (EIA 731) level 3� Jan. 2001

�Began focus on integrated process improvement�Began transition to CMMI®

� June 2002 – May 2003 CAM Appraisal�OSP: Target profile 5 for CMMI®-SE/SW/IPPD/SS�Projects: Target profile 3 for CMMI®-SE/SW

� August 2003 – SCAMPISM Appraisal�Achieved maturity level 3 using CMMI®-SE/SW

® CMM is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.


LM Maritime Systems & Sensors Tactical Systems2004 Initiatives

� Implement CMMI®-SE/SW maturity level 4� Streamlining of references and guidance documents which

supplement the OSP� Conduct LM-IEP gap analysis� Risk management methodology best practice� Populate Process Asset Library (PAL)� Self audit process compliance� Mechanical engineering guidebook� Airworthiness manual� Work product templates� Cost Estimation Relationships Guide


LM Maritime Systems & Sensors Tactical SystemsSoftware Productivity

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

1979 1984 1989 1994 1999 2004

Year

Sour

ce S

tate

men

ts p

er H

our

Level 3CMM®

Level 4CMM®

Level 3CMMI®


LM Maritime Systems & Sensors - SyracuseProcess Improvement Credentials

� SW-CMM® level 5 (CBA IPISM) in Dec. 1999� Systems Engineering Capability Model (EIA/IS

731) level 3+ (CAM) in Dec. 1999� Focus on integrated process improvement

including hardware began in 2000� Transition to CMMI® began in 2000� CMMI®-SE/SW/IPPD target profile 4 (CAM) in

Nov. 2002� CMMI®-SE/SW maturity level 4 (SCAMPISM) in

May 2004® SW-CMM and CMMI are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.SM SCAMPI is a service mark of Carnegie Mellon University.


LM Maritime Systems & Sensors - Syracuse2004 Functional Excellence Initiatives

� LM21 Business Excellence� Design for Six Sigma (DFSS)� Technical Assurance� Integrated Product Development� Digital Dashboard� CMMI®-SE/SW Maturity Level 4� Continuous Process Improvement

�Process Development�Quantitative management


LM Maritime Systems & Sensors - SyracuseSoftware Productivity

Software Engineering Productivity History(Completed, Current/In-Process, Projected-Goal, BQM)

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007

SLO

Cs

/ Sta

ff D

ay

Completed Log Best-Fit

SWCMML3

SWCMML4

SW SECMCMM L3+L5

CMMICAML4

CMMISCAMPIL4


LM Maritime Systems & Sensors – Undersea SystemsProcess Credentials

� Systems & Software Engineering*� Software CMM® Level 4 (CBA IPI) - June 1995� Software CMM® Level 5 (CBA IPI ) - February 1999� CMMI® & EIA-731 target profile 3 (CAM) - October 2001� CMMI®-SE/SW/IPPD/SS target profile 5 (CAM) - October 2002

� Quality Management� ISO 9001:1994 - September 1995� AS9000 - November 1997� Defense Contract Management Agency (DCMA) ISO 9001

Qualified - December 1997� ISO 9001:2000 - December 2000� AS9100A – December 2002

Certificate No.10061

Certificate No.7092

Certificate No.9227

Certificate No.E00015

Certificate No.EN10002

*Assessed programs comprise over 80% of the Undersea Systemsdevelopment programs, and all parts of the development cycle.

*Assessed programs comprise over 80% of the Undersea Systemsdevelopment programs, and all parts of the development cycle.


LM Maritime Systems & Sensors – Undersea SystemsProcess Chronology

1970sTop-down Structured ProgrammingDesign & Code Inspections

1980-2Functional DecompositionSW Engineering WorkshopAdvanced Design Workshop

1983SW Management Workshop

1984Ada Workshop

1985Requirements Inspections

1986FSC Practices & Measurements

1988SW Technology Steering GroupOrganizational Operating Procedures

1990SW Engineering Process Group FormedFirst SW-CMM® Assessment (Level 3)Formal Estimation Procedures

1991Market Driven QualityReuse Focus

1992Defect Prevention Process

1993Integrated TeamsStandard Development EnvironmentIntegrated Process Group

1994Automated MetricsProcess Coordination Group (PCG)

1995-6Integrated Process LibraryISO 9001 RegistrationSoftware CMM® Level 4

1997-9ISO 14001 Registration, AS9000 &

DCMA ISO 9001 qualificationSoftware CMM® Level 5

2000->ISO 9001: 2000 CertificationEIA-731 Level 3AS9100A CertificationCMMI®-SE/SW/IPPD/SS Target

Profile 5 (CAM)


LM Maritime Systems & Sensors – Undersea SystemsSoftware Product Quality

Software Product Quality (Defects Per Million Delivered Source Statements)

0

500

1,000

1,500

2,000

2,500

3,000

1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Program Completion Year

Def

ects

Per

MD

SS

CMM L3, 9/90CMM L4,6/95

CMM L5,2/99CMMI L3, 10/01

CMMI L5, 10/02

Note: CMM® levels were achieved via CBA IPI. CMMI® levels indicate theresult of CAM assessments using CMMI®-SE/SW/IPPD/SS with a leadappraiser outside of Undersea Systems.


S ig ma De fe c ts /MS1 690,000.02 308,537.03 66,807.04 6,210.05 233.06 3.4

0

1 0 0 ,0 0 0

2 0 0 ,0 0 0

3 0 0 ,0 0 0

4 0 0 ,0 0 0

5 0 0 ,0 0 0

6 0 0 ,0 0 0

7 0 0 ,0 0 0

8 0 0 ,0 0 0

1 .0 0 2 .0 0 3 .0 0 4 .0 0 5 .0 0 6 .0 0

S ig m a N u m b e r

Def

/Mill

ion

Parts

5 Sigma

Our quality rate is 20times better than theaverage industry rates.

LM Maritime Systems & Sensors – Undersea SystemsSoftware Quality

MS = MillionSource Statements

Product Quality Level is in Five Sigma Range


Reference: “A Correlational Study of the CMM® and Software Development Performance”Lawlis, Flowe & Thordahl, CROSSTALK, September 1995

SCATTER PLOTS BETWEEN “1” AND “3” RATINGS ARE INDUSTRY SAMPLES.LOCKHEED MARTIN DATA HAS BEEN ADDED IN RED.

LM Maritime Systems & Sensors – Undersea SystemsCost & Schedule Estimates vs. Actuals at Maturity Level 5

DATA FROM 8 PROGRAMS IN 2Q 2003

Cost Performance Index (CPI) & Schedule Performance Index (SPI)

0

1

2

3

1 2 3 4,50

1

2

3

1 2 3 4,5

+

++++

+

+

++

++

+++

+++=+++= =

==++

+++

++++++

++++

+ ==

====

CPI

SPI

CMM® Rating

+ +++

CMM® Rating


• Consistency•More likely to get the same results each time within a small rangeTHIS IS A VERY REAL MEASURE OF QUALITY IN TODAY’S MARKETPLACE

• Predictability•Much easier to estimate work•Better able to assess impact of change

• Manageability•Able to measure and track progress•Better able to change and improve•Better able to control costs and meet schedule

•IT ALL ADDS UP TO•IMPROVED QUALITY•IMPROVED PROFITABILITY

Continuous process improvement is rooted in Manassas’culture...and the benefits are so ingrained in our approach, we tend

to take them for granted:

LM Maritime Systems & Sensors – Undersea SystemsProcess Maturity Benefits


Example of Defect Discovery Profile forAnother LM Organization

Example Defect Discovery Goals & Actuals ToDate

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

HLD LLD C CT/UT/ST RT/ESI/SWIT I&T Latent

Verification Stage

Nor

mal

ized

Def

ects

Per

K

SLO

C

Low er Range Values

Goal, Expected Value

Upper Range Values

Actuals90%Confidence Range


Summary

� The trend previously reported continues:�Benefits derived during SW-CMM® implementation

continued to be realized as CMMI® maturity evolves

� Allocating benefits to their sources is difficultwhen implementing multiple models/standards andinitiatives

� SW-CMM® and CMMI® are viewed as significant(but not sole) contributors to process improvementbenefits to date


Contact Information

Joan WeszkaLockheed MartinCorporate Engineering & TechnologySystems & Software Resource Center(301) 240-7013 / Fax [email protected]


Acronyms� ARC – Appraisal Requirements for CMMI®

� CAM - Continuous Appraisal Method� CMM® - Capability Maturity Model� CMMI® - Capability Maturity Model Integration� ESLOC or ESS – Equivalent SLOC/SS; a normalized value derived from

new development, plus SLOC/SS that are modified, retained, ported, etc.� IPPD - Integrated Product and Process Development� LM - Lockheed Martin� LM-IEP - Lockheed Martin Integrated Engineering Process� OSP – Organizational Standard Process� PA - Process Area� SLOC – Source Line of Code� SS – Source Statement (sometimes called a “Logical SLOC”)

lockheed martin benefits continue under cmmi · lockheed martin’s integrated engineering process...

Documents