it performance improvement with cobit and the sei cmm

I N F O R M A T I O N S Y S T E M S C O N T R O L J O U R N A L , V O L U M E 3 , 2 0 0 5

IT Performance Improvement With COBIT and the SEI CMM

By Debra Mallette, CISA, CSSBB, SEI CMM and CMMI Assessor,

and Managed Change™ Master, and Monica

Jain, CSQA

The promise of business performance improvement ismillions of US dollars in annual cost savings andproduct quality improvements. The risk is that even a

company that dramatically improves its efficiency and productquality can fail miserably. The challenge is that manyexecutives realize that radical changes are needed, but theyhave to increase today’s level of performance while makingthose changes. Improving the performance of the ITorganization can accelerate the business to greaterperformance. That is the promise of improving ITperformance. The risk when the IT organization does notdeliver is that the whole organization suffers, losing assets,sales, customers and momentum, or even goes under.

Improving and sustaining business performance requiresresources. These are indirect, overhead or risk mitigation costsand show up on the business’s income statement as overhead.The IT organization represents a significant part of thisoverhead. With constant pressure to decrease overhead, the IT organization represents an attractive target for cost reduction.At the same time, almost all of IT’s operational and maintenancecosts sustain performance and mitigate risks. Sustaining current performance, continuously reducing costs, decreasingexposure to risk and safely improving performance are the IT challenges.

Maturity Models and PerformanceImprovement

Maturity models are guidelines for process performanceimprovement and can be applied by business and IT. Adoptersof maturity models claim results that include significantreduction in defects, reduced project cycle times, increasedproductivity, improved employee satisfaction, increasedcustomer satisfaction, reduced costs and reduced exposure torisks. These results are reproducible using the model to assessthe current state and compare it to that of the industry (best-in-class) to identify opportunities. Implementation costs arecontrollable and the costs, risks and optimum methods forimplementation can be learned from the experiences of others.The Software Engineering Institute’s Capability MaturityModel (SEI CMM) for software engineering has been widelypublicized and adopted, particularly by large engineeringorganizations serving the US government. The IT GovernanceInstitute’s Control Objectives for Information and relatedTechnology (COBIT) is generally accepted as the de facto

standard for IT. Both incorporate continuous improvementprinciples, so that by putting the processes and controls inplace, the foundation is built for continuously improvingperformance, reducing costs and decreasing risk over time.

When most business people think performanceimprovement, they generally think Six Sigma. General ElectricCorporation’s highly publicized Six Sigma implementationprogram has been attributed with myriad performanceimprovements reported in the company’s annual report. SixSigma is not a maturity model; rather, Six Sigma refers to anumeric description of variation, statistical methods that bringprocesses under measurable control applied in business processimprovement projects, and a strategic program to drivebusinesswide performance improvement to the bottom line.Perhaps less publicized is that Six Sigma implementationexperts recommend not attempting a Six Sigma improvementprogram until the foundation maturity level 3 is laid (i.e., processes are defined and used repeatedly). COBIT and the SEI CMM can be used to bring the business and IT to a level of maturity and performance that can be acceleratedtoward more improvement using Six Sigma.

The DecisionDeciding which model to use is a classic cost-benefit trade-

off. Each maturity model has a particular focus of control fromwhich the improvement benefit is derived. The SEI CMMfocuses on practices that bring software engineering, such asIT application software engineering, under control while COBITprocesses are aimed at a broader range of IT practices. Adecision matrix mapping benefits, costs and alternatives clearlycommunicates the trade-offs and can be used in making thedecision and in implementation communications. Figure 1 is adecision matrix showing evaluation criteria and maturity modeloptions for SEI CMM, COBIT or the two combined.

The SEI CMM is comprised of five levels of maturity(figure 2). Each level is a conceptual step or stage of processdefinition resulting in control, effectiveness and efficiency inproducing software. The starting point is initial (maturity level1), an ad hoc approach. Progression is expected through therepeatable (maturity level 2), defined (maturity level 3) andmanaged (maturity level 4), culminating in the optimizing(maturity level 5) level. The key practice areas (KPAs) aregrouped by level. To be assessed as repeatable, the organizationmust implement all of the level 2 KPAs and show evidence ofhaving met the goals and objectives for those practices. Thelevel 2 KPAs shown in figure 2 are: requirementsmanagement, software project planning, software projecttracking and oversight, software subcontract management,software quality assurance and software configuration

Copyright © 2005 Information Systems Audit and Control Association. All rights reserved. www.isaca.org.


management. To be assessed as defined, all of the KPAs forlevel 3 must be implemented with evidence of having met thegoals and objectives for those areas, as well as continuing tomeet the goals and objectives for level 2 KPAs.

COBIT COBIT comprises 34 IT processes organized into four

domains: Plan and Organize, Acquire and Implement, Deliverand Support, and Monitor and Evaluate. The COBITmanagement guidelines contain the maturity model, processdescription, information criteria and IT resources, whichindicate the improvement potential, critical success factors, keygoal indicators and key performance indicators for eachprocess. The COBIT framework, detailed control objectives andaudit guidelines improve the IT organization’s level of control,mitigating risks and sustaining performance. The managementguidelines can be used with this knowledge base and COBITOnline® benchmarking to prioritize and guide improvement.

Decision Criteria: Organizational FitThe value of any model in driving performance

improvement depends on whether the model will identifyopportunities for improvement appropriate to the organization.COBIT can be used to identify weaknesses and opportunitiesfor improvement in efficiency, effectiveness, confidentiality,integrity, compliance and reliability. COBIT can also be used tooptimize management of people, applications, technology,facilities and data. These are IT opportunities. ImplementingSEI CMM KPAs delivers improvements in effectiveness andefficiency of people, applications and technology. There arefew SEI CMM references to processes or goals to assureconfidentiality or data integrity. Security, business continuityand disaster recovery risk mitigation practices are largelymissing from the SEI CMM.

Decision Criteria: Size of Target Population and Model Complexity

Performance improvement is fundamentally organizationalchange to orchestrate people using processes and technology.The constraint in organizational change is on the ability ofevery person to know, understand, believe and do/manage thework using the improved processes. These are the targets of thechange. Management must sponsor the change, change agentslead the change and targets make the change. Target, historicaland cultural risks must be mitigated. The larger the targetpopulation, the more costly and risky the change.

The target population for the SEI CMM in IT is theapplication software engineers, their managers and ITmanagement—a subset of the IT organization. It has fewertargets than COBIT, and fewer people are required to understandthe model. This is good, because the size and intricacy of theSEI CMM are intimidating. The focus is on precision andthoroughness of the assessment process; assessments can take aweek or more. Assessors are expected to be trained and certified.

Figure 1—Maturity Model Options

Decision Criteria SEI CMM COBIT SEI CMM and COBITOrganizational fit Excellent for software engineering, Excellent for IT Depends on software

including IT software engineering engineering population sizeSize of target population and Dependent on size of software Dependent on size of IT Synergy: COBIT can direct model complexity—impact on engineering population within IT. organization. Documentation is SEI CMM investment forcommunication, learning and use Assessment and gap analysis take concise and readable by IT most benefit. SEI CMM

weeks, and implementation takes professionals and IT management. target can be limited tomonths. Quality focus may be Assessment and gap analysis take software engineering indifficult for software engineers who days, and implementation takes context of COBIT for ITare focused on art rather than weeks or months. It may be organization. SEI CMMdiscipline. Cost of quality and quality less understandable for those experience reduces thelanguage may be less unfamiliar with language investment requiredunderstandable for IT management. of controls. to use COBIT.

Synergies across practices and Key process areas are grouped Internal audits operational Similarity of practiceswithin organization for implementation by maturity metrics can be leveraged enables leverage of

level. The integral processes to identify and target implementation guidancefoundation is synergistic for later improvement opportunities. across both models. Keyimplementation. Quality audits goal indicators and are synergistic with internal performance indicatorsaudit needs. supplement SEI CMM, and

SEI CMM integral practicessupplement COBITimplementation guidance.

Optimizing (5)

Managed (4)

Defined (3)

Repeatable (2)

Initial (1)

Peer reviews—PR Intergroup coordination—IC Software product engineering—SPE Integrated software management—ISM Training program—TP Organization process definition—OPDOrganization process focus—OPF

Software configuration management—SCM Software quality assurance—SQA Software subcontract management—SSM Software project tracking and oversight—PTO Software project planning—SPP Requirements management—RM

Software quality management—PR Quantitative process management—QPM

Process change management—PCM Technology change management—TCMDefect prevention—DP

Figure 2—SEI CMM MAturity Levels


Gap analysis takes weeks, implementation takes months, andthere may be a lag after implementation before seeing results.The quality focus may be difficult for software engineersconcerned with creativity rather than discipline, and the cost ofquality language requires management interpretation to derivefinancial performance expectations.

The target population to understand COBIT is the entire ITorganization and management, including businessmanagement. Fortunately, the COBIT documentation has beendesigned to address the needs of each of the target populations:Executive Summary for business management, ManagementGuidelines for IT management, Control Objectives for processimplementers and Audit Guidelines for auditors. The entireCOBIT 3rd Edition© package consists of fewer than 500 pages. Itis clear and concise and takes a relatively short period of timeto read and comprehend. Self-assessments for prioritizationand gap analysis take days, implementation takes weeks andresults are almost immediately apparent. A drawback formanagement is the emphasis on control and risk mitigationrather than performance opportunity. COBIT’s language ofcontrol is also not generally well understood by engineeringorganizations without previous exposure to audit or financialcontrols. Creating awareness of the need for controls and riskmitigation in those not familiar with the concept can get in theway of understanding the value.

Decision Criteria: SynergiesSynergies are opportunities to leverage processes to reduce

costs and risks. The SEI CMM key practice areas identified asintegral practice areas are designed to be enablers for thefunctional practice areas with which they are staged and higherlevels of maturity. Functional practice areas in the set ofmaturity level 2 KPAs include requirements management,software project planning, project planning tracking andoversight, and software subcontract management. Integralpractice areas are software configuration management andsoftware quality assurance. Integral practice areas are enablersand risk mitigators for the functional practice areas and thehigher levels of maturity. For example, software configurationmanagement, an integral practice area, is an enabler forrequirements management, enabling requirements traceabilityand mitigating risks to software quality by controlling changes.SCM also enables software product engineering, a level 3KPA. Software quality assurance, which includes softwareaudits, is critical to assuring that all the level 2 KPAs remainunder control, and is an enabler for bringing new KPAs undercontrol. Control for the new KPAs is assured by expanding thescope of the audits.

COBIT leverages internal audit to sustain the processcontrols and give guidance on what to improve to identifyopportunities. In addition, the COBIT processes are organizedinto planning, doing and monitoring based on the plan-do-check-act continuous improvement cycle.

A correlation analysis of SEI CMM and COBIT overlaphelps in understanding the possible synergies between themodels. This analysis was performed using a three-passapproach. The first pass established a high-level mapping ofCOBIT processes to SEI CMM KPAs to the practice and goallevel. The COBIT process descriptions were compared with SEICMM KPAs grouped by maturity level, looking for significantmatching process words, such as technology, project plan,

project tracking and oversight, quality management, audit,training, process documentation, configuration and change.These resulted in high-level correlation (see figure 3). Thesecond pass was for more inclusive correlation based onsimilarities in the activities’ intent and goals, and the third passexamined the potential for fulfilling COBIT detailed controlobjectives using SEI CMM practices.

Figure 4 shows the information graphically. The KPAs aresorted by their SEI CMM level (shown in figure 2). The counttotals are shown in the bars, with a total bar for each capabilitymaturity model level 2, level 3, and levels 4 and 5 (left X-axis)correlated to each COBIT process (Y-axis) with the percentcoverage (right Y-axis) of COBIT detailed control objectivessuperimposed.

SEI CMM Best Practice Guidance for COBIT Processes

COBIT is best used to decide what and how much to improvethe IT processes, while best practice models such as the SEICMM provide better guidance for how to implement theimprovements. There is a greater depth and precision ofguidance available from the SEI CMM for control of softwareengineering. Figure 5 shows the COBIT processes that willreceive the most benefit from SEI CMM guidance. For example,PO11 manage quality benefits from KPAs, including level 2KPA software quality assurance and level 4 KPA softwarequality management and quantitative process management.

Using COBIT and SEI CMM to Lead Process Improvement

Recommended steps include:1. Identify opportunities for improvement. The opportunities

could be identified by looking at internal audit findingsmapped to COBIT control objectives and a COBIT assessmentand/or benchmark.

2. Evaluate the expected benefit from the improvement. TheCOBIT key goal indicators and the “why do it” statementfrom COBIT can be used if process measures are not alreadyavailable.

3. Use correlations and mapping of SEI CMM key practices toCOBIT control objectives to identify control objectives that aremet and strengthened using SEI CMM practices. COBITcontrol objective correlation to SEI CMM practices indicatewhere the SEI CMM KPAs have a higher probability of givingmore accurate and precise guidance than using COBIT alone.

4. Decide whether the expected benefit justifies a full SEICMM assessment if not already available.

5. Base the implementation strategic and tactical plans onassessed opportunities linked to best practices. Set goals andmilestones to reach an IT-wide balanced maturity level.Establish priorities for processes to improve based on thedesired improvement and the planning and monitoringprocesses to create the feedback loops foundational tosustaining performance and generating additionalopportunities. As an example of step 5, use maturity level 2 as the target.

The IT organization goal is to sustain performance and realizeimprovement with the minimum investment beginning with theSEI CMM staged guidance. The IT organization implementationplan should address AI1 identify automated solutions, AI2


Figure 3—Correlation Matrix: COBIT With Correlation to SEI CMM KPAs

COBIT Processes SEI CMM KPAs SEI CMM KPAs COBIT Detailed Percent of Percent ofHigh-level Correlated to Control COBIT KPAs toCorrelation COBIT Through Objectives Fulfilled With COBIT

Activity and Intent Fulfilled SEI CMMPlan and Organize

PO1 Define a strategic plan IC TCM 5 of 8 63% 11%PO2 Define the information architecture 0 of 4 0% 0%PO3 Determine technological direction TCM TCM 4 of 5 80% 6%PO4 Define the IT organization and relationships IC OPF,OPD, 6 of 15 40% 28%

IC, TCM,SSMPO5 Manage the IT investment TCM 1 of 3 33% 6%PO6 Communicate management aims and direction PCM 6 of 11 55% 6%PO7 Manage human resources 0 of 8 0% 0%PO8 Ensure compliance with external requirements RM 1 of 6 17% 6%PO9 Assess risks SPP, ISM SPP, PTO, ISM 6 of 8 75% 17%PO10 Manage projects SPP, PTO, SPP, PTO, ISM, 14 of 14 100% 28%

ISM SQA, SPEPO11 Manage quality SQA, OPF, SQA, OPF, SQM, 16 of 19 84% 33%

SQM, TP, ISM TP, ISM, QPMAcquire and Implement

AI1 Identify automated solutions RM, TCM RM, SPE, TCM 4 of 18 22% 17%AI2 Acquire and maintain application software SPE, SSM, SCM SPE, SSM, SCM, RM 6 of 17 35% 22%AI3 Acquire and maintain technology infrastructure SCM, TCM, PCM SCM, TCM, SSM 3 of 6 50% 17%AI4 Develop and maintain procedures ISM, OPF, OPD OPF, OPD, PCM, 3 of 4 75% 28%

SPE, ISMAI5 Install and accredit systems SPE SPE, ISM 6 of 14 43% 11%AI6 Manage changes SCM, PCM, SCM 5 of 8 63% 6%

TCMDeliver and Support

DS1 Define and manage service levels 0 of 7 0% 0%DS2 Manage third-party services SSM SSM 6 of 8 75% 6%DS3 Manage performance and capacity 0 of 9 0% 0%DS4 Ensure continuous service SPP,ISM 3 of 13 23% 11%DS5 Ensure systems security 0 of 21 0% 0%DS6 Identify and allocate costs SPP, PTO SPP, PTO, ISM 3 of 3 67% 17%DS7 Educate and train users OPD, TP OPD, TP,SPE 2 of 3 67% 17%DS8 Assist and advise customers SQA 2 of 3 67% 6%DS9 Manage the configuration SCM SCM 6 of 8 75% 6%DS10 Manage problems and incidents DP DP 3 of 5 60% 6%DS11 Manage data SPP, PTO, ISM SPP, PTO, ISM 3 of 30 10% 17%DS12 Manage facilities 0 of 6 0% 0%DS13 Manage operations IC 0 of 8 0% 0%

Monitor and EvaluateM1 Monitor the processes QPM, PCM QPM, PCM 4 of 4 100% 11%M2 Assess internal control adequacy SQA SQA 3 of 4 75% 6%M3 Obtain independent assurance SQA, PR SQA, PR,SSM 6 of 8 75% 17%M4 Provide for independent audit SQA 4 of 8 50% 6%

Legend: SEI CMM KPAs Used in Correlation Matrix

DP: Defect preventionIC: Intergroup coordinationISM: Integrated software managementOPD: Organization process definitionOPF: Organization process focusPCM: Process change management

PTO: Project tracking and oversight QPM: Quantitative process managementRM: Requirements management SCM: Software configuration managementSPE: Software product engineeringSPP: Software project planning

SQA: Software quality assuranceSQM: Software quality managementSSM: Software subcontract management TCM: Technology change managementTP: Training program


acquire and maintain application software, PO11 manageprojects, PO10 manage quality and DS2 manage third-partyservices. The integral processes to sustain performance are AI6manage change and DS9 manage the configuration. AdditionalCOBIT planning and monitoring processes to sustainperformance and generate additional opportunities are PO1define the strategic plan, emphasizing capability improvementplanning, and M1 monitor the processes, so that the organizationrecognizes the expected process performance improvementresults from the capability maturity improvement projects. M2assess internal control adequacy also sustains the performanceand generates information that can be leveraged to identifyadditional opportunities.

Case Study An assessed SEI CMM level 3 engineering organization was

supported by an engineering IT organization self-assessed withCOBIT maturity levels averaging at level 2 and ranging to level4. The engineering IT organization found that less than 6percent of its resources were applied in those COBIT processesmost highly correlated with the SEI CMM. The project list andallocation of resources to projects accounted for approximately50 percent of the resources. The organization was familiar withcapability maturity models and adapted readily to using COBITto guide its capability improvement strategy. The managementteam performed the self-assessment and put an improvementprogram in place with goals to improve PO1, M1 and assetmanagement (not a COBIT process). While continuing to drivefor capability improvement, investing 2.5 full-time-equivalentemployees (FTEs) with approximately US $5,000 in travel-related expenses, the IT staff was cut in half, capitalexpenditures were eliminated, and the expense budget was cutby 60 percent. Through these reductions, the IT organizationwas able to maintain ISO 17999 compliance, a satisfactorylevel of internal control compliance, and performance withinservice level agreement targets for 3.5 to 4.5 out of five 9savailability, service request cycle time average closure of lessthan 24 hours and customer satisfaction of “very satisfied.”

SummarySustaining current performance while continuously reducing

costs, decreasing exposure to risk and carving out resources tosafely improve performance from a budget constantly targetedfor cost reduction is the IT challenge. Maturity models can tellwhere there are opportunities to improve the organization’sperformance. By using a maturity model, the organization cansafely and predictably reproduce the performance improvementresults of others with confidence in the approach and theexpected expenditure of resources and benefits to be derived.Using any model requires an investment in learning,assessment and implementation. Best practice maturity modelstell how to attain the improvements with the most precisionand accuracy and may require more investment because oftheir detailed and specialized guidance. Model synergies,including continuous improvement practices, leveragesustaining costs for higher returns. Using COBIT with SEICMM combines the best of both worlds to improve ITperformance and drive the results to the business bottom line.

ReferencesHarry, Mikel, Ph.D.; Richard D. Schroeder; Six Sigma: TheBreakthrough Strategy Revolutionizing the World’s TopCorporations, Random House, 1999

IT Governance Institute, COBIT Management Guidelines,COBIT Framework and COBIT Control Objectives, 2000

IT Governance Institute, COBIT Online, www.isaca.org/cobit

Keen, Peter G. W.; The Process Edge: Creating Value Where ItCounts, Harvard Business School Press, 1997

Kimpton, Clarence; Denys Martin; “Overview of Principal ITEvaluation Models: Tools for IT Auditors,” InformationSystems Control Journal, vol. 5, 2001

LaMarsh, Jeanenne; Changing the Way with Change: GainingControl over Major Organizational Change, Addison-WesleyPublishing, 1995

Martin, James; The Great Transition: Using the SevenDisciplines of Enterprise Engineering to Align People,Technology, and Strategy, American Management Association,1995

Paulk, M.C., et al; “Capability Maturity ModelSM forSoftware,” CMU/SEI-93-TR-24, Carnegie Mellon University,Software Engineering Institute, USA, 1993

35%

30%

25%

20%

15%

10%

5%

0

KPA

Coun

t

4.5

4

3.5

3

2.5

2

1.5

1

.5

0

SEI CMM Level 2 KPAs SEI CMM Level 3 KPAs SEI CMM Level 4 and 5 KPAs

Figure 4—COBIT and SEI CMM Correlation

76543210SE

I CM

M K

PA C

ount

COBIT Process

PO1

PO2

PO3

PO4

PO5

PO6

PO7

PO8

PO9

P10

PO11 AI

1

AI2

AI3

AI4

AI5

AI6

DS1

DS2

DS3

DS4

DS5

DS6

DS7

DS8

DS9

DS1

0

DS1

1

DS1

2

DS1

3

M1

M2

M3

M4

SEI CMM Level 2 KPAs SEI CMM Level 3 KPAs SEI CMM Level 4 and 5 KPAs

Figure 5—COBIT and SEI CMM KPA By Level


Debra Mallette, CISA, CSSBB, SEI CMM and CMMIAssessor, and Managed Change™ Masteris a process program manager for a large healthcare ITorganization. Her experience ranges across industries andorganizations. She has been published and has presented at theMotorola Software Engineering Symposium and SEI CMM’sSEPG. Her specialty is strategic capability improvement forenterprises making the transition to the information age. Shecan be contacted at [email protected].

Monica Jain, CSQAis a process consultant at Covansys Corporation, USA,specializing in technology and business consulting. Her areasof interest include implementation of CMM, CMMI, ITIL, andconducting audits and assessments. She has also cleared theITIL Foundation Certification examination conducted byEXIM UK. She can be contacted at [email protected] [email protected].

Information Systems Control Journal, formerly the IS Audit & Control Journal, is published by the Information Systems Audit and Control Association, Inc.. Membership in the association, a voluntaryorganization of persons interested in information systems (IS) auditing, control and security, entitles one to receive an annual subscription to the Information Systems Control Journal.

Opinions expressed in the Information Systems Control Journal represent the views of the authors and advertisers. They may differ from policies and official statements of the Information Systems Auditand Control Association and/or the IT Governance Institute® and their committees, and from opinions endorsed by authors' employers, or the editors of this Journal. Information Systems Control Journaldoes not attest to the originality of authors' content.

© Copyright 2005 by Information Systems Audit and Control Association Inc., formerly the EDP Auditors Association. All rights reserved. ISCATM Information Systems Control AssociationTM

Instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. For other copying, reprint or republication, permission must be obtained in writing from theassociation. Where necessary, permission is granted by the copyright owners for those registered with the Copyright Clearance Center (CCC), 27 Congress St., Salem, Mass. 01970, to photocopy articlesowned by the Information Systems Audit and Control Association Inc., for a flat fee of US $2.50 per article plus 25¢ per page. Send payment to the CCC stating the ISSN (1526-7407), date, volume,and first and last page number of each article. Copying for other than personal use or internal reference, or of articles or columns not owned by the association without express permission of theassociation or the copyright owner is expressly prohibited.

www.isaca.org

it performance improvement with cobit and the sei cmm

Documents