software acquisition management defense acquisition overview (1) software measurement sam executive...

Software Acquisition ManagementDefense Acquisition Overview (1)

Software Measurement

SAM Executive Seminar


What is “Performance Measurement”?

Performance Measurement: is the assessment of the efficiency and effectiveness of IT in support of an organization’s missions, goals, and quantitative objectives through the application of outcome-based, measurable, and quantifiable criteria, compared against an established baseline to activities, operations, and processes. US DoD Guide for Managing IT as an Investment

Rev 1.1


Enterprise Level

Functional Level

Program/Project Level

Results

Results

External Oversight

Results

Missions, Goalsand Objectives

Functional Requirements

and PerformanceMeasurementFramework

Linkage of Measures Rev 1.1

Levels of Measurement

Software Acquisition ManagementDefense Acquisition Overview (4)Rev 1.2

Spectrum of Performance Measurement

• Enterprise Level: focus is on on overall mission results. Measurement information is used on a cyclical (e.g., annual or quarterly) basis to choose policy directions and make mission decisions. Heavily involved in investment allocation of a project.

• Functional Level: focus is on unit results where information is needed to manage and improve operations. Measurement information is used on a periodic (e.g., quarterly or monthly) basis to report on performance of major organizational functions that cross multiple programs/projects or acquisitions. Heavily involved in selection, evaluation, and requirements approval of a project.

• Program/Project Level: focus is on activity and task information needed to make and execute tactical project and program management resource allocation decisions. Measures are performance oriented. In addition to use by the project manager and staff, these measures are combined, synthesized, and reported to the functional level. Heavily involved in day-to-day project management activities.

LessDetail

MoreDetail


Efficiency vs. Effectiveness

• Efficiency: this criteria demonstrates that an organization is employing the best use of available resources. For example:

– Are efforts completed within estimates?

– Were resources expended optimally?

• Effectiveness: this criteria demonstrates that an organization is doing the right things. For example:

– Achieving missions and goals

– Generating satisfied customers

– Producing work of high quality

Rev 1.1


What Program Managers Want to Know

• Is there really a problem?• What is the scope of the problem?• What is causing the problem?• Are there related problems?• Can I trust the data?• What should I expect--What will happen?• What are my alternatives?• What are recommended courses of action?• When can I expect to see results?

metrics!

2.3


“How do you know?”“What does that mean?”

“Can you show me?”

PM “Common-Sense”

Metrics

Categories of Project-Level Metrics

Rev 2.2

ProcessMetrics

CMM (SEI)SDCE (USAF)

SPICE (ISO), etc.

Quality Metrics

Error DensityComplexity

Reliability, etc.

Management Metrics

ProgressSize & Cost StatusResource Margins

Volatility, etc.

Metrics

Metrics

MetricsMetrics

Even more Metrics!

Metrics

Metrics

Metrics


Some Key Software Measurement Principles*

• Software measures should be driven by program-specific issues and objectives• The developer’s software process defines how the software is actually measured• Collect and analyze low level data• Use the measurement process as a basis for objective communications• Use a structured analysis process to trace measures to decisions• Measurement results must be considered in the context of other information from the program• The PM must have a measurement analysis that is independent of the software developer’s• Software measurement must be an integral part of program management throughout the life cycle•The primary measurement analysis focus should be at the single program level

PM Note: A myth exists that program managers can just collect data and plot graphs to form an effective measurement process. Although possibly true in a static, precedented software development environment, such not the case for the majority of DoD software acquisitions. The measurement must be dynamic because of the constantly changing issues and complexities in DoD contracting

*Courtesy of the PSM’s Practical Software Measurement Guidebook

Rev 2.1


Examples of Software Measures

Rev 4.4

• Software Size• Software Staffing• Software Complexity• Software Progress• Problem Report/Change Report Status• Build Release Content• Computer Hardware Resource Utilization• Milestone Performance• Scrap/Rework• Effect of Software Reuse • Requirements Volatility

Note: This is a set of management indicators shown here that might be used on a software development project....There is no intent to impose these indicators or to preclude using any other ones...the acquirer picks a subset driven by program risks and issues and in accordance with service policies.


What’s a “METER”?


Example of Mapping Project Information Needs

Project - Specific Issues Common Issue Areas

A gg ress ive S chedu leC on cu rren t A c tiv itiesC ritica l Dep en denc ies

C hang ing M iss ion O bjec tives

Q uestion ab le S ize E stim atesU nstab le R equirem en ts

R e liability O bjec tivesM a inta in ab ility R equ irem ents

Fixed B ud getS taff E xperience S taff A va ilab ility

Schedule and Progress

Product S ize and Stability

Product Quality

Resources and Cost

Practical Software and Systems Measurement

PSM Version 5.0c, 11PSM All rights reserved.




PSM Mapping ofInformationCategories,Concepts, andMeasures

In form ationCatego ries

M easurab leCo ncepts

Inform ation Category - M easurable Concept - M easure

S chedule and P rogress

R esources and C ost

P roduc t S ize and S tability

M ilestone Com pletionC ritica l Pa th P erform anceW ork U nit P rogress

Increm ental C apability

P ersonnel E ffo rt

P hysical S ize and S tability

Functional S ize andS tability

F inancial P erfo rm ance

E nvironm en t and S upport R esources

Pro spective M easures

M ilestone Da tesS lack T im eR equ irem ents TracedR equ irem ents TestedP roblem R eports O penedP roblem R eports C losedR eview s Com ple tedC hange R equests O penedC hange R equests ResolvedU nits D esigned U nits C odedU nits IntegratedTest Cases A ttem ptedTest Cases P assedA ction Item s OpenedA ction Item s Com ple tedC om ponen ts Integra tedFunctionality Integrated S ta ff Leve lD eve lopm ent EffortE xperience Leve lS ta ff TurnoverB CW S , B C W P, AC W PB udget C ostQ uan tity N eeded Q uan tity Availab leTim e Availab leTim e UsedD atabase S izeC om ponents InterfacesLines o f C odeR equ irem entsFunctional ChangesFunction Po in ts



In form ationCatego ries

M easurab leCo ncepts

Inform ation Category - M easurable Concept - M easure

S chedule and P rogress

R esources and C ost

P roduc t S ize and S tability

M ilestone Com pletionC ritica l Pa th P erform anceW ork U nit P rogress

Increm ental C apability

P ersonnel E ffo rt

P hysical S ize and S tability

Functional S ize andS tability

F inancial P erfo rm ance

E nvironm en t and S upport R esources

Pro spective M easures

M ilestone Da tesS lack T im eR equ irem ents TracedR equ irem ents TestedP roblem R eports O penedP roblem R eports C losedR eview s Com ple tedC hange R equests O penedC hange R equests ResolvedU nits D esigned U nits C odedU nits IntegratedTest Cases A ttem ptedTest Cases P assedA ction Item s OpenedA ction Item s Com ple tedC om ponen ts Integra tedFunctionality Integrated S ta ff Leve lD eve lopm ent EffortE xperience Leve lS ta ff TurnoverB CW S , B C W P, AC W PB udget C ostQ uan tity N eeded Q uan tity Availab leTim e Availab leTim e UsedD atabase S izeC om ponents InterfacesLines o f C odeR equ irem entsFunctional ChangesFunction Po in ts





PSM Mapping ofInformationCategories,Concepts, andMeasures(continued)

InformationCategories

MeasurableConcepts

Prospective Measures

Information - Category - Measure Mapping

Product Quality

Process Performance

Technology E ffectivenessCustomer Satisfaction

Functional Correctness

PortabilityUsabilityDependability-ReliabilityProcess Compliance

Process E fficiency

Process E ffectiveness

Technology Suitab ilityTechnology VolatilityCustomer Feedback

Customer Support

Supportability-

E fficiency

Maintainability

DefectsAge of DefectsTechnical Performance LevelTime to RestoreCyclomatic ComplexityU tiliza tionThroughputResponse Tim eStandards ComplianceOperator E rrorsMean Time to FailureReference Maturity RatingProcess Audit Find ingsProductivityCycle TimeDefects Conta inedDefects EscapingRework E ffortRework ComponentsRequirements CoverageBaseline ChangesSatisfaction RatingsAward FeeRequests for SupportSupport Time




software acquisition management defense acquisition overview (1) software measurement sam executive...

Documents