isa 201 intermediate information systems acquisition · - milestone progress - number of milestones...

ISA 201Intermediate

Information Systems Acquisition

Lesson 5Measurement and Analysis

3

• True or False: Measures are defined at the beginning of the program and remain constant throughout.

In-Class Quiz

• _____________ drive what you measure, analyze, and evaluate.

• A _____________ is a measure of a single attribute by a specific method; raw data collected.

Team 1

Team 2

Team 3

• True or False: The analysis process changes through a program based on life-cycle phase, acquisition strategy, and current activities.

• A change in requirements (size and stability) is a leading indicator of what other changes later in a program?

Team 4

Team 5

Measurement and Analysis

Lesson Overview

Today you will learn to:

Measurement and Analysis 4

Overall: Given a DoD IT/SW acquisition scenario and the Practical Software and Systems Measurement (PSM) methodology, create appropriate measures that allow program/enterprise leadership to make effective decisions.

•Describe the Practical Software and Systems Measurement (PSM) Principles, Practices, and Measurement Methodology.

•Given a DoD IT/SW scenario and the PSM methodology, identify information needs.

•Create measures based on an IT/SW program/enterprise’s information needs.

•Apply measurement results to support effective IT/SW program/enterprise decisions.

5Measurement and Analysis

• Fundamental Measurement Principles• Plan Measurement - Information Needs• Exercise 1• Plan Measurement - Select and Specify Measures• Exercise 2• Perform Measurement

Measures Lesson Overview


Project Decisions

Why Measure?

Project Measures

Measurement and Decision Making

InformedInfluencer

Fact-BasedDecision

Maker

WatchWatcher

ReactiveManager

Maximum

Maximum

Minimum

Minimum

DecisionAuthority

MeasurementInformation


• Data presented during the Challenger Launch decision.• Identifies “distressed” O rings by temperature.• Omits flights that did not have problems. • No indication of launch day temperature (31o).• Personnel working on O rings recommended no launch below 55o .• Data went through 5 management levels prior to decision.

Measurement and Decision Information

The right information at the right time is what we want to achieve. Information is the key. Not just data.


• Fundamental Measurement Principals

• Plan Measurement - Information Needs

• Exercise 1• Plan Measurement - Select and Specify Measures• Exercise 2• Perform Measurement



Measurement Process Model

McGarry, J., Card, D., Jones, C., Layman, B., Clark, E., Dean, J., & Hall, F. (2002). Practical Software Measurement.


Plan Measurement


The Plan Measurement activity is iterative throughout the life cycle


• Project/enterprise prioritized information needs define what measures are implemented and the focus of analysis

• These are based on the goals and objectives of the projects and the decisions that must be supported –what does the decision maker need to know?

• Must consider enterprise objectives, issues, and acquisition/technical mandates

• With limited program resources, measurement efforts must be focused on the critical information needs—these change over the course of the program

Information Needs


• Drive what you measure, analyze, and evaluate• Directly relate to established objectives• Define where you focus your attention• May be different for different organizational levels• Are usually dynamic• May have both positive and negative connotations

(opportunities and risks)• Are what you need to address to make and justify a

decision

Information Needs

Information needs are what you need to know

Measurement and Risk Management

• Current events• Narrower scope

of information and issues

• Objective data -easier to define

• Quantifiable relationships

• Defines triggers

• Future events• Expansive scope of

information and issues• Not easily

measureable• Wider uncertainty -

less tangible• Largely subjective data• Complex interactions

and interdependencies• Establishes priorities

AnalysisIntegration

DecisionProcess

RiskManagement

ProcessMeasurement

ProcessInformation Needs - Priority Data

Identified Risks - Exposure Data

14

• The measurement and risk management processes work together to identify, evaluate, and prioritize information needs

• Identified risks help to focus measurement requirements and measurement implementation priorities

• Measurement results help to identify new risks and evaluate the potential impact of risk

• Measurement analysis and risk analysis results are integrated to support the decision process

• Remember: not all information needs come from risks - Information needs defined outside of a formal risk

management process also need to be addressed

Measurement and Risk Management

15

Measurement Focus Across the Acquisition Lifecycle

Materiel Solution Analysis

Technology Development

Engineering & Manufacturing Development

Production & Deployment

Operation & Support

A B C

M/S ADefinition of stakeholder needs

Budget sufficiency

M/S BRequirements, architecture, design definition/volatility

Resource - facility sufficiency

Definition of TPMs

M/S CProgressQuality (defects)Financial performance

TPM performance

SustainmentQuality (defects)TPM performanceCybersecurity

"Be careful what you measure ... because that is what you're going to get."


• There are different “types” of information needs. In general these include: - Describing/quantifying IT process and product characteristics- Assessing the accomplishment of programmed activities and the

achievement of pre-defined thresholds- Quantifying the changes in outcomes due to a previous action(s)- Assessing the adequacy of the performance of the IT process or product

• The different “types” of information needs are supported by different measures and different analysis techniques

• Information needs are often written as questions: - Will this project meet its scheduled delivery date?- Are project requirements or story points being implemented as planned?- Is the project meeting budget targets?- Is the product meeting required quality thresholds?

Identifying Information Needs


• Strategic goals and objectives• Project goals and objectives• External requirements• Identified risks• Constraints and assumptions• Experience–high interest concerns• Planned decision points• Product acceptance criteria• Leveraged technologies

Defining Project–Specific Information Needs


Information Categories

Customer Satisfaction: Insight on how well the user’s expectations have been met

Technology Effectiveness: Insightinto suitability, technical performance, and stability of technologies

Process Performance: Insight into the maturity/ capability of the technical and management processes

Product Quality:Insight into the quality of the product

Size and Stability: Insight into changes in product function and structure

Resources and Cost: Insight into resources, such as people, dollars, facilities, and equipment

Schedule and Progress: Insight into achievement of milestones / events

Within each information category there are many

different information needs.


• Fundamental Measurement Principals• Plan Measurement - Information Needs

• Exercise 1• Plan Measurement - Select and Specify Measures• Exercise 2• Perform Measurement



• Review the case study material.• Identify three project-specific information needs that

apply to the Case Study project. One of these should be related to some aspect of program risk.

• Prioritize them according to importance (with “1” being the most important).

• Map your project specific information need (column #1) to the applicable PSM information category.

• Document your reasons for selecting each information need (reference facts from the write-up that support the selection) and what decision the need supports for the PM.

Group Exercise 1:Identify Information Needs


• Fundamental Measurement Principals• Plan Measurement - Information Needs• Exercise 1

• Plan Measurement - Select and Specify Measures

• Exercise 2• Perform Measurement



• A key aspect of the measurement process is defining and aligning the measures that support the decision maker’s information needs

• The measures that are actually implemented are a synthesis of user information needs and the ability to generate the measurement data

• The implemented measurement set is directly influenced by the instantiated program technical and management processes

• The decision maker needs to establish an “audit trail” between what is measured and the decisions that are made

• Most programs have data available from existing tools, etc. Data can be derived from existing processes and management activities. Some new data sources may need to be developed to support critical requirements.

Selecting and Specifying Measures


Information Model–Software Design


Entities and AttributesSpecific products and parameters to be measured

Measurement ConstructDocuments the detailed definition of a measure

Information ProductThe measures and interpretations

Work unit progress

Measurable ConceptDescribes possible ideas to satisfy information needs

What is the status of our design activity?

Information NeedCan be addressed by many different measures

Schedule and Progress of SW Design

• Design Plan• Records of completed

design units• Units to be completed (planned)• Status of the design units


Measurement Construct—Levels

Status of design units

Units completing design

% of the design package complete

Design Completion



• Measurement data can be derived from:- Existing processes, tools, and measurement activities- New data sources developed and integrated to support critical

requirements• Most programs have data available from existing tools and

measurement activities, including: - Problem report databases- Configuration management systems- Cost, financial management, and schedule control systems- Design tools- Risk management systems- Enterprise Resource Planning (ERP) Systems

Identifying Measurement Data


• The measures selected will vary based on the phase of the identified information needs, the phase of the program, and the available base measures

• Once selected, measures are document in a measurement plan

Mapping of Information Categories, Measurable Concepts, and Measures (ICM)

Reference full ICM Table

Prospective Indicators Sample Base Measures

Milestone Completion Is the project or service meeting scheduled milestones? Are critical tasks or delivery dates slipping?

- Milestone Progress - Number of milestones started and completed versus plan

Work Unit Progress Are specific activities and products completed as scheduled?

- Requirements Progress- Problem Reports Progress- Reviews Progress- Change Requests Progress- System Elements (Units) Progress- Test Cases Progress- Action Items Progress

- Requirements defined, traced, verified, validated- Problem reports discovered, closed- Reviews completed- Change requests opened, resolved- System elements designed, implemented, integrated, approved, qualified, accepted- Test cases developed, attempted, passed- Action items opened, completed

Schedule and Progress

MeasuresInformation Categories Measurable Concepts Questions Addressed

• Quantify the attributes of an enterprise, organization, product, project, etc.

• Helps to describe and categorize entities based on their characteristics (size, weight, color, quality, frequencies of occurrence, etc.)

• Provide a general context

Types of Measures:Characterization Measures

• Tell you if you are doing what you said you were going to do

• Largely relate to schedules and planned accomplishments

• Are what most measurement processes initially focus on

• Usually structured as expected vs. actual values

Types of Measures:Activity Measures

• Measure the change in a process, product, or enterprise attribute after an overt action is taken

• Address the impact(s) of a previous decision

• Encompass explicit exit criteria in the measurement definition

Types of Measures:Outcome Measures

• Measure the change in a process, product, or organizational attribute (outcome) against a defined threshold of required performance

• Requires that success factors be defined in measurable terms and linked to measured attributes

• Focus on effectiveness, efficiency, quality, and timeliness

Types of Measures:Performance Measures




• Exercise 1


• Exercise 2• Perform Measurement



• Select an information need from Exercise 1 and fill out the specification table.

• Refer to the Specification Example for Exercise 2 for additional information.

Group Exercise 2:Selecting Measures




• Exercise 1


• Exercise 2

• Perform Measurement



• A systematic method for collecting and transforming measurement data into usable information

• Like planning, performing must be flexible to adapt to changing information needs and evolving data

• Analysis is the primary task of this activity–it includes estimating, assessing the feasibility of plans, and tracking performance against plans

Perform Measurement


Analyze data to determine what actions need to be taken

Data

Evaluation Measures

AnalysisResults

Information

Questions

New Information NeedsProject andEnterpriseContext

AnalyzeData

Collect and Process

Data

Make Recommendations

Measurement Plan


• The measurement process supports three primary types of information driven analysis activities across the IT program life cycle:- Estimation and planning- Feasibility assessments- Performance evaluation

• The focus of the measurement process shifts based on:- Which of these activities are currently required- Life-cycle phase- Acquisition/development/maintenance strategies- Programmed tasks and activities

• Different measurement constructs and analysis techniques are used to generate the required information for each type of analysis

Analysis Requirements


Key IT Parameters Are Highly Interdependent

Program and systems IT

performance parameters and

associated information needs

are highly interrelated

Program decisions must consider these

relationships-the measurement and analysis

process helps to quantify them

Examples of current challenges include:• Acquisition process changes• Security/risk mandates• Resource constraints• Technology impacts• Life-cycle support focus

Key performance parameters include: • Product size and stability• Resources and cost• Schedule and progress• Product quality

The DOD IT Environment

changes rapidly making

defensible IT decisions critical

to project success

Decisions that result in a change

in one area almost always

impact another-strong cause and

effect relationships

exist


• Integrated analysis combines multiple indicators and focuses on the cause and effect relationships inherent between IT performance parameters

Integrated Analysis





• Exercise 1


• Exercise 2

• Perform Measurement



• Materials in this lesson are based on:- Practical Software and Systems Measurement (PSM)

project and materials - www.psmsc.com

- ISO/IEC/IEEE 15939, Systems and software engineering -Measurement process

- Practical Software Measurement, McGarry, J., Card, D., Jones, C., Layman, B., Clark, E., Dean, J., & Hall, F., 2002.

Sources and References

http://www.psmsc.com/

Summary


Today you learned to:

Overall: Given a DoD IT/SW acquisition scenario and the Practical Software and Systems Measurement (PSM) methodology, create appropriate measures that allow program/enterprise leadership to make effective decisions.

•Describe the Practical Software and Systems Measurement (PSM) Principles, Practices, and Measurement Methodology.

•Given a DoD IT/SW scenario and the PSM methodology, identify information needs.

•Create measures based on an IT/SW program/enterprise’s information needs.

•Apply measurement results to support effective IT/SW program/enterprise decisions.