driving innovative it metrics (project management institute presentation)

Driving Innovative IT MetricsFocusing on the CONDITIONS for Success

© 2013 Joe Hessmiller1

2© 2013 Joe Hessmiller

Schedule Check

You Are Here

Joe Hessmiller

• Over 30 years in IT; analyst, programmer, technical training manager, project manager, user interface designer, project management consultant, consulting sales and marketing manager.

• Worked with dozens of IT managers from Fortune 50 corporations and large government agencies.

• Current focus on software project risk management, specifically, identifying, monitoring and use of project risk “early warning signs”.


1. To establish the need for monitoring the metrics that really matter.

2. To identify the types of metrics that really matter.

3. Show how a familiar framework can be adapted for metrics identification (and communication).

4. Give you enough to use back at your office to improve your metrics program.


Objectives of Session

Agenda

1. The Innovative Metrics Opportunity

2. Why Do These Opportunities Still Exist?

3. What Metrics Should We Monitor?

4. Working With Conditions Data

5. Developing Innovative Metrics for Your Organization


Part One

The Innovative Metrics Opportunity(in Project Management)


7

• Despite Everything We’ve Tried, Project Success Rates Little Changed in 30 Years– McKinsey (17% threaten)– IBM (40% met 10X range)– KPMG (70% orgs)– Standish CHAOS Report

Source: http://calleam.com/WTPF/?page_id=1445

The Innovative Metrics Opportunity

8

• There Is Plenty of Opportunity for Improving Project Success Rates

Source: http://versionone.com/assets/img/files/ChaosManifesto2013.pdf



9

• Strategic Opportunity to Increase Market Competitiveness

Organizations with a mature PMO outperform those with an immature PMO by:

28% for on-time project delivery;

24% for on-budget delivery; and

20% for meeting original goals and business intent of projects.

Source: www.metier.com, According to PMI


http://www.metier.com/

10

Source: Paul D. Nielsen, “About Us: From Director and CEO Paul D. Nielsen,” Carnegie Mellon Software Engineering Institute, http://www.sei.cmu.edu/about/message/

• Operational Opportunity to Address Causes of Largest Project Cost ComponentAccording to the Carnegie Mellon Software Engineering Institute, “Data indicate that 60-80% of the cost of software development is in rework.”


Part Two

Why Do These “Opportunities” Still Exist?


12

• The “99% Complete” Syndrome– Everything is Fine…Everything is

Fine…– Oh, Oh, SURPRISE! We’re in Trouble!

• The real challenge to successful software project management is: – having the information

– that is very DIFFICULT to collect, analyze and act on

– in time to make a difference.

What You

Know

What YouDon’t Know…but Need ToKnow

Why Do These Opportunities Still Exist?

13

• It’s DIFFICULT to Have Project Managers Who Can Predict the Future



14

• It’s DIFFICULT to find project managers who can keep everyone on the same accurate, relevant and timely page.



15

• It’s DIFFICULT to Trust the Service Providers You Rely On When You Can’t Verify Their Behavior/Performance



SO…What Does an Innovative Project Manager Need to Do?

16

• Address these DIFFICULT Project Management Challenges

• By Having the Data That’s Really Needed to Be Successful

Part Three

What Metrics Should We Monitor?



• Tracking Progress–Backward Looking

• Managing Risk–Forward Looking


Tracking Progress Looking Backward

• Volume• Quality• Cost


What did we do?

• Alignment of IT Investments to Business Strategy• Cumulative Business Value of IT Investment• IT Spend Ratio – New Versus Maintenance• Critical Business Services

– Customer Satisfaction– Service Level Performance

• Operational Health– Outages– Security Incidents– Project Success Rate– Average Defect Rate

Source: Craig Symons, Forrester Research, 4-4-0820

© 2013 Joe Hessmiller

Tracking Progress Looking Backward – Enterprise Level


http://www.slideshare.net/anandsubramaniam/project-metrics-measures

Tracking Progress Looking Backward – Project Level


Managing RiskLooking Forward – Experts Agree on EWS

What should we do?

Managing RiskLooking Forward – Kappelman Research

• Kappelman Research– Derived List of

• Six People Factors• Six Process Factors

• For In-process Audits

23


Managing RiskLooking Forward – Dominant Dozen

• Project Management Institute– 10 Knowledge Areas– Things You Should

Know– Things You Should

Do

• For Gate Audits25

Managing Risk Looking Forward – Practitioners Too

• In Addition to Traditional –Key Performance Status

• What To Collect –Key Performance Conditions

• Intra-process Conditions• Inter-process Conditions

–Key Process/Practice Compliance


Summary: What Metrics Should We Monitor?

• How Collect– Intuition–MBWA–Survey Software–Purpose Designed

Software



• Pattern Recognition– Correlations Between Outcomes and

Precedent Conditions• Feedback Loops

– Correlations Drive KPI Algorithms

Need to Collect the Data for the Insights


The Future: PMBDProject Management Big Data

Part Four

Working with Conditions Data

(The Four Missing Metrics)


Three Important CONDITIONS to Monitor

• Expectations Management • Sponsor Involvement • Process Compliance

To Minimize Project Risk Factor

• Rework ProbabilityAt 40% of total project costs, Rework is the leading cause of projects running over budget and beyond schedule.


Managing Risk ESPR “The Missing Metrics”

The Four Missing Metrics

• SMART – Are expectations clear?

• SMPL – Is sponsor engaged?

• PAL - Are processes being

followed?

• PRPL – Are causes of

Rework being avoided?


SMART LevelTracks the clarity of assignments. The higher the SMART Level, the higher the level of understanding of what is expected. Therefore, less Rework and less management intervention required.

The SMART Level

8/10/2

010

8/21/2

010

9/1/2

010

9/12/2

010

9/23/2

010

10/4/2

010

10/15/2

010

10/26/2

010

11/6/2

010

11/17/2

010

11/28/2

010

12/9/2

010

12/20/2

0100.0

20.0

40.0

60.0

80.0

100.0

120.0

SMART IndexUpper Control LimitLower Control Limit

The SMART Level


SMART LevelMetric derived from participant feedback on the perceived Specificity, Measurability, Achievability, Relevancy and Timeframes for their assignments/deliverables.

SMART IndexWeek

SMART Index

Upper Control

Limit

Lower Control

LimitSpecific Measurable Achievable Relevant

Time-Based

8/10/2010 76.3 100 75 75 50 80 100 958/17/2010 76.3 100 75 75 50 80 100 908/24/2010 75.0 100 75 75 50 80 95 958/31/2010 82.5 100 75 80 70 85 95 95

9/7/2010 83.8 100 75 80 70 85 100 959/14/2010 82.5 100 75 90 70 85 85 909/21/2010 87.5 100 75 90 90 80 90 909/28/2010 92.5 100 75 100 90 80 100 9010/5/2010 88.8 100 75 100 90 75 90 95

10/12/2010 86.3 100 75 90 90 75 90 9510/19/2010 87.5 100 75 90 90 75 95 8510/26/2010 88.8 100 75 90 90 80 95 85

11/2/2010 92.5 100 75 95 95 80 100 9011/9/2010 92.5 100 75 95 90 85 100 95

11/16/2010 93.8 100 75 100 90 85 100 9511/23/2010 93.8 100 75 100 100 85 90 10011/30/2010 93.8 100 75 100 100 80 95 95

12/7/2010 88.8 100 75 90 90 85 90 9012/14/2010 92.5 100 75 90 90 90 100 9512/21/2010 95.0 100 75 95 95 95 95 9512/28/2010 92.5 100 75 90 80 100 100 100

Components of the SMART Level

The SMART Level


SMART LevelLooks like this project was back under control once the Relevance of the assignments were made clear to the staff.

8/10/2

010

8/21/2

010

9/1/2

010

9/12/2

010

9/23/2

010

10/4/2

010

10/15/2

010

10/26/2

010

11/6/2

010

11/17/2

010

11/28/2

010

12/9/2

010

12/20/2

01040.0

50.0

60.0

70.0

80.0

90.0

100.0

110.0

SMART IndexUpper Control LimitLower Control LimitSpecific MeasurableAchievableRelevantTime-Based

The SMART Level

SMART Level Influencers


SMPL Line

Tracks the participation level of the senior management and/or sponsor.

The SMPL Line

Senior Management Participation Level

Attention Needed

8/10/2

010

8/21/2

010

9/1/2

010

9/12/2

010

9/23/2

010

10/4/2

010

10/15/2

010

10/26/2

010

11/6/2

010

11/17/2

010

11/28/2

010

12/9/2

010

12/20/2

0100.0

20.0

40.0

60.0

80.0

100.0

120.0

SMPLUpper Control LimitLower Control Limit


SMPL LineIn this example, the level of participation in meetings, surveys, and other activities is compared to the planned participation by senior management


Components of the SMPL Line


SMPL LineIn this example, the components of the SMPL are shown separately to identify individual areas needing improvement.


8/10/2

010

8/23/2

010

9/5/2

010

9/18/2

010

10/1/2

010

10/14/2

010

10/27/2

010

11/9/2

010

11/22/2

010

12/5/2

010

12/18/2

0100.0

20.0

40.0

60.0

80.0

100.0

120.0

SMPLUpper Control LimitLower Control LimitStatus Meeting Atten-denace RateAPO Response RateDecisions Made/Delegated RateConfidence Level

SMPL Line Influences


PALMeasures likely level of process adherence based on conditions that would tend to lead to ‘short cuts’ on process..

Process Adherence Likelihood


PALThis table shows how problems with EVA lead to likely abandonment of process until project gets back on track. May be OK, must be identified to be managed.

Process Adherence Likelihood

Components of PAL Line


PAL

This illustrates the various components of PAL.

PAL Line Influences


Project Rework Probability LevelPRPL LineTracks the ‘probability’ of Rework based on changes in the conditions that are known to cause Rework.

8/10

/201

0

8/24

/201

0

9/7/

2010

9/21

/201

0

10/5

/201

0

10/1

9/20

10

11/2

/201

0

11/1

6/20

10

11/3

0/20

10

12/1

4/20

10

12/2

8/20

100.0

5.0

10.0

15.0

20.0

25.0

PRPL

Upper Control Limit

Lower Control Limit

Attention Needed

The PRPL Line


PRPL Line

SME Involvement, Team Confidence Level, Technical Capability and Requirements Stability are components of PRPL Line.

Project Rework Probability Level (PRPL) Week PRPL

Upper Control

Limit

Lower Control

Limit

SME Involvement

Technical Capability

Requirements Stability

Confidence Level

8/10/2010 5.0 20 0 100 100 80 1008/17/2010 10.0 20 0 80 100 80 1008/24/2010 12.5 20 0 80 100 80 908/31/2010 22.5 20 0 70 90 70 80

9/7/2010 17.5 20 0 80 90 70 909/14/2010 16.3 20 0 80 90 75 909/21/2010 11.3 20 0 80 90 85 1009/28/2010 23.8 20 0 80 60 85 8010/5/2010 23.8 20 0 80 60 85 80

10/12/2010 23.8 20 0 80 60 85 8010/19/2010 18.8 20 0 65 90 85 8510/26/2010 21.3 20 0 70 90 70 85

11/2/2010 16.3 20 0 80 90 70 9511/9/2010 16.3 20 0 80 90 70 95

11/16/2010 16.3 20 0 80 80 80 9511/23/2010 16.3 20 0 80 80 80 9511/30/2010 17.5 20 0 80 80 80 90

12/7/2010 12.5 20 0 80 90 80 10012/14/2010 8.8 20 0 85 90 90 10012/21/2010 5.0 20 0 90 100 90 10012/28/2010 0.0 20 0 100 100 100 100

Project Rework Probability Level

Components of PRPL Line


PRPL Line

SME Involvement, Team Confidence Level, Technical Capability and Requirements Stability influence rework probability.

Project Rework Probability Level

8/10/2

010

8/21/2

010

9/1/2

010

9/12/2

010

9/23/2

010

10/4/2

010

10/15/2

010

10/26/2

010

11/6/2

010

11/17/2

010

11/28/2

010

12/9/2

010

12/20/2

0100.0

20.0

40.0

60.0

80.0

100.0

120.0

PRPLUpper Control LimitLower Control LimitSME InvolvementConfidence LevelTechnical CapabilityRequirements Stability

PRPL Line Influences


Part Five

Developing Innovative Metrics for Your Organization


Five “C”s of Effective Metrics

• Connected to important goals (G-Q-M)• Complete so that all the factors important to

achieving the goal are included• Current so that they reflect current goals• Communicated so that they drive behavior• Calibrated so that values are comparable

across organizations and over time.

Source: Bob Lewis, InfoWorld, 12-28-11


A Gauge for Every Condition

Automotive Engineers Long Ago Defined the Critical Measures for Safe, Effective Engine Operation.


The Basic Automobile Dashboard

Automotive Gauge

Odometer

Clock

Fuel Level

Speedometer

Tachometer

Oil Pressure

Oil Temperature

Water Pressure

Water Temperature

Voltmeter


What Questions Are We Asking?

Automotive Gauge Asks the Question

Odometer How far have we gone?

Clock How much time has it taken?

Fuel Level How far can we go?

Speedometer How fast are we going now?

Tachometer How intensely is engine working?

Oil Pressure Do we have enough ‘lubrication’ to keep parts working well together?

Oil Temperature How smooth are interactions?

Water Pressure Do we have enough coolant to keep the engine producing?

Water Temperature How effective is the coolant in keeping the engine cool?

Voltmeter Is enough energy being applied to the other important systems?


The Basic Measures

Automotive Gauge Asks the Question To MeasureOdometer How far? Deliverables Delivered

Clock How long? Duration

Fuel Level How much further? Input Units Available

Speedometer How fast? Deliverables per Unit of Time

Tachometer How intensely? Effort Intensity

Oil Pressure Do we have enough lubrication to smooth interactions?

Supply of Lubricant to Smooth Interaction Between Components

Oil Temperature How smooth are interactions? Ability of Lubricant to smooth Interaction Between Components

Water Pressure Do we have enough coolant to keep the engine producing?

Supply of Coolant to dissipate excess engine heat

Water Temperature How effective is the coolant in keeping the engine cool?

Ability of Coolant to dissipate engine heat

Voltmeter Is enough energy being applied to the other important systems?

Ability to Support other Control and Comfort Systems


Comparative Metrics

To MeasureAutomotive Metric IT Metric

Deliverables Delivered Miles Service Level Achieved, Function Point Delivered

Duration Hour Hour

Input Units Available Gallons Resource Hour

Deliverables per Unit of Time Miles Per Hour Earned Value Per Clock Hour

Effort Intensity RPM Hours Worked Per Week/Available Hours

Supply of Lubricant to Smooth Interaction Between Components

PSI Stakeholder Interaction Satisfaction

Ability of Lubricant to Smooth Interaction Between Components

Degrees Number of Open Issues from Stakeholder Interactions

Supply of Coolant to dissipate excess engine heat

PSI Duration to Close Issues/Number of Issues

Ability of Coolant to dissipate engine heat

Degrees Number of Escalated Issues

Ability to Support other Control and Comfort Systems

Volts On Time Process Deliverables (Status, Reporting, Training)


The More Complex the Project Environment…


• To establish the need for monitoring the metrics that really matter.

• To identify the types of metrics that really matter.• Show how familiar framework can be adapted

for metrics identification (and communication).• Give you enough to use back at your office to

improve your metrics program.


Objective of Session

driving innovative it metrics (project management institute presentation)

Leadership & Management

joe hessmiller http

innovative metrics opportunity

project managers

innovative project manager

project management institute

project management consultant

project success rates

backward project level