measurement news webinar

We want to thank all of you for joining us in todays Webinar.

We’ll be presen9ng a quick overview of the material published in Measurement News around building a credible Performance Measurement Baseline (PMB) and assessing program performance in ways beyond just the Earned Value numbers.

We’ll show the steps needed to development that PMB and assess program progress using the measures contained in the IMP and IMS.

These measures start with Measures of Effec9veness and Measures of Performance in the Integrated Master Plan.

The outcomes of Work Packages in the Integrated Master Schedule, can inform the Technical Performance Measures at the subsystem and system level in the IMP.

At all level reducible and irreducible risks play a cri9cal role in defining a credible PMB. The buy down of reducible risks and the margin needed to handle the irreducible risk must be represented in the IMP and the IMS.

1

CPM Webinar, 3 October 2014 Using Technical Progress to Inform Earned Value Performance Glen B. Alleman, Niwotridge, L.L.C. Tom Coonce, Ins9tute for Defense Analyses Rick Price, Lockheed Mar9n

2

The use of Earned Value Management is guided by EAI-‐748-‐C, where 32 Guidelines define the processes needed to have a compliant Earned Value Management System (EVMS).

Early in 748-‐C, it states we should objec9vely assess accomplishments at the work performance level. This means we need to measure the work performance where the work is performed.

This is logical. If we plan the work at a specific level – meaning the WBS level – then we should assess the performance of the work at the same level.

748-‐C also tells us that quality and technical content of the work are outside of the processes of Earned Value.

This sounds a bit odd, assessing the accomplishments of the work surely must involve assessing the technical and quality aspects of that work.

In this webinar, we’ll focus on connec9ng the dots between Earned Value Management and Technical Performance assessment.

But we’re going to go into more detail; around risk, because it’s a topic that inconsistently dealt with on many programs. Our Measureable News ar9cle, Building A Credible Performance Measurement Baseline Establishing a credible Performance Measurement Baseline: with a risk adjusted Integrated Master Plan and Integrated Master Schedule, starts with the WBS and connects Technical Measures of progress to Earned Value, addressed all measures of performance in more detail.


Using Earned Value Management processes is a cri9cal success factor for projects. EVM shows the Efficacy of your dollar.

§  For the dollar spent, did you get a dollar in value?

§  Did you get that dollar’s value on the day you planned to get that value?

§  If you didn’t get a dollar’s value for a dollar spent, when will the dollar of value be earned?

The answers to these ques9ons provide visibility into the performance of the project. But this assumes that the assessment of the value earned in exchange for the cost, has some connec9on to other abributes of the project.

§  Effec9veness of the project’s outcomes in accomplishing the mission.

§  Performance of the project’s outcomes in enabling the effec9veness.

If the outcomes perform as needed (fit for purpose) or aren’t effec9ve in accomplishing the mission (fit for use), then the earned value and it’s indices – Cost Performance Index and Schedule Performance Index – don’t match the technical performance of the project

3


4

Using DI-‐MGMT-‐81861 as our star9ng point, we need to show how to connect “objec9ve technical measures” with the Earned Value Performance metrics of the program.

Some of you may not be subject to 81861. It describes how to submit the monthly Integrated Program Management Report on a monthly basis, star9ng with the WBS cumula9ve performance to date and the current period performance.

These two measures should match. If not we need to find out why, which one is in error, and what the correc9ve ac9ons are to get them back in syn.

When they do match we then have visibility to all three variables of all projects. Cost, Schedule, and Technical Performance.

If they don’t match, we not have a place to look for the mismatch.


Here’s one look at the element of a project.

§  WBS – we all know about breaking down the work into the individual deliverables.

§  Cost – the budget needed to produce these deliverables.

§  Statement of Work – describing what needs to be delivered.

§  Technical Performance Measures – the assessment of the technical performance if each deliverable. There are other measures than just Technical Performance Measures.

•  Measures of Effec9veness •  Measures of Performance •  Key Performance Parameters •  Other …ili8es of the project –

maintainability, serviceability, … §  Integrated Master Plan and Integrated

Master Schedule – showing the planned increasing maturity of the deliverables and the planned work to make this increasing maturity appear on the needed date

§  Risk – both reducible and irreducible risk to the success of the project

5


The elements of the Performance Measurement Baseline begin with the Statement of Work (SOW) which may include a Concept of Opera9ons. These document say what the system must be able to do. The government frequently defines a standard WBS per 881C, but it is up to the contractors to refine the WBS to deliver products that sa9sfy the requirements enumerated in the SOW and CONOPS. The Contractors’ WBS contains all the products that must be delivered and the CWBS Dic9onary defines the scope of each product and defines the criteria that says when the product meets the Technical and Opera9onal Requirements . The Integrated Master Plan (IMP) is an Event-‐Based “plan” describing the maturity of each Major Deliverable. These deliverable are comprised of systems and subsystems to meet the system’s Measures of Effec9veness (MOEs) that are ar9culated in the Capability Development Document (CDD) by way of Key Performance Parameters. Work performed in the Integrated Master Schedule and it’s Work Packages produces the systems and subsystems the meet the Accomplishment Criteria. Risk Management is shown in yellow to reflect the fact that the planning and implementa9on ac9vi9es must adjust for and manage the risks that can jeopardize delivery the required capabili9es. The chart also shows that Technical Performance Measures informing earned value. TPMs show the appropriate technical progress at the work package level of detail in the IMS. TPMs must be met in order for the subsystems and systems to meet the acceptance criteria. Use of the TPM permits the Program Manager to keep the program on track toward accomplishing the required system capabili9es. Cost and schedule progress depend on this informa9on to provide a complete and consistent progress picture.

6


This chart and the next one describe 6 steps needed to create the credible Performance Measurement Baseline.

Step 1: As started earlier, all planning begins with a thorough analysis of the SOW, SOO and CONOPS and this results in a comprehensive CWBS and Dic9onary that defines the criteria of what cons9tutes acceptable deliverables. The WBS is paramount in defining the products.

Step 2: The IMP shows the progression of maturity of those deliverables by assuring that the systems and subsystems meet the MOEs and MOPs as defined by the KPPs.

Step 3: Major “reducible risks” that threaten successful deliverables must be addressed in the top-‐level plan. The PM needs to decide whether the risk will be mi9gated and his plans for doing so will be reflected later as he develops the IMS. The PM may need to change logic, add resources or 9me or all three within the IMS For those risks that are not mi9gated, the PM must set aside Management Reserves. We describe how this should be done on the next slide.

7


Step 4: Aler the IMP is created and major discrete risks plans are established, the contractor must develop the IMS that contains work packages and task ac9vi9es needed to create the subsystems and system deliverables that meets the Accomplish Criteria. This results in a cost-‐phased plan from contract start to contract comple9on.

Step 5: The cost-‐phase plan must next be adjusted for irreducible risks. These are risks that occur in the normal process of developing such systems where the PM can not take specific ac9on to mi9gate the risks. Rather, the PM can add a schedule cushion or Margin within the phased plan to accommodate these risks. This is done by performing a Monte Carlo simula9on on the plan with the known dura9on distribu9ons within the IMS ac9vi9es. More on this later. When completed, this results in a Performance Management Baseline

Step 6: Aler the schedule margin has been added to the IMS, the PM must calculate the Management Reserve using the PMB and the specific irreducible risks that were not mi9gated, to determine the Management Reserves. To do this, the PM performs a Monte Carlo Simula9on on the PMB to determine the confidence level of the plan and calculates the amount of management reserve needed to bring the contract budgeted baseline (CBB) to an acceptable confidence level.

8


As state earlier, TPMs show the appropriate technical progress at the work package level of detail in the IMS and they are the objec9ve measures of progress.

They must be met in order for the subsystems and systems to meet the acceptance criteria. Cost and schedule progress depend on this informa9on to provide a complete and consistent status picture at any 9me.

9


The WBS is the star9ng point of our efforts.

The WBS tells us what is to be delivered on the program and with the WBS Dic9onary, SOW, SOO, ConOps and other documents, what the objec9ve technical measures are for these deliverables.

In some sense the WBS appears to be simple. In fact it’s hard work to assemble the WBS in the proper manner. MIL-‐STD-‐881C is the star9ng place. But reading 881C needs to take into considera9on the specific of the program under management.

The WBS is the architectural structure of the program. It shows the rela9onship between the skeleton parts of the program.

10


The WBS as a document needs to have integrity is what is says and what it contains.

This integrity starts with the MECE principle. As an informa9on organizing processes – and that’s what the WBS is, MECE

1.  Mutually exclusive -‐ Informa9on should be grouped into categories so that each category is separate and dis9nct without any overlap

2.  Collec9vely exhaus9ve -‐ All of the categories taken together should deal with all possible op9ons without leaving any gaps.

As an accoun9ng system – McKinsey is where MECE started – it makes sense. No double coun9ng and no overlooking informa9on

For the WBS it means each deliverable is properly decomposed from it’s parent. No cross connec9ons between deliverables. No missing elements for the deliverables

11


The “goodness” of the WBS is assessed by the visibility to the deliverables, their rela9onships to each other, their parents and children in the MECE manner.

No widows, nor orphans, no rela9onships between cousins, only parent childe. And the parent is a signal parent .

12


Risk management starts with the WBS.

The WBS Dic9onary contains a descrip9on of the outcomes of the work performed in the IMS to produce the outcomes in the WBS.

Risks are assigned to describe the impediments to producing the outcomes that fulfill the MOEs, MOPs, and TPMs.

Each of the risks is entered into the risks register. The probability of occurrence, the probability of the impact, the cost of the impact, the cost to mi9gate the risk, are all in the Risk Register.

With these probabili9es, and the period over which they are in effect is then the basis of the Monte Carlo Simula9on of the risk exposure to the program for the reducible risks from the WBS.

13


The Integrated Master Plan is a unique approach to defining what Done looks like for a program. It takes the elements of the program defined in the WBS through increasing maturity levels to produce the end item deliverables of your contract.

In a tradi9onal approach the work is planned with ac9vi9es or collec9ons of ac9vi9es. These are linked in a sequence to produce outcomes from the work efforts.

What is missing is a structured assessment of the increasing maturity of the products produced by this work in units of measure that address the effec9veness and performance of the deliverables.

The IMP approach is especially useful for development efforts where increasing maturity of system design & integra9on efforts leads to procurement, fabrica9on, assembly and test of a system that meets stakeholder needs. This is the basic premise of Systems Engineering.

14


Our first step here is to recognize the difference between a Plan and a Schedule.

The PLAN is a procedure used to achieve an objec9ve. It is a set of intended ac9ons, through which one expects to achieve a goal.

The SCHEDULE is the sequence of these intended ac9ons placed against 9me, needed to implement the PLAN.

But we don’t what to mix the PLAN with the SCHEDULE.

The PLAN is the strategy for the successful comple9on of the project. It describes “What” needs to be accomplished to meet the technical requirements and goals of the program.

In the strategic planning domain, a PLAN is a hypothesis that needs to be tested along the way to confirm where we’re headed.

The SCHEDULE builds off the plan, providing the order of the work, the expected 9me to do the work, and an appropriate level of detail needed to execute the PLAN. It provides status of work done and the forecast of work to be done.

We need both. PLANS without SCHEDULES are not executable.

SCHEDULES without PLANS have no stated mission, vision, or descrip9on of success other than the execu9on of the work.


The hierarchical structure of an IMP depicted here helps understand the rela9onship of the elements of the IMP and the rela9onship of schedule to the Plan. The Events are the top level of the hierarchy and represent key assessment points for the increasing maturity of the system. They are typically sequen9al. For most DoD programs, events may include a System Requirements Review (SRR), a System Design Review (SDR), a Preliminary Design Review (PDR), a Cri9cal Design Review (CDR), a Test Readiness Review (TRR), and End Item Delivery. These may not apply directly to your program but I would encourage you to assess what the key maturity assessment events would be in your domain.

Each event is then broken down into accomplishments against each product or process defined in the WBS that represent the required maturity level established by the event.

Criteria are then developed for each accomplishment that provide objec9ve measurements defining what’s required to assess an accomplishment as done.

The schedule (the IMS) then takes all of the IMP elements and defines work packages and tasks at an appropriate level of detail needed to reflect the needed integra9on between efforts, enabling accurate forecas9ng and

16


Here are the Measures of Effec9veness (MOEs), Measures of Performance (MOPs), and specific Key Performance Parameters (KPPs) for the turn-‐of-‐the century Wright Brothers Heavier than Air Military Flyer. This informa9on was extracted from the actual Army Signal Corps’ Request for Proposal in December 1907.

The MOEs describe capabili9es in the targeted opera9ng environment and the MOPs get more specific about how the plaoorm must perform, e.g., it must fly at an average of 40 miles per hour in s9ll air and travel at least 125 miles.

The two KPPs deal with training and sustainment.

The Wright Brothers had six months to design and build the aircral and promised to do so for $25,000. It was a fairly ambi9ous schedule and geqng the aircral to meet the performance requirements meant designing and building aircral to maximize thrust and minimize drag and maximize lil.

Weight obviously had to be controlled and kept within limits so that MOE and MOPs could be met. In subsequent slides, we show how weight could have been used as a Technical Performance Measure to inform cost and schedule progress as the plane was designed and built.


Risks that come for probability distribu9on, that is there is a probability that something can happen, are reducible.

We can spend money to reduce the probability of occurrence.

18


There are two types types of uncertainty inherent in any type of development endeavor: those the are specific and reducible, i.e., one can do something about them to reduce or eliminate them and those that are irreducible, ones that naturally occur is such ac9vi9es the specifics of which can not be iden9fied or mi9gated.

These two uncertain9es are also categorized as Epistemic (for specific events) and Aleatory (naturally occurring).

During step 3, the PM iden9fies the specific risk events that could jeopardize the system in delivering the requirements. The informa9on about these risk events are recorded in the risk register.

At a minimum, they include a descrip9on of the risk, the WBS elements which could be affected, the probability of occurrence and the technical, cost and schedule consequences if they occur.


Once the specific reducible risk are iden9fied, the contractor must decide which risks will be mi9gated, the cost of doing so, and the remaining risk exposure aler the mi9ga9on ac9ons have been taken.

Risk mi9ga9on ac9ons must be included in the IMS. With budget and defined outcomes. These outcomes are the reduc9on of the risk.

20


This is an example of a risk mi9ga9on plan “buy down” over 9me.

Each numbered step represents a specific ac9on defined to reduce either the probability or impact of the risk.

These ac9ons are described in the Risk Management Plan, represent contract scope and are budgeted, scheduled, and tracked in the IMS and PMB just like any other contract scope.

Measuring performance against a risk reduc9on plan can be a form of technical performance measurement.

21


With the Integrated Master, the reducible risks from the WBS, let’s build the Integrated Master Schedule.

The IMS tells us what work must be performed in what order to produce the outcomes for the Criteria that complete the Accomplishments to increase the maturity of the deliverables assessed at each Event.

22


The Integrated Master Schedule is the road map to producing the outcomes of the project.

This picture is likely no9onal, but illustrates the core concept of a Master Schedule. No9ce the longest work ac9vity – training the astronauts. The other work was technical and based on standard engineering processes. New materials had to be developed, processes to assemble the materials, and tes9ng them.

But the human factors were unprecedented.

The IMS is the mechanism of delivering the Criteria needed to sa9sfy the Accomplishments back in the pyramid chart where the ver9cal integra9on was shown.

The founda9on of the program's success is a credible, risk adjusted IMS.

We’ll see how this is done in this sec9on.

23


The term IMS is known in many domains. But the Integrated Master Plan (IMP) is just now coming to domains outside the US Department of Defense.

The IMP is an event–based, plan of a hierarchy of Program Events, with each Event supported by Accomplishments, and each Accomplishment supported by a specific Criteria to be sa9sfied for its comple9on. The three elements of the IMP are:

1.   Event – an assessment point at conclusion of significant program ac9vi9es;

2.   Accomplishment – is the planned result(s) prior to or at comple9on of an Event that defines a level of the program's progress;

3.   Criteria – provides a defini9ve evidence that a specific Accomplishment has been completed at the conclusion of the Work Package.

The Integrated Master Plan (IMP) is expanded to a 9me–based Integrated Master Schedule (IMS) to produce a networked, mul9–layered schedule showing all the detailed tasks required to accomplish the work effort contained in the IMP.

The IMS flows directly from the IMP and supplements it with addi9onal levels of detail. It incorporates all of the IMP events, accomplishments, and criteria; to these ac9vi9es it adds the detailed tasks necessary to support the IMP criteria along with each task’s dura9on and its rela9onships with other tasks.

This network of integrated tasks, when 9ed to the start date (e.g., contract award), creates the task and calendar–based schedule that is the IMS.

24


Tom spoke about the reducible risks the result from the reducible uncertainty. These risks – the reducible ones – can be bought down by spending 9me and money.

Irreducible uncertainty and the resul9ng risk can’t be bought down. We can’t pay money to make the irreducible uncertainty go away.

By defini9on it is always there. It’s part of program.

We can only handle irreducible risk with margin. Cost margin, technical margin, or schedule margin.

25


Irreducible uncertainty starts with the natural variability of the underlying processes.

The natural randomness of the work ac9vi9es, the technical performance of a product or process.

This is the Deming variance. Spending money in an abempt to reduce it is a waste – Muda in Japanese

As well ambiguity about the variances in a processes is irreducible. This is a natural ambiguity that more money and 9me can’t address. So margin is needed to protect the project from these naturally occurring variances and ambigui9es.

The last class of irreducible risk is know risk, but a risk we can’t do anything about. That is it is un-‐mi9gable. There is a possibility that the comet will hit the building we work in. It is possible but not likely probable. But seemingly other low probability events have been in the news.

The probability that an earthquake will trigger a tsunami that will breach a seawall built to a specific height and flood the basement of the nuclear power sta9on where the auxiliary power – the site power – generators were places because it was too expensive to place them on the 3rd floor of the equipment room. The result was the lose of site power and the inability to restart the reactor when it scrammed and was cooling too fast and was going the warp the fuel rods.

All known but un-‐mi9gateable – at least at the 9me of the design basis.

Aler the fact of course, it was mi9gable. So tes9ng the “mi9gatability” of the risk is cri9cal.


One star9ng point for irreducible risks is Reference Class Forecas8ng. RFC can be applied to reducible risks as well. Bent Flybjerg’s picture here are the steps to developing a reference class forecast. Reference class forecas9ng was originally developed to compensate for the type of cogni9ve bias iden9fied in pioneering work on planning and decision making carried out by Kahneman and Tversky (1979a) and Kahneman 1994. Kahneman and Tversky found that errors of judgment are olen systema9c and predictable rather than random, manifes9ng bias rather than confusion, and that any correc9ve prescrip9on should reflect this. They also found that many errors of judgment are shared by experts and lay people alike.

Finally Kahneman and Tversky found that errors remain compelling even when one is fully aware of their nature. Thus awareness of a perceptual or cogni9ve illusion does not by itself produce a more accurate percep9on of reality, according to Kahneman and Tversky (1979b: 314).

Awareness may, however, enable one to iden9fy situa9ons in which the normal faith in one's impressions must be suspended and in which judgment should be controlled by a more cri9cal evalua9on of the evidence. Reference class forecas9ng is a method for such cri9cal evalua9on. Human judgment, including forecasts, are biased. Reference class forecas9ng is a method for unbiasing forecasts.

27


The cri9cal concept in irreducible uncertainty and the resul9ng risk is that every node in the IMS is a probabilis9c process.

The fancy term is a stochas9c process. That is it behaves sta9s9cally and resul9ng in a naturally occurring variability that cannot be reduced.

When these stochas9c processes are connected in a network, their individual behavior produces other – many 9mes in predictable – behaviors.


For an example of schedule margin needed to protect from the natural varia9ons, let’s look at a case study developed for a training course.

Here the contract due date was August 31 1908. For the Wright Brother’s delivery of their Flyer

The determinis9c schedule shows a date of August 4 1908. The 80% confidence date for the deliver, given the naturally occurring variances in the work dura9on was August 14 1908.

The result is a 10 day schedule margin between the determinis9c date and the 80% confidence date for the probabilis9c comple9on.

The steps needed to produce the IMS, model the probabilis9c distribu9ons and run the Monte Carlo are beyond our short 9me here.

So we’ll use this as a no8onal example for now.

29


Once you’ve determined an acceptable confidence level for your schedule and the amount of schedule margin needed to achieve that confidence, then you need to decide where to place it in your schedule and how to manage it. Some prefer to hold it all at the end item delivery milestone. That can be effec9ve but it can also be hard to communicate rules for acceptable consump9on. An alternate approach is shown here with margin baselined at key points in the process. Good candidates for placement of margin are high convergence points-‐where many thing come together (integra9on points) or where many things get tested together; these are olen “points of discovery” in complex systems where things don’t go as planned; these represent irreducible uncertainty that likely can only be managed with margin.

How you manage these is just as important as where you place them. In the graphic, no9ce the dividing line between green and yellow is not a straight line (9me-‐based) degrada9on. It’s stair-‐stepped with each step 9ed to re9rement of irreducible risk associated with comple9on of key tests. With this approach, all team members knew if we fell behind (fell into the yellow) we had to regain the 9me back (work a Saturday). We measured performance to this plan and implemented correc9ve ac9on twice to “Stay Green”. Forcing correc9ve ac9on to stay green enabled on-‐9me delivery of this planetary spacecral in a 9ghtly constrained schedule.

30


With each of the elements in Steps 1 through 5, let’s assemble them into our Performance Measurement Baseline.

The Performance Measurement Baseline (PMB) is a 9me-‐phased budget plan for accomplishing work against which contract performance is measured. It includes the budgets assigned to scheduled control accounts and the applicable indirect budgets. For future effort, not planned to the control account level, the PMB also includes budgets assigned to higher level Contractor Work Breakdown Structure (CWBS) elements, and to Undistributed Budgets (UB). It does not include Management Reserve (MR).

32


When we started this Webinar we men9oned connec8ng the dots.

Well here’s the dots and this is how they are connected.

This chart speaks for itself, with the dots connected to the lines.

We’ve included the no9on of con9ngency here. The Department of Energy uses con9ngency at the agency or field office level. The Department of Defense doesn’t formally do this.

33


The Wright Brothers proposed to build a Heavier than Air Flying machine that met specific performance and measures of effec9veness. In order deliver the system that met these requirements, weight had to be controlled.

On the next slides, we use weight as the TPM to inform cost and schedule progress during this six month design/build schedule.

34


Let’s re-‐emphases that point that technical progress needs to drive the reported cost and schedule progress so that stakeholders have a clear and consistent status picture and provide leading indicators so that management can take correc9ve ac9on.

Keeping the program green is the day-‐to-‐day focus and that ensure that the system is delivered on 9me, within the resources and accomplishes the required capabili9es.

Our Measureable News white speaks to these execu9on process in more detail by having Earned Value measures be informed by objec9ve Technical Performance

35


Here is how technical performance informs the cost and schedule status and in turn, how the reported cost and schedule informa9on informs the technical progress.

In this picture, weight is above the upper design threshold for the Wright Brothers’ Flyer and the report performance is consistent because it falls below the planned Budgeted Cost of Work Scheduled.

36


Here’s the summary of the steps needed to create a credible plan – one that has been risk-‐adjusted for both reducible and irreducible risks, and quan9fiably protected with Management Reserves to cover the known risks that were not mi9gated.

The chart also reminds us that cost and schedule progress must be based on objec9ve technical measures of performance and these are typically manifested in Technical Performance Measures (TPMs).

37


Here are some references that have guided our efforts in connec9ng the elements of a credible Performance Measurement Baseline.


38


39

40


measurement news webinar

Technology