pitfalls of project estimating: an applied lesson from dr. deming’s funnel experiment john miller,...

Pitfalls of Project Estimating: An Applied Lesson from Dr. Deming’s Funnel Experiment

John Miller, PMP, CLSSBB4/28/10

• Introduction• Dr. Deming’s Funnel Experiment• What is a Process?• Statistical Measures• Estimating Methods• Concepts• Averages and Standard Deviation• Data Points to Data Trends• Control Charts• Lessons from the Funnel Experiment• Using Excel to Avoid Common Mistakes

J. Miller, 4/28/10

Contents

IntroductionParametric and evidenced-based estimating tools have become very sophisticated.

Many essential benefits of these tools can inexpensively be provided using Excel, but project managers need to be aware of risks associated with improperly adjusting estimating metrics.

Dr. Deming’s Funnel experiment serve’s as a model for improperly adjusting estimating metrics. A simple example will demonstrate how statistically valid estimating methods can be duplicated in Excel.

Dr. Deming’s Funnel Experiment“A experiment that demonstrates the effects of tampering [with a process]. Marbles are dropped through a funnel in an attempt to hit a flat-surfaced target below. The experiment shows that adjusting a stable process to compensate for an unstable result or an extraordinarily good result will produce output that is worse than if the process had been left alone.”Donna C. Summers, Quality Management 2nd Edition, p 546

J. Miller, 4/28/10

http://www.symphonytech.com/funnelexp.htm

J. Miller, 4/28/10

DR. Deming’s Funnel Experiment Funnel Comparison to Project Estimating Funnel = Estimating process

Target = Goal -- project time goal or cost goal or resource goal

Marble = Actual project or estimate that creates measurable output – cost, schedule, etc. Each marble drop = actual project or part of project

Moving the Funnel = Adjusting the project resources after each project to try to meet the target on the next project. (Ex. Adding people or tools, if schedules are being missed, removing resources if budgets are being missed)

Moving the Target = Adjusting the time or cost estimate estimating process after each project based on the results from the last project.

J. Miller, 4/28/10

DR. Deming’s Funnel Experiment Funnel Comparison to Project Estimating Each of the four rules is discussed below. Rule 1: Leave the funnel fixed over the target. Why don't we simply adjust the funnel after each drop so the next drop will be closer to the target? Don’t change the estimating process, accept the variation, the difference between the estimate and actual projects.

Rule 2: For every drop, the marble will come to rest a distance "z" from the target. Rule 2 is to move the funnel a distance -z from its last position. Move the funnel based on the funnels last position. Chang the estimating process (metrics) after each project ordifference between estimate and actual.

Rule 3: Move the funnel a distance -z from the target after each drop of the marblethat ends up a distance z from the target. Note that Rule 2 moves the funnel based on thefunnel's last position. Rule 3 moves the funnel a distance from the target. Rule 4: Rule 4 is simply to set the funnel over where the last drop came to rest.

What is a Process?• A series of steps or actions that convert input to output.

• Project estimating is a process.

• Processes are described by statistical measures – numbers that describe “groups”.

• Groups are made of data points.

• What do we need to better understand this group of data points?Centering – MeanSpread - Stand DeviationShape – Normality, data distribution(assume all distributions here are Normal)

• We need data.

J. Miller, 4/28/10

Statistical Measures

• Dimensional attributes• Product or Process metrics such as:

# or % defects % on time deliveries Efficiency (productivity, ratios) # units of output / unit of time (may also include inputs) Time / # units of output xx units / hr Turn Around Time (TAT), xx Points / unit Cost/Unit

Which can be averages?

J. Miller, 4/28/10

J. Miller, 4/28/10

Estimating MethodsPMBOK 3rd

6.4.2 Activity Duration Estimating: Tools and Techniques.1 Expert JudgmentActivity durations are often difficult to estimate because of the number of factors that can influence them, such as resource levels or resource productivity. Expert judgment, guided by historical information, can be used whenever possible. The individual project team members may also provide duration estimate information or recommended maximum activity durations from prior similar projects. If such expertise is not available, the duration estimates are more uncertain and risky.

.2 Analogous EstimatingAnalogous duration estimating means using the actual duration of a previous, similar schedule activity as the basis for estimating the duration of a future schedule activity. It is frequently used to estimate project duration when there is a limited amount of detailed information about the project for example, in the early phases of a project. Analogous estimating uses historical information(Section 4.1.1.4) and expert judgment. Analogous duration estimating is most reliable when the previous activities are similar in fact and not just in appearance, and the project team members preparing the estimates have the needed expertise.

J. Miller, 4/28/10

Estimating Methods PMBOK 3rd

6.4.2 Activity Duration Estimating: Tools and Techniques.3 Parametric EstimatingEstimating the basis for activity durations can be quantitatively determined by multiplying the quantity of work to be performed by the productivity rate. For example, productivity rates can be estimated on a design project by the number of drawings times labor hours per drawing, or a cable installation in meters of cable times labor hours per meter. The total resource quantities are multiplied by the labor hours per work period or the production capability per work period, and divided by the number of those resources being applied to determine activity duration in work periods.

.4 Three-Point EstimatesThe accuracy of the activity duration estimate can be improved by considering the amount of risk in the original estimate. Three-point estimates are based on determining three types of estimates:• Most likely. The duration of the schedule activity, given the resources likely to be assigned, their productivity, realistic expectations of availability for the schedule activity, dependencies on other participants, and interruptions.• Optimistic. The activity duration is based on a best-case scenario of what is described in the most likely estimate.• Pessimistic. The activity duration is based on a worst-case scenario of what is described in the most likely estimate.An activity duration estimate can be constructed by using an average of the three estimated durations. That average will often provide a more accurate activity duration estimate than the single point, most-likely estimate.

J. Miller, 4/28/10

Estimating Methods .1 Expert JudgmentExpert judgment, guided by historical information, can be used whenever possible. .2 Analogous EstimatingAnalogous estimating uses historical information and expert judgment.

.3 Parametric EstimatingEstimating the basis for activity durations can be quantitatively determined by multiplying the quantity of work to be performed by the productivity rate.

.4 Three-Point Estimates An activity duration estimate can be constructed by using an average of the three estimated durations.

• MeanThe mean is the average data point value within a data set. To calculate the mean, add all of the individual data points then divide that figure by the total number of data points.

• Published estimating methods overlook critical statistical attributes.

What’s missing?

Estimating Methods – Averages

J. Miller, 4/28/10

What’s missing? Mean

If I use the “average” of past projects to estimate future projects, what is virtually guaranteed?

Standard Deviation

Concepts

• Statistical measure – numbers that describe groups. A numerical value, such as standard deviation or average, that characterizes the sample or population from which it was derived.

• What do we need to better understand this group of data points? Centering – Mean Spread - Stand Deviation Shape – Normality, data distribution

(assume all distributions here are Normal)

J. Miller, 4/28/10

Averages and Standard Deviation

• MeanThe mean is the average data point value within a data set. To calculate the mean, add all of the individual data points then divide that figure by the total number of data points.

• Standard DeviationA statistic used to measure the variation in a distribution. Standard deviation is a measure of the spread of data in relation to the mean. It is the most common measure of the variability of a set of data.

J. Miller, 4/28/10

Averages and Standard DeviationStandard Deviation ExampleSuppose we wished to find the standard deviation of the data set consisting of the values3, 7, 7, and 19.

Step 1: find the arithmetic mean (average) of 3, 7, 7, and 19(3 + 7 + 7 + 19) / 4 = 9. Step 2: find the deviation of each number from the mean,

3 − 9 = − 6 7 − 9 = − 2 7 − 9 = − 2 19 − 9 = 10.

Step 3: square each of the deviations, which amplifies large deviations and makes negative values positive,

( − 6)2 = 36( − 2)2 = 4( − 2)2 = 4(10)2 = 100.

Step 4: find the mean of those squared deviations,(36 + 4 + 4 + 100) / 4 = 36.

Step 5: take the non-negative square root of the quotient (converting squared units back to regular units), so, the standard deviation of the set is 6.

J. Miller, 4/28/10

Standard Deviation Discovered

30

36 42 48 54 60 66-6 0 6 12 18 24

1 = 6s2 s = 2(6) = 123 s = 3(6) = 184 s = 4(6) = 245 s = 5(6) = 306 s = 6(6) = 36

s1

s2

s3

s4

s5

s6

- 1 = - 6s- 2 s = 2(-6) = -12- 3 s = 3(-6) = -18- 4 s = 4(-6) = -24- 5 s = 5(-6) = -30- 6 s = 6(-6) = -36

s-6

s-5

s-4

s-3

s-2

s-1

+6 +12 +18 +24 +30 +36-36 -30 -18 -12 -6-24

Assume: Mean = 30, Std Dev = 6

J. Miller, 4/28/10

Standard Deviation Discovered

30

36 42 48 54 60 66-6 0 6 12 18 24

Assume: Mean = 30, Std Dev = 6

J. Miller, 4/28/10

Standard Deviation is Critical

Project A Data points are 1, 50 ,99Mean = 50StdDev = 49

Project BData points are 49, 50, 51Mean = 50StdDev = 1

50 60 70 80 90 1000 10 20 30 40

Assume:There are two projects, each using the Three Point Estimate Method

Project A Project BOptimistic = 1 Optimistic = 49Most likely = 50 Most likely = 50Pessimistic = 99 Pessimistic = 51

J. Miller, 4/28/10

5149 991

Normal Distribution with Std Dev

J. Miller, 4/28/10

Estimating Methods PMBOK 3rd

6.4.3 Activity Duration Estimating: Outputs.1 Activity Duration EstimatesActivity duration estimates are quantitative assessments of the likely number of work periods that will be required to complete a schedule activity. Activity duration estimates include some indication of the range of possible results. For example:• 2 weeks ± 2 Points to indicate that the schedule activity will take at least eight Points and no more than twelve (assuming a five-Point workweek).• 15 percent probability of exceeding three weeks to indicate a high probability—85 percent—that the schedule activity will take three weeks or less.

Let’s take a closer look at making a valid estimate…

Data Points to Data TrendsJ. Miller, 4/28/10

Point 1

Point 3

Point 2

Point 4

Point 6

Point 5

Point 7


P1 P3P2 P4 P6P5 P7


Point 1

Point 2

Point 4

Point 3

Point 5

Point 6

Point 7


P1 P3P2 P4 P6P5 P7


P1 P3P2 P4 P6P5 P7

UCL

LCL

?

Data Points to Data Trends

X

J. Miller, 4/28/10

Process Exercise

What about when we don’t have data for all process steps?

What if we had a way to estimate the data?

Creating a Control Chart1. Define the metric: time/unit of output, delta of actual vs. planned time or cost, etc2. Must be homogeneous3. Data collection planning

1. Who will collect data?2. What aspect of the process will be measured?3. Where, or at what point in the process will the measurement be taken?4. When or how frequently will the data be collected?5. Why is this sample being taken?6. How will the data be collected?7. How many samples will be taken?

4. Collect the data

Step1

Step2

Step3

Step4

Step5

Step6

Step7

Step8

Step9

Step10

#Steps

Total System

Time

MeanStep

DurationProject 1 45 32 78 12 67 54 87 43 54 87 10 559 55.9Project 2 35 67 9 64 55 75 33 58 8 396 49.5Project 3 56 45 29 54 39 66 63 70 8 422 52.8Project 4 43 43 56 31 66 40 69 45 75 9 468 52.0Project 5 41 67 9 59 56 71 43 67 78 9 491 54.6Project 6 60 45 14 54 67 76 65 74 8 455 56.9Project 7 54 39 63 61 59 44 60 7 380 54.3Project 8 41 74 13 63 73 49 59 71 8 443 55.4Project 9 51 45 72 12 65 87 46 56 85 9 519 57.7

Project 10 49 34 60 8 67 56 75 7 349 49.9

What if data isn’t collected for every project or period?

Creating a Control Chart

As long as the units are the same, the model works!

Step1

Step2

Step3

Step4

Step5

Step6

Step7

Step8

Step9

Step10

#Steps

Total System

Time

MeanStep


Project 10 49 34 60 8 67 56 75 7 349 49.9

OR

Step1

Step2

Step3

Step4

Step5

Step6

Step7

Step8

Step9

Step10

#Steps

Total System

Time

MeanStep


Project 10 49 34 60 8 67 56 75 7 349 49.9

J. Miller, 4/28/10

Step1

Step2

Step3

Step4

Step5

Step6

Step7

Step8

Step9

Step10

#Steps

Total System

Time

MeanStep


Project 10 49 34 60 8 67 56 75 7 349 49.9

Total # Projects 10 10 10 10 10 10 10 10 10 10Recorded # Projects 8 8 9 9 9 9 9 7 8 7

% of Projects Reporting 80.0% 80.0% 90.0% 90.0% 90.0% 90.0% 90.0% 70.0% 80.0% 70.0%Mean 49.9 39.3 64.7 15.2 61.9 54.3 75.4 43.3 60.3 77.1

Standard Deviation 6.6 5.1 10.1 8.6 5.2 9.4 7.3 5.0 4.5 6.6Variance 44.1 25.9 102.0 74.4 27.1 88.0 53.0 24.9 19.9 43.8


Sum of Means 541.4Sum of Variances 503.3

Sq Rt of Variances 22.4

- 3StdDev

- 2StdDev

- 1StdDev Mean

+ 1StdDev

+ 2StdDev

+ 3StdDev

-3* 22.4 -2* 22.4 -1* 22.4 1* 22.4 2* 22.4 3* 22.4-67.2 -44.8 -22.4 541.4 22.4 44.8 67.2474.2 496.6 519.0 541.4 563.8 586.2 608.6

Creating a Control ChartTotal # Projects 10 10 10 10 10 10 10 10 10 10

Recorded # Projects 8 8 9 9 9 9 9 7 8 7% of Projects Reporting 80.0% 80.0% 90.0% 90.0% 90.0% 90.0% 90.0% 70.0% 80.0% 70.0%

Mean 49.9 39.3 64.7 15.2 61.9 54.3 75.4 43.3 60.3 77.1Standard Deviation 6.6 5.1 10.1 8.6 5.2 9.4 7.3 5.0 4.5 6.6

Variance 44.1 25.9 102.0 74.4 27.1 88.0 53.0 24.9 19.9 43.8

5. Calculate the centerline – Sum of Means

6. Calculate the Upper and Lower Control Limits.

UCL = Sum of Means + 3 σ

LCL = Sum of Means - 3 σ

σ = Sq Rt of Sum of Variances

541.4

608.6

474.2

After calculating the control limits, place the center line and control limits on the chart

Creating a Control Chart- 3

StdDev- 2

StdDev- 1

StdDev Mean+ 1

StdDev+ 2

StdDev+ 3

StdDev-3* 22.4 -2* 22.4 -1* 22.4 1* 22.4 2* 22.4 3* 22.4

-67.2 -44.8 -22.4 541.4 22.4 44.8 67.2474.2 496.6 519.0 541.4 563.8 586.2 608.6

J. Miller, 4/28/10

A Guide to the Project Management Body of Knowledge (PMBOK® Guide) Third Edition2004 Project Management Institute, Four Campus Boulevard, Newtown Square, PA 19073-3299 USA


J. Miller, 4/28/10

Determine if the Process is Stable Over Time

• Check the date for trends• Check the date for patterns

Learn how to decide if the process is stable and if the data can be used for further analysis. Data is analyzed using a control chart.

(1) Point beyond UCL and LCL (beyond 3 sigma)

Process Stable Over Time?

J. Miller, 4/28/10

(2) Seven consecutive points on the same side of center line.


J. Miller, 4/28/10

(3) Trend: 6 consecutive points steadily increasing or decreasing:


J. Miller, 4/28/10

(4) Repeating pattern and cycles.


J. Miller, 4/28/10

191715131197531

1400

1200

1000

800

600

Observation

Indiv

idual Valu

e

_X=1000.0

UCL=1367.5

LCL=632.4

191715131197531

600

450

300

150

0

Observation

Movin

g R

ange

__MR=138.2

UCL=451.6

LCL=0

1

1

I-MR Chart of Northeast

Can the above data be used for any further analysis and decision making?

Observations: No point is beyond UCL and LCLThere is no pattern repetitionThere is no continuous increase or decrease of 6 data pointsThere is NO 7 consecutive points on the same side of center line

So, this data can be used for future predictions


J. Miller, 4/28/10

191715131197531

1400

1200

1000

800

600

Observation

Indiv

idual V

alu

e

_X=1025.9

UCL=1298.0

LCL=753.8

191715131197531

300

200

100

0

Observation

Movin

g R

ange

__MR=102.3

UCL=334.3

LCL=0

1

1

I-MR Chart of Northwest

Can the above data be used for any further analysis and decision making

Observation : There are points beyond UCL and LCLThere is no pattern repetitionThere is no continuous increase or decrease of 6 data pointsThere is 7 consecutive points on the same side of center line

This tells, there is a special cause influencer, which is causing this to happen. Before making any predictions for the future, need to analyze the special cause


J. Miller, 4/28/10

191715131197531

1600

1400

1200

1000

Observation

Indiv

idual V

alu

e

_X=1195.9

UCL=1534.5

LCL=857.3

191715131197531

400

300

200

100

0

Observation

Movin

g R

ange

__MR=127.3

UCL=416.0

LCL=0

I-MR Chart of Southwest

Can the above data be used for any further analysis and decision making :

Observation : No point is beyond UCL and LCLThere is no pattern repetitionThere is no continuous increase or decrease of 6 data pointsThere is 7 consecutive points on the same side of center line

So, Before using the data for future predictions, need to analyze the cause for data points to be on same side (during the period) and if required separate those points from the rest, in making the decision


J. Miller, 4/28/10

J. Miller, 4/28/10

Apply Lessons from the Funnel Experiment Rule 1: Leave the funnel fixed over the target. Why don't we simply adjust the funnel after each drop so the next drop will be closer to the target? Don’t change the estimating process, accept the variation, the difference between the estimate and actual projects.

Rule 2: For every drop, the marble will come to rest a distance "z" from the target. Rule 2 is to move the funnel a distance -z from its last position. Move the funnel based on the funnels last position. Chang the estimating process (metrics) after each project ordifference between estimate and actual.

Rule 3: Move the funnel a distance -z from the target after each drop of the marblethat ends up a distance z from the target. Note that Rule 2 moves the funnel based on thefunnel's last position. Rule 3 moves the funnel a distance from the target. Rule 4: Rule 4 is simply to set the funnel over where the last drop came to rest.

Determine if the Process is Stable Over Time

• Check the date for trends• Check the date for patterns

J. Miller, 4/28/10

Using Excel to Avoid Common Mistakes

Questions?

ContactJohn H. Miller

[email protected]

mailto:[email protected]

pitfalls of project estimating: an applied lesson from dr. deming’s funnel experiment john miller,...

Documents

estimating process target

pitfalls of project

project resources

project estimatingfunnel

project estimatingeach

project ordifference

project managers

estimating process metrics