reactor royce, cantor v2-16-9

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

Steer with Predictive AnalyticsLess Surprises, More Insight

Walker Royce Murray CantorChief Software Economist Distinguished EngineerIBM Software Group IBM Software Group

@weroyceusibmcom @murraycantor

[email protected] [email protected]

1

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

2

Deliver business critical software product release within 12 months

Challenge

SCHEDULE

6 MONTHS 12 MONTHS 18 MONTHS

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

Jerry’s probable outcome…Reality

3

LATE SCRAP & REWORK

SCHEDULE


0%

100%

PR

OG

RE

SS

INTEGRATION BEGINS

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

4

Program parameters are also delivery predictionsCOST, SCHEDULE, EFFORT, QUALITY…


PR

OB

AB

ILIT

Y

Insight

SCHEDULE

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

5

There is roughly a 50% chance of making the date


PR

OB

AB

ILIT

Y

Coin Flip

SCHEDULE

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

6

Move out the date to improve likelihood of shipping?


PR

OB

AB

ILIT

Y

Option 1Absolutely Absolutely NOT!NOT!

SCHEDULE

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

7

Decrease time estimate by sacrificing quality or content?


PR

OB

AB

ILIT

Y

Option 2Absolutely Absolutely NOT!NOT!

SCHEDULE

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

8

Reduce the variance to improve likelihood of shipping

Optimize


PR

OB

AB

ILIT

Y

SCHEDULE

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

9

Measure validated learning | Reduce the variance

Uncertainty in stakeholder satisfaction

Optimize

Uncertainty in Plans, Scope and Design

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

10

Economic Governance

The moral of this story…

Variance Reduction

Smarter Analytics

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

A (Fictional) Case Study The Setting: Product planning for KillerApp V2 for US

Treasury property auctions

– KillerApp v1 met with unprecedented success

– Since its release and success, many state governments have indicated interest in variants specific to their local laws and needs

The Opportunity

– Sales reports that they can close $1B in new business with a promise for features supporting state government auctions, including mobile, analytics, and social media support

The Catch

– An aggressive startup has plans to release a similar capacity in August, 2014

• The first to market will have a huge competitive advantage

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

KillerApp’s Product Development Team Structure

Alice, Team Lead

Peter, Product Manager

Joe, Team Lead

12

• The server team develops using hybrid process• Agile with a target date

• They must manage technical risks carefully to meet challenging delivery target

• Is their planned scope of work appropriate to their timeline, resources?

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

The server team has identified work they would like to complete, the task owners, and preliminary estimates

Analytics capability

Story 1Mobile support

Story 2

Owner: John

Task 1.1

[1w,2w,3w]

Owner:John

Task 1.2

[2w,5w,7w]

Owner: Alice

Task 1.3

[3w,7w,11w]

Owner: John

Task 2.1

[1w,1.5w,3w]

Owner: Claire

Task 2.2

[2d,1w,6w]

Owner: Claire

Task 2.3

[1w,2w,3w]

The team needs to finish its work by the integration milestone.Is it a good bet that it will?

Task 1.1.7

2d

5d

6w

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

Is the server team making a good “bet”? The conversation begins...

Where did these estimates come from?

– Should we bet the business on this data?

– Did tampering occur?

User expertise

– One developer’s experience

– Multiple developers’ experiences

Learning from prior history

– This team’s prior history (Machine Learning)

– Models that run over prior projects (e.g., COCOMO, SLIM, SEER)


Alice, Team Lead

Predictive analytics facilitate critical conversations between stakeholders.To have value, they must be trustworthy and explainable.

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

Is the server team making a good “bet”? The conversation continues...

Does it reflect realistic expectations?

– What is causing the variance (uncertainty)? What actions can we take to reduce it?

– Did you account for new work being identified in flight?

Variance is coming from

– Estimating large-scale pieces of work

– Uncertainty around reusability of components

15


Alice, Team Lead

Predictive analytics have value only if the data have value.They are not magic. They require hard work to provide value.

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

Once we trust the data, we can ask what it says about the outcome. But there are many different possible schedules,

leading to different outcomes Valid schedules come from

– Task dependencies

– Implied orderings due to owners

Actual schedules depend on when owners finish tasks (uncertain)

– NB: This is not critical path analysis

Schedule 1

John

Claire

AliceTask 1.1[1w,2w,3w]

Task 1.2[2w,5w,7w]

Task 2.1[1w,1.5w,3w]

Task 2.3[1w,2w,3w]

Task 2.2[2d,1w,6w]

Task 1.3[3w,7w,11w]

Schedule 2

John

Claire


Task 1.2[2w,5w,7w]

Task 2.1[1w,1.5w,3w]Task 2.3[1w,2w,3w]

Task 2.2[2d,1w,6w]

Task 1.3[3w,7w,11w]

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

There are many different possible outcomes, depending on the actual schedule and the time the tasks take

Schedule 1

John

Claire


Task 1.2[2w,5w,7w]

Task 2.1[1w,1.5w,3w]

Task 2.3[1w,2w,3w]

Task 2.2[2d,1w,6w]

Task 1.3[3w,7w,11w]

Schedule 2

John

Claire


Task 1.2[2w,5w,7w]

Task 2.1[1w,1.5w,3w]Task 2.3[1w,2w,3w]

Task 2.2[2d,1w,6w]

Task 1.3[3w,7w,11w]

What are the possible outcomes?How likely are they?

Is this set of work a good bet?

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

Is the server team likely to finish their planned work by 2/13?

[3w,7w,11w]Task 1.3

Alice

John

Claire

[2w,5w,7w]Task 1.2

[1w,1.5w,3w]Task 2.1

[2w,5w,7w]Task 2.2

[1w,2w,3w]Task 1.1

[1w,2w,3w]Task 2.3

Sim 1 8d

Task 2.1

10d

Task 2.3

10d

Task 1.1

10d

Task 2.2

35d

Task 1.3

25d

Task 1.260d

Sim 2 5d

Task 2.1

5d

Task 2.3

5d

Task 1.1

14d

Task 2.2

15d

Task 1.3

10d

Task 1.254d

Sim 3 15d

Task 2.1

15d

Task 2.3

15d

Task 1.1

35d

Task 2.2

55d

Task 1.3

35d

Task 1.2170d

10/1 1/1512/1

38% 62%

For each random variable (task duration), sample a value from its

distribution

Different values will be sampled in each simulation, leading to different outcomes

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

Server team has been working for a month. Will they deliver on time?

2/13Delivery

8/1Start

9/1Now

Pre

dic

ted

Del

ive

ryD

ate

Ra

ng

e

Date of Prediction

2/13

1/1

3/1

2/1

4/1• Best case: 1/1• Most likely: 2/11• Worst case: 2/28

• On-time delivery: 87%

• Best case: ?• Most likely: ?• Worst case: ?

• On-time delivery: ?

19

We have new learning: actuals, evidence of how the team is actually performing

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

Data mining can help us learn from the past to predict the future

Data mining (aka machine learning, statistical inference) is about learning correlations between inputs and outcomes from sample data (actuals)

– Classification prediction seeks to learn a model that predicts the classification of artifacts

– Numeric prediction seeks to learn a model that predicts a numeric value for an artifact of interest

20

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

Techniques for Learning From Actuals1. Leverage actual distribution

- The actuals form a distribution

- We can sample from this distribution during Monte Carlo simulation

2. Predicting based on relationships between estimates and actuals (regression)

– If estimates and actuals are strongly correlated, regression analysis can be useful in predicting actuals from estimates

3. Predicting based on learned relationships between multiple variables and actuals

4. Bayesian Networks

- After 50 tasks complete, we have new evidence:

• DB tasks: effort = [11d, 16d, 42d], type = 20%

• Analytics tasks: effort = [4d, 16d, 20d], type = 75%

• Mobile tasks: effort = [1d, 2d, 4d], type = 5%

- We can update the conditional probabilities to reflect this learned evidence

21P

rob

abil

ity

Actual Task Effort

Node 0n: 4591; %: 100.0Predicted: 5.5 E8

Node 0n: 4591; %: 100.0Predicted: 5.5 E8

Effort

Node 1n: 1911; %: 41.6

Predicted: 5.0 E8

Node 1n: 1911; %: 41.6

Predicted: 5.0 E8

Node 2n: 2680; %: 58.4

Predicted: 5.9 E8

Node 2n: 2680; %: 58.4

Predicted: 5.9 E8

Attribute: OwnerImprovement: 1.8 E15

Alice; David John; Claire

Node 3n: 832; %: 31.1

Predicted: 5.6 E8

Node 3n: 832; %: 31.1

Predicted: 5.6 E8

Node 4n: 1848; %: 68.9

Predicted: 6.1 E8

Node 4n: 1848; %: 68.9

Predicted: 6.1 E8

Attribute: OwnerImprovement: 2.7 E14

Claire John

Task duration

Task duration

Task owner

Task owner

Task type

Task type

Task expected

effort

Task expected

effort

Qualification for task

Qualification for task

Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

Server team has been working for a month. It is not going well for them.

It is no longer a good bet that the server team will deliver the committed scope on time

2/13Delivery

8/1Start

9/1Now

Pred

icte

d D

eliv

ery

Dat

e Ra

nge

Date of Prediction

2/13

1/1

3/1

2/1

4/1 • Best case: 1/1• Most likely: 2/11• Worst case: 2/28


• Best case: 1/17• Most likely: 2/23• Worst case: 4/7


Sa

fe a

rea

– n

o g

rap

hic

s h

ere

Sa

fe a

rea

– n

o g

rap

hics h

ere

Lessons for us all: Making better bets

Eliminate some of the long, red tail

– Understand and remediate sources of uncertainty that are putting the deadline at risk

Change the scope

– Drop or postpone work

– Never negotiate estimates!

Change the resources available

– Beware Brooks’ Law!

Change the time

– Negotiate a later deadline

Make changes, rerun Monte Carlo simulation,check results

– Repeat until the outcome is acceptable10/1 12/1812/1

87% 13%

Task 1.1.7

2d

5d

6w

• Break down the task further• Do some work now to eliminate variance-inducing

questions

reactor royce, cantor v2-16-9

Technology