agile metrics - modeling, simulation, and data mining
DESCRIPTION
Modeling, Simulation & Data Mining: Answering Tough Cost, Date & Staff Forecasts QuestionsTRANSCRIPT
Modeling, Simulation & Data Mining: Answering Tough Cost, Date
& Staff Forecasts Questions
Troy Magennis (Focused Objective)
Larry Maccherone (Rally)
Pain Point
My Boss “Needs” A Date…
Getting Quantitative
Evidence
Assessing & Communicating
Risk / Uncertainty
Arm my teams (and yours) with the tools and techniques
to solve these problems
My Mission
2 Minutes About Larry
• Larry is a Pisces who enjoys skiing, reading and wine (red, or white in outdoor setting)
• We have a lot in common… over to Larry!
Metrics & Measurement
Why measure?
Feedback
Diagnostics
Forecasting
Lever
When to NOT take a shot
Good players?
• Monta Ellis
– 9th highest scorer (8th last season)
• Carmelo Anthony (Melo)
– 8th highest scorer (3rd last season)
The ODIM Framework
better Measurement
better Insight
better Decisions
better Outcomes
What is normal?
Cumulative -> 0.1% 2.3% 15.9% 50.0% 84.1% 97.7% 99.9%
Roughly -> 85% 98%
Are you normal?
You will be wrong by…
• 3x-10x when assuming Normal distribution
• 2.5x-5x when assuming Poisson distribution
• 7x-20x if you use Shewhart’s method
Heavy tail phenomena are not incomprehensible… but they cannot be
understood with traditional statistical tools. Using the wrong tools is incomprehensible.
~ Roger Cooke and Daan Nieboer
Bad application of control chart Control is an illusion, you infantile
egomaniac. Nobody knows what's gonna happen next: not on a freeway, not in an airplane, not inside our own bodies and
certainly not on a racetrack with 40 other infantile egomaniacs.
~Days of Thunder
Time in Process (TIP) Chart A good alternative to control chart
Collection
• Perceived cost is high
• Little need for explicit collection activities
• Use a 1-question NPS survey for customer and employee satisfaction
• Plenty to learn in passive data from ALM and other tools
• How you use the tools will drive your use of metrics from them
Summary of how to make good metric choices
• Start with outcomes and use ODIM to make metrics choices.
• Make sure your metrics are balanced so you don’t over-emphasize one at the cost of others.
• Be careful in your analysis. The TIP chart is a good alternative to control chart. Troy’s approach is excellent for forecasting. We’ve shown that there are many out there that are not so good.
• Consider collection costs. Get maximal value out of passively gathered data.
Data visualization is like photography. Impact is a function of perspective, illumination, and focus.
~Larry Maccherone
Flaw of Averages, Risk & Monte Carlo Sim
A model is a tool used to mimic a
real world process
A tool for low-cost experimentation
Monte Carlo Simulation
Monte Carlo Simulation?
Performing a simulation of a model multiple times using
random input conditions and recording the frequency of
each result occurrence
Scrum
Backlog This Iteration Deployed
2 5
8
Run Sim Total Iterations
1 3
2 2
3 5
4 3
5 4
6 2
… …
Kanban
Backlog Design Develop Test Deployed
2
1 – 2 days 1 – 5 days 1 – 2 days
Run Time Total
1 5
2 4
3 3
4 9
5 5
6 6
… …
Result versus Frequency (50 runs)
More Often
Less Often Result Values – For example, Days
15 10 20
Fre
qu
en
cy o
f R
esu
lt
1
5
10
15
20
25
Result versus Frequency (250 runs)
More Often
Less Often Result Values – For example, Days
15 10 20
Fre
qu
en
cy o
f R
esu
lt
1
5
10
15
20
25
Result versus Frequency (1000+ runs)
More Often
Less Often Result Values – For example, Days
15 10 20
Fre
qu
en
cy o
f R
esu
lt
1
5
10
15
20
25
Key Point
There is NO single forecast result
There will always be many possible results, some more likely
Time to Complete Backlog
50% Possible Outcomes
50% Possible Outcomes
When pressed for a single number, we often give the average.
Like
liho
od
Time to Complete Backlog
95% Outcomes 5%
Monte Carlo Simulation Yields More Information – 95% Common.
Like
liho
od
Key Point
“Average” is NEVER an option WARNING: Regression lines
are most often “average”
But, I.T. gets worse
1 2 3
Planned Backlog
Perf. Issues
Vendor Delay
Time to Delivery
Like
liho
od
Promised New Average
50% Possible Outcomes
Key Point
Risks play a BIG role in forecasts
Yes, more than backlog.
Velocity is NOT Linear nor is defect rate, scope-creep, story
expertise requirements, team skill, etc.
Date for likelihood
Likelihood (0-100%)
Key Point Forecasts should be presented with the
right amount of uncertainty
PAIN POINT Demo: Forecasting… My Boss “Needs” a Date…
In this demo
• Basic Scrum and Kanban Modeling
• How to build a simple model
– SimML Modeling Language
– Visual checking of models
– Forecasting Date and Cost
– The “Law of Large Numbers”
Demo: Finding What Matters Most Cost of Defects & Staff Analysis
Actively Manage
Ignore for the moment
Sensitivity Report
Staff Skill Impact Report
Explore what staff changes have the greatest impact
Key Point
Modeling helps find what matters
Fewer estimates required
In this demo
• Finding what matters most
– Manual experiments
– Sensitivity Testing
• Finding the next best 3 staff skill hires
• Minimizing and simplifying estimation
– Grouping backlog
– Range Estimates
– Deleting un-important model elements
Demo: Finding the Cost / Benefit of Outsourcing
Outsourcing Cost & Benefits
• Outsourcing often controversial
– Often fails when pursued for cost savings alone
– Doesn’t always reduce local employment
– An important tool to remain competitive
– I.Q. has no geographic boundaries
• Many models
– Entire project
– Augmentation of local team
Build Date & Cost Matrix
1 x Estimates
1.5 x Estimates
2 x Estimates
1 x Staff Best Case
1.5 x Staff Midpoint
2 x Staff Worst Case
Benefit = (Baseline Dev Cost – New Dev Cost) - Cost of Delay + Local Staff Cost Savings
$(150,000)
$(100,000)
$(50,000)
$-
$50,000
$100,000
$150,000
1 1.5 2
1x Multiplier
1.5x Multiplier
2x Multiplier
NOT LINEAR & NOT YOUR PROJECT
In this demo
• Model the impact of various outsourcing models
New Project Rules of Thumb…
• Cost of Delay plays a significant role
– High cost of delay project poor candidates
– Increase staffing some compensation
• Knowledge transfer and ramp-up time critical
– Complex products poor candidates
– Captive teams better choices for these projects
• NEVER as simple as direct lower costs!
Assessing and Communicating Risk
Speaking Risk To Executives
• Buy them a copy of “Flaw of Averages” • Show them you are tracking & managing risk • Do
– “We are 95% certain of hitting date x” – “With 1 week of analysis, that may drop to date y” – “We identified risk x, y & z that we will track weekly”
• Don’t – Give them a date without likelihood
• “February 29th 2013”
– Give them a date without risk factors considered • “To do the backlog of features, February 29th, 2013”
We spend all our time estimating here
1 2 3
**Major risk events have the predominate role in deciding where deliver actually occurs **
Plan Performance Issues
External Vendor Delay
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
Key Points
• There is no single release date forecast
• Never use Average as a quoted forecast
• Risk factors play a major role (not just backlog)
• Data has shape: beware of Non-Normal data
• Measurement → Insight → Decisions → Outcomes : Work Backwards!
• Communicate Risk early with executive peers
Call to action
• Read these books
• Download the software FocusedObjective.com
• Follow @AgileSimulation
• Follow @LMaccherone
Please Submit an Eval Form!
We want to learn too!
BEST PRACTICES
Model (a little)
Visually Test
Monte-Carlo Test
Sensitivity Test
The Model Creation
Cycle
Baseline
Make Single
Change
Compare Results
Make Informed
Decision(s)
The Experiment
Cycle
Best Practice 1
Start simple and add ONE input condition at a time.
Visually / Monte-carlo test
each input to verify it works
Best Practice 2
Find the likelihood of major events and estimate delay E.g. vendor dependencies,
performance/memory issues, third party component
failures.
Best Practice 3
Only obtain and add detailed estimates and opinion to a
model if Sensitivity Analysis says that input is material
Best Practice 4
Use a uniform random input distribution UNTIL sensitivity
analysis says that input is influencing the output
Best Practice 5
Educate your managers’ about risk. They will still want a “single” date for planning, but let them decide 75th or
95th confidence level (average is NEVER an option)
SIMULATION EXAMPLES
Return to main presentation…
unlikely
certain
Forecasts Return to main presentation…
unlikely
certain
Forecasts
50% Possible
Outcomes
50% Possible Outcomes
Return to main presentation…
Actively Manage
Ignore for the moment
Sensitivity Report Return to main presentation…
Staff Skill Impact Report
Explore what staff changes have the greatest impact
Return to main presentation…
Return to main presentation…
Focused Objective
• Risk Tools for Software Dev
• Scrum/Agile Simulation
• Kanban/Lean Simulation
• Forecasting Staff, Date & Cost
• Automated Sensitivity Analysis
• Data Reverse Engineering
• Consulting / Training
• Book
We Use & Recommend: EasyFit
• MathWave.com
• Invaluable for
– Analyzing data
– Fitting Distributions
– Generating Random Numbers
– Determining Percentiles
Free Images: MorgueFile.com
• http://www.morguefile.com/corporate/about
– Calendar: http://www.opticgroove.com.au
– Calculator: http://www.therising-sun.us
– Dice: [email protected]