stop flying blind! quantifying risk with monte carlo simulation

Stop Flying Blind!Quantifying Startup Product Risks with Monte Carlo Simulation.Sam McAfee, Principal - @sammcafee

Slides available at bit.ly/riskiest_assumption

Who is this guy?• Dot-com boom survivor. • Ran a dev shop for 10 years after that. • Agile has always been the “norm”. • Read ‘4 Steps’ before #LeanStartup. • Participated in 3 different startups. All dead. • Ran Change.org engineering. • Learned LeanUX from @clevergirl. • Wore plaid and facial hair before it was cool.

Tweet every word: @sammcafee

A Typical ScenarioMaybe you’ve experienced something like this?

The CEOA typical scenario

Our competitors have it, so we should

have it too!

“ ”

Product ManagementA typical scenario

It takes too long to get new features out.

“ ”

EngineeringA typical scenario

We have to refactor all the things we rushed out the door last time.

“ ”

How do we prioritize competing objectives?

Understanding RiskIt’s kind of important for startups.

Why Is Risk Important?Understanding Risk

Startups are risky!You are participating in one of the highest risk

business endeavors there is.

Risk is a business term.You are in business. Let’s use terminology that

business people use to make decisions.

Ignore it at your peril.Choosing to ignore risk will not protect you from it.

What Is Risk?Understanding Risk

Risk = Impact * Probability

Stop Using Your Gut!Understanding Risk

All people, including experts and managers,

are very bad at assessing the probabilities of events.

— Doug Hubbard

“ ”

Risk = Impact * Probability

How Do We Define Impact?

Hint: For startups, it can probably be found on the bottom line.

Economic framework.Allows you to compute the impact of any change in

the system into a single unit of measure.

Start with your P&L.Your CEO or finance people already have a

framework for you to start with. Include them.

Apply risk scenarios.Calculate the impact of likely scenarios on the total

product life-cycle profits, or similar KPI.

Cost of DelayScenario 1

Cost of Delay Scenarios• Projects will have different cost of delay curves.

• What is the effect on total life-cycle profits for each unit of delay?

• Delay in product development is overwhelmingly affected by queues between steps rather than job duration at a given step.

• Cost of Delay enables you to calculate the cost of queues.

Sequence of WorkScenario 2

Work Sequence Scenarios• The order in which you do work can have a substantial impact on

the total cost of queues.

• If cost of delay is homogeneous, do the shortest job first.

• If job duration is homogeneous, do the highest cost of delay first.

• You can combine job duration and cost of delay using weighted shortest job first.

Tangent: which is the shortest job?

• Job duration is an estimate.

• Estimates are probabilistic, not deterministic.

• Use cycle time and throughput to compute a probability distribution for likelihood of job duration.

Capacity vs. Queue CostScenario 3

Capacity Utilization Scenarios• You can reduce the cost of queues by adding additional capacity.

Obviously, additional capacity has a cost too.

• Additional capacity increases transaction costs, but lowers holding (delay) costs.

• Is the cost of adding additional capacity justified by the gains in product development throughput and lower queue costs?

How Do We Define Probability?

Use your metrics, Luke!

Use business metrics.You are probably already tracking tons of data about

how you acquire new customers and how they enter

and leave your product funnels.

Value each conversion.Use your historical data to calculate values for each

step in a product funnel.

Note areas of variance.The steps in your funnel that have consistent,

regular conversion rates are unlikely to change.

They represent lower information content.

How Do We Value Information?

Hint: It’s probabilistic too.

Information Has Value

• 10,000 visitors, 10% sign-up, $20/month.

• Each visitor is worth $2/month (10% x $20/month).

• A test to increase rate to 15% is worth:

• An additional $10/month * probability of test success.

Experiments Have Value

• Experiments attempt to capture new information.

• Experiment value = expected benefit - cost of running the test.

• Expected benefit = increase in KPI * probability of success.

• Cost of running the test = cost of delay * job duration.

Your backlog is a prioritized list of

experiments.

Monte Carlo SimulationsIt’s not just for insurance geeks anymore!

Monte Carlo Simulation• A series of dependent variables, each with probability

distributions.

• Randomly selects a value from each variable, and computes output. Repeats thousands of times.

• End result is another probability distribution, in the unit of measure that you care about.

• It’s 2014. You can do this in a spreadsheet, in about an hour.

Basic funnelUsing time series data from your funnel, create

probability distributions for each step toward

capturing revenue.

Basic histogramAfter 5,000 or so simulations, display a histogram of

the expected output.

1

2

1

2

References

+

Thank you!

Slides available at bit.ly/riskiest_assumption