rational is practical: sh · pdf filepalantir economic solutions ltd john schuyler denver,...

OPTIMIZER’S CURSENov. 4, 2008

(revised 7-Nov-08)

Seminar Hosted by

Palantir Economic Solutions Ltd

John SchuylerDenver, Colorado, USA

PetroSkillsand

Decision Precision

Copyright © 2006-2008 by John R. Schuyler. All rights reserved.

© 2008 PetroSkills, LLC. All rights reserved. 2

Optimizer’s CurseOptimizer’s Curse

Background

Optimizer’s Curse phenomenon

Implications for Portfolio Forecasts

An Approach for Correcting this Bias

– Portfolio forecasts

– Individual project value estimates

Discussion and Wrap-Up



Background








BackgroundBackground

Risk, Uncertainty and Investment Decision-Making in the Upstream Oil and Gas Industry– PhD Dissertation by Fiona Macmillan– University of Aberdeen, Oct. 2000

“Best Practices in Project Evaluation and Influence on Company Performance”– John Schuyler– Surveyed some former class participants– Journal of Petroleum Technology, Aug. 1997


InterestsInterests

Shareholder value creation:measurement and forecasting

Capital investment decisionsWhat criteria to use?– PV discount rate– Adjusting for market risk or correlation– Risk tolerance

Modeling the project, portfolio, and enterprise


Present thinking –rocket science?Present thinking –rocket science?

Most companies are using PV discount rates that are too high.Most companies are too conservative, when the diversified investor wishes them to be essentially risk-neutral.– Scaling up the typical investor’s decision

policy to the corporate level

– Market Value Discount adjustment

Consider the company’s portfolio in establishing decision policyOptimizer’s curse: Yet one more detail


Interesting PhenomenaInteresting Phenomena

Lemon Market

Survivorship Bias

Brown’s Note

Winner’s Curse


Lemon MarketLemon Market

Used cars in the market are either ‘good’ or ‘bad’ (lemons).

The owner of a car knows the condition of his car.

<asymmetric information>

The buyer of a car doesn’t know the condition and must therefore assume that a car for sale is of ‘average’ quality.


Lemon MarketLemon Market

There are few good cars on the market– The owner of a good used car cannot get a high

enough price to induce him to sell.

Buyers are disappointed, on average, because the cars available and purchased are mostly lemons.

Described by George Akerlof (1940-), who shared the 2001 Nobel Prize in Economics

Excluding candidates causes the bias.


Survivorship BiasSurvivorship Bias

Failed companies are excluded from studies because they no longer exist.

With mutual funds, 90% accurately claim to have performed in the top quartile of their peers. The other 3/4 of funds have closed.

Publication bias: scientific journals tend to publish studies where something was found out.– Studies where “nothing happened” tend to be rejected

or never submitted for publication

– “File drawer problem”Ignoring all data causes the bias.


Brown’s NoteBrown’s Note

Brown, K. C. (1974)“A note on the apparent bias of net revenue estimates for capital investment projects”

“Projects approved tend to be those where

revenue is overstated and costs and/or

investment are understated.”

J. Finance 29 1215–1216; cited by Smith & Winkler


Winner’s CurseWinner’s Curse

At an auction, the winner will tend to overpay (unless he recognizes the winner’s curse when bidding)

Auction analyses usually assume

– Common-value auction, where value similar for all bidders but uncertain at the time of the auction

– Similar bidding methods

– Similar bidding strategy



The winning bidder is usually the one with the highest value estimate.

Even if value estimates are unbiased, on average

– The person who is randomly most-optimistic this time will estimate highest.

– The person with the highest estimate will likely bid highest and win the auction.

Ranking bids causes the bias.



Experienced, successful buyers at auctions have learned to shade their bids

– bid lower than what they think the item is worth

Longstanding experiences teaches:

– Bid lower when the value of the item is more uncertain;

– Bid lower as the number the number of bid competitors increases.



“Winner’s curse” term coined in the 1950s in the context of bidding for an oil field.

Capen, E. C., R. V. Clapp, and W. M. Campbell, 1971, "Competitive Bidding in High-Risk Situations," Journal of Petroleum Technology (now JPT), June, p. 641-53.

Richard Thaler, 1992, The Winner's Curse: Paradoxes and Anomalies of Economic Life



Background








Smith and Winkler’s PaperSmith and Winkler’s Paper

The Optimizer’s Curse: Skepticism andPostdecision Surprise in Decision Analysis

– James Smith and Robert WinklerFuqua School, Duke University

– Management Science

– March 2006


Their ConclusionsTheir Conclusions

Portfolio results tend to be well below their forecast mean

“A decision maker who consistently chooses alternatives based on her estimated values should expect to be disappointed on average, even if the individual value estimates are conditionally unbiased.”

Caused by screening and ranking

Same phenomenon as winner’s curse?

Selecting best candidates causes the bias.


Why DA Values are OverstatedWhy DA Values are Overstated

Estimates may be unbiased– No biases in input judgments

– No biases introduced by calculations

Unbiased: The hallmark of a good evaluation

– If the decision-maker or company wants to be conservative, then that is best addressed by a risk policy (using a utility function).


Why DA Values are OverstatedWhy DA Values are Overstated

The individual project value estimates may be unbiased, on average,

Yet …

Any screening or optimizing effort will preferentially retain cases that tend to be optimistic, e.g. selecting projects with:

– EMV > 0

– DROI > cutoff hurdle



Background








Portfolio Forecast BiasPortfolio Forecast Bias

Brown (1974) observed:

A project is more likely to be accepted if its revenues have been overestimated and its costs underestimated.

Journal of Finance, vol. 29, 1215-1216

Corollary:

If a project has been accepted, then it’s value tends to have been overestimated.


Our paperOur paper

OPTIMIZER'S CURSE:REMOVING THE EFFECT OF THIS BIAS IN PORTFOLIO PLANNING

by John Schuyler and Timothy Nieman– Society of Petroleum Engineers paper no. 107852,

Proceedings, 2007 SPE Hydrocarbon Economics and Evaluation Symposium held in Dallas, Texas, U.S.A., 1–3 April 2007

– Reviewed, edited and republished in SPE Projects, Facilities & Construction, March 2008


Example in the paper and articleExample in the paper and article

Revenue=Lognormal(μ = 100, σ = 50) $million

CapInv= Lognormal(μ = 40, σ = 20) $million

OpExp= Lognormal(μ = 40, σ = 20) $million


EMV average is $20 million

CapInv and OpExp distributions are each correlated to Revenue with a 0.5 Spearman rank correlation coefficient (ρs).

The E/A are beta distributions and independent.

Example in the paper


-35.2848.6583.93Those with DROI > 1

-17.5935.7853.37Those with EMV > 0

0.0419.9919.95All Candidates

AverageError

AverageActual

AverageEstimate

Project Values$millions

Value overstated by 49%

Value overstated by 72%

Example in the paper



Background








A calculation approach – Minimize biasA calculation approach – Minimize bias

Actuals vs. Estimates

0.1

1.0

10

100

0.1 1.0 10 100

Estimate EUR

Act

ual E

UR

Unbiased estimateswill appear balancedabout the 45° line.


Checking ObjectivityChecking Objectivity

Random Errors in Estimation

.4 .6 .8 1.0 1.2 1.4 1.6 1.8 2.0 2.2

Ratio Estimate/Actual

Freq

uenc

yMean= 1.0 if unbiased


A calculation approachA calculation approach

Error functions

– Strive to ensure unbiased judgments and ‘normal’ unbiased calculations.

– Also want to improve precision

Estimate/Actual

μ = 1



As project values can be negative, we decomposed value into three PV components:– Capital Expenditure– Revenue– Operating Maintenance Cost

Brute force Bayesian calculations– Generate a “cloud” synthetic dataset of actual

values and estimates– Get conditional values by partitioning the cloud



A population of similar project is judged, also along the dimensions of (all PVs):– Capital Expenditure (CapEx)– Revenue (Revenue)– Operating Maintenance Cost (O&Mcost)

EMV = Revenue – CapEx – O&Mcostexpected monetary value

DROI = EMV / PV CapExdiscounted return on investment



Background








Example Spreadsheet Model - PortfolioExample Spreadsheet Model - Portfolio

Zoom-in shots follow


Reserves Analogy with Economic CutoffReserves Analogy with Economic Cutoff

0 80 160

240

320

400

0 80 160

240

320

400

.55

___

___

.45

___

.65

.35

.10

μ=137

μ =160

No minimum economic field size

Minimum economic field size = 83Bcf



Background








Example Spreadsheet Model - ProjectExample Spreadsheet Model - Project

Zoom-in shots follow



This portion of the worksheet is the same



EMV


Mean-Reversion BiasMean-Reversion Bias

Most of the effect, seen here, we’re calling “mean-reversion bias.”

As filtering becomes more exclusive, the OC-bias begins to dominate.

MR-bias Corrected ValuesOriginal Estimates

means


Projects near the population meanProjects near the population mean

0

10

20

30

40

50

20 25 30 35 40 45 50

Unadjusted Project Estimate

Adj

uste

d P

roje

ct E

stim

ate

Project revisions ⎯ reversing themean-reversion bias effect ⎯are adjusted toward this point


The Most ImportantDA Publication in Decades?The Most ImportantDA Publication in Decades?

Optimizer’s curse seems analogous to– Winner’s Curse– Survivorship Bias

How did we miss this??No easy solution --- yet

If your job requires you to objectively forecast either projects or portfolios, here’s one more detail to deal with.


Open IssuesOpen Issues

Might people already be self-correcting for the OC effect?– Data for E/A comparisons are usually from funded

projects

Is it really necessary to describe the parent population?– If so, assessments for these should be reasonably

consistent:

Project type population

E/A distribution

Project’s NPV distribution



How does a real options analysis affect the optimizer’s curse?

– Real options in a project almost always increase project value and will never decrease project value:

There is a higher value estimate

Unsure whether ROA will increase or decrease the OC-correction



How does a real options analysis affect the optimizer’s curse?

– Most projects are not evaluated with options, so the typical project estimate can be increased by real options analysis.

E E’increases with ROA added

– Yet value engineering through project execution increases actual value regardless of whether these options were recognized in project evaluation

A A’increases in implementation as decision opportunities are recognized


Wrap-UpWrap-Up

Comments?

Questions

☺

Follow-up questions?

Contact info. for John in U.S.: 303-693-0067 (GMT-7)

[email protected]

Links for copies of the slides are posted temporarily at www.maxvalue.com/palantir.htm

rational is practical: sh · pdf filepalantir economic solutions ltd john schuyler denver,...

Documents