“fooled by randomness”...“fooled by randomness” - improving decision making with limited...

SocietyofPetroleumEngineers2017Dis8nguishedLecturerProgramwww.spe.org/dl

JimGouveiaP.Eng.Partner Rose&AssociatesLLP

“FooledbyRandomness”-ImprovingDecisionMakingWithLimitedData

1

q  Asprofessionalswearecon8nuouslychallengedtomakeinformeddecisionswithlimiteddatasets.

q  Ourexploita8onofUnconven8onalresourcesina8meofbudgetrestraints,lowcommodityprices,andcompe88vepressureshasdriventhedesiretogettherightanswersassoonaspossible.

q  Ourdecisionson“sweetspots”,newtechnologiesandindeednewplaysareoMenbasedsimplyonthearithme8caverageoftheresultsfromafewwells.

q  Wherewehaveerredasanprofessionisinhonouringlimiteddatasetswithoutconsidera8onoftherepresenta8venessofthedata.

“Fooled by Randomness”

2

Outline

q  BackgroundonAggrega8onPrinciples

q  ReviewAggrega8onCurvesandtheirapplica8ontolimiteddatasets

q  ReviewtheuseofSequen8alAccumula8onplotsforreal8mevalida8onofourdistribu8ons

q  Conclusions&Recommenda8ons

3

•  Thereisanequalprobabilityofrollinga1toa10.

•  90%ofthe8mewewillrealizeanoutcomethatequalsorexceeds2.

•  10%ofthe8mewewillrealizeanoutcomethatequalsorexceeds10.

•  Thera8ooftheP10(high)totheP90(low)is5.

•  Weknowthedistribu8onofadieisdiscreteuniform,andthatwithrepeatedtrialstheaverageoutcomewillbe5.5.

ConsideraTensidedDie

Aggrega8onPrinciples101

4

•  Whatisourconfidencethatwewillrealizethe

meanoutcomeof5.5,aMer1dieroll,5dice

rolls,10dicerolls?

•  Whatifwedevelopedanewtechnologythat

wouldimprove“Die”performanceby20%.

•  Howmanydicerollswouldweneedtoconfirm

theeffec8venessofthenewtechnology?

Aggrega8onPrinciples

0=10

5

•  Whatwouldyouconcludeifonyourfirst

trialofthenewtechnologyyourolleda5?

•  Shouldyoufeelbeberorworseaboutthenewtechnology?

•  Couldweconcludethatthetechnologyfailed?

•  Letsreviewapragma8csta8s8calapproachto

providequicksolu8ons.


0=10

6


0=10

•  Wearereasonablycertainwewillrolla2ormore90%ofthe8me.TheP10:P90ra8ois5.0

•  Rollfivedice.Dividesumby5,repeat.Wewillaverage2ormore99.86%ofthe8me.TheP90oftheaggregatedoutcomeis3.8TheP10:P90ra8ois1.9

•  Rolltendice.Dividesumby10,repeat.TheProbabilityofaveraginga2ormoreis99.999%.ThisisnotaP90!TheP10:P90ra8ois1.6

SingleDieOutcome

FiveDiceAveragedoutcome

TenDiceAveragedoutcome

2.0 10

3.8 7.2

4.3 6.7

7

NumberofTimesTheDieisRolled

WithIncreasingDiceRollsTheVarianceDecreases

P90

P10

8

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

5.50

6.00

6.50

7.00

7.50

8.00

8.50

9.00

9.50

10.00

0 5 10 15 20 25 30 35 40 45 50

AverageOutcomeVa

lue

NumberofDiceNumberofTimesTheDieisRolled

Aggrega8onChartsRevealHowTheVariance

DecreasesWithIncreasingDiceRolls

AggregateP10AggregateP50AggregateP90

AverageOutcomeVa

lue

9

InThisAggrega8onChartTheOutcomesAre

NormalizedasaFunc8onofTheMean


Percentageofth

eMeanVa

lue

0%

20%

40%

60%

80%

100%

120%

140%

160%

180%

0 5 10 15 20 25 30 35 40 45 50

Percen

tageo

fTh

eM

ean

NumberofDiceNumberofTimesTheDieisRolled1010

•  Thekeydriversofeconomicvalua8onsaMerproductpricearetypically:

o  Reserveso  Rateo  Capitalcosto  CycleTime

•  Aseachoftheaboveisbasedonmul8plica8veprocessesthey

canbewellfibedwithlognormaldistribu8ons,with“spiked”endmembers.

•  Let’sreviewanexampleofaggrega8onusingalognormaldistribu8onfores8matedul8materecovery(EUR),onaperwellbasis.

Aggrega8onAppliedtoSubsurfaceParameters

11

1 5 25

P90 P10

1Well

5-WellAverage

25-WellAverage

TheImpactofAggregaKonona5&25wellprogramEUR–MMscf

Aggrega8ngEURDistribu8ons-P10:P90Ra8oof4

12

Percentageofth

eMeanVa

lue

WellCount

Thesecurvesarebasedonsamplingageologicsubsetwithaknownmean.


SPE175527byMcLane&Gouveia

Ageologicsub-setisaregionofrockthathasanalogousgeologicaldeposi8onalproper8es,withrela8velyconsistentbulkrockandfluidproper8es.


13

Percentageofth

eMeanVa

lue

WellCount

Withlargerwellcounts,theP90andP10approachtheunderlyinggeologicsubsetmeanthatthe

Aggrega8oncurveswerebasedon.



14

•  Therealityisthatbudgetsandcompe88vepressuresforceourhandsinmakingdecisionswithlimiteddata.

•  Understandingtheinherentuncertaintyinourdataisnotintendedtopreventdecisionmaking.

•  Thegoalshouldbeabeberunderstandingoftheinherentuncertaintyinourdatasetsandthenmakingdecisionswithknowledgeoftheirrepresenta8veness.

•  Let’sreviewhowAggrega8oncurveswillguideus.


15

•  NorthAmericanexperiencehasdemonstratedahighdegreeofcongruenceinP10:P90ra8osforhorizontalwellswithcommonhorizontallengthsandcomple8ons.

•  P10:P90ra8osof4to5arecommonforasingleOperatorwitha

consistentcomple8ontechniqueinlateralsof5,000feet(1500+m)and20+fracturestages.

•  P10:P90ra8osof2to3arebeingexperiencedwhenlateralsaredrilledusing3DSeismicdrivenGeo-steeringandconsistentcomple8ontechniqueinlateralsof3000m+.

•  Thetechniquerequiresustoassumelognormalityandthevariancefromanalogousreservoirswithsimilarhorizontalwelllengthsandcomple8ontechniques.

Applica8onofAggrega8onCurves

16

•  Resourceplaysshowrepeatabledistribu8ons,yearoveryearforagivengeologicsubset(SocietyofPetroleumEvalua8onEngineersMonograph3).

•  Caveatstothisapproach:o  Horizontalwelllengthisconsistentornormalized.o  Drillingandcomple8ontechniquesareanalogous.o Wearereasonablycertainthatthe“averaged”geologydoesnotvarysignificantlywithinthegeologicsubset.

•  Inemergingplaystheaggrega8oncurvescanbeusedtoboundtherangeofthegeologicsubsetmeanasafunc8onofwellcount.Acri8calinsightforearlydecisionmaking.

Applica8onofAggrega8onCurves

17

Aggrega8onDeriva8veApplica8onsAggrega8onCurves(TrumpetPlots)–Usedtoillustratereduc8oninuncertainty(variance)asafunc8onofincreasingwellcountusingknownanalogs• Func8onofAnaloguncertainty(P10:P90ra8o),andwellcountAggrega8onCurves–Reverseengineeredtodeterminetherangeofthemeanbasedonaknownsamplesizeandarithme8cmean• VariancebasedonAnalogs

ConfidenceCurves–Usedtocommunicateconfidenceinoutcomes• BasedonmeanandvarianceinProduc8onTypeCurves.• HelpsdefinePre-developmentstagegatethresholdsbasedonanalogdata• Confidenceisafunc8onofuncertainty(P10:P90ra8o),yourtarget(objec8ve)andthenumberofwellstobedrilled(samplesize).PilotDesign–HowmanywellsandwhataveragerateSequen8alAccumula8onPlots–Usedtoassessvalidityofforecasts• TracksactualsagainsttheforecastedP10andP90oftheaggrega8oncurves• Providesearlyfeedbackaboutthevalidityoftheforecastedparameter

40%

50%

60%

70%

80%

90%

100%

110%

120%

130%

140%

150%

160%

170%

180%

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Percentageofth

eMeanVa

lue

WellCount


87%

111%

Applica8onofAggrega8onCurves–KnownMeanof1,200Bopd

DetermininguncertaintyinthemeanbasedonsamplesizeP10:P90=4

TheP50~TheMean=1,200

A25wellprogramwilldelivera:P10=1,332bopdP50=1,200bopdP90=1,056bopd

Falher‘H’–PeakDailyGasRateofTheFirst24Wells

P90 = 5. 6 MMscf/d P50 = 11.2 MMscf/d P10 = 22.5 MMscf/d Arithmetic Avg =12.8 MMscf/d P10:P90 ratio = 4

q  Based on the 24 well sample we observe that the distribution is well fit with a lognormal distribution.

q  The P90 and P10 of a randomly sampled individual well is 5.6 and 22.5 MMscf/d respectively.

q  The arithmetic mean of the 24 well sample is 12.8 MMscf/d.

What is the uncertainty in the mean of this Geologic subset given the 24 well arithmetic mean of 12.8 MMscf/d?

21

18

Percentageofth

eMeanVa

lue

WellCount


22

DeterminingTheUncertaintyinTheMean

-BasedonSampleSize

Percentageofth

eMeanVa

lue

WellCount


Percentageofth

eMeanVa

lue


-BasedonSampleSize

BasedontheP10:P90ra8oof4aggrega8oncurve,90%of

the8methegeologicsubsetmeanwillbe86%ormore

ofthe24wellarithme8csamplemean.

24

86%

2323

Percentageofth

eMeanVa

lue

WellCount


Percentageofth

eMeanVa

lue

Basedonthisaggrega8oncurve,only10%ofthe8me

willthegeologicsubsetmeanbe115%ormoreofthe24

wellarithme8csamplemean.

24

115%


-BasedonSampleSize

2424

Percentageofth

eMeanVa

lue

WellCount


Percentageofth

eMeanVa

lue

14.7

11

12.8

86%

115%

AMer24wells,weare80%confidentthatthegeologicsub

setmeanwillbebetween11and14.7MMcfd.


-BasedonSampleSize

2525

P90 = 5.56 MMcf/d P50 = 11.18 MMcf/d P10 = 22.46 MMcf/d P10:P90 ratio = 4 Avg. = 12.8 MMcf/d

Falher‘H’–PeakDailyGasRate

Mean ~ P40

80% confident that the geologic sub-set mean will be between 11 and 14.7 MMcf/d

26

•  Employershaveanexpecta8onthattheirprofessionalscanforecasttheresultsoffutureprogramsbasedonpriorresults.

•  Withincreasedsamplesizethearithme8cwellaveragewillconvergeonthetruegeologicalsubsetmean.Withlimitedwells,thebestwecandoisevaluatetheuncertaintyinthemeanofthesubset.

Forecas8ngBasedonLimitedSamples

Mean=12.8MMcfd

MMcfd

2727

•  E&P professionals often ignore the uncertainty in the mean value. As a consequence forecasted aggregation will converge on the mean of the sampled wells.

•  This simple aggregation does not honour the irreducible uncertainty based on the original 24 well sample set.

Aggregation Curve Application –

Arithmetic Mean = 12.8 MMcfd

28 28

BuildingConfidenceCurvesfromAggrega8on-Confidenceofexceeding140targetvswellcount

1well50%outcomes>140

10wellaggregate76%outcomes>140

20wellaggregate86%outcomes>140

5wells67%outcomes>140

#wells probof>140BOPD

1  50%5  67%10  76%20  86%

5 10 20

30

1020

908070605040

Confi

dence-%

0

10

20

30

40

50

60

70

80

90

100

0 5 10 15 20 251NumberofWellsinProject

#wells probof>140BOPD

1  50%5  67%10  76%20  86%

BuildingConfidenceCurvesfromAggrega8on-Confidenceofexceeding140targetvswellcount

VariableRateConfidenceCurves-Confidenceofexceedingatargetvswellcount

5 10 20

30

1020

908070605040

1NumberofWellsinProject

175

150

140125

100

BuiltusingCrystalBallExcelfunc8on“CB.GetCertaintyFN”

Confi

dence-%

Toincreasetheconfidenceofdelivery:•  Increasenumberofwells•  Reducetarget

IndesigningPilotsthekeyques8onsshouldbe:•  Howmanywellsshouldbeinourpilot?• Whatconfidencelevelinthedatadowerequire?• Whatratedoweneedtovalidatebeforeproceedingtothenextstageofdevelopment?

LetsreviewanexampleofPilotdesign

DecisionMakinginUnconven8onalReservoirs

PilotDesign:Step1–BuildAnalogDistribu8ons

TheperwellPeakratedatacanbepresentedviathelogcumula8veprobabilityplotorafrequencyplot.

140bopdistheP50ofthedistribu8on.Weknowthat50%ofthe8meanewwellwillmeetorexceedthattarget.50%ofthe8meanewwellwillsamplelessthanorequalto140bopd.

•  Thesevaluesaresetbythefirm’sleadership.ForexampleallnewplaysmustprovideafullcycleRORof15%ormore.

•  TheAssetteamwilldevelopacurveofdailyaverageproduc8onratedividedbypeakrateversus8mefortheirselectedanalog.

•  Thepeakrateisthenreducedinthedevelopmentmodelun8lthefullcycleeconomicreturnequalstherequired15%ROR.

•  ThisisthereverseengineeredPeakratethatwewilltestagainstinourPilotdesign.

•  Wewilluseabreak-evenrateof100bopdtobuildanexample.

PilotDesign:Step2–ReverseEngineeraMinimumEconomicTargetofPeakRateinbopd.

•  Howmanywells,andatwhatperwellaveragepeakratewillourPilotneedtoproducetovalidatea50%and90%confidencethatthenewzonewillmeetorexceedManagement’sexpecta8ons

•  Ifdecisionsaretobemadeonan50or90%confidenceinterval,wewillneedtoreverseengineertherequired“Pilottarget”basedonanaveragedperwelloutcomefromourpilot.

•  Wewillneedtoassumelognormalityandthevariance(P10:P90ra8o)basedonourbestavailableanalog(s).

•  The“Up8ck”ofthe“break-evenrateof100bopdwillbeafunc8onofthepilotwellcurveandthederivedvaluefromtheP90andP50aggrega8oncurves.

PilotDesign:Step3

•  EntertheP90andP50Aggrega8oncurvesatthenumberofpilotwellsbeingevaluated.Readofftherespec8ve%ofthemeanvalues.

•  TheP50=97%ofthemean

•  TheP90=70%ofthemean

PilotDesign:Step4

PercentageoftheM

eanVa

lue

WellCount


70%

97%

•  Tobe50%confidentthatourdevelopmentwillmeetorexceedtheminimumthresholdvalueof100bopd,the5wellpilotaveragemustmeetorexceed(100/0.97)103bopd.

•  Tobe90%confidentthatourdevelopmentwillmeetorexceedtheminimumthresholdof100bopd,the5wellpilotaveragemustmeetorexceed(100/0.7)142.9bopd.

PilotDesign-The“NoRegrets”Rates

•  Wedeterminedthatweneedtorealizeanaveragerateofatleast103bopd/welltobe50%confidentthatthenewareawouldrealizeabreakevenROR.Ques8onis,whatifyoutested100bopd,wouldyouwalkknowingthereisclosetoa50%chanceyouwillbeembarrassed?

•  Thisbringsaboutaconsidera8onofwhatwewillrefertoasthe“NoRegretsRate”.Thatproduc8onrateatwhichyouwouldhavenoregretswalkingaway,knowingthatitmightactuallywork.Therearetwokeyfactorstoconsider:

o  Determininganoregretsvolumeonabreak-evenbasis

o  Determiningthelikelihoodoftheplaymee8ngyoudesiredeconomichurdlesgiventhatitfailedtomeetyour50%breakevenrate.

Determininganoregretsvolumeonabreak-evenbasis

• Weneedafrankdiscussionwithourdecisionmakersbeforethewellresultsandouremo8onscomeintoplay.

• Youwillneedtohaveaddi8onalaggrega8oncurvestoshowa25%confidencelevelplusothervaluestheymayrequest.R&Arecommendstheaddi8onofaP25andP10value.

• Rerunthenumbersusingyourcorporatehurdleratesinaddi8ontothe“break-even”RORvalues.

PilotDesign-The“NoRegrets”Rates

PilotDesign-“ChallengesinPilo8ngNewTechnology”

•  Baseforecastuncertaintyo  InSAGDwellwehaveobserveda+/-20%varianceo  AtthePadlevelthiscandecreaseto+/-5%o  Measurementprora8onissuesamplifiedatthewelllevel

•  Timedelaytoseeimprovementso  Impactofforecastuncertaintyiscompounded

o  Impa8ence

•  Incrementalchangesaresmallrela8vetootheruncertain8es

•  Withsuchalargedegreeofinnateuncertaintyhowdoweassureourmanagementteamthatourprogramsareontrack?

•  WecanusetheAggrega8oncurvestodetermineour80%confidenceintervalsasafunc8onofwellcount.

•  Byployngouractualresultsagainstthe80%confidencebandswearegenera8ngwhatarereferredtoas“Sequen8alAccumula8onPlots”.

•  Thisgraphicalapproachprovidesanearlyindica8onofpossibleissuesandfacilitate“real-8me”earlydecisionmaking.

MakingBeberDecisionsBasedonLimitedData

Sequen8alAccumula8onPlots(SAP)

•  SAPPlotsarebasedontheAggrega8onCurves•  Ratherthanpresen8ngdataastheaveragewellinamul8plewellprogram,thedataispresentedasthetotalP10-P50-P10valueforthefullwellcount

•  Usefulforcomparingactualtoexpectedoutcomes–  Sothatexpecta8onscanbeadjustedand,ifnecessary,theprogramcanbehaltedifunderperforming

Accumula8ng

1

2

3

4

6

P10:

P90RATIO

WeconverttheAggrega8oncurve,whichpresentstheaveragewellasafunc8onofwellcount,totheaccumula8ngtotals,inthisplot,bymul8plyingthe%ofthemeanfromtheaggrega8oncurvebythemeanvalueandbythetotalnumberofwells.

5

WellCount

Accum

ula8

ngM

Mcfd

40%

50%

60%

70%

80%

90%

100%

110%

120%

130%

140%

150%

160%

170%

180%

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

DeterminingTheUncertaintyinTheMeanBasedonSampleSizeP10:P90 =4(P1,P99 spike)

Percen

tageofthe

MeanVa

lue

WellCount


Accumula8ng

P10:P90Ra8o

1

2

3

4

5

6 P10:

P90RATIO

46

Sequen8alAccumula8onPlot-P10:P90Ra8o=4

Thebluebarsareeachindividualwell’speakdailygasrateusingtherighthandYaxis.

Accum

ulaK

ngM

Mcfd Increm

entalMMcfd

4747

PeakDailyGasRate-Falher‘H’

CaseStudy–NorthernMontney,WCSBTheIni8alAnalogProduc8onTypeCurveWasBasedonRegionalHeritageMontney.IsThisTypeCurveRepresenta8ve?

GreendotsareMontneyHorizontalsattheendof2012.

NorthernMontneyArea

RegionalMontneyArea

48

AnalogueBest3Mo.Avg.GasRate

BasedonRegionalHeritageMontneyselectedanaloguewells

P10/P90=4.2

49

Sequen8alAccumula8onPlot

ForecastbasedonRegionalHeritageAnaloguewells.ActualsbasedoncasestudyOperator’s Northern Montney wells coming on-stream from the period from08-2009through12-2009(nextwellon-streamin07-2010).

50

Forecast based on Regional Heritage Analogue wells. Actuals based on casestudy Operator’s NorthernMontneywells coming on-stream from the periodfrom08-2009through03/2011(nextwellson-streamin08/2011).

51

Sequen8alAccumula8onPlot

ComparisonofBest3Mo.Avg.GasRate

Basedonactualresults-to-datefromcasestudyOperator’sNorthernMontneyrevisedistribu8on to reflect mean from case study Operator’s Northern Montney wellswithsamevariance(P10/P90=4.2)fromanaloguewells.

Mean=3,206

Mean=4,137

52

Sequen8alAccumula8onPlot(RevisedMeantoReflectResults-To-Date)

ForecastbasedonrevisedmeanusingcasestudyOperator’sNorthernMontneyactualresults-to-date.ActualsbasedoncasestudyOperator’sNorthernMontneywellscomingon-streamfromtheperiodfrom08-2009through03/2011.

53

54

57

0

10

20

30

40

50

60

70

80

90

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

CumulativeRa

te-MMcfd

WellCount

Actual ForecastP90 ForecastP50 ForecastP10 CumActual

Actuals-MMcfd

4

6

2

8

1

5

7

3

Sequen8alAccumula8onPlot(RevisedMeantoReflectResults-To-Date)

ForecastbasedonrevisedmeanusingOperator’sNorthernMontneyfirst31wells.Actualsarefromthenext16wellsdrilledwithrevisedtechnology.

54

HowtoErode$1BillioninShareholderValue

Forecastsarebasedonthebreakeventypewellcurve

AllCaseStudyOperator’sWells

•  AsProfessionalswetendtorelyontheobserveddatawithoutacknowledgingandunderstandingtherepresenta8venessofthesampleddata.

•  Allowingsta8s8cstospeakforthemselvesrequireslargewellcountsthatareoMennotprac8calinhighcostcompe88veplays.

•  Aggrega8oncurvesarepragma8capproachesthatprovideinsigh{ulillustra8onsoftheinnateuncertaintyinourlimiteddatasets.

•  Theobservedvarianceindrillingprogramsdoesnotalwaysimplythatthingsarechanging.

Conclusions

56

SocietyofPetroleumEngineers2017Dis8nguishedLecturerProgramwww.spe.org/dl

JimGouveiaP.Eng.Partner Rose&AssociatesLLP

“FooledbyRandomness”-ImprovingDecisionMakingWithLimitedData

57

“fooled by randomness”...“fooled by randomness” - improving decision making with limited...

Documents