nba nn project 15-16 update
TRANSCRIPT
NBANeuralNet2015-16Update
BenTaborskyApril19,2016
Contents
1. ReviewofModel2. 2015-16Performance3. Analysis
1.ReviewofModel
AnNBANeuralNet• AboutayearagoItrainedaneuralnettopredicttheoutcomesofbasketballgames.Thisyear,Itestedit
forreal,beLngonrealbasketballgameswithrealmoney.Theseslidesareanupdateonmyexperience,starNngwithareallyquickreviewofthemodel.
• Aneuralnet’sstructureissortofbasedonbrainbiology.ItisaseriesofconnecNngnodesthat“fire”likeneuronsthroughtheir“dendrites,”basedonthevaluesofinputsitreceivesonits“axonterminals.”
• NeuralnetworksarecoolandusefulbecausetheycanmatchanyfuncNonwiththeirnonlineariNes,andthey’reagreatwaytolearnaboutcomplicatedpaTernswithoutimposingalotofassumpNonslikeyoumighthavetowith,forexample,linearregression.
• ItturnedouttobepossibletousejustarelaNvelysmallamountofdatatotrainaneuralnetworkthatachievedaleveloferrorcloseto,butnotquiteaslowas,theVegasmoneyline.
• Ithoughtitwascloseenoughtowonder,ifthepredicNonsareverydifferent,isthisaglitchintheneuralnetoraretheoddsmakersmissingsomething?
Predic'on AverageCEErrorRandomProbability 1.00050%HomeVictory 0.69360%HomeVictory 0.671NetTrainingError 0.609NetValidaNonError 0.608WestgateMoneyLine 0.581
NetErroronBacktestYears(adj) 0.595
BeLngStrategyBacktest• ToanswerthisquesNon,Imadeupastrategy:
– Whenthedifferencebetweenthemodel’spredicNonandVegas’predicNonisgreaterthan15percentagepoints,makeabet
– TheneuralnethasahardNmewithverybadhometeams,andnevergivesthemlessthana27-28%victoryprobability.SoIignorethefirstruleifVegasgivesahometeamlessthana40%chance.• InteresNngnote:theseveryimprobablebetsgenerallybreakevenbecauseyouonlyhavetowinoneortwotomakeup
forlosses,andsomeNmesGoldenStatelosesatMilwaukee.Buttheyaddalotofuglyvariance.– ThefourseasonsItestedwerenotusedtotraintheneuralnet.
• Accordingtothesetests,themodelseemedtoberightogenenoughtobeconsistentlyprofitable.
AnnualAverageProfit 1632AnnualStandardDevia'on 962Informa'onRa'o 1.70MaxDrawdown 891
2014-15OutofSamplePerformance• Ofcourse,nobodyshouldbelieveabacktestbyitself,becausethereareallkinds
ofbiasesthatcouldseeptheirwayin.Forexample,ItriedhardtocomeupwithagoodsystemforguessingwhatplayersI’dhaveknownwereplayingbeforeeachgame.ButIalsoreallywantedthemodeltowork.
• Soinmid-February2015Ifrozeallthemodel’sparametersand“paper-traded”usingitspredicNonsunNltheendofthatseason,tryingtobeasrealisNcaspossible.
• NotanoverwhelminglyposiNveresult,butalsonotinconsistentwiththebacktest.– Mostimportantly,itdidn’tloseatonofmoney.
2.2015-16Performance
PuLngMyMoneyWhereMyMouthIs• Agertheoutofsampletestwasnotacatastrophe,Idecidedtogivethemodelarealtestinthe
2015-16season.• So,offIwenttospendthewinterinsnowyLakeTahoe,whereIcouldsnowboardstraightfrommy
liTlestudiotothecasino.– UnNlIbrokemyarm.
• Istuckwiththestrategyasdescribed:
– Aslongasthehometeamisgivena40%+chanceatvictory,makeabetifthemodeldisagreeswiththelinebymorethan15percentagepoints.
– AfewweeksintobeLngIrealizedthatthemodel’spredicNonsweredoingverywellinthe30-40%rangeforhomevictorysoIloweredthefloorto30%onaprovisionalbasisfortherestoftheseason(andmadehalf-sizebets,thoughthat’snotreflectedintheresultsIshowhere).
– Never,everletpersonalfeelingsoropinionsoverridethemodel.Thiscanbehard.• Model,youthinktheClippershavea38%chancetobeattheWarriors?Really?INOAKLAND?Oh,dear.
LovelyLakeTahoe
Vicodinandsausagefingers
PerformanceWith$100betsoneverymodelpredicNon,thecumulaNveprofitfortheyearlookslikethis:
Breakdowns• BetsarecategorizedaccordingtoVegas’esNmateofthehometeam’svictory
probability
BetsonHomeTeam
BetsonAwayTeam
3.Analysis
SomeNmesyoujustgetlucky!• Themodel’sresults,presentedoverthelastfewslides,lookpreTygood,butthereisalwaysachancethatamonkeythrowingdartsatanNBAschedulecoulddoaswell.
Isthatwhathappened?• Tofindouthowwellyoucouldexpecttodowithapproximatelythesamenumberofbets,but
chosenrandomlyinsteadofbythemodel,Icreated1000setsofrandombets.– Onaveragetheylost$500.ThismakessensebecauseVegastakestransacNoncostsaliTleover4%.– 4%transacNoncost*$100*124bets=$496– Theydidhavehugevariance,withastandarddeviaNonofalmost$1500– So,assumingthatthemodeldoesNOThaveexplanatorypowerandtheresultswererandom,itsprofitof
$2644is2.1standarddeviaNonsabovethemean,andturnsouttofallinthe97thpercenNle– Inotherwords,it’spossiblebutunlikelythattheperformancecanbeexplaineden'relybyluck
Histogramof1000randombetseasonsFrequency(y)vsProfit(x)ProfitshownistheboTomofthebucket
Timeseriesof50randombetseasons
OtherPossibleExplanaNons
• ThemodelwasalmostcertainlybeTerthanrandombeLng–butdiditjustluckilybetontheunderdogorfavoritealot?
• Whataboutcertainbet-winningteams?
FavoritesandUnderdogs• OntheBreakdownsslide,it’spreTyclearthatalotofthemodel’sprofitscamefrombeLngon
underdogs.So,didithappentodothisinanamazingyearforunderdogs?• ThechartsbelowshowwhattheprofitwouldhavebeenlikebeLngontheunderdogorfavoritein
everygame.• UnderdogsdidmildlybeTer.Butsayingthiswouldnothavebeenagoodstrategyisan
understatement.• ...waitasec,whyaretheyBOTHnegaNve?
– 1,157games*2betseach*4%transacNoncost*$100=$9256– AllUnderdogsProfit:-$4476– AllFavoritesProfit:-$4962– TotalProfit:-$9438
ProminentTeams• ThemodelmadealiTleoverhalfofitsprofitsbybeLngagainstHouston20NmesandonCharloTe
15Nmes(actually13Nmesfor,twiceagainst).• Wasthisthesourceoftheluck?
– Bothofthesewouldhavebeendecentbetsrepeatedoverandoverallseason,thoughwithHoustonthegainscameallatonceinthebeginning.
– WithregardtoHouston,themodelcapturedthreeofthefourbigupsetsthatmadeuptheearlyseasongains.BytheendItmanaged$816ingainsin20bets,beaNngtheAlwaysBetAgainststrategybyaboutathird.
– AsforCharloTe,themodelgained$653,out-earningtheAlwaysBetOnstrategywhileusingmanyfewerbets.
Profitsfrom73betsonCharloTeProfitsfrom73betsagainstHouston
Othergoodbets?
74betsontheWarriors.Notbad! ButnotnearlyasgoodasbeLngontheNets...
Conclusion
• TesNngtheNBAneuralnetinareal-lifesituaNonresultedinasuccessfulbeLngseason!
• Whiletheperformancemayormaynotbereplicable,itisunlikelythatitwasduetoluck.
• BeLngisverystressful.• Approximately60%rangeofmoNonrecoveredinlegwrist.