tom junk - sfu department of statistics and actuarial...
TRANSCRIPT
![Page 1: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/1.jpg)
BanffChallenge2
TomJunkFermilab
BIRSSta6s6csinHEPWorkshopJuly2010
1TomJunkBanffChallenge2
![Page 2: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/2.jpg)
TomJunkBanffChallenge2 2
CommonStandardsofEvidence
Physicistsliketotalkabouthowmany“sigma”aresultcorrespondstoandgenerallyhavelessfeelforp‐values.
Thenumberof“sigma”iscalleda“z‐value”andisjustatransla6onofap‐valueusingtheintegralofonetailofaGaussian
Double_tzvalue=‐TMath::NormQuan6le(Double_tpvalue)
1σ⇒15.9%
Tip:mostphysiciststalkaboutp‐valuesnowbuthardlyusethetermz‐value
Folklore:95%CL‐‐goodforexclusion3σ:“evidence”5σ:“observa6on”Someargueforamoresubjec6vescale.
€
pvalue =1− erf zvalue / 2( )( )
2z-value (σ) p-value
1.0 0.159
2.0 0.0228
3.0 0.00135
4.0 3.17E-5
5.0 2.87E-7
![Page 3: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/3.jpg)
BanffChallenge2Problem#1–StackedplotshownHEP‐style
3TomJunkBanffChallenge2
• Observeddatashownaspointswithsqrt(n)errorbars(yes,theconven6on’scrazybutthat’sthewaywedoit.)• Signalpredic6onshownstackedontopofthebackgroundpredic6on.UsefulbecausewecancomparethethedatawithH0andH1withjustoneplot.
DiscriminantVariable
Even
ts
nbackground=10000nsignal=210ndata=9815
![Page 4: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/4.jpg)
TomJunkBanffChallenge2 4
€
−2lnQ ≡ LLR ≡ −2ln L(data | s+ b, ˆ θ )
L(data |b, ˆ ˆ θ )
Problem1,nosystema6cuncertainty1MillionsimulatedexperimentsforH0and1MillionsimulatedexperimentsforH1
Nuisanceparametersalwaysattheirnominalvalues
p‐value=5.95x10‐4z‐value=3.24‐2lnQobs=1.98
hatsdon’tmaierheresincethere’snofit.
![Page 5: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/5.jpg)
TomJunkBanffChallenge2 5
€
−2lnQ ≡ LLR ≡ −2ln L(data | s+ b, ˆ θ )
L(data |b, ˆ ˆ θ )
Problem1,withsystema6cuncertainty1MillionsimulatedexperimentsforH0and1MillionsimulatedexperimentsforH1
p‐value=1.91x10‐5z‐value=4.11‐2lnQobs=‐15.43
nowdotwofitspersimulatedexperiment‐‐fitforallnuisanceparameters,rateandshape
Eachpseudoexperimentgetsrandomlyfluctuatednuisanceparameters(“prior‐predic6veensemble”)
![Page 6: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/6.jpg)
TomJunkBanffChallenge2 6
ATricktoUseonly1MSimulatedH0Experiments
• Fitthedistribu6onof‐2lnQ(H0)toasumoftwoGaussians–canintegratethatanaly6callywitherf’s.• Needtocheckfitquality.Arealjobwouldbetoes6matetheuncertainty(extrapola6onuncertaintyifneedbe).• Forarealdiscoveryofapar6cle,we’djustusetheneededCPU.Maybethefiiergetsstuckoncein1x107experiments–needtoknowthat.
ThesumoftwoGaussiansisagoodapproxima6onherebutapooroneiftheproblemismorediscrete–onebin,forexample,orlotsoflows/bbinsandoneveryhighs/bbinwithjustafewexpectedeventsinit.
![Page 7: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/7.jpg)
TomJunkBanffChallenge2 7
AProblemwithProblem#2
DiscriminantVariable
Even
ts
nbackground=10000nsignal=200ndata=98439209dataeventsinthevisiblepartofthehistogram
634observedeventsoutof9843areintheupperoverflowbin(!)(~6.44%ofthem)Backgroundtemplatemodelsthis.
IdiscouragetheuseofROOT’sover‐andunderflowbinsforseveralreasons:
1)Theyarenot(usually)ploied.Hardtovalidatethemifyoucannotseethem2)TheyarenotincludedinTH1::Integral()orinfSumwwhendumped.Soscalingbydividingbytheintegralandmul6plyingbythedesiredyieldwon’tgetitright.
rootaccumulatesentriesbeyondthehistogramedgesinunderflowandoverflowbins,andtreatsthemasspecialbins(why?)Sugges6ontoallstudents:constrainallselecteddatatobeinvisiblebins(maxandmin). Problems1and3havenoentriesinthe
underfloworoverflowbins.
SoIsolvedaproblemthatisslightlydifferentandpossiblymoreinstruc6ve.
![Page 8: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/8.jpg)
TomJunkBanffChallenge2 8
Problem2
€
−2lnQ ≡ LLR ≡ −2ln L(data | s+ b, ˆ θ )
L(data |b, ˆ ˆ θ )
Nosystema6cs:‐2lnQ=17.46z‐value=0.20p‐value=0.42
Withsystema6cs:‐2lnQ=‐17.33z‐value=4.1p‐value=1.75x10‐5
DiscriminantVariable
Even
ts
GOFnotevaluatedwithoutsystema6cs–preiypoorthough.Showsthattheno‐systema6csinterpreta6onisincorrect.
nbackground=10000nsignal=200ndata=9843
![Page 9: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/9.jpg)
TomJunkBanffChallenge2 9
Problem2’sFit
Notperfectonthetail,probablyjustneedtorunmorepseudoexperiments
![Page 10: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/10.jpg)
TomJunkBanffChallenge2 10TomJunkBanffChallenge2 10
Problem3
€
−2lnQ ≡ LLR ≡ −2ln L(data | s+ b, ˆ θ )
L(data |b, ˆ ˆ θ )
Nosystema6cs:‐2lnQ=‐43.1z‐value=7.3p‐value=1.4x10‐13
Withsystema6cs:‐2lnQ=‐21.2z‐value=4.44p‐value=4.5x10‐6
DiscriminantVariable
Even
ts
nbackground=80nsignal=72ndata=134
![Page 11: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/11.jpg)
TomJunkBanffChallenge2 11
CombiningProblems1+2+3Jointfits–correlatedsystema6cuncertain6esinarealproblem.Wearetoldtodecorrelatethenuisanceparametersbetweenchannels.
Nosystema6cs:
‐2lnQcomb=‐2lnQ1‐2lnQ2‐2lnQ3
Withsystema6cs–spoiledabitbythedifferentfits,ifnuisanceparametersarecorrelated.Inthiscasethesumrules6llworksbecausealldataandallnuisanceparametersareindependent.
GOFispoorforbothhypotheses–seeprob.2.Largesensi6vity.Nosystema6cscanruleoutbothH0andH1.
Nosystema6cs:‐2lnQ=‐23.7z‐value=7.0p‐value=1.2x10‐12
Withsystema6cs:‐2lnQ=‐53.96z‐value=7.5p‐value=3.6x10‐14
![Page 12: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/12.jpg)
TomJunkBanffChallenge2 12
Combina6onSignificancesareaBitofanExtrapola6onwithjust1MSimulatedOutcomes
![Page 13: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/13.jpg)
TomJunkBanffChallenge2 13
Es6matesofSensi6vity• Well,1Millionsimulatedexperimentsisn’tenough–canget1Millionmoreincombina6onbyaddingthe‐2lnQ’sfrom1,2,and3’stogether.
• Wilks’sTheoremprobablyisagoodapproxima6onheretoo.
• Importancesamplingcouldbeusedtoimproveprecisionintails
• Fordiscovery,we’duserealCPUasthesysteam6cswillbecorrelatedandtheremaybeasinglebinaddingadiscretecomponenttoit.
• Ourfavoritesensi6vityes6mate:pmed,signalisthemedianexpectedp‐valueassumingasignalispresent.1Mpseudoexperimentsnotquiteenough.
• Astand‐in:the“o‐value”(namedbytheCDFKarlsruhesingletopteam,butwe’duseditbefore.
• Medianscanbeusedinsteadofmeans,andtheσ’sareRMS’softhe‐2lnQdistribu6on.
€
o − value =−2lnQ bkg − −2lnQ s+b( )
σ bkg2 +σ s+b
2
![Page 14: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/14.jpg)
TomJunkBanffChallenge2 14
€
o − value =−2lnQ bkg − −2lnQ s+b( )
σ bkg2 +σ s+b
2
Problem <‐2lnQ>b RMSb <‐2lnQ>s+b RMSs+b o‐value
1nosyst 41.9 12.3 ‐46.3 14.3 4.7
2nosyst 19.1 8.6 ‐20.0 9.1 3.1
3nosyst 49.5 11.9 ‐68.9 19.5 5.2
123nosyst 110.6 19.1 ‐135.2 25.8 7.6
1syst 21.2 8.9 ‐28.1 13.5 3.0
2syst 12.8 6.6 ‐16.7 9.3 2.6
3syst 21.6 9.5 ‐25.6 12.2 3.0
123syst 55.5 14.6 ‐70.3 20.5 5.0
Toagoodapproxima6on,o‐valuesaddinquadratureforthecombina6on.Trueforthisproblem,butnottrueingeneral.
![Page 15: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/15.jpg)
TomJunkBanffChallenge2 15
BackupMaterial
![Page 16: Tom Junk - SFU Department of Statistics and Actuarial Sciencelockhart/richard/banff2010/junk_bc2.pdf · Tom Junk Banff Challenge 2 2 Common Standards of Evidence Physicists like](https://reader036.vdocument.in/reader036/viewer/2022071301/60a3b6ccf5b54f6a805361cd/html5/thumbnails/16.jpg)
TomJunkBanffChallenge2 16
Fitting Nuisance Parameters to Reduce Sensitivity to Mismodeling
Means of PDF’s of -2lnQ very sensitive to background rate estimation.
Still some sensitivity in PDF’s residual due to prob. of each outcome varies with bg estimate.