why do wouter (and atlas) put asymmetric errors on data points ? what is involved in the cls...

Why do Wouter (and ATLAS) put asymmetric errors on data points ?

What is involved in the CLs exclusion method and what do the colours/lines mean ?

ATLAS J/Ψ peak (muons)Excluding SM Higgs masses

LEP exclusion Tevatron exclusion

Why do you put an error on a data-point anyway ?

ATLAS J/Ψ peak (muons)

Estimate of underlying truth (model value)

Poisson distribution


Probability to observe n eventswhen λ are expected

λ=4.90

Number of observed events

#observed #observed Lambda hypothesisLambda hypothesis

fixedfixedvaryingvarying

€

P(n | λ ) =λne−λ

n!

€

P(0 | 4.9) = 0.00745

€

P(2 | 4.9) = 0.08940

€

P(3 | 4.9) = 0.14601

€

P(4 | 4.9) = 0.17887

Poisson distribution: properties


properties

the famous √N

(1) Mean:

(2) Variance:

(3) Most likely value: first integer ≤ λ

http://www.nikhef.nl/~ivov/Statistics/Poisson.pdf

€

P(n | λ ) =λne−λ

n!

€

⟨x⟩= λ

€

⟨(x − ⟨x⟩)2⟩= λ

Lambda known expected # events

λ=0.00 λ=1.00

λ=4.90λ=5.00

Large number of events

λ=40.0

Unfortunately this is not what you wanted to know …

What you have: What you want:

€

P(Nobs | λ )

€

P(λ |Nobs)

From data to theory

Likelihood: Poisson distribution“what can I say about the measurement (Number of observed events) given an expectation from an underlying theory ?”

This is what you want to know: “what can I say about the underlying theory given my observation of a given number of events ?”

€

P(λ |Nobs) = P(Nobs | λ )P(λ )

λ (hypothesis)

Nobs known (4) information on lambda

“Given a number of observed events (4): what is the most likely / average / mean underlying true vanue of λ ?”

#observed #observed Lambda hypothesisLambda hypothesis

fixedfixed

P(N

ob

s=

4|λ

)

varyingvarying

Normally you plot -2log(Likelihood)

Likelihood:

€

P(4 | 0) = 0.00000

€

P(4 | 2) = 0.09022

€

P(4 | 4) = 0.19537

€

P(4 | 6) = 0.13385

Properties of P(λ|N) for flat P(λ)

properties

(1) Mean:

(2) Variance:

(3) Most likely value: λmost likely = x

http://www.nikhef.nl/~ivov/Statistics/Poisson.pdf

€

P(λ |Nobs) = P(Nobs | λ )P(λ )

€

⟨λ⟩ =x +1

€

⟨(λ − ⟨λ ⟩)2⟩= x +1

Assuming P(λ) is flat

This is normally presented as likelihood curve

λ (hypothesis)

P(N

ob

s=

4|λ

)-2

Log

(P(N

ob

s=

4|λ

))

Likelihood

-2Log(Prob)

4.002.32

-1.68

6.35

+2.35

sigma: ΔL=+1 ΔL=+1

68.4%

Pdf for λ

ATLAS J/Ψ peak (muons)

€

4−1.68+2.35

So, if you have observed 4 eventsyour best estimate for λ is … :

CLS method

http://www.nikhef.nl/~ivov/Statistics/thesis_I_v_Vulpen.pdf

Chapter 7.4

http://www.nikhef.nl/~ivov/Statistics/

http://www.nikhef.nl/~ivov/Statistics/thesis_I_v_Vulpen.pdf

Your Higgs analysis

Discriminant variable Discriminant variable

Higgs

SM

Hebben we nou de Higgs gezien of niet ?

Higgs

SM

SM+Higgs

Scaled to correct cross-sections and 100 pb-1

Can also be an invariant mass plot

Approach 1: counting


tellen tellen

Experiment 1 Experiment 2

Origin # events

SM 12.2

Higgs 5.1

MC total 17.3

Data 11

Origin # events

SM 12.2

Higgs 5.1

MC total 17.3

Data 17

Expectations

If the Higgs is NOT there:On average 12.2 events

If the Higgs is there:On average 17.2 events

Experiment 2:17 events observed

Experiment 1:11 events observed

SM SM + Higgs

Discovery

- Only look at what you expect from Standard Model background- Given the SM expectation: if probability to observe as many events you have observed (or more) is smaller than 5.7 10-7

SM hypothesis is very unlikely reject SM discovery !

- Only look at what you expect from Standard Model background- Given the SM expectation: if probability to observe as many events you have observed (or more) is smaller than 5.7 10-7

SM hypothesis is very unlikely reject SM discovery !€

Ppoisson (N |NSM )dN < 5.7Nobs

∞

∫ 10−7

Test hypotheses: rules for discovery

In the hypothesis that there is NO Higgs (SM hypothesis):

What is the probability to observe as many events as I have observed …OR EVEN MORE

If P < 5.7 10-7 reject SM

P(N≥33|12.2) = 6.35 10-7

P(N≥34|12.2) = 2.24 10-7

P(N≥33|12.2) = 6.35 10-7

P(N≥34|12.2) = 2.24 10-7

Integrate this plot

SM + HiggsSM

Question 1: did you make a discovery ?

See previous slide:

Yes

Discovery No discovery

No€


∞

∫ 10−7€

Ppoisson (11 (or17) |12.2)dN > 5.711 (or 17)

∞

∫ 10−7

Question 2: did you expect to make a discovery:

If the Higgs is NOT there:On average 12.2 events

If the Higgs is there:On average 17.2 events

If you observe exactly the number of events you expect (assuming the Higgs is there), it is not unlikely enough to be explained by the SM

NO discovery expected

If you observe exactly the number of events you expect (assuming the Higgs is there), it is not unlikely enough to be explained by the SM

NO discovery expected

SM SM + Higgs

€

Ppoisson (N |12.2)dN = 0.0717

∞

∫

Question 3: At what luminosity do you expect to make a discovery ?

Lumi x 1Lumi x 1

NSM = 12.2

NHiggs = 5.1

NSM = 122.0

NHiggs = 51.0

Lumi x 10Lumi x 10

Lumi x 12.5Lumi x 12.5

NSM = 152.5

NHiggs = 63.75

no

no

yes

SM + Higgs

SM + HiggsSM

SM

€

Ppoisson (N |12.2)dN = 0.0717

∞

∫

€

Ppoisson (N |122)dN = 5.5 10−6

173

∞

∫

€

Ppoisson (N |152.5)dN = 5.2 10−7

216

∞

∫

Discovery or not

It is not likely you get exactly the number of events you expect.

You can be lucky … or unlucky.

From simple counting to the real thing in 3 steps

1) Introduce X (Likelihood ratio) test statistic

2) From simple counting to weighted counting (a real analysis)

3) Toy Monte-Carlo (fake experiments)

Hypothesis testing: likelihood ratio

frequently used: X=-2ln(Q)

Hypothesis 1: the Standard Model without the Higgs boson

Hypothesis 2: the Standard Model with the Higgs boson

Definieer een statistic (= variabele) die onderscheid maakt tussen de 2 hypotheses.Note: kan vanalles zijn: # events of Neural net output.

Ex: counting experiment

€

Q =Ls+b

Lb

€

Q =Ppoisson (n | λ s+b)

Ppoisson (n | λ b)

Likelihood ratio

Likelihood ratio: counting

Counting experiment

N events left after some a selection of cut on discriminant

Note: X = 0 means hypoteses equally likely

Used in plots:

More SM+Higgs like More SM like

100.000 SM experiments

100.000 SM + Higgs experiments

€

Q

€

Q =P(N | s +b)

P(N | b)

€

=e−(s+b)(s +b)n /n!

e−bbn /n!

€

=e−s (s +b)n

e−bbn

€

X = −2ln(Q)

14 events observed

Variabele transformatie

Likelihood ratio: counting

Counting experiment


Note: X = 0 means hypoteses equally likely

Used in plots:

More SM+Higgs like More SM like

100.000 SM experiments

100.000 SM + Higgs experiments

€

Q =P(N | s +b)

P(N | b)

€

=e−(s+b)(s +b)n /n!

e−bbn /n!

€

=e−s (s +b)n

e−bbn

€

X = −2ln(Q)

€

P(14 |12.2) = 0.093

€

P(14 |17.3) = 0.076

€

X = 0.420

14 events observed

€

P(15 |12.2) = 0.076

€

P(15 |17.3) = 0.087

€

X = −0.278

15 events observed

Likelihood ratio

Counting experiment Weighted counting experiment

Eveny event has a weight according to a NN output or discriminant called pi : Signal: S(pi) and Background B(pi)

B(pi)

S(pi)+B(pi)


tellen

€

Q =e−(s+b)(s +b)n /n!

e−bbn /n!

€

Q =e−(s+b)(s +b)n /n!

e−bbn /n!⋅

sS(pi) +bB(pi)

s+bi=1

n

∏B(pi)i=1

n

∏

Many possible experiments


tellen tellen

Experiment 1 Experiment 2

1) Experiment condensed in 1 variable Note: Each experiment (read ATLAS) yields only ONE value of Q see 2 slides ago for counting example 2) Do Toy-MC experiments to study distribution of Q Note: Two distributions: for SM and SM+Higgs hypothesis

Toy Monte Carlo experiment

SM toy experiment: Draw for each bin i a random number from Poisson with μ= λSM (i)

SM+Higgs toy experiment: Draw for each bin i a random number from Poisson with μ= λSM(i)+ λSM+Higgs(i)

λSM(i)+ λSM+Higgs(i)

λSM(i)

The Higgs does not exist: 100,000 toy-experiments (SM)The Higgs exists: 100,000 toy-experiments (SM+Higgs)

With 1 and 2 sigma bands for SM hypothesis

Note (again): each experiment will produce 1 (one) number in this plot

Different masses … different cross-sections

Small Higgs cross-section Large Higgs cross-section

Two hypotheses are more apart if: 1) cross-section of Higgs is larger 2) Higgs is more different from SM

LEP plots

LEP paper Fig 1

Cross-section drops as function of mass

dummy

dummy

dummy

Expectation for Q or -2ln (Q): toy experiments

Probability that background resultsin the numer observed or (even) more

If 1-CLb < 5.7 10-7 we can say we reject the SM hypothesis discovery !

The famous 5 sigma

€

1- CL b = Pb(X ≥ Xobs) = Pb(X)dXX obs

∞

∫

Probability that background results in the numer observed or less€

CL b = Pb(X ≥ Xobs) = Pb(X)dXX obs

∞

∫Clb = confidence level in the background

SMSM

SM+HiggsSM+Higgs

Discovery

€


∞

∫ 10−7

€

1 − CL b < 5.7 10−7

Do you expect to discover Higgs with at this mass ?

Average SM+Higgs experiment: 1-CLb = 2 10^-7So yes, you expect to make a discovery IF 10xSM

The one 2-sigma is not the other 2-sigma

2.X sigma discrepancy at mh ~ 97 GeV Far away form what you expect from Higgs1.X sigma away at mh = 114 GeV Exactly what you expect from Higgs

No 5 sigma discovery what Higgs hypotheses can we reject

No discovery

No 5 sigma deviation found … what now ?

Trying to say something on the hypothesis that the Higgs exists exclusion

Exclusion

€

CL s =CL s+b

CL b

< 0.05

- Look at what you expect from Standard Model +Higgs - Given the SM + Higgs expectation: if probability to observe as many events you have observed (or less) is smaller than 5% SM+Higgs hypothesis is not very likely reject SM+Higgs

- Look at what you expect from Standard Model +Higgs - Given the SM + Higgs expectation: if probability to observe as many events you have observed (or less) is smaller than 5% SM+Higgs hypothesis is not very likely reject SM+Higgs

Expectation for Q or -2ln (Q): toy experiments

If CLs < 0.05 we are allowed to rejectthe SM+Higgs at 95% confidence level

The famous 95% confidence level

Probability that signal hypothesis results in the numer observed or less

€

CL b = Pb(X ≥ Xobs) = Pb(X)dXX obs

∞

∫

Cls = confidence level in the signal

SMSM

SM+HiggsSM+Higgs

€

CL s+b = Ps+b(X ≥ Xobs) = Ps+b(X)dXX obs

∞

∫

€

CL s =CL s+b

CL b

Extra Normalisation:

This is why it is called modified frequentist

CLs mean SM-only expeciment is 0.13 > 0.05 so NO !

Question 2: did you expect to be able to exclude ?

Question 3: At what luminosity do you expect to make a discovery ?

Lumi = 1x normal lumi

CLs = 0.13 no exclusion for average SM-only experiment

Lumi = 2x normal lumi

CLs = 0.034 exclusion for average SM-only experiment

#SM = 100 #H = 10

#SM = 200 #H = 20

A scan:

Luminosity / nominal luminosity

CLs

CLs = 0.05

CLs = 0.13

CLs = 0.66

CLs = 0.046

2 sigma up

1 sigma down

Si: If you would have a 1 sigma downward fluctuation, i.e. you see less events than you expect there is less room for a SM+Higgs hypothesis. In this case you would have been able to exclude it.

You expect to be able to exclude at Lumi / Lumi nominal = 1.70

Question 4: At what Higgs xs do you expect to make a discovery ?

Higgs XS = 1x normal Higgs XS

CLs = 0.13 no exclusion for average SM-only experiment

Higgs XS = 2x normal Higgs XS

CLs = 0.006 exclusion for average SM-only experiment

#SM = 100 #H = 10

#SM = 100 #H = 20

A scan:

Higgs XS / nominal Higgs XS

CLs

CLs = 0.05

CLs = 0.13

CLs = 0.66

CLs = 0.046

2 sigma up

1 sigma down

You expect to be able to exclude at Higgs XS / Higgs XS nominal = 1.40

A projection along the CLs = 0.05 line

Hig

gs

XS

/ n

om

inal H

igg

s X

S

Nominal luminosity

SM only (mean)

At what Higgs XS scale factordo you expect to be able to exclude the Higgs hypothesis ?

SM only (1 sigma up)

SM only (2 sigma up)

SM only (2 sigma down)

SM only (1 sigma down)

1.4

Hig

gs

XS

/ n

om

inal H

igg

s X

S

1.4

You can now scan over Higgs masses

The important thing is of course what you actually measured

Finito!

why do wouter (and atlas) put asymmetric errors on data points ? what is involved in the cls...

Documents

n events

events observedexperiment

given number of events

large number of events

higgs sm hypothesis

likely average mean

sm expectation

unlikely reject sm discovery