c. acerbi and b. szekely - parmenides foundation · testing expected shortfall c. acerbi and b....

Testing Expected Shortfall

C. Acerbi and B. Szekely

MSCI Inc.

Workshop on systemic risk and regulatory market risk measures

Pullach, Germany, June 2014

Carlo Acerbi and Balazs Szekely Testing Expected Shortfall June 2014 1 / 59

Outline

1 Motivation and goals

2 Testing settingBasel VaR backtestThree tests for ES. Plus one

3 Results

4 ConclusionsPost Scriptum

3 Results

Motivation

in the VaR/ES debate, backtesting has always been the main problemwith ES. See for instance Yamai and Yoshiba (01)last obstacle for the adoption of ES in Basel N, finally occurred in 2013

but model testing still based on VaR

rich literature on VaR backtesting: Basel I (96), Kupiec (95),Christoffersen (98), Berkowitz (00), Engle and Manganelli (04), amongothersfew works on ES backtesting: noticeably Kerkhof and Melenberg (04)Angelidis and Degiannakis (06)

Why is it difficult to test ES?Fundamental reasons? Practical aspects? Power of the test? Model risk?

Motivation

in the VaR/ES debate, backtesting has always been the main problemwith ES. See for instance Yamai and Yoshiba (01)last obstacle for the adoption of ES in Basel N, finally occurred in 2013

but model testing still based on VaR

rich literature on VaR backtesting: Basel I (96), Kupiec (95),Christoffersen (98), Berkowitz (00), Engle and Manganelli (04), amongothersfew works on ES backtesting: noticeably Kerkhof and Melenberg (04)Angelidis and Degiannakis (06)

Why is it difficult to test ES?Fundamental reasons? Practical aspects? Power of the test? Model risk?

Confusion

The nice thing about VaR is it’s more or less transparentlyback-testable. You know what you’re getting. With ES it’s all cloudedup with assumptions about distribution and arbitrary choices. Whenhave you breached it? What exactly are you testing? When you gointo the tail you are never quite sure...

RISK Magazine, last week

The drama of non–elicitability of ES

Gneiting (11): VaR is elicitable, ES is notThis negative result may challenge the use of the ES functional as apredictive measure of risk, and may provide a partial explanation for thelack of literature on the evaluation of ES forecasts, as opposed toquantile or VaR forecasts

elicitability is a subtle concept: x = arg minx E[S(x ,Y )]

What most people understoodES is not backtestable, at all

a magnum champagne bottle gift for the VaR nostalgicpanic followed

ES cannot be back-tested because it fails to satisfy elicitability ... If youheld a gun to my head and said: ‘We have to decide by the end of theday if Basel 3.5 should move to ES, or do we stick with VaR’, I wouldsay: ‘Stick with VaR’

Paul Embrechts, Imperial College, 2013

ES cannot be back-tested because it fails to satisfy elicitability ... If youheld a gun to my head and said: ‘We have to decide by the end of theday if Basel 3.5 should move to ES, or do we stick with VaR’, I wouldsay: ‘Stick with VaR’certainly not a VaR fanatic! → Paul Embrechts, Imperial College, 2013

Examples of elicitable statistics

the mean is elicitable

x = arg minm

EX [S(m,X )] S(m, x) = (X −m)2

a α–quantile is elicitable

qα = arg minq

EX [S(q,X )] S(q, x) = (x − q)(α− (x − q < 0))

when α = 1/2 we retrieve the median

M = arg minµ

EX [S(µ,X )] S(µ, x) = |x − µ|

there is no scoring function S that elicits ES

ES = arg minc

EX [S(c,X )] S(c, x) does not exist

Something is not quite right

if elicitable means backtestable isn’t it a bit strange thatbanks have always backtested VaR but never by exploiting its elicitability?even standard deviation is not elicitable?Kerkhof and Melenberg, back in (04), had found that

...contrary to common belief, ES is not harder to backtest than VaR ifwe adjust the level of ES. Furthermore, the power of the test for ES isconsiderably higher than that of VaR.

as a matter of fact, others reacted quite differentlyES is not elicitable. So, what? Dirk Tasche

3 Results

Setting

we look at ES backtesting from a regulatory point of viewprofit–loss: independent (but not i.i.d.) Xt ∼ Ft , the real distributions,t = 1, . . . ,T (= 250)Pt predicted (model) distributionsVaR and ES (with Basel confidence levels)

VaRβ = −P−1(β) α = 1%

ESα =−1α

∫ α

0P−1(q)dq α = 2.5%

we assume Pt continuous and strictly monotonic (just for simplicity,inessential here). Then

ESα = −E[X |X + VaRα < 0]

the assumption can be easily removed at the cost of heavier notation

ES estimators

standard estimator of ESα for N i.i.d. draws Xi ∼ P

ESα,N

(X ) = − 1Nα

[Nα]∑i

Xi:N + (Nα− [Nα])X[Nα+1:N]

coherent ∀N, α, consistent, asymptotically normal, known variancegeneralizes the idea of ‘average of the Nα worst cases’ to Nα /∈ Nbut biased. It always underestimates risk for finite N. No unbiasedestimator known for unknown Pconditional estimator; assuming VaRα is known exactly

ESα,N

(X ) = −∑N

i=1 Xi 1Xi+VaRα<0∑Ni=1 1Xi+VaRα<0

is unbiased

ES estimators

standard estimator of ESα for N i.i.d. draws Xi ∼ P

ESα,N

(X ) = − 1Nα

[Nα]∑i

Xi:N + (Nα− [Nα])X[Nα+1:N]

coherent ∀N, α, consistent, asymptotically normal, known variancegeneralizes the idea of ‘average of the Nα worst cases’ to Nα /∈ Nbut biased. It always underestimates risk for finite N. No unbiasedestimator known for unknown Pconditional estimator; assuming VaRα is known exactly

ESα,N

(X ) = −∑N

i=1 Xi 1Xi+VaRα<0∑Ni=1 1Xi+VaRα<0

is unbiased

Hypothesis testing

Goaltesting VaRt and ESt predictions against observed profit–loss realizations xt

H0 : Pt = FtH1 : Ft is riskier than Pt

ESFt > ESP

we test only in the direction of risk underestimationmore specific H1’s in the following, for computing test power

Model–free testWe avoid any assumption on the nature of the predicted distributions Pt (nolocation-scale family, no parametric models, ...)We do not assume asymptotic convergence of any statistics either

Hypothesis testing

Goaltesting VaRt and ESt predictions against observed profit–loss realizations xt

H0 : Pt = FtH1 : Ft is riskier than Pt

ESFt > ESP

we test only in the direction of risk underestimationmore specific H1’s in the following, for computing test power

Model–free testWe avoid any assumption on the nature of the predicted distributions Pt (nolocation-scale family, no parametric models, ...)We do not assume asymptotic convergence of any statistics either

3 Results

Basel test for VaR exceptions (96)

H0: bt = 1xt+VaRt<0 ∼ i .i .d .Bernoulli(β), ∀ttest statistic: B =

∑T bt ∼ B(T , β): yearly number of exceptionsone expects E[B] = Tβ

H1: VaRβPt

= VaRβ′

Ftwith β′ > β ⇔ B ∼ B(T , β′)

one says that coverage is not 1− β = 99% but only 1− β′ (say 98%)traffic–light system: yellow zone from 95% significance level and red zonefrom 99.99%

Basel VaR test: power vs coverage

Figure: ‘Fundamental review of the trading book: a revised market risk framework’,Basel Committee 2013

Basel VaR test: traffic light system

Criticism

Basel test addresses only unconditional coverageindependence of time arrival should be tested separatelyChristoffersen (98): likelihood ratio test for conditional coverage

LRcc = LRuc + LRind

in most practical cases, however, independence testing is left to visualinspection, which helps interpreting exception clusters. Basel did notintroduce any independence formal testin the following we assume that independence is tested separately. Wefocus on unconditional ES–coverage

Criticism

LRcc = LRuc + LRind

Criticism

LRcc = LRuc + LRind

Visual inspection

3 Results

Test 1

test ES after having tested VaRfrom

Xt + ESt

∣∣∣∣Xt + VaRt < 0]= 0

denoting It = 1Xt+VaRt<0, introduce

Test statistic 1

Z1(~X ) =

∑Tt=1

Xt ItESt∑T

i=1 It+ 1

EH0[Z1] = 0. ES underestimated if Z1 < 0the test averages over exceptions; insensitive to an excess of exceptionsZ1 defined as a pure number to sterilize changes in portfolio–size

Test 1

test ES after having tested VaRfrom

Xt + ESt

∣∣∣∣Xt + VaRt < 0]= 0

denoting It = 1Xt+VaRt<0, introduce

Test statistic 1

Z1(~X ) =

∑Tt=1

Xt ItESt∑T

i=1 It+ 1

EH0[Z1] = 0. ES underestimated if Z1 < 0the test averages over exceptions; insensitive to an excess of exceptionsZ1 defined as a pure number to sterilize changes in portfolio–size

Computing a p–value

under H0, the distribution PZ1 of Z1(~X ) is simulated by drawingindependent Xt ∼ Pt , ∀tthe realization Z1(~x) provides a p–value p = FZ1(Z1(~x))acceptance/rejection based on a chosen significance level, say 5%type–2 probabilities and test power are computed based on specificalternatives H1

Main difficultyStorage of the α–tail of each distribution Pt , to simulate Z1 under H0.Technologically elementary, but a challenge for auditing

the observations in this slide apply to all the tests proposed in thefollowing

Computing a p–value

under H0, the distribution PZ1 of Z1(~X ) is simulated by drawingindependent Xt ∼ Pt , ∀tthe realization Z1(~x) provides a p–value p = FZ1(Z1(~x))acceptance/rejection based on a chosen significance level, say 5%type–2 probabilities and test power are computed based on specificalternatives H1

Main difficultyStorage of the α–tail of each distribution Pt , to simulate Z1 under H0.Technologically elementary, but a challenge for auditing

the observations in this slide apply to all the tests proposed in thefollowing

Test 2

direct test for ESfrom the unconditional expectation

E[−Xt Itα

]= ESα,t

introduce

Test statistic 2

Z2(~X ) =T∑

Xt ItTαESt

EH0[Z2] = 0. ES underestimated if Z2 < 0the test averages over all days; it detects an excess of exceptions

Z2 = (Z1 − 1)∑

t ItTα

Test 2

direct test for ESfrom the unconditional expectation

E[−Xt Itα

]= ESα,t

introduce

Test statistic 2

Z2(~X ) =T∑

Xt ItTαESt

EH0[Z2] = 0. ES underestimated if Z2 < 0the test averages over all days; it detects an excess of exceptions

Z2 = (Z1 − 1)∑

t ItTα

Test 3

direct test for ESconsider the r.v.’s Ut = Pt(Xt). Under H0, Ut ∼ i.i.d U(0,1)Berkowitz (01) proposes to test for uniformity the tail of the empiricaldistribution of the xt

We use this pseudo–uniform sample ~U to estimate ES

Test statistic 3

Z3(~X ) = − 1T

T∑t=1

EST ,α

(P−1t (~U))

T ,α(P−1

t (~V ))

where ~V ∼ i.i.d U(0,1)EH0[Z3] = 0. ES underestimated if Z3 < 0notice that the denominator is not ESt but a finite sample estimate, tocompensate for bias. Analytical expressions available for any Pt

Test 3

direct test for ESconsider the r.v.’s Ut = Pt(Xt). Under H0, Ut ∼ i.i.d U(0,1)Berkowitz (01) proposes to test for uniformity the tail of the empiricaldistribution of the xt

We use this pseudo–uniform sample ~U to estimate ES

Test statistic 3

Z3(~X ) = − 1T

T∑t=1

EST ,α

(P−1t (~U))

T ,α(P−1

t (~V ))

where ~V ∼ i.i.d U(0,1)EH0[Z3] = 0. ES underestimated if Z3 < 0notice that the denominator is not ESt but a finite sample estimate, tocompensate for bias. Analytical expressions available for any Pt

Test 4

similar to Berkowitz (01), we can directly test the tail density, via the ES ofthe uniform distribution

Test statistic 4

Z4(~X ) =ES

T ,α(~U)

T ,α(~V )

] − 1

where ~V ∼ i.i.d U(0,1)EH0[Z4] = 0. Risk underestimated if Z4 < 0not a test of ES of the model, but a generic test of the tail density

Test 4

similar to Berkowitz (01), we can directly test the tail density, via the ES ofthe uniform distribution

Test statistic 4

Z4(~X ) =ES

T ,α(~U)

T ,α(~V )

] − 1

where ~V ∼ i.i.d U(0,1)EH0[Z4] = 0. Risk underestimated if Z4 < 0not a test of ES of the model, but a generic test of the tail density

Observations

tests 2 and 3 can naturally be extended to all spectral measurestest 1 can be extended to simple spectral measures, with piecewiseconstant spectrum

3 Results

H0: Student-t; H1: scaled distributions

H0: Ft = Pt , Student-t distributionH1: Ft(x) = Pt(x/γ), scaled distribution (γ > 1)

H0: Student-t, ν = 100; H1: scaled distributions

H0: Student-t; H1: ES–coverage 95%, 90%

H0: Ft = Pt , Student-t distributionH1: Ft(x) = Pt(x/γ), again scaled distribution, but labeled in terms of EScoverageESP

α = ESFα′ , with α′ = 5%,10%

analogous to the Basel VaR coverage tables

H0: Student-t, ν = 100; H1: ES–coverage 95%, 90%

;Carlo Acerbi and Balazs Szekely Testing Expected Shortfall June 2014 35 / 59

H0: Student-t, ν = 100; H1: ν = 10,5,3;

H0: Ft = Pt , Student-t distributionH1: Student-t distribution with lower νnotice that the standard deviation is larger σ =

√ν/(ν − 2)

H0: Student-t, ν = 100; H1: ν = 10,5,3;

H0: Student-t, ν = 10; H1: ν = 5,3;

H0: Normalized Student-t, ν = 100; H1: ν = 10,5,3;

H0: Ft = Pt , Student-t distribution with σ = 1H1: Normalized Student-t distribution with lower ν and σ = 1

H0: Normalized Student-t, ν = 10; H1: ν = 5,3;

H0: Normalized Student-t; H1: fixed VaR 97.5%

H0: Ft = Pt , Student-t distribution with σ = 1H1: Normalized Student-t distribution with lower ν and σ = 1the distribution are offset in such a way to have all the same VaR 97.5%alternative hypotheses built to analyze test 1

H0: Normalized Student-t; H1: fixed VaR 97.5%

H0: Student-t, ν = 100; H1: fixed VaR 97.5%

H0: Student-t, ν = 10; H1: fixed VaR 97.5%

H0: Norm. Student-t, ν = 100; H1: fixed VaR 97.5%

H0: Norm. Student-t, ν = 10; H1: fixed VaR 97.5%

Summary of results

all tests for ES 97.5% generally display more power than the Basel testfor VaR 99% in identical conditionstest 1 is subordinated to testing VaR, but has strong power for modelmisspecifications in the tailtest 2 and test 3 excel in different cases. Test 2 is more powerful onscaled distributions. Test 3 is more powerful on distributions with differenttail index

Test 2: a very practical test

Test 2 has critical levels that are almost invariant with respect to the tailproperties, in a range ν = [5,+∞) that spans all realistic cases of afirmwide bank portfolioit allows to define a traffic light system that does not require the collectionof the entire α–tail of Pt , but just the three numbers xt , ESt and It

Critical levelsTest 1 Test 2 Test 3

significance 5% 10% 5% 10% 5% 10%ν = 3 -0.43 -0.27 -0.82 -0.59 -0.49 -0.32ν = 5 -0.26 -0.17 -0.74 -0.55 -0.30 -0.22ν = 10 -0.17 -0.12 -0.71 -0.53 -0.21 -0.16ν = 100 -0.12 -0.08 -0.70 -0.53 -0.15 -0.12Gaussian -0.11 -0.08 -0.70 -0.53 -0.15 -0.11

3 Results

Our results

ES is backtestable; this is certainly not a new result, but – surprisingly –it’s worth reaffirming itwe propose three tests for ES: the novelty of these tests is that they arenon–parametric and contain no model assumptions. For this reason theyrepresent valid proposals for regulatory purposesall of these tests display superior power to the standard Basel VaRbacktesting methodologythe main difficulty with backtesting ES is that you need to store the tail ofall predictive distributions Pt . If this is not a conceptual problem andcertainly no more a technological one either, this is still a challenge for anauditable process. This is the only difference between backtesting ESand VaRone of the proposed tests displays a remarkable stability of the criticallevels, which provides an opportunity to set up practical tests for whichthe storage of the predictive distributions is not needed

Our results

Elicitability

Elicitability of VaR has no relevance in the regulatory debateElicitability allows you to compare models which forecast the exact sameprocess, based on point forecasts only. But to score the performance of amodel against an absolute significance level, one still needs (or at leastwe don’t see how one would not) either model assumptions or recordingall predictive distributionsIt’s no coincidence that VaR in banks is backtested without exploiting itselicitability

3 Results

By the way, ES is elicitable

well, not exactly but consider the scoring function

S(v ,e, x) = αe2/2−ev(α−(x+v < 0))+(ex−2(v2−x2))(x+v < 0)+2αv2

then you have{VaR,ES} = arg min

v ,eEF [S(v ,e,Y )]

the only condition is that 4VaR > ES, which is always true in non–crazycasesthis means that you can set up a contest among models that forecastjointly VaR and ESwe could call it joint elicitability of VaR and ESLambert, Pennock, Shoham (08) call this property 2–elicitability andprove it for variance and mean

v ,eEF [S(v ,e,Y )]

General score function

most general scoring function, for all W

SW (v ,e, x) = αe2/2 + W αv2/2− αev+(e(v + x) + W (x2 − v2)/2

)(x + v < 0)

withES < W VaR

A scoring function of VaR and ES

Thanks!

c. acerbi and b. szekely - parmenides foundation · testing expected shortfall c. acerbi and b....

Documents