tsuyoshi nagafuji
TRANSCRIPT
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 1/35
Validating AMA frameworks
- A Regulator’s Experience in Japan
2nd International Conference on Operational RiskSao Paulo, Brazil, June 5, 2009
Tsuyoshi NagafujiFinancial Services Agency, Japan
This presentation does not necessarily expressestablished views or policies of the FSA.
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 2/35
2
Overview : Objective
Sharing my experience in validating and approving Japanese
banks’ AMA applications.
¾Presenting what we have done or what we are actually doing inJapan, rather than what we hope to do.
¾Focusing on the factors that remain until the final stage for
application, which banks find difficult and time consuming to
address.
9Model – sensitivity analysis / stress testing: Do you know all
the possibilities for strange behavior?
9Scenarios – rules and documentation: Have you done your best to exclude subjectivity?
9Use test: Are you actually using the framework? Is it really
working?
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 3/35
3
Overview : Method
I am using a realistic “model of AMA models" as an example, inconsideration of anonymity.¾ I am using “The model of AMA models” that I presented at the
“Operational Risk Scenario Analysis Workshop” held at Bank of Japan, the central bank, in 2006*.* The model presented here is the same as the one I presented in 2006, but the descriptionis simplified. Please see “Quantification of Operational Risk Using Scenario Data (Nagafuji,2006)” for the details.
¾ The model is extremely simplified but still retains someaspects of typical AMA models used by Japanese banks.9 The model is based on real internal data and real scenario
data from major Japanese banks.
9 The model has a similar structure to typical Japanesemodels.
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 4/35
4
Overview : Outline
Presentation Overview (5 minutes)
1. Context (5 minutes)
2. Sample Model (10 minutes)
3. Validation of the Sample Model (15 minutes)
Concluding Remarks (5 minutes)
Q & A (5 minutes)
(Total: 45 minutes)
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 5/35
5
Overview1. Context
2. Sample Model3. Validation
Concluding Remarks
Appendix
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 6/35
6
0
20
40
60
80
100
120
H S B C C i
t i R B
S J P M B O
A
C r e d i t A
g I C B C S C
H B O
C
P a r i b
a s
B a r c l
a y s
C C B C H B
O S
U n i C
r e d i t I N
G
W a c h o
v i a R a
b o
B r a d
e s c o I t a
u
U n i b a
n c o
B B
Context: Japanese Banking Industry
Consists of three "Mega” banks and many smaller banks.
Foreign banks play a very small role.
World banks by Tier I capital ($billion) (Source: The Banker, July 2008)
MUFG$82 bil
Mizuho$49 bil6
1518
SMFG$44
… … … … … … … …
2838
48 50 66 81 89 94
Top 20 Banks Top 21 – 100
<Japanese banks>98 banks in the Top 1,000 list. <Brazilian banks>14 banks in the Top 1,000 list.
(Now merged)
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 7/35
7
Context: Op Risk in Japanese Banks Operational risk losses are extremely small.
Total annual lossamount
(Average Dollar Amount by year,percentages of total assets)
Loss frequencies(# of losses greater or equal to
$20,000, per year, per total assets of $1 billion)
* Both figures are medians of the banks that participated in the exercise
(Source) 2004 U.S. LDCE, 2007 Japan LDCE (See “Appendix: References aboutJapanese AMA implementation" for detail).
0 0.5 1 1.5 2
Japanese
banks
US banks
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07
Japanese
banks
US banks
About 1/40 About 1/20
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 8/35
8
Context: Application Timetable
■ Typical validation process
■ Currently one banking group has been approved for the AMA.
1. PreparationBanks are encouraged to develop their framework
to a practical level and use it for their internalpurposes before going into the parallel run.
2. Parallel Run (At least one year)
Two capital calculations are verified through visitsand regular discussions.
3. ApprovalBanks that do not meet the requirement stay at
stage 2 or go back to stage 1.3. Approval
OK?
1. Preparation
Ready?
2. Parallel Run
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 9/35
9
Overview
1. Context
2. Sample Model
3. Validation
Concluding Remarks
Appendix
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 10/35
10
Sample Model: Overview
Several major banks are treated as if they were a single big bank. 9 Their internal loss data and scenario data are put into a very
simple loss distribution model (LDA).
LDA Model Results
Scenario Data
Internal LossData
Bank A
Bank B
Bank C
Bank X
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 11/35
11
Sample Model: Quantification Model
■ Monte Carlo simulation (100,000 simulations)
¾ Frequency: Poisson distribution
λ = Frequency of Scenarios
+ Frequency of Real Loss Data
¾ Severity: Empirical distribution
■ Single unit of measure (= top of the house calculations)
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 14/35
14
Sample Model: Scenarios (2/4)
■ Common Scenarios 1: EarthquakesLosses by historical earthquakes are estimated for each bank and then aggregated.
Description (year ,magnitude of earthquake)
Frequency(once in Xyears)
Severity(largest=100)
Details
Earthquake in Tokyo 1,200 100 Earthquake greater than any of those below is assumed.Keian (1649, 7.1) 49Genroku (1703, 8.2) 85
Ansei Edo (1855, 6.9) 55Meiji Tokyo (1894, 7.0) 47Great Kanto (1923, 7.9) 82
Hoei (1707, 8.4) 57 Ansei (1854, 8.4) 50Nobi (1881, 8.0)
400 each
55
(Frequency) 8 large-scale earthquakes between 1600 and1925 in Tokyo, Nagoya and Osaka are listed, assuming eachwill occur once every 400 years.(Severity)<Buildings> The damage to the building, furniture and the
opportunity cost due to interruption of business arecalculated based on the earthquake intensity and quakeresistance of the buildings.
<Systems> Extra work cost, damage to the machines andequipment and the opportunity cost due to businessinterruption are calculated. Damage to the computer center and paralysis of the head office functions are assumed.
<Other> Declines in the value of the loans (includingimpairment of the value of collateral) are not factored in.
Tokyo (1926) - Aichi(1997) (61 earthquakes) 77 each
Average0.4
(Frequency) 61 earthquakes occurred between 1926 and 97(of intensity 5- or higher) are listed, assuming each will occur once every 77 years.(Severity) as shown above.
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 15/35
15
Sample Model: Scenarios (3/4)
■ Common Scenario 2: Failure in the settlement system A scenario was created where “a failure occurs in the computer systems commonly
used by the banks once every twenty years, causing total damage of JPY 20 billion.”
The f ollowing scenarios from a bank were referred to in creating this scenario.Frequency Severity Details
Once inseveraldecades
JPY several billions($US tens of millions)
A failure in the accounting system or in the domesticnetwork, which would take 12 hours for full recovery.
Once in
severaldecades
JPY several
hundred millions($US several millions)
1) A failure occurs in the communication infrastructure, or, 2)
there is a flaw in the emergency handling procedures,causing interruption of the settlement operation for half aday. The compensation for damage paid to securitiesexchanges as clearing agents in charge of settlement of thegovernment bonds is included.
Once in
severaldecades
JPY several billions
($US tens of millions)
Foreign exchange / settlement operations are not performed
for a full day due to a system failure
Once everyseveral years
JPY severalhundred millions
($US several millions)
A failure occurs in the Zengin System just after 9:00 am. Thesystem recovers at around noon. However, the settlementoperation is erratic during that day.
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 16/35
16
Sample Model: Scenarios (4/4)
■ Independent Scenarios<a> Actual scenarios collected from banks are used as they were.
<Scenarios Used>
Major scenarios (scenarios for larger amounts of losses)BIS event
types # of scenarios ExamplesInternal Fraud 30 Fraud in the market trading functions, withdrawal of customer fundsExternal Fraud 3 Swindles, compromised online bankingEmployment 5 DiscriminationClients,
Products
30 Lender’s liability, inappropriate advice to customers, failure to
explain the risks, etcPhysicalassets
11 Terrorist attacks
Systems 12 Failure in the accounts transfer system, including interruption of theaccounting system
Process 38 Failure in bond settlement (overseas), improper identity verification,
error in cash transfer, etcTotal 129
<b> Made-up scenarios for banks that did not have scenarios are also used.
→ Some scenarios from <a> scaled by the total assets of each bank
.
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 19/35
19
Overview1. Context
2. Sample Model3. Validation
Concluding Remarks Appendix
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 20/35
20
Validation of the Sample Model: Overview
Three major points that banks have trouble with in completingtheir AMA application.
(1) Model: Sensitivity analysis / stress testing(2) Scenarios: Rules and documentations
(3) Use test: Are you actually using the framework?
Model Risk Amount
Scenarios
Internal Losses
BEICFs
External LossData
(1)
(2)
(3) Use test
Sensitivity analysis/
stress testing: Do youknow all the possibilitiesfor strange behavior?
Rules and documentations:
Have you done your best toexclude subjectivity?
Are you actually using
the framework? Is itreally working?
Scenarios
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 21/35
21
Validation: Model (1/3)
Statistical integrity of the model is essential, but has not been adetermining factor at the last stage.
<The sample model>
9 LDA model
Independence between frequency and severity should be accounted for.
9 Choice of distributions and estimation methods
Those may not be great discussion points as the sample model uses empirical
distribution.9 Granularity
Independence among the data points should be accounted for.
Æ As long as assumptions in the model are clarified and accountedfor, this factor is not a decisive one.
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 22/35
22
Validation: Model (2/3)
Instead, sensitivity analysis / stress testing has been a greatchallenge at the last stage.9No model is free from “strange” (counter-intuitive) behavior.9Comprehending all the possibilities of “strange” behavior is
time consuming, especially when the model is complex.
<The sample model>
Massive losses have a large impact.
A small change in frequency for a massive loss may have a large impact.<Simplified example> (Frequency: Poisson, Severity: Empirical)
Data set 1) and data set 2) give completely different risk, although the onlydifference is the frequency of a single big loss!
Losses EL 99.9%
1) One JPY100 billion Loss (Once in 999 years)
+ 100 JPY 10,000 Losses (Each once in 10 years)
0.1
billion
100
billion
2) One JPY100 billion Loss (Once in 1000 years)
+ 100 JPY 10,000 Losses (Each once in 10 years)
0.1
billion
0.0003
billion
3 / 1,000,000
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 23/35
23
Validation: Model (3/3)
Comprehending unusual (counter-intuitive) behavior of a modelis essential for regulatory and internal purposes.
We request banks:
9 To comprehend possible “strange” behaviors of their model.
9To address those “strange” behaviors. Accept them (Management should fully understand the
consequences). Revise and reconstruct their models.
Æ May take a lot of time.
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 24/35
24
Validation: Scenarios (1/3)
Estimation of frequency and severity is essential, but not adetermining factor at the last stage.Æ Scenarios for the sample model must include “once in 1000
years” events, as the model uses empirical distribution for severity.
ÆWe verify this through9 Checking the logic.9 Checking facts that scenarios are based on.9 Benchmarking scenarios between banks.
<Examples> Earthquakes:
Is the use of past earthquakes appropriate?What is included as losses from earthquakes?Is the latest seismological knowledge utilized?
System failures:
Are the statistics on computer failures utilized?What is the accuracy of the statistics?
Are the statistics used appropriately?
Æ As it is impossible and inappropriate to press one specificview, this factor has not been a decisive one at the last stage.
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 26/35
26
Validation: Scenarios (3/3)
We request banks to:
9Do their best to set rules that ensure the same estimateregardless of who the estimator is.
9Fully document and account for subjective judgments thatremain.
This looks easy to accomplish, but turns out to be verychallenging, because:
9Rules can be set only after experience is accumulated.
9This is often neglected until the last stage.
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 27/35
27
Validation: Use test (1/3)
“Use test” is a strong tool to improve an AMA framework. Thus, itis very challenging and often becomes a determining factor at thelast stage.
Our “use test” in Japan is not special.¾ Banks should show that they use their AMA framework in their
day-to-day risk management (= The framework is notexclusively for regulatory purposes).
¾ Use test is based on the idea that supervisors can be moreconfident with an AMA framework that is “really used”.
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 28/35
28
Validation: Use test (2/3)
Banks decide how to demonstrate their compliance with the usetest (= Banks decide how to use their AMA framework).
Many Japanese banks choose to base their “risk management
cycle” on their AMA model.
Evaluate riskreductionmeasures based
on the model.
Implementrisk reductionmeasures
based on themodel.
Verify the
results usingthe model.
Risk reduction measures・ Introducing double
checking・Computerize operations ・Restricting operations ・…
3. See
2. Do.Plan
Model/
Risk
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 29/35
29
Validation: Use test (3/3)
“Use test” imposes improvement on their overall AMA framework.
¾It imposes improvement on the model. 9 The model needs to be practically free from “counter-intuitive” behavior 9
The model needs to be sensitive enough.
¾Understanding by management and business units is essential.
Thus, it often becomes a determining factor at the final stage.
¾When the AMA framework does not meet the use testrequirement, it needs modification.
¾When the modification is drastic, banks are required to take the“use test” again, which needs at least half a year to complete.
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 30/35
30
Overview1. Context
2. Sample Model3. Validation
Concluding Remarks Appendix
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 31/35
31
Concluding Remarks: In Theory…
■ In theory, capturing 99.9% risk is the single most importantrequirement of AMA models.9 Do distribution assumptions capture 99.9% risk?9 Are scenarios representing once-in-1000-year events?
■ However, this requirement does not turn out to be the remainingfactor at the last stage.9Choice of distribution or estimation of scenarios boils down to
subjective judgments, which are argumentative, but notdecisive.9Banks that cannot address these issues cannot enter the
parallel run.
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 32/35
32
Concluding Remarks: In Practice…
■ The following practical factors have often played a decisive roleat the last stage.
1. Comprehending unusual (counter-intuitive) behavior of themodel and preparing for it.
2. Setting rules and perfecting documentation to minimize thesubjectivity of scenarios.
3. Meeting the “Use test” requirement
■ Those factors ensure workable framework both for internal and
regulatory purposes.
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 33/35
33
Concluding Remarks: Challenges for Banks
■ We do not press banks to have an ideal or very sophisticatedframework. Rather, we ask them to have a practical, reliableframework to meet requirements for regulatory purposes.
■ After all, it is up to banks to build an AMA framework that trulyenhances their risk management.
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 34/35
34
ご清聴ありがとうございました
Questions?
For further questions, feel free to contact:
7/28/2019 Tsuyoshi Nagafuji
http://slidepdf.com/reader/full/tsuyoshi-nagafuji 35/35
35
Appendix: References about Japanese AMA implementation
Sample model used in this presentation9 Quantification of Operational Risk Using Scenario Data (July, 2006)
http://www.boj.or.jp/en/type/release/zuiji_new/data/fsc0608be3.pdf
Losses of Japanese banks (Page 7 of this presentation)9 Results of the 2007 Operational Risk Data Collection Exercise (August, 2007)
http://www.fsa.go.jp/en/news/2007/20070810-2.pdf 9 A research paper comparing the loss data between the U.S. and Japan (April, 2008)
http://www.boj.or.jp/en/type/release/adhoc/data/risk0804a.pdf (For the results of U.S. LDCE, see http://www.bos.frb.org/bankinfo/qau/papers/pd051205.pdf )
Others
9 Use of External Data for Operational Risk Management Workshop (April, 2008)http://www.boj.or.jp/en/type/release/adhoc/fsc0804a.htm
9 The Effect of the Choice of the Loss Severity Distribution and the Parameter Estimation Methodon Operational Risk Measurement (December, 2007)
http://www.boj.or.jp/en/type/ronbun/ron/research07/ron0712c.htm
9 Discussions on Further Advancing Operational Risk Management (Part1: June 2006, Part2:
August 2006)Part1: http://www.boj.or.jp/en/type/release/zuiji_new/fsc0608c.pdf Part2: http://www.boj.or.jp/en/type/release/zuiji_new/fsc0612a.pdf
9 Operational Risk Scenario Analysis Workshop (July 2006)http://www.boj.or.jp/en/type/release/zuiji_new/fsc0608a_add.htm