the advanced irb forum monday 23 june kevin ryan

THE ADVANCED IRB FORUM

Monday 23 June

Kevin Ryan

AREAS TO BE COVERED

• Issues in validating Internal Ratings systems

• How are supervisors reacting – emerging thoughts

• The FSA Consultative Paper

– Sections on Validation, External Models & Data, Data Quality, Assessment Horizon & PD estimation

NOT TO BE COVERED

Detailed ‘how to do it’ to get your IRB systems approved by the FSA

• We do not know enough at this stage– “We recognise a lot of work remains in this

complex area with many challenges for firms and us”

• Responsibility of firms to validate their rating systems

– Firms ‘validate’; supervisors ‘approve’ (or certify)

STYLISED SUPERVISORS’ VIEW OF RATING SYSTEMS

• Data and validation a challenge!

• Good work has been done, but not been part of the

mainstream activity of the firms, so– Even in the best firms, few people understand the issues

– Knowledge and oversight by senior management and control

functions quite low

– Resources for improvements can be difficult to get, so trade-offs

made

– Limited appetite for fresh thinking or challenging assumptions

• Appetite for short cut solutions, eg agency ratings

A MODEL IS PROVED BY ITS PERFORMANCE ?

SMALL SAMPLE SIZE

In the common binomial test, construct a

confidence interval around estimated PD

• Suppose a PD estimate is 100bp, and you want

to be 95% confident that actual PD is between

80 and 120bp


SMALL SAMPLE SIZE

In the common binomial test, construct a

confidence interval around estimated PD

• Suppose a PD estimate is 100bp, and you want

to be 95% confident that actual PD is between

80 and 120bp

You need 9500 borrowers in that grade


SMALL SAMPLE SIZE

Using the common binomial test, what levels of

true PD are consistent with zero observed

defaults, at a 95% confidence level?

• If you have 500 borrowers in a grade, any PD up to

1.25%

• If you have 80 in a grade, a PD as high as 6% or 7%(Binomial test doesn’t work if PD x Borrowers below 5)


DEFAULTS ARE CORRELATED

• The distribution you can expect from defaults is

many periods with actual defaults below the mean,

and fewer periods (typically clustered together) with

actual defaults well above the mean

IMPLICATIONS

• Confidence bounds wider than binomial

• More difficult to interpret


IMPLICATIONS

• “Regardless of the efforts used by banks, …there

will still remain some portfolios where there will

likely never be sufficient default data to calibrate

PDs with any degree of statistical significance”Internal Ratings Validation Study

• “We’re going to need to be pragmatic for some

years to come”

WHAT CAN BE DONE?

BROADER APPROACH TO VALIDATION, TO

SUPPLEMENT OUTCOMES ANALYSIS

• Logic and conceptual soundness of the approach

• Statistical testing prior to use

• Monitoring of process – are the methods being

applied as intended

• Benchmarking – compare to relevant alternativesDavid Wright, FRB, 19 June

WHAT CAN BE DONE?

MORE ON OUTCOMES ANALYSIS, INCLUDING

• More exploration and use of tolerance levels by firms,

which requires

• Better understanding of distribution of expected defaults

• Supervisory interest in information content of transition

matrices

• What can be learnt from other industries?– Insurance experience of modeling rare events?

FSA APPROACH – FSA CP

KEY POINTS

• Accountability

– Firms responsibility to validate & submit documentation with IRB

application

– Application to be signed by chief executive

• We expect he will have similar questions to supervisors

• Independence

– Approved by senior committee

– Independent staff to participate


KEY POINTS

• Scope

– To cover all portfolios, but depth will vary with significance

• Materiality considerations

– Methods may vary, especially by portfolio

• If more reliance can be placed on back-testing we should need less

additional evidence

• If less reliance on back-testing, more additional evidence is needed


KEY POINTS

• Scope

– Must assess the accuracy of the overall output of the system

• Not just the inputs

• Will need to take account of overrides and judgmental adjustments

to any underlying statistical models

– Based on statistical analysis of rating system, related

internal data, and third party and publicly available

information


• Coverage

– Full documentation

– Clarity on what rating system aiming to predict, and expected

distributions

– Take full account of adjustments between unbiased estimates

and those used in regulatory capital calculation

• Conservatism

• Cyclical effects

• Double default effects


• Coverage

– Clearly set out standards of control

– Clearly set out limitations of approach

• Assume that there will be candour

– Include work to demonstrate both ability to rank into

grades (discriminate) and estimate a PD etc (calibrate)

– Include steps to be taken in event quantitative tests for

power and accuracy are breached


• Discrimination/ranking

– Firm must justify that system shows a ‘high degree of

power in line with industry norm for portfolios of that nature’

– We do not specify what test should be used, eg Gini or

other, or its level

• But we consult on such a test

– Expect firms to strive for best model they can, other things

being equal


• Discrimination/ranking - issues

– Some see discrimination as key to model building

– Some research that high discrimination needed to achieve

accurate PD measures

– But levels dependent on features of portfolio, number of

defaults

– High is good, but may be over-fitting

– Firms must use targets, but reluctant to admit to them

– Importance of expertise to interpret


• Calibration

– Standardised ‘scorecard’ proposed

– Firm to justify its estimate against own historic experience and

external sources

– Take account of factors which may expect to lead to differences;

eg conservatism, cycle effects

– We do not specify tests for assessing accuracy

• But we consult on such a test

– If actuals not consistent with estimates, firms must justify

differences and/or take steps to improve


• Data quality

– “Data quality was not seen as the major obstacle to validation in the

long term”Internal Ratings Validation Study

– Some evidence that cleaning the data produces more powerful models,

while greater accuracy self evident

– Missing defaults?

– “We recognise that the data accuracy challenge for IRB is significant” FSA CP

– “We propose quantifiable targets to cover completeness and accuracy

that will rise over time”


• External models and data

– In principle supportive as can supplement internal data and

models; also external vendors may operate to higher

standards than firms

– But external models carry explicit risk of black box, with

limitations not known and application to inappropriate portfolios

– Limit to how much vendors will reveal

– We are trialing ‘vendor grid’ which would standardise

information to be provided – aimed at reducing risks


• Pooled data

– In principle supportive as, like external data, can

supplement internal experience, and without

commercial confidentiality issues of external vendors

– “Issues and challenges facing pooling initiatives

• Consistent data definitions for all

• Legal restrictions – data protection and confidentiality

• Potential increase in systemic risk”Internal Ratings Validation Study


• Expert judgment

– “There is a feeling that not enough time has been spent on

discussing acceptable validation techniques for these types

of systems”Internal Ratings Validation Study

– Will look to accommodate, but validation challenges increase

– Are there some cases where it is not feasible to produce

quantified estimates?

– Or use ranking with conservative estimates of losses

BENCHMARKING

• A complement to back-testing

• Conceptually two types, although differences between them

may be blurred in practice

– Relative benchmarks – compare estimates between firms to

identify outliers

– Absolute benchmarks - compare with an external benchmark

given some credibility

• We propose some element of benchmarking in our proposed

approaches to discrimination (industry standard) and

calibration (comparison with external sources)

BENCHMARKING

POSSIBLE INITIATIVES

• Private sector services to benchmark ratings and/or

estimates

• Some international interest in requiring firms to rate ‘test

portfolios’

• FSA considering targeted benchmarking at obligor level

where back-testing difficult and amounts large

– Very early stage of thinking but would welcome feedback

– Could be run by industry, FSA or other body

SOME VALIDATION CHALLENGES

• Scale of task given possible number of models

– Materiality to firm v materiality to the market

– Importance of consistency

• Need for firms and supervisors to increase expertise

– Even statistical models require expert judgment

– How much do supervisors need to know?

• Can we give enough guidance to allow objective self-

assessment?

• What is the scope and appetite for improved standards?

SOME VALIDATION CHALLENGES

• What is the supervisory standard?

– Can we identify a definite pass or a definite fail?• Benchmarks which must be beaten to qualify, or if beaten are sufficient

• Implications for expert judgment approaches, or methods which

‘perform’ less well?

• Other objectives may be avoidance of systemic risk, and allowing entry

to stimulate competition

– How do we incentivise firms to improve?• Rising standards or Pillar 2 requirements

– Too much conservatism will take away incentives

• LGD and EAD?

the advanced irb forum monday 23 june kevin ryan

Documents