model validation
TRANSCRIPT
© 2015 Protiviti Inc.
CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party.
TREPP BANKING SYMPOSIUM MODEL RISK MANAGEMENT
OCTOBER 9, 2015
TABLE OF CONTENT
• MODEL DEFINED
• EFFECTIVE CHALLENGE
• TOP ISSUES
• THREE LINES OF DEFENSE
• VALIDATION TASK & TECHNIQUES
• ROLE OF INTERNAL AUDIT
• QUESTIONS
© 2015 Protiviti Inc.
CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party.
A Model Defined A model is a computer-based mathematical construct that processes data inputs and produces
outputs used in business decision making
3
A model is not a decision-maker
A model is only as good as the assumptions and inputs
that go into it
Even a great model will “fail” if it is used inappropriately
Even a great model that has been used appropriately will
need to be updated for environmental changes
© 2015 Protiviti Inc.
CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party.
Financial Models Differ From Other User Developed Tools
4
• Provided by vendors or developed in house
using a wide range of platforms.
• An approximate representation of the real
world that can be used to calculate prices,
risks, reserves, and strategies for financial
products.
• An estimate based on a set of assumptions
about the behavior of the people,
organizations, risks, and other models
participating in the marketplace
• A model may reside within a user developed
tool.
A Quantitative Financial Model is:
• Also known as an End User Tool,
Spreadsheet or just tool.
• End user tools tabulate known or agreed
values.
• If a calculation is objectively true with no
possibility of an incorrect assumption, it is not
a model.
A User Developed Tool is:
Model Validation and Audit should provide effective challenge around controlling risks associated with
all business tools.
© 2015 Protiviti Inc.
CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party.
Effective Challenge In Model Risk Management Effective challenge for model risk involves three guiding principles: independence from model
developers, competent resources, and an influence within the organization that values and
supports the function’s role.
5
Senior Management & Board Oversight
Risk Identification & Assessment Risk Measurement Risk Reporting,
Monitoring & Management
Effective Challenge
Re
so
urc
es Organizational Structure
Risk Culture
Policies & Practices
Independence/Incentives
• The effective challenge function (e.g.
Model Validation, Governance
Committees, Internal Audit) or model
risk should be distinctly separate
from the model development process
• Challenge should also be
appropriately supported with well-
designed compensation practices
Competence
• Individuals in the challenge function
require technical expertise and
modeling skills to effectively validate
model assumptions, limitations,
usage, and outcomes
• Understanding of business
processes and financial knowledge
enhance the challenge function’s
value in risk management
Influence
• Effective challenge has explicit
authority and responsibilities
supported by policies and procedures
as well as Senior Management’s
commitment
• The challenge function has stature
and visibility within the organization
and is a respected partner to the
business
© 2015 Protiviti Inc.
CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party.
Guidance For Effective Challenge of Model Risk Management The OCC/FED supervisory guidance on model risk management outlines key principles and
control mechanisms for model risk, including effective challenge.
6
• Developers should consider models’ impact on business performance from user feedback and should factor in
suggestions and criticism from sources independent of the business
• Existence of effective challenge is required to identify model limitations and assumptions
Model Development, Implementation, and Use
• Validation of models should be performed by a function independent of the developers
• Each model should be reviewed at least annually to assess material changes and whether validation activities
are sufficient.
• Models should be validated for conceptual soundness including the validity of assumptions, limitations, and
outcomes
• Models from 3rd party vendors should be evaluated for appropriateness to the institution’s situation and require
ongoing monitoring by the institution and the vendor
Model Validation
• Board and Senior Management should support effective challenge through establishment of a robust model risk
framework that includes policies and procedures outlining the roles and responsibilities for model validation and
quality assurance functions
• Risk Management functions are responsible for managing model validation and review processes to ensure
effective challenge occurs
• Internal Audit evaluates the design and effectiveness of controls for model risk management and communicates
audit findings to the business for remediation
• Use of external resources for model validation, compliance, or audit requires clear lines of communication to
internal parties for identified issues
Model Governance, Policies, and Controls
© 2015 Protiviti Inc.
CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party.
Top 10 Issues In Model Validation
7
Data constraints 1
Lack of explanation for business judgment 2
Inadequate model documentation 3
Invalid or unrealistic assumptions 4
Too many performance testing exceptions 5
Difficulty in choosing the appropriate level of conservatism 6
Lack of alternative model analysis 7
Inadequate validation of vendor models 8
Inadequate understanding of modeling software functionalities 9
Model risk has inadequate stature in the organization 10
© 2015 Protiviti Inc.
CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party.
Model Governance & Validation
2nd Line of Defense
Line of Business
1st Line of Defense
Industry Leading Framework for Model Risk Management
8
Model Governance and
Model Validation
Model Governance Committee(s)
Board of Directors
Model Owners and Model Users
Internal Audit
3rd Line of Defense
Internal
Audit
© 2015 Protiviti Inc.
CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party.
Industry Leading Practices
9
First Line of Defense Model Developer
Second Line of Defense Risk Management /
Model Validation Group
Third Line of Defense Internal Audit
The Issuance of OCC bulletin 2011-12 / FRB SR 7-11 has prompted a shift in all banks to
place heightened emphasis on Model Risk Management, including model validation
The largest banks have begun to appoint a Chief Model Risk Officer
Medium-sized banks have adopted model risk policies designed to drive a structured
model risk governance process with a focus on effective challenge
Some banks with assets over $10 billion now have a dedicated model validation function
Model Risk Management Infrastructures focuses on three levels of control
Executive Management and The Board of Directors are expected to review and approve
a Bank’s model risk governance process
© 2015 Protiviti Inc.
CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party.
Roles in Model Risk Management Model validation and internal audit ensure controls are designed adequately and implemented
effectively to reduce the likelihood of erroneous model output or incorrect interpretation of model
results.
10
Activity
Players Development Implementation Production Reporting Validation
Model
Developers
• Code
• Document
assumptions,
theory, and
testing
• Conduct user
acceptance
tests (UAT)
• Monitor
performance
• Support user
interpretation
of model
results
• Provide code
and
documentation
Independent
Validators
• None (maintain
independence)
• Review
implementation
• Periodic
reviews
• Ensure model
reports are
used
appropriately
• Evaluate
• conceptual
soundness,
outcomes
analysis,
• ongoing
monitoring
Internal
Audit
• Evaluate
adequacy of
development
guidelines
• Test
completeness
of development
documentation
• Test systems
development
life cycle
(SDLC)
• Evaluate
model risk
monitoring
controls
• Evaluate
completeness
of reports
• Evaluate design
adequacy of
validation
guidelines
• Test
effectiveness of
validations
© 2015 Protiviti Inc.
CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party.
Validation Tasks
11
• Interview model developers and other key stakeholders to obtain an understanding of the models
• Review and assess whether the documentation is sufficient to allow a third party to reconstruct the
model if necessary
Interviews & Model
Documentation
Review
• Review developmental evidence for the model
• Review assumptions
• Conduct impact analysis for underlying assumptions
• Assess theoretical limitations
• Assess whether theoretical and conceptual framework aligns with the purpose
Model framework,
Theory & Design
• Review calculations within ranges of industry practices and regulatory requirements
• Independently replicate calculation components
• Independently replicate the model calculation process and benchmark outputs
Process, Code
& Math
• Review the model control environment by testing a sample of model inputs if applicable
• Review the reasonableness of the calculation of input variables
• Review and evaluate missing data handling process
• Review and evaluate the performance of input model if applicable
• Assess the reasonableness of key parameter inputs and assumptions
• Assess the alignment of qualitative and quantitative assumptions, supporting business rationale,
and statistical analysis, if applicable
Model Input Quality
• In-sample, out-of-sample and out-of-time sample validation tests
• Multiple testing approaches may be applied for different types of models
Performance
Testing
© 2015 Protiviti Inc.
CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party.
Common Validation Techniques There are a variety of techniques that can be applied in the performance of model validations.
Some of the various techniques are described below. Many of these techniques are applied in
combination with other testing approaches.
12
Techniques Description
Back Testing Compare actual outcomes with model predicted outcomes using statistical measures.
Sensitivity Analysis Evaluate the sensitivity of changes in model output in relation to changes in input (macro
variables, interest rates, home prices, etc).
Benchmarking Compare internal model results to other like models (e.g. industry models).
Model Replication Replicate design, attributes, and mathematical form of models. Compare outputs between
parallel replica and the actual model.
Cash Flow
Replication
Estimate cash flows outside of the model validating that costs, expenses, instrument
attributes, and assumptions have been properly considered.
Calibration Testing
Where model inputs are not available. Calibrating the score or log-odds to observed default
rates over various time periods. Calibrations techniques such as determining calibration of
FICO scores to default rates.
Assumption &
Theory Review Justify the reasonableness of assumptions and modeling simplifications.
© 2015 Protiviti Inc.
CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party.
Industry Model Validation Approaches Below is an illustration of where certain techniques maybe useful given model type or subject
matter. The appropriateness of specific procedures must be determined by qualified personnel
and in the context of the model’s design and intended use.
13
Model Types Back
Testing
Sensitivity
Analysis
Bench-
marking
Model (or
Modular)
Replication
Cash Flow
Replication
Calibration
Testing
Assumption
Theory &
Review
Scorecards a a a
Interest Rate a a a a a Forecasting
Models a a a a
Valuations a a a a
Marketing a a a
Operational Risk
Models a a a a
Economic Capital
Models a a a
Roll Rate Models a a a
Pricing Models a a a a a a
Liquidity Models a a a a
AML Thresholds a a a
© 2015 Protiviti Inc.
CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party.
Model Audit Approach—3rd Line Review
14
Process Test of Design Test of
Effectiveness Sample Size (#)
Organization structure/roles & responsibilities N/A N/A
Risk reports submitted to the Board and/or
Senior Management
Development standards & documentation
Identification
Risk rating
Usage
Performance metrics & thresholds
Issues closure / extension
Change management
Adjustments / overrides
Annual review
Validation
© 2015 Protiviti Inc.
CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. 15
Questions?