stochastic modeling - model risk - sampling error - scenario reduction

SoA Stochastic Modelingfor Leading Edge Actuaries 1

Stochastic Modeling forLeading Edge Actuaries

Model Risk, Sampling Error,& Scenario Reduction

Ron Harasym MBA, CFA, FSA, FCIAVice-President & Chief Risk Officer


Outline of Presentation

What is Model Risk?

Model Risk General Principles.

Sampling Error & Model Risk Issues.

Scenario Reduction Techniques.

Final Thoughts


Models in General

A model is an imitation and simplification of a real world system --a tool that provides statistical estimates and not exact results.

Models can be computational/statistical (based on directmathematical representations), judgment-based, or a combinationthereof.

Models provide assistance in product design and pricing,valuation, forecasting, risk management, financial reporting, andperformance management.

Considerations for evaluating a model are provided in Appendix A.

All in all, Management wants results that are Accurate,Explainable, and Predictable.


Model Risk is a general term referring to the possibility of errorand/or loss resulting from the use of models.

Model Risk This risk has a number of components:

Model misspecification Assumption misspecification Inappropriate use or application Inadequate testing, validation, and documentation Lack of knowledge or understanding, user and/or management Inadequate systems structure and change management controls Error and negligence

Definition of Model Risk


Model misspecification incorrect recipe => incorrect model

Assumption misspecification incorrect ingredients => incorrect assumption

Inappropriate use or application birthday cake for retirement => Ln model for bond returns

Inadequate testing, validation, and documentation

Lack of knowledge or understanding, user and/or management

Inadequate systems structure and change management controls don’t follow recipe or write down modifications

Error and negligence

Example of Model Risk: Cake Baking


Ownership Each model must have an owner or “custodian”.

The custodian is accountable for the use, maintenance, riskidentification, control, and documentation of the model.

Model Risk General Principles


Vetting New models, and changes to existing models, must be vetted

and approved prior to production use. Vetting consists of three components:

1. Model testing (Ensuring that model outputs are consistentwith the input assumptions – i.e. that the model is functioningas planned.)

2. Model validation (Ensuring that, given model calibration,output is consistent with economic or business reality. Modelcould pass testing, but fail validation.)

3. Expert user validation (Internal or external peer review, toascertain whether the model being tested will, in fact, beappropriate for its intended use. Like a second opinion.)



Segregation of Duties Each model must be vetted by someone other than the

developer or custodian of the model.

Compliance audits must be performed by someone other thanthe developer or custodian.



Inventory The inventory should include:

Model and Name of custodian

Name of developer (internal or external)

Date model was developed / vetted / implemented

The general purpose of the model, and how it is intended to beused.



Documentation The custodian of the model must retain all available model

documentation. Documentation should include:

What the model does, and how it functions (i.e. the model’s intended business use, theoutput produced, underlying theory, mathematical premises, methodology, etc.)

What the inputs and source systems are, data types, parameters, …etc. Limitations of the model (what it can/cannot do; what adjustments must be made to inputs

or outputs in order to use them; … etc.) Instructions for use or maintenance of the system/model. Main assumptions, and sensitivity to key assumptions Calibrations necessary, including procedure and frequency for calibrating parameters. How the model is to be back-tested. The degree to which the model was subject to independent review and verification, and

when this occurred. History of dates that changes were made to the model, differences between versions, and

who reviewed/approved them. Who owns the source code and location there of.



Identification and Control of Risk Identification and assessment of model risk starts on day one

in model use life cycle, must be continuous, and is theresponsibility of all parties involved.

The custodian of the model is ultimately responsible foridentifying and assessing model risk, and putting in placeprocesses and procedures to control it.



Model Review There must be an appropriate frequency of review of the model.

Need to consider: The magnitude of the model’s financial and non-financial impacts,

The degree of knowledge related to the inputs and assumptions forthe model,

The model sensitivity to initial / long-term to market conditions,

The extent of knowledge about the model itself.

Note that regulatory requirements may require internal /external review for actuarial models.

Need to consider who performs the review.



Backtesting and Performance Attribution Backtesting should be done on a regular basis to compare model

expected results versus actual results.

Performance Attribution needs to be performed to help explain resultsas well as to assist in long-term model development and enhancement.

The frequency of backtesting depends on the same factors as in ModelReview.

Backtesting reports should be produced and retained, and materialdeviations escalated according to established processes.



Model Risk & Exposure to Sampling Error

A significant risk inherent in stochastic modeling is theexposure to sampling error.

Need to consider if the calibration requirement applies to themodel or to the scenarios used for applied modeling purposes.

Caveat emptor!


Present Value of Cash Flows ($M) under Various Scenario Sets

-$300

-$200

-$100

$0

$100

$200

$300

$400

$500

$600

1 - 1

0000

1 - 1

000

1001

- 20

00

2001

- 30

00

3001

- 40

00

4001

- 50

00

5001

- 60

00

6001

- 70

00

7001

- 80

00

8001

- 90

00

9001

- 10

000

Scenario Set

CTE(95) CTE(80) CTE(60) CTE(0)


Percentage Error from Base under Various Scenario Sets

-20%

-15%

-10%

-5%

0%

5%

10%

15%

20%

25%

1 - 1

0000

1 - 1

000

1001

- 20

00

2001

- 30

00

3001

- 40

00

4001

- 50

00

5001

- 60

00

6001

- 70

00

7001

- 80

00

8001

- 90

00

9001

- 10

000

Scenario Set

% E

rror


Note: Assume 10,000 scenariosproduce the correct result. Ii.e. Base = 1-10000 scenario set.


Use of Representative Scenarios

Stochastic modeling is computationally intensive. Variance reduction techniques, converge on the mean of the

distribution efficiently, but compromise the distribution of therisk factors in the process.

The information content of the “tail” is no longer credible. Article in the July 2002 NAAJ, written by Yvonne Chueh, details

the use of representative scenario techniques for interest ratesampling.

2003 CIA Stochastic Symposium article, Efficient StochasticModeling Utilizing Representative Scenarios: Application toEquity Risks, written by Alastair Longley-Cook, details use forequity scenario sampling.


Present Value of Cash Flows ($M) using Representative Scenario Sets

-$300

-$200

-$100

$0

$100

$200

$300

$400

$500

$600

1 - 1

0000

Rep

500

0

Rep

250

0

Rep

200

0

Rep

125

0

Rep

100

0

Rep

625

Rep

500

Rep

400

Rep

250

Rep

200

Rep

125

Rep

100

Scenario Set



Percentage Error from Base under Various Representative Scenario Sets

-20%

-15%

-10%

-5%

0%

5%

10%

15%

20%

25%

1 - 1

0000

Rep

500

0

Rep

250

0

Rep

200

0

Rep

125

0

Rep

100

0

Rep

625

Rep

500

Rep

400

Rep

250

Rep

200

Rep

125

Rep

100

Scenario Set

% E

rror


Note: Assume 10,000 scenariosproduce the correct result. i.e.Base = 1-10000 scenario set.


Advantages of Scenario Reduction

Allows for a reduction in scenario sample size while preserving theprobability distribution.

May reduce, but does not eliminate, sampling error.

Scenario reduction algorithms can be independent of the form ofthe scenario generator and the asset/liability models.

Assists in sensitivity testing.

A quick way of estimating tail risk when pressed for time.


Limitations of Scenario Reduction

Some algorithms involve the estimation of the present value orfuture value of a stream of cash flows that are unknown inadvance.

When a metric is developed to measure similarity ordissimilarity between scenario paths, the continuity property isdesirable.

The continuity means that if two paths are close in the domain ofa function, the corresponding function outputs will be similar.

The condition is difficult to verify or satisfy due to its mathematicalcomplexity.

In some case, sampling errors could be too big to provide areasonable replication of the true distribution.


Wrap-Up

Questions & Answers!


Appendix A: Model Considerations

Considerations when Evaluating a Model

Reliance on Internal / External Consultant ExpertiseAre the individuals upon whom we are relying experts in the field?

Has the model been reviewed by or opined upon by experts in the field?

Are there significant differences of opinion among experts concerning aspects of the model thatcould be material to the use to which the model is being put?

How reliant are we on the consultants to run the model?

What is the long-term support capability of the consultant?

What training is available for us?

What level of documentation is provided?

Are we provided details on the technical algorithms employed?

Is the model a black-box or open?

Annual licensing or one-time fee? Annualized Run Rate?

What is the cost/rate for initial customization?

What arrangements are available for future customization?



Understanding of General Concepts of the ModelWhat are the general principles used in developing the model?

Is the model based on disputed or new principles or theory?

What are the principal components of the model and how do these components interrelate withinthe model?

To what extent has the model been tested, validated and subjected to independent review andtesting?

Structural Systems IssuesWhat platform does the model run on?

What are the computational demands of the model?

Do we have the computational capability?

What are the input/output data demands of the model?

Do we have the data capacity?



Structural Model IssuesIs the model useful off-the-shelf or is customization required?

How much customization is required?

With respect to change, is the model flexible or inflexible?

Do we have access to the source code?

What is the cost for customization?

Data InputWhat user data input and what level of detail is required to produce the expected model output?

What model calibration is required?

Do we have control over model calibration or are we dependent on others?

How reliant are we on external data sources?



Appropriateness of the Model for the Intended ApplicationIs the model appropriate for the particular analysis for which it is to be used?

Does the model quantify the results in a manner useful to us?

Will modifications be needed to the model to produce the desired output?

Is the model output consistent with the intended use of the model?

What adjustments or additional work are required in order to make use of the model results?

What assumptions and additional work are needed to use the model output?

Does the historical data used by the model adequately represent the range of reasonablyexpected outcomes?



Model ValidationWhat is the quality and availability of the user input data?

Is the model output reasonable?

How do the model results compare to those produced by other models? Other methods?

Is backtesting sensible/rational/feasible? If so, how well does the model backtest?

Are the relations between various output results consistent and reasonable?

Is the sensitivity of the model output to variations in assumptions reasonable?

Model LimitationsWhat limitations does the model have? Are these critical? Are the results materially sensitive to

these?

What is the projected lifespan of the model?

stochastic modeling - model risk - sampling error - scenario reduction

Documents

model validation

model testing

ln model

model outputs

model calibration

model risk general principles

risk management

soa stochastic modeling