challenges in validating catastrophe models

Challenges in Validating Catastrophe ModelsDr. Paul Rockett

Casualty Actuaries in Europe, 31 May 2013

31 May 2013

Agenda

► Introduction► Validation tools► Validating catastrophe models► The bigger picture► Summary

Challenges in Validating Catastrophe Models

31 May 2013

Introduction


► Tools designed to evaluate the risk profile of a risk or set of risks exposed to catastrophe events

31 May 2013

What are catastrophe models?


Event Set Module

Hazard Module

Vulnerability Module

Financial Analysis Module

Exposure Module

IntensityLo

ss R

atio

Event Set Module

Hazard Module

Exposure Module

Vulnerability Module

Financial Analysis Module0

0.01

0.02

0.03

0.04

0.05

0.06

0 0.2 0.4 0.6 0.8 1

Prob

abili

ty D

ensit

y

Damage Ratio

Damage Distribution

Insurance Cover

Over Limit

Deductible

► They combine the frequency of catastrophe events with their severity to produce annual exceedance probability (EP) curves

► EP Curves can be thought of like VAR. SII SCR is a VAR measure corresponding to the 99.5% conf level of non-exceedance in 1-year

Convolution

31 May 2013

What are catastrophe models?


31 May 2013

What do they cover?

► There are catastrophe models covering► Earthquakes► Windstorms, Tropical Cyclones► Storm Surge, River Floods, ...► Severe Convective Storm (Tornado, Hail, ...)► Bushfire, wild fire, ...

► And man-made catastrophes► Terrorism, Explosion, Conflagration► Pandemic

► Most important models for the insurance industry are US Hurricane, US Earthquake and Europe Windstorm

► But it varies by company


31 May 2013

What are catastrophe models used for?


Catastrophe Models

Capital model inputs

Reinsurance decision

making

Pricing risksAccumulation management

Estimate post event loss

size

AIR

EQECAT

RMS

Brokers

Consultants

Marketplace – e.g.

OASIS LMF

Who builds catastrophe models?

InsurersCat

Modelling Companies

Reinsurers Others

31 May 2013 Challenges in Validating Catastrophe Models

Why validate catastrophe models?

► Extreme events and subsequent losses e.g.► Hurricane Katrina► Hurricane Ike► Christchurch Earthquakes

Solvency II directive:► Article 124

(Validation Standards) ► Article 229 TSIM18

(Validation Process)

► Catastrophe models can drive the Solvency Capital Requirement

► Limited data► Many assumptions

UncertaintyMaterial

Why Validate?

Regulatory pressureHistory


Validation Tools


Solvency II Validation tools Tools for use by all firms (Article 230 TSIM19)

Purpose: Modelling and

estimation errors, Any model

weaknesses

Applications Back-testing Goodness of fit Dependency

assumptions Supplement with

qualitative analysis

Purpose: Impact of single or

multiple events

Applications Challenge results

and parameters Dependencies and

tails of distributions Understanding of

the risk profile Supports capital

allocation decisions Exceptional but

plausible large-loss events

Reverse stress testing

Limitations of the model

Purpose: Are capital

requirements produced robust?

Applications Sensitivity of

results to changes in the key underlying assumptions

Stability of the model

Results against experience Model robustness Stress & scenario

testing

Purpose: Are risks covered

by the model complete?

Explain Profits and Losses

Applications Explains actual

results using the model.

Assess level of unexplained profits and ability to reflect risk profile of the business

Profit & loss attribution


Solvency II Validation tools: Further (optional) tools set out by EIOPA

Purpose: Reinforce the

appropriateness of Pillar I and II results

ApplicationsConsider results of peers to investigate

whether alternative modelling approaches could improve quality

Note differences in company risk profiles

Purpose: Improve

comparisons between different models or versions

ApplicationsHypothetical portfolios of assets and/or liabilities chosen by the

company to analyse outcomes from different versions of internal models

Care needs to be taken to avoid creating systemic risk

Purpose: Analyse how the

results of the model have changed

Applications Reconciles results Validates that the

model is working as expected

Benchmarking Analysis of change Hypothetical portfolio

Purpose: Additional

qualitative validation tools

ApplicationsUse test Using the model in

BAU demonstrates sufficient level of comfort that the results are appropriate for use

Qualitative reviewWith dialogue and review, gain comfort: Theoretical basis of

the internal model Any part of the

internal model

Qualitative review


Validating Catastrophe Models


► Each component needs validation

Validating catastrophe models

► Main focus is on model output

Event Set

Hazard

Vulnerability

Financial Analysis

Exposure DataEP Curves

ELTsExposure

Input Cat Modules Output


► Key Aspects

► Typical ApproachQualitative review inc. data audit

► Validation ChallengeEnsuring processes in place track all the exposure information, and that it passes quality control checks before modelling

Exposure data

Geospatial Resolution

Risk Attributes

Valuation Data Heuristics

Event Set

Hazard

Vulnerability

Financial

Exposure Data

EP Curves

ELTs

Exposure

O

utpu

t

C

at M

odul

es

Inpu

t


► Key Aspects

► Typical ApproachBack-testing, Sensitivity Testing, Benchmarking

► Validation ChallengesNeed an understanding of the physical mechanism to define suitable tests

Event set module

Event Frequency

Clustering of Events

Event Characteristics

Event Set

Hazard

Vulnerability

Financial

Exposure Data

EP Curves

ELTs

Exposure

O

utpu

t

C

at M

odul

es

Inpu

t


► Key Aspects

► Typical ApproachBack-testing, Qualitative Review

► Validation ChallengeSensitivity testing very difficult

Hazard module

Event Set

Hazard

Vulnerability

Financial

Exposure Data

EP Curves

ELTs

Exposure

O

utpu

t

C

at M

odul

es

Inpu

t

Event Footprints

Hazard Distribution

Spatial Correlation


► Key Aspects

► Typical ApproachBenchmarking, Audit

► Validation ChallengeVery involved to properly check

Exposure module

Geocoding Data Recognition

Event Set

Hazard

Vulnerability

Financial

Exposure data

EP Curves

ELTs

Exposure

O

utpu

t

C

at M

odul

es

Inpu

t


► Key Aspects

► Typical ApproachQualitative Review, Sensitivity Testing

► Validation ChallengeLimited data

Vulnerability module

Event Set

Hazard

Vulnerability

Financial

Exposure Data

EP Curves

ELTs

Exposure

O

utpu

t

C

at M

odul

es

Inpu

t

Vulnerability Functions

Secondary Modifiers


► Key Aspects

► Typical ApproachQualitative Review, Audit

► Validation ChallengeUnderstanding the real level of uncertainty

Financial module

Event Set

Hazard

Vulnerability

Financial

Exposure Data

EP Curves

ELTs

Exposure

O

utpu

t

C

at M

odul

es

Inpu

t

Implementation Uncertainty Aggregation


► Key Aspects

► Typical ApproachBack-testing, Sensitivity Testing, Qualitative Reviews, Benchmarking

► Validation ChallengeLimited data

Output

Event Set

Hazard

Vulnerability

Financial

Exposure Data

EP Curves

ELTs

Exposure

O

utpu

t

C

at M

odul

es

Inpu

t

EP Curves Event Loss Tables

Historical Event Losses


Bigger Picture


► Validating catastrophe models should be seen within the context of internal model validation

► Each aspect needs validation

Cat Model

Data Quality

Accumulation

Pricing

Capital Modelling

Corporate Governance

Model Risk Management

Understanding

Non-modelled

Risks

Scope of validation


How to validate non-modelled perils?

► Key Aspects

► Typical ApproachQualitative Review

► Validation ChallengeSubjective

Limited Geographic Coverage

Missing Lines of Business

Missing Sub-Perils

Non-modelled Risks



Understanding


Corporate governance

► Key Aspects

► Typical ApproachQualitative Review, Audit

► Validation ChallengeChecking the right structures, processes and controls are in place to ensure the IM is robust

Materiality (SCR, Pricing, Accumulation)

Processes and Controls

Documentation Change Management


Non-modelled Risks



Understanding

31 May 2013

Understanding cat models


Cat Team

Capital Modelling and Accumulation

Management Teams

Audit

Board

Knowledge of Catastrophe Models

► ChallengeImproving knowledge flow

Non-modelled Risks



Understanding

31 May 2013

Model Risk Management. Is the cat model really reasonable?

► Scenario: After a validation exercise, model C is selected for the IM.

► Model C is revised. The company doesn’t approve.

► Validation ChallengeRobustness of the IM


0 50 100 150 200 250

Model AModel BModel C

Return Period

Loss

(€)

Non-modelled Risks



Understanding

Summary

Validating catastrophe models is challenging ►Catastrophe models are complex, and comprise a number of sub-modules

►Each sub-module is subject to significant uncertainties

►Data to validate them is generally limited, but there are tools to help

Validation needs to be seen in a broad context ► Internal Model perspective includes adequate understanding by stakeholders,

corporate governance, model risk management and non-modelled risks

► It also considers the interface between cat models and capital models, pricing and accumulation management tools

►Effort needs to be proportional to materiality

Challenges in Validating Catastrophe Models31 May 2013

Paul RockettSenior ManagerT: +44 (0)20 7951 1098 M: +44 (0)78 2408 4806E: [email protected]

Contact


mailto:[email protected]


Reliance►This document is provided on the basis that you agree that Ernst & Young LLP (including its partners and staff) accepts no responsibility and shall have no liability in contract, tort or otherwise in relation to the contents of this document and that any use you make of this document is entirely at your own risk.►The information in this presentation pack is confidential and contains proprietary information of Ernst & Young LLP. It should not be provided to anyone other than the intended recipients without our written consent.►The information in this pack is intended to provide only a general outline of the subjects covered. It should not be regarded as comprehensive or sufficient for making decisions, nor should it be used in place of professional advice.►Accordingly, Ernst & Young LLP accepts no responsibility for loss arising from any action taken or not taken by anyone using this pack.►The information in this pack will have been supplemented by matters arising from any oral presentation by us, and should be considered in the light of this additional information.

31 May 2013

Thank you

challenges in validating catastrophe models

Documents

ey onscreen dark

ey onscreen white

white background

ey presentations

custom menu

onscreen projection

dark grey background

colors dropdown menu