august tbs pmo l&p team meeting - acswacsw.us/spring17/zimmerman.pdf · the last slide wasn’t...
Post on 10-Nov-2018
213 Views
Preview:
TRANSCRIPT
August TBS PMO L&P Team Meeting
Model Governance – Transamerica ApproachDarin Zimmerman
Director, Actuarial Integrity
Actuarial Club of the Southwest Spring Meeting
June 15 & 16, 2017
3
Valuing Controls is a paradigm shift from the Actuarial Wild West
► We are financial engineers at heart and engineers are always trying to improve efficiency
• Faster
• Cheaper
• Less Waste
► Controls are engineered inefficiency
• Slower
• More costly
• Redundant
Engineering 101: What is the engineer’s primary concern?
Why are we here today?
We have do this to prevent
costly mistakes!
Safety
Agenda
Introduction
Phase 1 – Model Validation & Change Management
Phase 2 – Actuarial Integrity & In Control (current state)
Phase 3 – It’s TIME to Modernize! (future state)
6
Started with Model Validation
Created a model Inventory, then risk-rated our models into low, medium, and high
► Inforce size and financial volatility
► Model complexity
► Product Novelty (Whole life has 75 years of data, nursing home rider on annuities maybe 10)
Validated (First round used consultants)
► Methodology
► Assumptions
► Model documentation
► SOX control procedures
Then discussed “findings” and documented SLIME (slide 13)
Base-lining (“KC0”) & Validation is first-line function
► We relied on second-line
Phase 1: Model Validation
7
Then we created the Change Management Office (“CMO”):
(has since been renamed the Model Governance Team)
Design and execute a controlled model change management process (workflow)
Oversee that change process was followed (the component verification control)
Build secure repository for models
► A stop-gap on way to true production environment
► Will include production, test, and development regions
► IT controlled and administered
► Models not run in this environment, but secure file cabinet with versioning capabilities
Maintain Model Governance Systems
Phase 1: Creation of the CMO
9
The last slide wasn’t our first iteration
► We started with a paper form
► The last section was peer review
► This was the consulting firm view
► Didn’t work for us
Four critical elements for a successful peer review
1. Subject Matter Expert (SME understands the work)
2. Independence from the work (Can’t review your own work)
3. Independence of the reviewer (Can’t review your boss’s work)
4. Time / Capacity to actually perform the review (easily overlooked, but critical)
Phase 1: A Few Words on Peer Review
10
Phase 1 CMO Systems (Change Management Workflow)
CMO - Actuary
Review technical actuarial
requirements
Reviewer approve?
MO or Developer
Create tech specs from requirements
documentation
MO or Developer
Complete model changes/coding/
unit testing/ docu
Peer of MO or Developer
Review coding and unit testing
Reviewer approve?
CollaborationCMO, MO,
Developer, IT
Develop UAT plan
MO or Developer
Workflow Form Communication
Model TesterExecute UAT
Reviewer approve?
Model Owner
Present results to governance & Corp
Risk for approval
CMO - Actuary
Final review for proper governance and documentation
Reviewer approve?
CMO - Operations
Promote model to production; update SLIME & change log
No
Yes
Model Supervisor
Acceptance of UAT results
Governance
Accepts and reflect in meeting minutes
Transamerica Management
Approves Model change
Model Owner
Submit TBS Model Change Request
Form
No
Yes
No
Yes
No
Yes
Collaboration MO, SBU, IT, AMERC
BusinessRequirements In
itia
tio
nD
eve
lop
me
nt
Test
ing
Go
ve
rna
nce
Model Owner
Acceptance of UAT ResultsCMO – Actuary
Execute CVC
11
Phase 1 Question: “Does this need a ticket?”
Data
Logic
Assumptions
Routine Production
Set up
Non-routine Change
Management
Model Start Date, Policy Inventory, Asset inventory, Bridge Scenario, Sensitivity Multipliers, RBC Factors
IT Changes to Admin Systems & Record Layout
Economic Scenario FilesUpdating Expense & Mortality Tables
Expenses, Mortality, Morbidity, Surrender, Static & Dynamic Lapse, Competitor Rate, Crediting Strategy, Reinvestment Strategy, Long-term Mean Reversion
Product Data File Setup, User Defined Function Coding, Database Coding, Changes to Spreadsheet Formulas Changes to Macros and VBA FormulasChanges to Table Names & Dimensions
DAC Spreadsheets
SOX Controls: Change Management or Production?
12
1. Risk Model Catalog (RMC)
2. SVN – Secure Repository
3. Change Management Workflow
RMC comprised of:
► The Model Inventory
► The SLIME log
► The Assumption and Methodology Repository
The Model Inventory Captures Essential Model information
► Model Name
► Risk Rating (based on complexity, novelty, and materiality)
► Model Owner and Model Supervisor
► Commercial projection system used
► Model Groupings
Phase 1: Model Governance Systems
13
SLIME Log
► Simplifications
► Limitations (in the modeling platform)
► SOX control Issues (as in execution process)
► Maintenance (update assumption, new regulations)
► Errors
• Actuarial Error: Does model do what it was intended to do?
• Accounting Error: Is the number materially correct?
SLIME is an unfortunate acronym: It is not a list of problems to be fixed: it is a list of opportunities to improve model precision; most are not cost justified
Phase 1 Model Governance Systems
14
The Assumptions and Methodology Repository
► Not just an inventory, but future state vision is a source for populating models at runtime to ensure we are using the properly authorized assumption set for a particular run date.
Three Basic Elements
1. Assumption definition (name, type, model, etc.)
2. Review information (multiple histories)
3. Implementation (multiple histories)
Phase 1 Model Governance Systems
15
We had to be precise with our language► Management wanted a “secure environment”
► In the short-term we could only deliver a “secure repository”
► Significant IT involvement to achieve the former
Currently we are creating an SVN region for our experience studies (Technically Phase II)
► Technically, they meet our definition of model, but were scoped out because of resource constraints and that wasn’t really the objective of Phase I
Phase 1 Model Governance Systems (SVN – Secure Repository)
17
Ensure no unintended changes were introduced – compare the current production model with the previous production model
► Use Beyond Compare to verify each file
• Supposed to change and did (GOOD)
• Not Supposed to change and didn’t (GOOD)
• Changed when wasn’t supposed to (BAD)
• Didn’t change when it was supposed to (BAD)
► Compare Spreadsheets with Excel 13
► Compare other MS files with Beyond Compare
► Use other platform tools for Axis, ALFA, MoSes, ArcVal, Poly
► Maintain database of reports for routine and Non-routine
Phase 1 Model Governance Systems Component Verification Control
19
Coincided with Transamerica Functional Reorganization► Actuarial Functional area still has 3 Valuation Legs and 3 Pricing Legs (based on product line)
► Appointed Actuary Team is centralized
► Actuarial Integrity Team is centralized (Corporate was removed, but if any group is Corporate Actuarial, we are it)
► All of the above report to one Chief Actuary of Transamerica
Actuarial Integrity is responsible for:► Actuarial Model Integrity (1st line KC0 function)
► Model Governance (formerly CMO); with additional independent Actuarial Review of assumptions and methodologies
► Experience Analytics (Experience Studies, Predictive Analytics, Assumption Setting, Controls)
► Assumption and Methodology Governance
Phase 2 – Creation of Actuarial Integrity Area
20
Transamerica reviewed Model Governance controls in 2016
► Enhanced controls associated with:
• Assumption Approvals (AMR)
– Introduce manual controls to reconcile governance to workflow
• Audit Trails for Unintended Changes
– Introduce procedures to track file migration from directories to SVN
• Introduce Enhanced Reporting to Promote Ownership of Model Governance Responsibilities
– Changes in SLIME
– Model Inventory
– Looking to assign Model Liaison
– Documentation is KEY!
» Being able to follow the study, recommendation, how you set the assumption, rationale for expert judgment, etc.
» TA has started to look at technical writers, so far with mixed results
Phase 2 – In Control
21
Phase 2 – Committee Governance Approval Structure
Supposedly Congress only does two things: Nothing and Over-react
24
Transamerica’s
Initiative to
Modernize our
Enterprise
T
I
M
E
Initiative to
Modernize our
Enterprise
25 25
Phase 3: Modernization: ATOM and One-MDLC
Actuarial Target Operating Model (“ATOM”) and One-Model Development Life Cycle (“One-MDLC”) are two of nine projects in the
Modernization program, focused on future state actuarial systems and target operating model, and development of a Model
Development Life Cycle that is consistent across TA
ATOM –Outcomes
• Decision-making framework
• List and description of business capabilities
• Tool inventory categorized by product, process and
calculation framework
• High level architecture supporting actuarial work
• Future state operating model
• Timetable for platform and tool review
One-MDLC –Outcomes
• Model and assumption governance framework
• Creation of an IT controlled production environment
• System selection for governance workflow and inventory
management
• Rollout effort plan/roadmap for other processes
• Assumptions store design and build
• Expanded reporting capabilities
26 26
Phase 3: Process, Controls & Governance – key pieces
The conceptual target operating model is founded on using a repeatable process to leverage key pieces for the different product lines
Actuarial Governance
Process
Realign roles and
responsibilities of actuarial
processes to the right ability
Change Management
Mold change
management process
around the new
environments
Systems Management
Continuous review and
management of actuarial
systems and tools
.
Controlled
Environments
Development, testing,
production, and post-
production in a
controlled setting
top related