the business of predictive modeling
DESCRIPTION
The Business of Predictive Modeling. December 17, 2013 Christine Hofbeck, FSA, MAAA Centroid Analytics, LLC. AGENDA. PART I -- INTRODUCTION PART II – MODELING 101 (Basic Steps) PART III – “GOLDEN QUESTION” PART IV – OPERATIONAL CONSIDERATIONS. INTRODUCTION. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/1.jpg)
The Business ofPredictive Modeling
December 17, 2013
Christine Hofbeck, FSA, MAAACentroid Analytics, LLC
![Page 2: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/2.jpg)
AGENDA
PART I -- INTRODUCTION
PART II – MODELING 101 (Basic Steps)
PART III – “GOLDEN QUESTION”
PART IV – OPERATIONAL CONSIDERATIONS
2
![Page 3: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/3.jpg)
INTRODUCTION
Predictive modelling [sic] is the process by which a model is created or chosen to try to best predict the probability of an outcome. -- Wikipedia
In practice:
3
VISUALIZECustomers
Most Profitable Lines
OTHERyou are only limited by
your creativity
Identify patterns/ segment
risks
Develop business
rules
Improved decision making
![Page 4: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/4.jpg)
Potential Applications
4
OPTIMIZE Operational Efficiency Distribution ChannelsClaims Management
Pricing / Reserves
DATA
VISUALIZECustomers
Most Profitable Lines or Products
Target Marketing
OTHERyou are only limited by
your creativity
MINIMIZERisk
Fraud
![Page 5: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/5.jpg)
Potential Applications (Life)
5
1. Triage UW decisions; implement STP for (more) applicants
2. Decrease purchase of traditional UW requirements by determining when they may not be necessary
3. Identify & target customers more likely to buy
4. Identify customers more likely to lapse – intervene if profitable, allow unhealthies to lapse
5. Inforce book management
6. Identify most desirable agents
7. Smart customer handling
![Page 6: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/6.jpg)
Predictors
A predictive model is made up of a number of predictors (“independent variables”), which are data elements likely to influence future behavior or results (“dependent variable”).
6 SEEK PARSIMONY
DON’T USE ONE VARIABLE DON’T USE ALL VARIABLES the mean predicts the future but doesn’t tell us why…(“underfit”)
exactly replicates the past… cannot predict the future (“overfit”)
![Page 7: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/7.jpg)
BASIC STEPS (Modeling 101)
7
Define & Scope
Data Prep
Model Build
Model Validation
Implementation
Review & Refine
![Page 8: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/8.jpg)
Define & Scope
8
What is the budget?
Consider IT, staff, data purchase,
training, etc.
Do we have the systems capacity to
implement?
How long do we have to build? To
implement?
Insource or outsource?
How will the results be used?For whom/what
are we trying to predict this? (“unit
of exposure”)
Exactly what are we trying
to predict?
![Page 9: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/9.jpg)
Data Prep* sometimes the most time intensive step of modeling
9
MODELING DATASET
INTERNAL DATAo # yearso accuracyo ability to accesso primary key
EXTERNAL DATAo match rateo cost – to modelo cost – to useo frequency of update
Consider both expected & unexpected relationships – creativity in data exploration can be the key
to your competitive edge!
![Page 10: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/10.jpg)
Data Prep (cont’d)1. COMBINE various data sources2. CONVERT to desired exposure unit or format3. CORRECT inaccurate data4. INSPECT to remove variables:
- Too many blank values that cannot be imputed- All/most values the same- Data cannot be relied upon- Data will not be captured going forward- Legal advice not to use
5. BUCKET (“bin”) values10
![Page 11: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/11.jpg)
Model Build (cont’d)
UNIVARIATE ANALYSIS – test each variable one by one to see which ones may be predictive.
11
MULTIVARIATE ANALYSIS – examine multiple variables in different groups to obtain the best, USABLE results – remember parsimony!
INTERACTIONS – which variables can be combined into a “mega variable” to improve results (i.e., does 1+1 = 1.5? does 1+1 = 3?)
Complicate the model (add variables, interactions) and simplify the model (remove variables, bin)
to find the preferred combination.
![Page 12: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/12.jpg)
Model Build (cont’d)
Various tests can be used to determine variable inclusion:
12
STATISTICAL CONSISTENCY JUDGMENT
P-valuesCramer’s VConfidence
intervalsType III tests
Apply business knowledge to
assess whether suggested
relationships make sense
Of patterns -
Over timeOver random
parts of a dataset
![Page 13: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/13.jpg)
Model Validation
ACTUAL vs. EXPECTED-- how close did we get?
Generally, a subset of the data is withheld during the modeling process for validation:
Model validation graphs are useful for communicating model performance to non-technical audiences.
13
OUT OF TIMEwithhold most
recent data
OUT OF SAMPLEwithhold randomly
generated % of records
![Page 14: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/14.jpg)
Model Validation – Sample Chart
14
1 2 3 4 5 6 7 8 9 100.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
ActualExpectedO
utco
me
Decile
![Page 15: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/15.jpg)
Implementation
BUSINESS RULESWhat decisions will be made based on the prediction?
May vary by location, business, rate group, etc.
SYSTEM BUILDScoring engine (collects data & calculates predictions)
Decision tool (executes business rules)User interface
TRAININGAnyone who will interact with the model must
understand what it does and why15
![Page 16: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/16.jpg)
Review & RefineREPORTING
How close did we get to the goal? How far did we exceed it?
Multiple reporting packages required for varied audiences, for example:
Executives – highlights in aggregate by zone, business unit, product
Actuaries – detailed results by variable, state, rate group Marketing – by broker/agent, location Underwriting – by underwriter as a performance measure
Frequency of update – weekly, monthly, quarterly, yearly?
Method of calculation – automated? ad hoc?
![Page 17: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/17.jpg)
Review & Refine (cont’d)MODEL UPDATES
WHY? As target customer is attained, characteristics of inforce
book will change Business goals/strategies may change New data may become available Tolerance for certain characteristics may change
HOW? Update current variable relativities (“recalibrate”) Start over - search for more predictive variables (“recast”)
HOW OFTEN?17
![Page 18: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/18.jpg)
Advantages of Modeling Over
18
1. Many additional and often unconventional variables may be examined
2. Modeling a particular variable controls for the effects of other included variables – we don’t risk double counting or attributing effects to the wrong variables
3. Traditional approaches segment data into smaller categories which impact credibility
4. Interactions are introduced
The above advantages can lead to improved accuracy, enhanced business and strategic benefits, more reliable
assumptions, improved risk mitigation, etc.
Traditional Approaches
![Page 19: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/19.jpg)
THE “GOLDEN QUESTION”
Through brainstorming, feedback loops, and data review, determine what single characteristic (“golden question”) will define your target
19
![Page 20: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/20.jpg)
OPERATIONAL CONSIDERATIONS
1. Executive & cross-functional support
2. Time/cost versus depth of investigation
3. Strategic modeling process
4. Cross-functional involvement throughout build
5. Thorough training
20
![Page 21: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/21.jpg)
Executive & Cross-Functional Support
21
If target users don’t support the model, they will resist using it.
Gaining complete support can be difficult:1. Resistance to change2. Concern that model results will highlight
current deficiencies3. Lack of understanding of predictive models
![Page 22: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/22.jpg)
Support (cont’d)
22
We’ve always done it this way, and it’s worked.
I don’t see a reason to change
anything.
I will have to take
on additional work associated with
new processes. My workflow will
double (triple).
I don’t know how to explain this to a broker/agent so I don’t want
to use it.
I found one outlier so the
model must be wrong.
I already have an established plan. I
know who our target customer is. The model will suggest
that my current methodis incorrect, which will reflect poorly on my
performance/reputation.
My position will be eliminated if a model is now used to select risks. My expertise
must not be important to the company.
![Page 23: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/23.jpg)
Time/Cost vs. Depth of Investigation
23
The process of building and implementing a model can typically be quite lengthy – longer than most expect
OR
Remember that a simple model does not necessarily indicate a simple study!
• Simpler Study (3-12 months)• Results more conservative• Perhaps internal data only• Generous binning• Limited interactions• May be appropriate if goal is
a general sense of direction
• More thorough investigation• Additional time• Additional development cost• Possible greater payoff
through enhanced segmentation and data exploration
![Page 24: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/24.jpg)
Strategic Modeling ProcessTARGET PREDICTION/USEo Ensure target is appropriate for the intended useo While many ideas are interesting, you may wish to focus on those
which are actionable
STATISTICAL SIGNIFICANCE vs. ULTIMATE IMPACTo The most statistically significant model may not be the most
impactfulo Consider ease of implementation, repeatability, updateso Identify when “less is more”!
FLEXIBILITYo Allow for unexpected insights which could lead to unanticipated
changes in business strategy or processo Sometimes the insights gained from the journey will prove more
important than the planned goal
![Page 25: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/25.jpg)
Cross-Functional InvolvementData, product & IT experts, legal advisors, and model
users must remain engaged throughout the model build
25
Insight from functions
Insight to functions
Eases training and implementation
Keep modelers apprised of changes in strategy
Legal considerations around certain variables
![Page 26: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/26.jpg)
Thorough Training
26
The model isn’t done when it’s done.
Who will provide the training?Who is most appropriate to provide training?
Modeling teamGeneral training team
Functional expertsConsulting team*
Other
No clear answer – but this must be thoughtfully considered and appropriately executed to reap the full benefits of the model
which was built
*Consider what information may be shared (non-proprietary)
![Page 27: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/27.jpg)
Discussion/Q&A
27
Remember…Modeling is a complete business strategy
NOT just a mathematical process
So how will YOU use predictive modeling to improve your business?
![Page 28: The Business of Predictive Modeling](https://reader035.vdocument.in/reader035/viewer/2022062517/5681364e550346895d9dcf19/html5/thumbnails/28.jpg)
Christine Hofbeck, FSA, MAAACentroid Analytics, LLC
[email protected] 908.884-4103 (c) 908.574-5351 (w)www.centroidanalytics.com