combining logit and ensemble modeling for increased customer churn detection
TRANSCRIPT
![Page 1: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/1.jpg)
#pawcon@pythongeert
Opening Keynote:
Le Mariage ParfaitCombining Logit and Ensemble Modeling for Increased Customer Churn Detection
PAW London Oct 29 – 30, 2014
Geert Verstraeten, PhDProgram ChairPAW London
![Page 2: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/2.jpg)
#pawcon@pythongeert
prediction
#pawcon@pythongeert
our domain is about predicting the future, but we’re not the community that uses a crystal ball
![Page 3: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/3.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
prediction
instead, by knowing what lies behind us, we make an estimated guess about what’s ahead of us
![Page 4: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/4.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
prediction
example 1: predicting the stock marketkeynote john elderPAW london 2013
![Page 5: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/5.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
prediction
example 2: predicting player injuriesleicester tigers rugby teamPAW london 2012
![Page 6: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/6.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
prediction
example 3: predicting breakdown of sewerage network in walesPAW london 2012
![Page 7: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/7.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
prediction
in many cases, such as targeted marketing and credit scoring, decision makers are playing a game of guess who
predictive analytics can help decision makers win the game
![Page 8: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/8.jpg)
#pawcon@pythongeert
big data
#pawcon@pythongeert
we add business value based on analysis of all kinds of data, including big data
![Page 9: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/9.jpg)
#pawcon@pythongeert
technologytrigger
inflatedexpectations
disillusionment
plateau of productivity
#pawcon@pythongeert
hype cycle
while big data is at the peak of the hype cycle, predictive analytics is in the plateau of productivity
![Page 10: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/10.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
tools
there’s a lot of tools out there, i.e. algorithms that we use to predict -we’ll only explain the basic tools used in our study
![Page 11: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/11.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
people who listen to radiohead are claimed to be more intelligent than those who listen to beyoncé
![Page 12: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/12.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
SAT score
rankradiohead
musicthatmakesyoudumb.virgil.gr
MassachusettsInstitute of Technology
Harvard College
Yale UniversityUniversity of
Chicago
WellesleyCollege
University ofSouthernCalifornia
PrincetonUniversity
1
2
3
4
5
6
7
8
9
10
1300 1350 1400 1450 1500 1550 1600
linear regression explained using somedata from this study
![Page 13: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/13.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
linear regression
Y = 𝛼 + 𝛽 ∗ X
rank radiohead
Y = 𝛼 + 𝛽1 ∗ X1 + 𝛽2 ∗ X2 +...
SAT score
linear regression explained using somedata from this study
![Page 14: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/14.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
SAT score
rankradiohead
MassachusettsInstitute of Technology
Harvard College
Yale UniversityUniversity of
Chicago
WellesleyCollege
University ofSouthernCalifornia
PrincetonUniversity
1
2
3
4
5
6
7
8
9
10
1300 1350 1400 1450 1500 1550 1600
logistic regression explained using somedata from this study
![Page 15: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/15.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
SAT score
probabilityradioheadin top 3
MassachusettsInstitute of Technology
Harvard College
Yale University
University ofChicago
WellesleyCollege
University ofSouthernCalifornia
PrincetonUniversity
1
0
1300 1350 1400 1450 1500 1550 1600
logistic regression explained using somedata from this study
![Page 16: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/16.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
logistic regression
p = 1
1+𝑒−( )
Probability radiohead in top 3
SAT score
p = 1
1+𝑒−(𝛼+ 𝛽1∗X
1+ 𝛽
2∗X
2+...)
𝛼 + 𝛽 ∗ X
![Page 17: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/17.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
SAT score
# under-
graduates
MassachusettsInstitute of Technology
Harvard College
Yale University
University ofChicago
WellesleyCollege
University ofSouthernCalifornia
PrincetonUniversity
1300 1350 1400 1450 1500 1550 1600
18000
16000
14000
12000
10000
8000
6000
4000
2000
0
University of Pennsylvania
75%
50%
0%
decision trees explained using somedata from this study
![Page 18: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/18.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
decision tree
50%
25% 75%
0% 50%
SAT < 1,450 SAT >= 1,450
#ugrads < 9,000 #ugrads >= 9,000
decision trees explained using somedata from this study
![Page 19: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/19.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
multiplicitythis is multiplicity photography
![Page 20: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/20.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
multiplicity
in predictive analytics, multiplicity of good models means that several models often deliver comparable results
![Page 21: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/21.jpg)
#pawcon@pythongeert
all models are wrong, but some are usefulGeorge Box
#pawcon@pythongeert
simplicity is theultimate sophistication
Leonardo Da Vinci
multiplicity
![Page 22: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/22.jpg)
#pawcon@pythongeert
ensemble techniques
#pawcon@pythongeert
youtube.com/watch?v=gBnvGS4u3F0
explaining ensembles through a video with penguins and a whale
![Page 23: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/23.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
bagging
ID Target
1 1
2 0
3 1
4 0
5 0
ID Target
1
5 0
5 0
3 1
1 1
1 1
ID Target
2
2 0
4 0
1 1
4 0
1 1
explaining ensembles without a video with penguins and a whale
![Page 24: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/24.jpg)
#pawcon@pythongeert
random forests
50%
25% 75%
0% 50%
#pawcon@pythongeert
random forests are an ensemble of trees
![Page 25: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/25.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
today
Véronique Van VlasselaerFraud Detection
Phil Winters
Internet of Things
Ajay Kelkar
Cross-selling
Arthur Von Eschen
Cheating Detection
Duncan Ross
Data for Good
the program of PAW london 2014wednesday oct 29
![Page 26: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/26.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
tomorrow
Dieter Debels
Buying flows
Lukas Vermeer
PersonalisedRecommendations
Gergely Kalmár
Online Behavior
Aaron Ferguson
Malicious Use of Anonymizers
Tom Khabaza
Time-seriesData Mining
the program of PAW london 2014thursday oct 30
![Page 27: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/27.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
keynotes
David Hand
The Improbability Principle
John Elder
The Peril of Vast Search
Dean Abbott
The Revolution in Retail Customer Intelligence
keynotesPAW london2014
![Page 28: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/28.jpg)
#pawcon@pythongeert
Geert
@pythongeert
Verstraeten
pythonpredictions.com
pawcon.com/london
my personal mariage parfait
![Page 29: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/29.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
retention
retention is an important marketing problem for many organizations andindustries
![Page 30: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/30.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
predicting churn
churn prediction is a common application where the number of churners is typically small compared to the number of customers that stay
![Page 31: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/31.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
predicting churn
oversampling typically does not work great
![Page 32: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/32.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
predicting churn
mixed results for undersampling
![Page 33: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/33.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
our study
can we combine
undersamplingbagginglogistic regression
and improve logistic regression?
![Page 34: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/34.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
our studythe idea was to apply bagging, but in a way that we always keep the churners and only a proportion of non-churners
![Page 35: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/35.jpg)
#pawcon@pythongeert
Kristof
@KCoussement
Coussement
ieseg.fr
pythonpredictions.comthe mariage parfait of kristofcoussement - professor of marketing in france - who performed the research
![Page 36: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/36.jpg)
#pawcon@pythongeert
the prize
#pawcon@pythongeert
the prize needed to convince kristof to spend his time on this study – ‘a research bet’
![Page 37: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/37.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
11 real-life european churn datasets in retail, telco and financial services
variables 15 – 236 customers 876 – 74,122incidence 1.8% – 47%
our study
A Bagging-Based Undersampling Strategy for Classification: A Customer Churn Prediction Application – IFORS 2014 Barcelona
![Page 38: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/38.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
we compared• logistic regression• logit + bagging• logit + undersampling• logit + undersampling + bagging• random forests
our study
A Bagging-Based Undersampling Strategy for Classification: A Customer Churn Prediction Application – IFORS 2014 Barcelona
‘u-bagging’
![Page 39: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/39.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
• logistic regression with correlation-based feature selection filter
• bagging uses 100 samples• undersampling balanced sampling
• 5 x 2-fold cross-validation AUC• wilcoxon signed-rank test & friedmann test
with bonferroni-dunn post-hoc test
details
A Bagging-Based Undersampling Strategy for Classification: A Customer Churn Prediction Application – IFORS 2014 Barcelona
![Page 40: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/40.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
AUC
probability that a randomly selected
churner will be ranked higher than a randomly selected
non-churner
![Page 41: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/41.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
results
-0.005
0
0.005
0.01
0.015
0.02
0.025
1 2 3 4 5 6 7 8 9 10 11
bagging increasesperformance significantly when compared to logistic regression
Dataset
AUCincrease
![Page 42: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/42.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
resultsundersampling significantly decreases performancewhen compared to logistic regression
Dataset
AUCincrease
-0.025
-0.02
-0.015
-0.01
-0.005
0
0.005
1 2 3 4 5 6 7 8 9 10 11
![Page 43: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/43.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
results
Dataset
AUCincrease
0
0.002
0.004
0.006
0.008
0.01
0.012
0.014
0.016
0.018
0.02
1 2 3 4 5 6 7 8 9 10 11
u-bagging increasesperformance significantly when compared to logistic regression
![Page 44: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/44.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
resultsu-bagging is competitive to bagging
Dataset
AUCincrease
-0.015
-0.01
-0.005
0
0.005
0.01
1 2 3 4 5 6 7 8 9 10 11
![Page 45: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/45.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
resultsu-bagging is competitive to random forests
Dataset
AUCincrease
-0.08
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
1 2 3 4 5 6 7 8 9 10 11
![Page 46: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/46.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
resultsu-bagging has top average ranking on the 11 real-life churn datasets
Average Ranking
1.6
2.2
3.4
3.5
4.4
1 2 3 4 5
u-bagging
bagging
logistic regression
random forests
undersampling
![Page 47: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/47.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
resultsu-bagging is much more efficient than bagging when incidence is low
Number of observations
100
10,000
10
1,000
logistic regression
bagging
undersampling (5% incidence)
u-bagging (5% incidence)
![Page 48: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/48.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
afterthoughts
![Page 49: Combining Logit and Ensemble Modeling for Increased Customer Churn Detection](https://reader034.vdocument.in/reader034/viewer/2022051400/55a0fee21a28ab132e8b4665/html5/thumbnails/49.jpg)
#pawcon@pythongeert
#pawcon@pythongeert
a more efficient approach to bagging
undersampling vs oversampling
afterthoughts
multiplicity of good models
simplicity is the ultimate sophistication
parallel processing in big data