Getting Started With Data Mining

Dan SteinbergN Scott Cardell

Mykhaylo GolovnyaNovember, 2011Salford Systems

Basic Concepts

Data Mining

•Predictive Analytics•Machine Learning•Pattern Recognition•Artificial Intelligence•Business Intelligence•Data Warehousing

Data Mining

•Statistics •Computer science•Database Management•Insurance•Finance•Marketing•Electrical Engineering•Robotics•Biotech and more

Cont.•OLAP•CART•SVM•NN•CRISP-DM•CRM•KDD•Etc.

What Everyone Hears

Data mining is the search for patterns in data using modern highly automated, computer intensive methods

◦ Data mining may be best defined as the use of a specific class of tools (data mining methods) in the analysis of data

◦ The term search is key to this definition, as is “automated”

The literature often refers to finding hidden information in data

Data Mining Defined

• Study the phenomenon • Understand its nature• Try to discover a law• The laws usually hold for a long time

Science

• Collect some data• Guess the model (perhaps, using science)• Use the data to clarify and/or validate the model• If looks “fishy”, pick another model and do it

again

Statistics

• Access to lots of data• No clue what the model might be• No long term law is even possible• Let the machine build a model• And let’s use this model while we can

Data Mining

The Three Sisters

Quest for the Holy Grail- build an algorithm that will always find 100% accurate models

Absolute Powers- data mining will finally find and explain everything

Gold Rush- with the right tool one can rip the stock-market and become obscenely rich

Magic Wand- getting a complete solution from start to finish with a single button push

Doomsday Scenario- all conventional analysts will eventually be replaced by smart computer chips

Data Mining Mythology

This is known as “supervised learning”

◦ We will focus on patterns that allow us to accomplish two tasks Classification Regression

This is known as “unsupervised learning”

◦ We will briefly touch on a third common task Finding groups in data (clustering, density estimation)

There are other patterns we will not discuss today including

◦ Patterns in sequences◦ Connections in networks (the web, social networks, link analysis)

Patterns Searched For

CART® (Decision Trees, C4.5, CHAID among others)

MARS® (Multivariate Adaptive Regression Splines)

Artificial Neural Networks (ANNs, many commercial)

Association Rules (Clustering, market basket analysis)

TreeNet® (Stochastic Gradient Tree Boosting)

RandomForests® (Ensembles of trees w/ random splits)

Genetic Algorithms (evolutionary model development)

Self Organizing Maps (SOM, like k-means clustering)

Support Vector Machine (SVM wrapped in many patents)

Nearest Neighbor Classifiers

Major Data Mining Tools

(Insert chart)

In a nutshell: Use historical data to gain insights and/or make predictions on the new data

The Essence of Machine Learning

Given enough learning iterations, most data mining methods are capable of explaining everything they see in the input data, including noise

Thus one cannot rely on conventional (whole sample) statistical measures of model quality

A common technique is to partition historical data into several mutually exclusive parts

◦ LEARN set is used to build a sequence of models varying in size and level of explained details

◦ TEST set is used to evaluate each candidate model and suggest the optimal one

◦ VALIDATE set is sometimes used to independently confirm the optimal model performance on yet another sample

Partitioning Historical Data

Historical DataLearn Build a Sequence of Models

Test Monitor Performance

Validate Confirm Findings

Optimal Model Selection

Analyst needs to indicate where the TEST data is to be found ◦ Stored in a separate file

◦ Selected at random from the available data

◦ Pre-selected from available data and marked by a special indicator

Other things to consider◦ Population: LEARN and TEST sets come from different populations

(within-sample versus out-of-sample)

◦ Time: LEARN and TEST sets come from different time periods (within-time versus out-of-time)

◦ Aggregation: logically grouped records must be all included or all excluded within each set (self-correlation)

Notes on Partitioning

Any model is built on past data!

Fortunately, many models trace stable patterns of behavior

However, any model will eventually have to be rebuilt:◦ Banks like to refresh risk models about every 12 months

◦ Targeted marketing models are typically refreshed every 3 months

◦ Ad web-server models may be refreshed every 24 hours

Credit risk score card expert Professor David Hand, University of London maintains:◦ A predictive model is obsolete the day it is first deployed

Shelf Life of Models

Model Evaluation

Model evaluation is at the core of the learning process (choosing the optimal model from a list of candidates)

Model evaluation is also a key part in comparing performance of different algorithms

Finally, model evaluation is needed to continuously monitor model performance over time

In predictive modeling (classification and regression) all we need is a sample of data with known outcome; different evaluation criteria can then be applied

There will never be “the best for all” model; the optimality is contingent upon current evaluation criterion and thus depends on the context in which the model is applied

General Notes

(insert graph)

One usually computes some measure of average discrepancy between the continuous model predictions f and the actual outcome y◦ Least Squared Deviation: R= Σ(y-f)^2◦ Least Absolute Deviation: R= Σ Iy-fI

Fancier definitions also exist◦ Huber-M Loss: is defined as a hybrid between the LS and LAD losses◦ SVM Loss: ignores very small discrepancies and then switches to

LAD-style

The raw loss value is often re-expressed in relative terms as R-squared

Regression Performance

There are three progressively more demanding approaches to solving binary classification problems

Division: a model makes the final class assignment for each observation internally◦ Observations with identical class assignment are no longer discriminated ◦ A model needs to be rebuilt to change decision rules

Rank: a model assigns a continuous score to each observation◦ The score on its own bears no direct interpretation

◦ But, higher class score means higher likelihood of class presence in general (without precise quantitative statements)

◦ Any monotone transformation of scores is admissible

◦ A spectrum of decision rules can be constructed strictly based on varying score threshold without model rebuilding

Probability: a model assigns a probability score to each observation◦ Same as above, but the output is interpreted directly in the exact probabilistic terms

Approaches to Binary Classification

Depending on the prediction emphasis, various performance evaluation criteria can be constructed for binary classification models

The following list, far from being exhausting, presents some of the frequently used evaluation criteria◦ Accuracy (more generally- Expected Cost)

Applicable to all models

◦ ROC Curve and Area Under Curve Not Applicable to Division Models

◦ Gains and Lift Not Applicable to Division Models

◦ Log-likelihood (a.k.a Cross-Entropy, Deviate) Not Applicable to Division and Rank Models

The criteria above are listed in the order from the least specific to the most

It is not guaranteed that all criteria will suggest the same model as the optimal from a list of candidate models

Binary Classification Performance

Most intuitive and also the weakest evaluation method that can be applied to any classification model

Each record must be assigned to a specific class

One first constructs a Prediction Success Table- a 2 by 2 matrix showing how many true 0s and 1s (rows) were classified by the model correctly or incorrectly (columns)

The classification accuracy is then the number of correct class assignments divided by the sample size

More general approaches will also include user supplied prior probabilities and cost matrix to compute the Expected Cost

The example below reports prediction success tables for two separate models along with the accuracy calculations

The method is not sensitive enough to emphasize larger class unbalance in model 1

(insert table)

Classification Accuracy

The classification accuracy approach assumes that each record has already been classified which is not always convenient

◦ Those algorithms producing a continuous score (Rank or Probability) will require a user-specified threshold to make final class assignments

◦ Different thresholds will result to different class assignments and likely different classification accuracies

The accuracy approach focuses on the separating boundary and ignores fine probability structure outside the boundary

Ideally, need an evaluator working directly with the score itself and not dependent on any external considerations like costs and thresholds

Also, for Rank models the evaluator needs to be invariant with respect to monotone transformation of the scores so that the “spirit” of such models is not violated

Why Other Evaluation Criteria?

The following approach will take full advantage of the set of continuous scores produced by Rank or Probability models

Pick one of the two target classes as the class in focus

Sort a database by predicted score in descending order

Choose a set of different score values◦ Could be ALL of the unique scores produced by the model ◦ More often a set of scores obtained by binning sorted records into equal

size bins

For any fixed value of the score we can now compute:◦ Sensitivity (a.k.a True Positive): Percent of the class in focus with the

predicted scores above the threshold◦ Specificity (a.k.a False Positive): Percent of the opposite class with the

predicted scores below the threshold

We then display the results as a plot of [sensitivity] versus [1-specificity]

The resulting curve is known as the ROC Curve

ROC Curve

(insert graph) ROC Curves for three different rank models are shown

No model can be considered as the absolute best in all times

The optimal model selection will rest with the user

Average overall performance can be measured as Area Under ROC Curve (AUC)◦ ROC Curve (up to orientation) and AUC are invariant with respect to the focus

class selection ◦ The best attainable AUS is always 1.0◦ AUC of a model with randomly assigned scores is 0.5

AUC can be interpreted ◦ Suppose we randomly and repeatedly pick one observation at random from the

focus class and another observation from the opposite class ◦ Then AUC is the fraction of trials resulting to the focus class observation having

greater predicted score than the opposite class observation◦ AUC below 0.5 means that something is fundamentally wrong

Comparing Models on ROC

The following example justifies another slightly different approach to model evaluation

Suppose we want to mail a certain offer to P fraction of the population

Mailing to a randomly chosen sample will capture about P fraction of the responders (random sampling procedure)

Now suppose that we have access to a response model which ranks each potential responder by a score

Now if we sample the P fraction of the population targeting members with the highest predicted scores first (model guided sampling), we could now get T fraction of the responders which we expect to be higher than P

The lift in P(th) percentile is defined as the ratio T/P

Obviously, meaningful models will always produce lift greater than 1

The process can be repeated for all possible percentiles and the results can be summarized graphically as Gains and Cumulative Lift curves

In practice, one usually first sorts observations by scores and then partitions sorted data into a fixed number of bins to save on calculations just like it is usually done for ROC curves

Direct Marketing Evaluation

(insert graphs and tables)

Gains and Lift Curves

(insert graphs) Lift in the given percentile provides a point measure of performance

for the given population cutoff◦ Can be viewed as the relative length of the vertical line segment connecting

the gains curve at the given population cutoff

Area Under the Gains curve (AUG): Provides an integral measure of performance across all bins◦ Unlike AUC, the largest attainable value of AUG is (1-p/2), P being the fraction

of responders in the population

Just like ROC-curves, gains and lift curves for different models can intersect, so that performance-wise one model is better for one range of cutoffs while another model is better for a different range

Unlike ROC-curve, gains and lift curves do depend on the class in focus◦ For the dominant class, gains and lift curves degenerate to the trivial 45-

degree line random case

Gains, Lift, and AUG

ROC, Gains, and lift curves together with AUC and AUG are invariant with respect to monotone transformation of the model scores◦ Scores are only used to sort records in the evaluation set, the actual score

values are of no consequence

All these measures address the same conceptual phenomenon emphasizing different sides and thus can be easily derived from each other◦ Any point (P,G) on a gains curve corresponds to the point (P,G/P) on the lift

curve◦ Suppose that the focus class occupies fraction F of the population; then any

point (P,G) on a gains curve corresponds to the point {(P-FG)/(1-F),G} on the ROC curve

It follows that the ROC graph “pushes” the gains graph “away” from the 45 degree line

Dominant focus class (large F) is “pushed” harder so that the degeneracy of its gain curve disappears

In contrast, rare focus class (small F) has ROC curve naturally “close” to the gains curve

All of these measures are widely used as robust performance evaluations in various practical applications

Key Relations

When the output score can be interpreted as probability, a more specific evaluation criterion can be constructed to access probabilistic accuracy of the model

We assume that the model generates P(X)-the conditional probability of 1 given X

We also assume that the binary target Y is coded as -1 and +1 (only for notational convenience)

The Cross-Entropy (CXE) criterion is then computed as (insert equation)◦ The inner Log computes the log-odds of Y=1◦ The value itself is the negative log-likelihood assuming independence of

responses◦ Alternative notation assumes 0/1 target coding and uses the following

formula (insert equation)◦ The values produced by either of the formula will be identical to each other

Model with the smallest CXE means the largest likelihood and thus considered to be the best in terms of capturing the right probability structure

Evaluating Probability Structure

The example shows true non-monotonic conditional probability (dark blue curve)

We generated 5,000 LEARN and TEST observations based on this probability model

We report predicted responses generated by different modeling approaches◦ Red- best accuracy MART model

◦ Yellow- best CXE MART model

◦ Cyan- univariate LOGIT model

Performance-wise◦ All models have identical accuracy but the best accuracy model is substantially

worse in terms of CXE

◦ LOGIT can’t capture departure from monotonicity as reported by CXE

A Simple Example

MARS

MARS is a highly-automated tool for regression

Developed by Jerome H. Friedman of Stanford University ◦ Annals of statistics, 1991 dense 65 page article◦ Takes some inspiration from its ancestor CART®◦ Produces smooth curves and surfaces, not the step-functions of CART

Appropriate target variables are continuous

End result of a MARS run is a regression model◦ MARS automatically chooses which variables to use◦ Variables are optimally transformed ◦ Interactions are detected◦ Model is self-tested to protect against over-fitting

Can also perform well on binary dependent variables ◦ Censored survival model (waiting time models as in churn)

Introduction

Harrison, D. and D. Rubinfeld.Hedonic Housing Prices and Demand for Clean Air. Journal of Environmental Economics and Management v5, 81-102, 1978

506 census tracts in city of Boston for the year 1970

Goal: study relationship between quality of life variables and property values

◦ MV- median value of owner-occupied homes in tract (‘000s)◦ CRIM- per capita crime rates◦ NOX- concentration of nitrogen oxides (pphm)◦ AGE- percent built before 1940◦ DIS- weighted distance to centers of employment◦ RM- average number of rooms per house◦ LSTAT- percent neighborhood ‘lower socio-economic status’◦ RAD- accessibility to radial highways◦ CHAS- borders Charles River (0/1)◦ INDUS- percent non-retail business◦ TAX- tax rate◦ PT- pupil teacher ratio

Boston Housing Dataset

(insert graph) The dataset poses significant challenges to

conventional regression modeling

◦ Clearly departure from normality, non-linear relationships, and skewed distributions

◦ Multicollinearity, mutual dependency, and outlying observations

Scatter Matrix

(insert graph)

A typical MARS solution (univariate for simplicity) is shown above

◦ Essentially a piece-wise linear regression model with the continuity requirement at the transition points called knots

◦ The locations and number of knots were determined automatically to ensure the best possible model fit

◦ The solution can be analytically expressed as conventional regression equations

MARS Model

Finding the one best knot in a simple regression is a straightforward search problem◦ Try a large number of potential knots and choose one with the best R-squared ◦ Computation can be implemented efficiently using update algorithms; entire

regression does not have to be rerun for every possible knot (just update X’X matrices)

Finding k knots simultaneously would require n^k order of computations assuming N observations

To preserve linear problem complexity, multiple knot replacement is implemented in a step-wise manner:◦ Need a forward/backward procedure◦ The forward procedure adds knots sequentially one at a time

The resulting model will have many knots and overfit the training data◦ The backward procedure removes least contributing knots one at a time

This produces a list of models of varying complexity◦ Using appropriate evaluation criterion, identify the optimal model

Resulting model will have approximately correct knot locations

Challenge: Searching for Multiple Knots

(insert graphs)

True conditional mean has two knots at X=30 and X=60, observed data includes additional random error

Best single knot will be at X=45, subsequent best locations are true knots around 30 and 60

The backward elimination step is needed to remove the redundant node at X=45

Example: Flat Top Function

Thinking in terms of knot selection works very well to illustrate splines in one dimension but unwieldy for working with a large number of variables simultaneously◦ Need a concise notation easy to program and extend in multiple

dimensions

◦ Need to support interactions, categorical variables, and missing values

Basis functions (BF) provide analytical machinery to express the knot placement strategy

Basis function is a continuous univariate transform that reduces predictor influence to a smaller range of values controlled by a parameter c (20 in the example below)◦ Direct BF: max(X-c, 0)- the original range is cut below c

◦ Mirror BF: max (c-X, 0)- the original range is cut above c◦ (insert graphs)

Basis Functions

The following model represents a 3-knot univariate solution for the Boston Housing Dataset using two direct and one mirror basis functions

(insert equations)

All three line segments have negative slope even though two coefficients are above zero

(insert graph)

Example: Solution with Three BFs

MARS core technology:◦ Forward step: add basis function pairs one at a time in conventional step-

wise forward manner until the largest model size (specified by the user) is reached Possible collinearity due to redundancy in pairs must be detected and eliminated For categorical predictors define basis functions as indicator variables for all

possible subsets of levels To support interactions, allow cross products between a new candidate pair and

basis functions already present in the model

◦ Backward step: remove basis functions one at a time in conventional step-wise backward manner to obtain a sequence of candidate models

◦ Use test sample or cross-validation to identify the optimal model size

Missing values are treated by constructing missing value indicator (MVI) variables and nesting the basis functions within the corresponding MVIs

Fast update formulae and smart computational shortcuts exist to make the MARS process as fast and efficient as possible

MARS Process

OLS and MARS regression (insert graphs)

We compare the results of classical linear regression and MARS◦ Top three significant predictors are shown for each model

◦ Linear regression provides global insights

◦ MARS regression provides local insights and has superior accuracy

All cut points were automatically discovered by MARS

MARS model can be presented as a linear regression model in the BF space

Boston Housing- Conventional Regression

Nearest Neighbor Classifier

One of the oldest Data Mining tools for classification The method was originally developed by Fix and Hodges (1951) in an

unpublished technical report

Later on it was reproduced by Agrawala (1977), Silverman and Jones (1989)

A review book with many references on the topic is Dasarathy (1991)

Other books that treat the issue:◦ Ripley B.D. 1996. Pattern Recognition and Neural Networks (chapter 6)◦ Hastie T, Tibshirani R and Friedman J. 2001. The Elements of Statistical Learning

Data Mining, Inference and Prediction (chapter 13)

The underlying idea is quite simple: make the predictions by proximity or similarity

Example: we are interested in predicting if a customer will respond to an offer. A NN classifier will do the following:◦ Identify a set of people most similar to the customer- the nearest neighbor◦ Observe what they have done in the past on a similar offer◦ Classify by majority voting: if most of them are responders, predict a responder,

otherwise, predict a non-responder

General Comments

(insert graphs) Consider binary classification problem

Want to classify the new case highlighted in yellow

The circle contains the nearest neighbors (the most similar cases)◦ Number of neighbors= 16◦ Votes for blue class= 13◦ Votes for red class= 3

Classify the new case in the blue class. The estimated probability of belonging to the blue class is 13/16=0.8125

Similarly in this example:◦ Classify the yellow instance in the blue class◦ Classify the green instance in the red class◦ The black point receives three votes from the blue class and another three

from the red one- the resulting classification is indeterminate

Illustration

There are two decisions that should be made in advance before applying the NN classifier◦ The shape of the neighborhood

Answers the question “Who are our nearest neighbors?”◦ The number of neighbors (neighborhood size)

Answers the question “How many neighbors do we want to consider?”

Neighborhood shape amounts to choosing the proximity/distance measure ◦ Manhattan distance◦ Euclidean distance◦ Infinity distance◦ Adaptive distances

Neighborhood size K can vary between 1 and N (the dataset size)◦ K=1-classification is based on the closest case in the dataset◦ K=N-classification is always to the majority class◦ Thus K acts as a smoothing parameter and can be determined by using a test

sample or cross-validation

The neighborhood and the Neighbors

NN advantages◦ Simple to understand and easy to implement◦ The underlying idea is appealing and makes logical sense◦ Available for both classification and regression problems

Predictions determined by averaging the values of nearest neighbors◦ Can produce surprisingly accurate results in a number of applications

NN have been proved to perform equal or better than LDA, CART, Neural Networks and other approaches when applied to remote sensed data

NN disadvantages ◦ Unlike decision trees, LDA, or logistic regression, their decision

boundaries are not easy to describe and interpret◦ No variable selection of any kind- vulnerable to noisy inputs

All the variables have the same weight when computing the distance, so two cases could be considered similar (or dissimilar) due to the role of irrelevant features (masking effects)

◦ Subject to the curse of dimensionality in high dimension datasets◦ The technique is quite time consuming. However, Friedman et. Al.

(1975 and 1977) have proposed fast algorithms

Pros and Cons of NN Classifiers

CART

Classification and Regression Trees (CART®)- original approach based on the “let the data decide local regions” concept developed by Breiman, Friedman, Olshen, and Stone in 1984

The algorithm can be summarized as:◦ For each current data region, consider all possible orthogonal splits (based on

one variable) into 2 sub-regions◦ The best split is defined as the one having the smallest MSE after fitting a

constant in each sub-region (regression) or the smallest resulting class impurity (classification)

◦ Proceed recursively until all structure in the training set has been completely exhausted- largest tree is produced

◦ Create a sequence of nested sub-trees with different amount of localization (tree pruning)

◦ Pick the best tree based on the performance on a test set or cross-validated

One can view CART tree as a set of dynamically constructed orthogonal nearest neighbor boxes of varying sizes guided by the response variable (homogeneity of response within each box)

CART® Overview

CART is best illustrated with a famous example- the UCSD Heart Disease study◦ Given the diagnosis of a heart attack based on

Chest pain, Indicative EKGs, Elevation of enzymes typically released by damaged heart muscle, etc.

◦ Predict who is at risk of a 2nd heart attack and early death within 30 days

◦ Prediction will determine treatment program (intensive care or not)

For each patient about 100 variables were available, including:◦ Demographics, medical history, lab results

◦ 19 noninvasive variables were used in the analysis Age, gender, blood pressure, heart rate, etc.

CART discovered a very useful model utilizing only 3 final variables

Heart Disease Classification Problem

(insert classification tree) Example of a CLASSIFICATION tree

Dependent variable is categorical (SURVIVE, DIE)

The model structure is inherently hierarchical and cannot be represented by an equivalent logistic regression equation

Each terminal node describes a segment in the population

All internal splits are binary

Rules can be extracted to describe each terminal node

Terminal node class assignment is determined by the distribution of the target in the node itself

The tree effectively compresses the decision logic

Typical CART Solution

CART advantages:◦ One of the fastest data mining algorithms available◦ Requires minimal supervision and produces easy to understand

models◦ Focuses on finding interactions and signal discontinuities ◦ Important variables are automatically identified◦ Handles missing values via surrogate splits

A surrogate split is an alternative decision rule supporting the main rule by exploiting local rank-correlation in a node

◦ Invariant to monotone transformations of predictors

CART disadvantages:◦ Model structure is fundamentally different from conventional

modeling paradigms- may confuse reviewers and classical modelers◦ Has limited number of positions to accommodate available

predictors- ineffective at presenting global linear structure (but great for interactions)

◦ Produces coarse-grained piece-wise constant response surfaces

CART®- Pros and Cons

(insert charts) 10-node CART tree was built on the cell phone dataset introduced

earlier

The root Node 1 displays details of TARGET variable in the training data◦ 15.2% of the 830 households accepted the marketing offer

CART tried all variable predictors one at a time and found out that partitioning the set of subjects based on the Handset Price variable is most effective at separating responders from non-responders at this point◦ Those offered the phone with a price>130 contain only 9.9% responders ◦ Those offered a lower price<130 respond at 21.9%

The process of splitting continues recursively until the largest tree is grown

Subsequent tree pruning eliminates least important branches and creates a sequence of nested trees- candidate models

Cell Phone Study: Root Node

(insert charts) The red nodes indicate good responders while the blue nodes indicate

poor responders

Observations with high values on a split variable always go right while those with low values go left

Terminal nodes are numbered left to right and provide the following useful insights◦ Node 1: young prospects having very small phone bill, living in specific cities are

likely to respond to an offer with a cheap handset

◦ Node 5: mature prospects having small phone bill, living in specific cities (opposite Node1) are likely to respond to an offer with a cheap handset

◦ Nodes 6 and 8: prospects with large phone bill are likely to respond as long as the handset is cheap

◦ Node 10: “high-tech” prospects (having a pager) with large phone bill are likely to respond to even offers with expensive handset

Optimal Model

(insert graph, table and chart)

A number of variables were identified as important◦ Note the presence of surrogates not seen on the main tree

diagram previously

Prediction Success table reports classification accuracy on the test sample

Top decile (10% of the population with the highest scores) captures 40% of the responders (lift of 4)

Variable Importance and Predictive Accuracy

(insert graphs)

CART has a powerful mechanism of priors built into the core of the tree building mechanism

Here we report the results of an experiment with prior on responders varying from 0.05 to 0.95 in increments of 0.05

The resulting CART models “sweep” the modeling space enforcing different sensitivity-specificity tradeoff

Sensitivity-Specificity Tradeoff

As prior on the given class decreases The class assignment threshold increases Node richness goes up But class accuracy goes down

PRIORS EQUAL uses the root node class ratio as the class assignment threshold- hence, most favorable conditions to build a tree

PRIORS DATA uses the majority rule as the class assignment threshold- hence, difficult modeling conditions on unbalanced classes.

In reality, a proper combination of priors can be found experimentally

Eventually, when priors are too extreme, CART will refuse to build a tree. ◦ Often the hottest spot is a single node in the tree built with the most

extreme priors with which CART will still build a tree.

◦ Comparing hotspots in successive trees can be informative, particularly in moderately-sized data sets.

General Rules

(insert graph)

We have a mixture of two overlapping classes

The vertical lines show root node splits for different sets of priors. (the left child is classified as red, the right child is classified as blue)

Varying priors provides effective control over the tradeoff between class purity and class accuracy

The Impact of Priors

Hot spots are areas of data very rich in the event of interest, even though they could only cover a small fraction of the targeted group◦ A set of prospects rich in responders

◦ A set of transactions with abnormal amount of fraud

The varying-priors collection of runs introduced above gives perfect raw material in the search of hot spots◦ Simply look at all terminal nodes across all trees in the collection and identify

the highest response segments

◦ Also want to have such segments as large as possible

◦ Once identified, the rules leading to such segments (nodes) are easily available

◦ (insert graph)◦ The graph on the left reports all nodes according to their target coverage and

lift ◦ The blue curve connects the nodes most likely to be a hot spot

Hot Spot Detection

(insert graph) Our next experiment (variable shaving) runs as follows:

◦ Build a CART model with the full set of predictors

◦ Check the variable importance, remove the least important variable and rebuild CART model

◦ Repeat previous step until all variables have been removed

Six-variable model has the best performance so far

Alternative shaving techniques include:◦ Proceed by removing the most important variable- useful in removal of

model “hijackers”- variables looking very strong on the train data but failing on the test data (e.g. ID variables)

◦ Set up nested looping to remove redundant variables from the inner positions on the variable importance list

Improving Feature Selection Process

(insert tree) Many predictive models benefit from Salford Systems patent on

“Structured Trees”

Trees constrained in how they are grown to reflect decision support requirements◦ Variables allowed/disallowed depending on a level in a tree

◦ Variable allowed/disallowed depending on a node size

In mobile phone example: want tree to first segment on customer characteristics and then complete using price variables ◦ Price variables are under the control of the company

◦ Customer characteristics are beyond company control

Constrained Trees

Various areas of research were spawned by CART

We report on some of the most interesting and well developed approaches

Hybrid models◦ Combining CART with linear and Logistic Regression◦ Combining CART with Neural Nets

Linear combination splits

Committees of trees◦ Bagging ◦ Arcing ◦ Random Forest

Stochastic Gradient Boosting (MART a.k.a TreeNet)

Rule Fit and Path Finder

Further Development of CART

Multi-Tree Methods

(insert images) Grow a tree on training data

Find a way to grow another tree, different from currently available (change something in set up)

Repeat many times, say 500 replications

Average results or create voting scheme◦ For example, relate PD to fraction of trees predicting default for a given

Beauty of the method is that every new tree starts with a complete set of data

Any one tree can run out of data, but when that happens we just start again with a new tree and all the data (before sampling)

Multi Tree Methods

Have a training set of size N

Create a new data set of size N by doing sampling with replacement from the training set

The new set (called bootstrap sample) will be different from the original:◦ 36.5% of the original records are excluded ◦ 37.5% of the original records are included once◦ 18% of the original records are included twice◦ 6% of the original records are included three times◦ 2% of the original records are included four or more times

May do this repeatedly to generate numerous bootstrap samples

Example: distribution of record weights in one realized bootstrap sample

(insert table)

Bootstrap Resampling

To generate predicted response, multiple trees are combined via voting (classification) or averaging (regression) schemas

Classification trees “vote”◦ Recall that classification trees classify

Assign each case to ONE class only

◦ With 100 trees, 100 separate class assignment (votes) for each record

◦ Winner is the class with the most votes

◦ Fraction of votes can be used as a crude approximation to class probability

◦ Votes could be weighted- say by accuracy of individual trees or node sizes

◦ Class weights can be introduced to counter the effects of dominant classes

Regression trees assign a real predicted value for each case◦ Predictions are combined via averaging

◦ Results will be much smoother than from a single tree

Voting Schemas

Breiman reports the results of running bootstrap aggregation (bagger) on four publicly available datasets from Statlog project

In all cases the bagger shows substantial improvement in the classification accuracy

It all comes at a price of no longer having a single interpretable model, substantially longer run time and greater demand on model storage space

(insert tables)

Bootstrap Aggregation Performance Gains

Bagging proceeds by independent, identically-distributed sampling draws

Adaptive resampling: probability that a case is sampled varies dynamically◦ Cases with higher current prediction errors have greater probability of being

sampled in the next round◦ Idea is to focus on these cases most difficult to predict correctly

Similar procedure first introduced by Freund & Schapire (1996)

Breiman variant (ARC-x4) is easier to understand:◦ Suppose we have already grown K trees: let m= # times case i was

misclassified (0≤m≤k) (insert equations)◦ Weight=1 for cases with zero occurrences of misclassification◦ Weight= 1+k^4 for cases with K misclassifications

Weigh rapidly becomes large is case is difficult to classify

Adaptive Resampling and Combining (ARCing a Variant of Boosting)

The results of running bagger and ARCer on the Boston Housing Data are reported below

Bagger shows substantial improvement over the single-tree model

ARCer shows marginal improvement over the bagger (insert table)

Single tree now performs worse than stand alone CART run (R-squared=72%) because in bagging we always work with exploratory trees only

Arcing performance beats MARS additive model but is still inferior to the MARS interactions model

Boston Housing Data

Boosting (and Bagging) are very slow and consume a lot of memory, the final models tend to be awkwardly large and unwieldy

Boosting in general is vulnerable to overtraining◦ Much better fit on training than on test data

◦ Tendency to perform poorly on future data

◦ Important to employ additional considerations to reduce overfitting

Boosting is also highly vulnerable to errors in the data◦ Technique designed to obsess over errors

◦ Will keep trying to “learn” patterns to predict miscoded data

◦ Ideally would like to be able to identify miscoded and outlying data and exclude those records from the learning process

◦ Documented in study by Dietterich (1998) An Experimental Comparison of Three Methods for Constructing Ensembles of

Decision Trees, Bagging, Boosting, and Randomization

Problems with Boosting

Random Forest

New approach for many data analytical tasks developed by Leo Breiman of University of California, Berkeley◦ Co-author of CART® with Friedman, Olshen, and Stone

◦ Author of Bagging and Arcing approaches to combining trees

Good for classification and regression problems◦ Also for clustering, density estimation

◦ Outlier and anomaly detection

◦ Explicit missing value imputation

Builds on the notions of committees of experts but is substantially different in key implementation details

Introduction to Random Forest

A random forest is a collection of single trees grown in a special way◦ Each tree is grown on a bootstrap sample from the learning set ◦ A number R is specified (square root by defualt) such that is noticeably

smaller than the total number of available predictors ◦ During tree growing phase, at each node only R predictors are randomly

selected and tried

The overall prediction is determined by voting (in classification) or averaging (in regression)

The law of Large Numbers ensures convergence

The key to accuracy is low correlation and bias

To keep bias low, trees are grown to maximum depth

The Algorithm

Randomness is introduced in two distinct ways

Each tree is grown on a bootstrap sample from the learning set◦ Default bootstrap sample size equals original sample size ◦ Smaller bootstrap sample sizes are sometimes useful

A number R is specified (square root by default) such that it is noticeably smaller than the total number of available predictors

During tree growing phase, at each node only R predictors are randomly selected and tried.

Randomness also reduces the signal to noise ratio in a single tree◦ A low correlation between trees is more important than a high signal when

many trees contribute to forming the model ◦ RandomForests™ trees often have very low signal strength, even when the

signal strength of the forest is high.

Randomness is introduced in order to keep correlation low

(insert graph)

Gold- Average of 50 Base Learners

Blue- Average of 100 Base Learners

Red- Average of 500 Base Learners

Error Correlation, Averaging, and Signal Strength

(insert graph)

Averaging many base learners improves the signal to noise ratio dramatically provided that the correlation of errors is kept low

Hundreds of base learners are needed for the most noticeable effect

Important to Keep Correlation Low

All major advantages of a single tree are automatically preserved

Since each tree is grown on a bootstrap sample, one can ◦ Use out of bag samples to compute an unbiased estimate of the accuracy◦ Use out of bag samples to determine variable importances

There is no overfitting as the number of trees increases

It is possible to compute generalized proximity between any pair of cases

Based on proximities one can◦ Proceed with a target-driven clustering solution◦ Detect outliers◦ Generate informative data views/projections using scaling coordinates ◦ Do missing value imputation

Interesting approaches to expanding the methodology into survival models and the unsupervised learning domain

Performance

RF introduces a novel way to define proximity between two observations:◦ For a dataset of size N define an NXN matrix of proximities

◦ Initialize all proximities to zeroes

◦ For any given tree, apply the tree to the dataset

◦ If case i and case j both end up in the same node, increase proximity proxij between i and j by one

◦ Accumulate over all trees in RF and normalize by twice the number of trees in RF

The resulting matrix provides intrinsic measure of proximity ◦ Observations that are “alike” will have proximities close to one

◦ The closer the proximity to 0, the more dissimilar cases i and j are

◦ The measure is invariant to monotone transformations

◦ The measure is clearly defined for any type of independent variables, including categorical

Proximity Matrix- Raw Material for Further Advances

TreeNet

TreeNet (TN) is a new approach to machine learning and function approximation developed by Jerome H, Friedman at Stanford University◦ Co-author of CART® with Breiman, Olshen and Stone◦ Author of MARS®, PRIM, Projection Pursuit, COSA, RuleFit™ and more

Also known as Stochastic Gradient Boosting and MART (Multiple Additive Regression Trees)

Naturally supports the following classes of predictive models◦ Regression (continuous target, LS and LAD loss functions)◦ Binary Classification (binary target, logistic likelihood loss function)◦ Multinomial classification (multiclass target, multinomial likelihood loss

function)◦ Poisson regression (counting target, Poisson Likelihood loss function)◦ Exponential survival (positive target with censoring)◦ Proportional hazard cox survival model

TN builds on the notions of committees of experts and boosting but is substantially different in key implementation details

Introduction to TreeNet

We focus on TreeNet because: It is the method introduced in the original Stochastic Gradient

Boosting article

It is the method used in many successful real world studies

We have found it to be more accurate than the other methods◦ Many decisions that affect many people are made using a TreeNet model◦ Major new fraud detection engine uses TreeNet◦ David Cossock of Yahoo recently published a paper on uses of TreeNet in web

search

TreeNet is a fully developed methodology. New capabilities include:◦ Graphical display of the impact of any predictor◦ New automated ways to test for existence of interactions◦ New ways to identify and rank interactions◦ Ability to constrain model: allow some interactions and disallow others.◦ Method to recast TreeNet model as a logistic regression.

TreeNet (aka MART)

Built on CART trees and thus◦ Immune to outliers

◦ Selects variables

◦ Results invariant with monotone transformations of variables

◦ Handles missing values automatically

Resistant to mislabeled target data◦ In medicine cases are commonly misdiagnosed

◦ In business, occasionally non-responders flagged as “responders”

Resistant to overtraining- generalizes very well

Can be remarkably accurate with little effort

Trains very rapidly; comparable to CART

Benefits of TreeNet

2007 PAKDD competition: home loans up-sell to credit card owners 2nd place◦ Model built in half a day using previous year submission as a blueprint

2006 PAKDD competition: customer type discrimination 3rd place◦ Model built in one day. 1st place accuracy 81.9% TreeNet accuracy 81.2%

2005 BI-CUP Sponsored by University of Chile attracted 60 competitors

2004 KDDCup “Most Accurate”

2003 “Duke University/NCR Teradata CRN modeling competition◦ Most Accurate and Best Top Decile Lift on both in and out of time samples

A major financial services company has tested TreeNet across a broad range of targeted marketing and risk models for the past two years◦ TreeNet consistently outperforms previous best models (around 10%

AUROC)◦ TreeNet models can be built in a fraction of the time previously devoted◦ TreeNet reveals previously undetected predictive power in data

TN Successes

Begin with one very small tree as initial model◦ Could be as small as ONE split generating 2 terminal nodes

◦ Typical model will have 3-5 splits in a tree, generating 4-6 terminal nodes

◦ Output is a continuous response surface regardless of the target type Hence, Probability modeling type for classification

◦ Model is intentionally “weak”- shrink all model predictions towards zero by multiplying all predictions by a small positive learn rate

Compute “residuals” for this simple model (prediction error) for every record in data◦ The actual definition of the residual in this case is driven by the type of the loss

function

Grow second small tree to predict the residuals from first tree

Continue adding more and more trees until a reasonable amount has been added ◦ It is important to monitor accuracy on an independent test sample

The Algorithm

(insert chart)

TreeNet: Sample Model Sequence

Trees are kept small (2-6 nodes common)

Updates are small- can be as small as .01,.001,.0001

Use random subsets of the training data in each cycle ◦ Never train on all the training data in any one cycle

Highly problematic cases are IGNORED◦ If model prediction starts to diverge substantially from observed data, that

data will not be used in further updates

TN allows very flexible control over interactions:◦ Strictly Additive Models (no interactions allowed)

◦ Low level interactions allowed

◦ High level interactions allowed

◦ Constraints: only specific interactions allowed (TN PRO)

Key Points

As TN models consist of hundreds or even thousands of trees there is no useful way to represent the model via a display of one or two trees

However, the model can be summarized in a variety of ways◦ Partial Dependency Plots: These exhibit the relationship between the

target and any predictor- as captured by the model◦ Variable Importance Rankings: These stable rankings give an excellent

assessment of the relative importance of predictors◦ ROC and Gains Curves: TN Models produce scores that are typically

unique for each scored record◦ Confusion Matrix: Using an adjustable score threshold this matrix displays

the model false positive and false negative rates

TreeNet models based on 2-node trees by definition EXCLUDE interactions ◦ Model may be highly nonlinear but is by definition strictly additive ◦ Every term in the model is based on a single variable (single split)

Build TreeNet on a larger tree (default is 6 nodes)◦ Permits up to 5-way interaction but in practice is more like 3-way interaction

Can conduct informal likelihood ratio test TN(2-node) versus TN(6-node) Large differences signal important interactions

Interpreting TN Models

(insert graphs)

The results of running TN on the Boston Housing Database are shown

All of the key insights agree with previous findings by MARS and CART

Example: Boston Housing

Slope reverses due to interaction

Note that the dominant pattern is downward sloping, but that a key segment defined by the 3rd variable is upward sloping

(insert graph)

Example of an Important Interaction

Algorithm Comparison

CART: Model is one optimized Tree◦ Model is easy to interpret as rules

Can be useful for data exploration, prior to attempting a more complex model◦ Model can be applied quickly with a variety of workers:

A series of questions for phone bank operators to detect fraudulent purchases Rapid triage in hospital emergency rooms

◦ In some cases may produce the best or the most predictive model, for example in classification with a barely detectable signal

◦ Missing values handled easily and naturally. Can be deployed effectively even when new data have a different missingness pattern

Random Forests: combination of many LARGE trees◦ Unique nonparametric distance metric that works in high dimensional spaces◦ Often predicts well when other models work poorly, e.g. data with high level

interactions◦ In the most difficult data sets can be the best way to identify important variables

Tree Net: combination of MANY small trees◦ Best overall forecast performance in many cases◦ Constrained models can be used to test the complexity of the data structure non-

parametrically ◦ Exceptionally good with binary targets

Advantages of Various Methods-1

Neural Networks, combination of a few sigmoidal activation functions ◦ Very complex models can be represented in a very compact form

◦ Can accurately forecast both levels and slopes and even higher order derivatives

◦ Can efficiently use vector dependent variables Cross equation constraints can be imposed. (see Symmetry constraints for

feedforward network models of gradient systems, Cardell, Joerding, and Li, IEEE Transactions on Neural Networks, 1993)

◦ During deployment phase, forecasts can be computed very quickly High voltage transmission lines use a neural network to detect whether there has

been a lightning strike and are fast enough to shut down the line before it can be damaged

Kernel function estimators, use a local mean or a local regression◦ Local estimates easy to understand and interpret ◦ Local regression versions can estimate slopes and levels◦ Initial estimation can be quick

Advantages of Various Methods-2

Random Forests:◦ Models are large, complex and un-interpretable◦ Limited to moderate sample sizes (usually less than 100,000

observations)◦ Hard to tell in advance which case Random Forests will work well on ◦ Deployed models require substantial computation

Tree Net◦ Models are large and complex, interpretation requires additional

work◦ Deployed models either require substantial computation or post-

processing of the original model into a more compact form

CART◦ In most cases models are less accurate than TreeNet◦ Works poorly in cases where effects are approximately linear in

continuous variables or additive over many variables

Disadvantages of Various Methods-1

Neural Networks: ◦ Neural Networks cover such a wide variety of models that no good widely-

applicable modeling software exists or may even be possible The most dramatic successes have been with Neural Network models that are

idiosyncratic to the specific case, and ere developed with great effort Fully optimized Neural Network parameter estimates can be very difficult to compute,

and sometimes perform substantially worse than initial statistically inferior estimates. (this is called the “over training” issue)

◦ In almost all cases initial estimation is very compute intensive

◦ Limited to very small numbers of variables (typically between about 6 and 20 depending on the application)

Kernel Function Estimators:◦ Deployed models can require substantial computation

◦ Limited to small numbers of variables

◦ Sensitive to distance measures. Even a modest number of variables can degrade performance substantially, due to the influence of relatively unimportant variables on the distance metric

Disadvantages of Various Methods-2

Breiman, L., J. Friedman, R. Olshen and C. Stone (1984), Classification and Regression Trees, Pacific Grove: Wadsworth

Breiman, L. (1996). Bagging predictors. Machine Learning, 24, 123-140.

Hastie, T., Tibshirani, R., and Friedman, J.H (2000). The Elements of Statistical Learning. Springer.

Freund, Y. & Schapire, R. E. (1996). Experiments with a new boosting algorithm. In L. Saitta, ed., Machine Learning: Proceedings of the Thirteenth National Conference, Morgan Kaufmann, pp. 148-156.

Friedman, J.H. (1999). Stochastic gradient boosting. Stanford: Statistics Department, Stanford University.

Friedman, J.H. (1999). Greedy function approximation: a gradient boosting machine. Stanford: Statistics Department, Stanford University.

References