On Sample Selection Bias and Its Efficient Correction via
Model Averaging and Unlabeled Examples
Wei Fan
Ian Davidson
A Toy Example
Two classes:red and green
red: f2>f1green: f2<=f1
Unbiased and Biased Samples
Not so-biased sampling Biased sampling
Effect on Learning
Unbiased 97.1% Biased 92.1% Unbiased 96.9% Biased 95.9%Unbiased 96.405% Biased 92.7% Some techniques are more sensitive to bias than others.
One important question: How to reduce the effect of sample selection
bias?
Ubiquitous
Loan Approval Drug screening Weather forecasting Ad Campaign Fraud Detection User Profiling Biomedical Informatics Intrusion Detection Insurance etc
1. Normally, banks only have data of their own customers2. “Late payment, default” models are computed using their own data3. New customers may not completely follow the same distribution.4. Is the New Century “sub-prime mortgage” bankcrupcy due to bad modeling?
Bias as Distribution Think of “sampling an example (x,y) into the training data” as
an event denoted by random variable s s=1: example (x,y) is sampled into the training data s=0: example (x,y) is not sampled.
Think of bias as a conditional probability of “s=1” dependent on x and y
P(s=1|x,y) : the probability for (x,y) to be sampled into the training data, conditional on the example’s feature vector x and class label y.
Categorization
From Zadrozny’04 No Sample Selection
Bias P(s=1|x,y) = P(s=1)
Feature Bias P(s=1|x,y) = P(s=1|x)
Class Bias P(s=1|x,y) = P(s=1|y)
Complete Bias No more reduction
Alternatively, consider D of the size can be sampled “exhaustively” from the universe of examples.
Bias for a Training Set
How P(s=1|x,y) is computed Practically, for a given training set D
P(s=1|x,y) = 1: if (x,y) is sampled into D P(s=1|x,y) = 0: otherwise
Realistic Datasets are biased?
Most datasets are biased. Unlikely to sample each and every feature
vector. For most problems, it is at least feature bias.
P(s=1|x,y) = P(s=1|x)
Effect on Learning
Learning algorithms estimate the “true conditional probability” True probability P(y|x), such as P(fraud|x)? Estimated probabilty P(y|x,M): M is the model built.
Conditional probability in the biased data. P(y|x,s=1)
Key Issue: P(y|x,s=1) = P(y|x) ? At least for those sampled examples.
Appropriate Assumptions More “good training examples” in “feature bias” than both
“class bias” and “complete bias”. “good”: P(y|x,s=1) = P(y|x) beware: it is “incorrect” to conclude that P(y|x,s=1) = P(y|x)
unless under some restricted situations that can rarely happen. For class bias and complete bias, it is hard to derive anything. It is hard to make any more detailed claims without knowing
more about Both the sampling process The true function.
Categorizing into the exact type is difficult. You don’t know what you don’t know.
Not that bad, since the key issue is the number of examples with “bad” conditional probability. Small Large
“Small” Solutions
Posterior weighting
Class Probability
Integration Over Model Space
Averaging of estimated class probabilities weighted by posterior
Removes model uncertainty by averaging
Prove that the expected error of model averaging is less than any single model combined.
What this says: Compute many models in different ways Don’t hang on one tree
“Large” Solutions
When too many base models’s estimates are off track, the power of model averaging will be limited.
In this case, we need to smartly use “unlabeled example” that are unbiased.
Reasonable assumption: unlabeled examples are usually plenty and easier to get.
How to Use Them
Estimate “joint probability” P(x,y) instead of just conditional probability, i.e., P(x,y) = P(y|x)P(x) Makes no difference use 1 model, but
Multiple models
Examples of How This Works
P1(+|x) = 0.8 and P2(+|x) = 0.4
P1(-|x) = 0.2 and P2(+|x) = 0.6 model averaging,
P(+|x) = (0.8 + 0.4) / 2 = 0.6 P(-|x) = (0.2 + 0.6)/2 = 0.4 Prediction will be –
But if there are two P(x) models, with probability 0.05 and 0.4
Then P(+,x) = 0.05 * 0.8 + 0.4 * 0.4 = 0.2 P(-,x) = 0.05 * 0.2 + 0.4 * 0.6 = 0.25
Recall with model averaging: P(+|x) = 0.6 and P(-|x)=0.4 Prediction is +
But, now the prediction will be – instead of + Key Idea:
Unlabeled examples can be used as “weights” to re-weight the models.
Improve P(y|x)
Use a semi-supervised discriminant learning procedure (Vittaut et al, 2002)
Basic procedure: Use learned models to predict unlabeled
examples. Use a random sample of “predicted” unlabeled
examples to combine with labeled training data Re-train the model Repeat until the predictions on unlabeled
examples remain stable.
Experiments
Feature Bias Generation
Sort the according to feature values “chop off” the top.
Class Bins
Randomly generate prior class probability distribution P(y). Just the number, such as P(+)=0.1 and P(-)=0.9
Sample without replacement from “class bins”
Class Bias Generation
+ -+-
Complete Bias Generation
Recall: the probability to sample an example (x,y) is dependent on both x and y.
Easiest simulation: Sample (x,y) without replacement from the
training data.
Feature Bias
0
10
20
30
40
50
60
70
80
90
100
Single CA sCA JA sJA Jax sJAx
Adult
SJ
SS
Pendig
ArtiChar
CAR4
Datasets
Adult: 2 classes SJ: 3 classes SS: 3 classes Pendig: 10 classes ArtiChar: 10 classes Query: 4 classes Donation: 2 classes, cost-sensitive Credit Card: 2 classes, cost-sensitive
Winners and Losers
Single model *never wins* Under feature bias winners:
model averaging *with or without* improved conditional probability using unlabeled examples
Joint probability averaging with *uncorrelated* P(y|x) and P(x) models (details in paper)
Under class bias winners: Joint probability averaging with *correlated* P(y|x) and
*improved* P(x) models.
Under complete bias: Model averaging with improved P(y|x)
Summary According to our definition, sample selection bias is ubiquitous. Categorization of sample selection bias into 4 types is useful for
analysis, but hard to use in practice. In practice, the key question is: relative number of examples with
inaccurate P(y|x). Small: use model averaging of conditional probabilities of several
models Medium: use model averaging of improved conditional
probabilities Large: use joint probability averaging of uncorrelated conditional
probability and feature probability
When the number is small Prove in the paper that the expected error of
model averaging is less than any single model combined.
What this says: Compute model in different ways Don’t hang yourself on one tree