a comparison of feature-based and feature-free case-based reasoning for spam filtering
DESCRIPTION
A Comparison of Feature-Based and Feature-Free Case-Based Reasoning for Spam Filtering. Derek Bridge University College Cork work done with Sarah Jane Delany Dublin Institute of Technology. Overview. Introduction Case-Based Spam Filtering Feature-Based Feature-Free Experiments I - PowerPoint PPT PresentationTRANSCRIPT
A Comparison of Feature-Based and Feature-Free Case-Based Reasoning
for Spam Filtering
Derek BridgeUniversity College Cork
work done with
Sarah Jane DelanyDublin Institute of Technology
Overview
• Introduction• Case-Based Spam Filtering
– Feature-Based– Feature-Free– Experiments I
• Case Base Maintenance– Competence-Based Editing– Experiments II
• Concept Drift– Incremental & periodic solutions– Experiments III
• Conclusions
Introduction
• From the Spamhaus project (www.spamhaus.org)
– “An electronic message is ‘spam’ IF:1) the recipient's personal identity and context are
irrelevant because the message is equally applicable to many other potential recipients;
AND2) the recipient has not verifiably granted deliberate,
explicit, and still-revocable permission for it to be sent.”
• “[It’s] about consent, not content”
• We focus on email spam
Spam Filtering
• Spam filtering is classification:– is an incoming email ham or spam?
• Spam filters– procedural
• whitelists, blacklists, challenge-response systems,…
– collaborative • sharing signatures
– content-based • rules, decision trees, probabilities, case bases,…
– hybrid.
Challenges of Spam Filtering• Spam is subjective and personal;• It is heterogeneous;• There is a high costs to false
positives (where ham is classified as spam); and
• It is constantly changing (‘concept drift’).
Overview
• IntroductionCase-Based Spam Filtering
– Feature-Based– Feature-Free– Experiments I
• Case Base Maintenance– Competence-Based Editing– Experiments II
• Concept Drift– Incremental & periodic solutions– Experiments III
• Conclusions
Case-Based Reasoning
Generalknowledge
Tested/RepairedCase
AdaptedCase
LearnedCase
RetrievedCase
Newproblem
PreviousCase
RETRIEVE
REVISE
RETAIN REUSE
[Aamodt & Plaza 1994]
Case-Based Reasoning
Generalknowledge
PreviousCase
MAINTAIN
Is Case-Based Reasoning (CBR) the answer?• Spam is subjective and personal;• It is heterogeneous;• There is a high costs to false
positives (where ham is classified as spam); and
• It is constantly changing (‘concept drift’). Users can have
individual case bases created from their own
emails
It is known that CBR handles disjunctive
concepts well
We can bias CBR away from false positivesCase bases can be
updated incrementally
Overview
• Introduction• Case-Based Spam Filtering
Feature-Based– Feature-Free– Experiments I
• Case Base Maintenance– Competence-Based Editing– Experiments II
• Concept Drift– Incremental & periodic solutions– Experiments III
• Conclusions
Email Classification Using Examples (ECUE)• ECUE uses Case-Based Reasoning (CBR) to
classify emails • A case base contains a user’s email (both
ham and spam)• ECUE classifies an incoming email using the
k-nearest neighbour algorithm:– It retrieves from the case base the k nearest
neighbours (the k that are closest or most similar)
– The cases it retrieves then vote to decide the class of the new email
– To bias away from false positives, ECUE uses unanimous voting.
Feature-Based ECUE
• Features extracted (fij )– words, characters, structural features
• Binary representation: fi1= 1 or fi1= 0
EmailEmailEmailEmail FeatureExtraction
Casebase
label class,,..., 21 iNiii fffe
Feature-Based ECUE
• Information Gain used to select the 700 most predictive features
EmailEmailEmailEmail FeatureExtraction
Casebase
FeatureSelection
Casebase
Feature-Based ECUE
EmailEmailEmailEmail FeatureExtraction
Casebase
FeatureSelection
Casebase
CaseSelection
Casebase
• Competence-Based Editing usedto edit case base
Runtime System
Feature-Based ECUE
EmailEmailEmailEmail FeatureExtraction
Casebase
FeatureSelection
Casebase
CaseSelection
Casebase
Classification
spam!
NewCase
Feature-Based ECUE
• The distance between cases is a count of the number of features that they do not share
• Naïve Bayes classifier thought to be among the best for spam filtering
• Feature-Based ECUE has comparable, and sometimes slightly better, accuracy than Naïve Bayes
Overview
• Introduction• Case-Based Spam Filtering
– Feature-BasedFeature-Free– Experiments I
• Case Base Maintenance– Competence-Based Editing– Experiments II
• Concept Drift– Incremental & periodic solutions– Experiments III
• Conclusions
Feature-Free ECUE
• Alternative to Feature-Based ECUE• Inspired by theory of Kolmogorov
Complexity– K(x) = size of smallest Turing machine
that can output x to its tape– K(x|y) = size of smallest Turing machine
that can output x when given y• Basis for distance measure
if K(x|y) < K(x|z) then y is more similar to x than z
[Li et al. 2003]
Feature-Free ECUE
• Approximate K(x) by C(x)C(x) = size of x after compression
• Text compression exploits intra-document redundancy
Case based reasoningCase b•d reasoning
Using Compression
• Consider length of two documents allowing for inter-document redundancy = len(gzip( + ))docX docY
= len(gzip( ))docX docY
= len( )docX docY
= C(xy)
Using Compression
• Consider length of two documents not allowing for inter-document redundancy
= len(gzip( )) + len(gzip( ))docX docY
= len( ) + len( )docX docY
= C(x) + C(y)
Compression-Based Dissimilarity (CDM)
• Max value ≤ 1 (furthest)Min value > 0.5 (nearest)
• HoweverCDM(x,x) ≠ 0; CDM(x,y) ≠ CDM(y,x); CDM(x,y) + CDM(y,z) ≥ CDM(x,z)
)()(
)(),(
yCxC
xyCyxCDM
[Keogh et al 2004]
Runtime System
Feature-Based ECUE
EmailEmailEmailEmail FeatureExtraction
Casebase
FeatureSelection
Casebase
Case BaseEdit
Casebase
Classification
spam!
NewEmail
Runtime System
Feature-Free ECUE
EmailEmailEmailEmailEmail
Casebase
Case BaseEdit
Casebase
Classification
spam!
NewEmail
EmailEmailEmail
Experiments I• Created 4 datasets of 1000 emails from
two years of email from two people– each dataset has 500 consecutive ham, 500
consecutive spam• 10-fold cross-validation • Settings:
– k = 3– Feature-based: 700 features– Feature-free: GZip as text compressor
• Measures:– FPRate = #false positives/#ham– FNRate = #false negatives/#spam– Err = (FPRate + FNRate) / 2
Results - % Error
5.7%
2.4%
4.0%
0.2%
9.8%
2.2%
13.2%
1.5%
Feature-Based Feature-Free (GZip)
Dataset 1
Dataset 2
Dataset 3
Dataset 4
Results - % False Positives
9.2%
1.4%1.4%
0.0%
1.0% 0.8%0.6%1.2%
Feature-Based Feature-Free (GZip)
Dataset 1
Dataset 2Dataset 3
Dataset 4
Overview
• Case-Based Spam Filtering– Feature-Based & Feature-Free– Experiments I
Case Base Maintenance– Competence-Based Editing– Experiments II
• Concept Drift– Incremental & periodic solutions– Experiments III
• Conclusions
Case Base Maintenance
• Case base editing algorithms– remove redundant cases, and– remove noisy cases.
• Their goal is to– reduce retrieval time but– maintain or even improve accuracy.
Competence Model
• For each case c, compute– coverage set of c
• cases that have c as one of their k-NN and which have same class as c
– liability set of c• cases that have c as one of their k-NN and
which have different class from c
xc x is in coverage set of c
y
y is in liability set of c
Competence-Based Editing
• Blame-Based Noise Reduction– For each case c with non-empty liability set
(taken in descending order of size of liability set),• if the cases in c’s coverage set can still be correctly
classified without c, then c can be deleted.
– This emphasises removal of cases that cause misclassifications.
• Conservative Redundancy Reduction– For each remaining case c (taken in ascending
order of size of coverage set)• retain c but delete the cases in c’s coverage set
– This retains cases close to class boundaries
Results - % Error
5.7%
3.8%2.4% 2.2%
9.8%
7.0%
2.2% 2.6%
Feature-Based (full)
Feature-Based
(edited)
Feature-Free(full)
Feature-Free(edited)
Dataset 1 Dataset 3
• Feature-based edited size = 75% and 65%• Feature-free edited size = 59% and 57%
Results - % False Positives
9.2%
3.4%
1.4% 1.0%1.0%2.2%
0.8% 0.4%
Feature-Based (full)
Feature-Based
(edited)
Feature-Free(full)
Feature-Free(edited)
Dataset 1 Dataset 3
Overview
• Case-Based Spam Filtering– Feature-Based & Feature-Free– Experiments I
• Case Base Maintenance– Competence-Based Editing– Experiments II
Concept Drift– Incremental & periodic solutions– Experiments III
• Conclusions
Concept Drift• The target concept is not static
– it changes according to season– it changes according to world events– people’s interests and tolerances
change– there is an arm’s race:
• ever more devious spamouflage!
• We need to investigate behaviour over time
Experiments III• Took ~10000 emails from two years of
email from two people in date-order• Created a case base for each person from
earliest 500 consecutive ham & earliest 500 consecutive spam
• Remaining ~9000 emails presented chronologically as test cases
• Same settings and measures as before– k = 3– Feature-based: 700 features– Feature-free: GZip as text compressor
Retention policies• CBR (and other lazy learners) can
easily incorporate the most recent examples– retain-all: store all new emails in the
case base– retain-misclassifieds: store a new email
if our prediction is wrong
Results - % Error
15.9%
2.3%
12.6%
3.2%
Feature-Free (GZip) Feature-Free (GZip):retain-misclassifieds
Dataset A Dataset B
• When we retain-misclassified cases, case bases increase in size by ~30%
Results - % False Positives
0.7%
1.5%
4.0%3.5%
Feature-Free (GZip) Feature-Free (GZip):retain-misclassifieds
Dataset A Dataset B
Retention• Bigger case base reduces efficiency• Obsolete cases may reduce accuracy• Obsolete features may reduce
accuracy
• Need a deletion policy
Incremental Solutions
• Consider add-1-delete-1– Case base size remains constant– retention policy
• retain-all• retain-misclassified
– forgetting policy• forget-oldest• forget-least-accurate
instance selection
instance weighting
Incremental Solutions
• Consider add-1-delete-1– Case base size remains constant– retention policy
• retain-all• retain-misclassified
– forgetting policy• forget-oldest• forget-least-accurate
Accuracy = #successes#retrievals
Results - % Error
15.9%
2.3% 1.7% 1.8% 1.9%
12.6%
3.2% 2.8%4.0% 3.0%
Feature-Free Feature-Free:retain-
misclassifieds,forget-oldest
Feature-Free:retain-all, forget-
oldest
Feature-Free:retain-
misclassifieds,forget-least-
accurate
Feature-Free:retain-all, forget-least-accurate
Dataset A Dataset B
Results - % False Positives
0.7%1.3% 1.7% 1.8%
2.4%
4.0%3.5%
4.2%
6.4%
5.0%
Feature-Free Feature-Free:retain-
misclassifieds,forget-oldest
Feature-Free:retain-all, forget-
oldest
Feature-Free:retain-
misclassifieds,forget-least-
accurate
Feature-Free:retain-all, forget-least-accurate
Dataset A Dataset B
Negative effect on FPs?
Periodic Solutions
• Periodic– Feature-based:
• retain-misclassified;• monthly, feature re-extraction, feature re-
selection, case base rebuild and case base edit
– Feature-free• retain-misclassified; • monthly, case base edit
Feature-Based ECUE
EmailEmailEmailEmail FeatureExtraction
Casebase
FeatureSelection
Casebase
Case BaseEdit
Casebase
Results - % Error
15.4%
4.5%
15.9%
2.3%
19.2%
6.1%
12.6%
2.6%
Feature-Based Feature-Based: retain-misclassifieds,
monthly reselect &edit
Feature-Free Feature-Free: retain-misclassifieds,
monthly edit
Dataset A Dataset B
Results - % False Positives
20.0%
2.0% 0.7% 0.9%
14.7%
2.4%4.0%
2.5%
Feature-Based Feature-Based: retain-misclassifieds,
monthly reselect &edit
Feature-Free Feature-Free: retain-misclassifieds,
monthly edit
Dataset A Dataset B
Overview
• Case-Based Spam Filtering– Feature-Based & Feature-Free– Experiments I
• Case Base Maintenance– Competence-Based Editing– Experiments II
• Concept Drift– Incremental & periodic solutions– Experiments III
• Conclusions
Feature-Free ECUE: Advantages• Accuracy
– lower error rate than traditional feature-based methods
– often lower false positive rate
• Costs– it uses the raw text– no need to extract, select or weight features– no need to update features as spam changes
• Concept drift– simple retention/forgetting policies can be
effective
Feature-Free ECUE: Disadvantages
• No justification factors to explain results or drive adaptation
• Higher computation time– Time to classify email (with cb of 1000)
Feature-free = 2 secs Feature-based = .01 sec
• Not a metric
Future Work
• Investigating algorithms to speed up retrieval time
• Application of measure to text other than emails
Thank you for your attention!
Spare slides
Normalized Compression Distance (NCD)
• Max value = 1 + ε (furthest)Min value = 0 (nearest)
• HoweverNCD(x,x) ≠ 0; NCD(x,y) ≠ NCD(y,x); NCD(x,y) + NCD(y,z) ≥ NCD(x,z)
))(),(max())(),(min()(
),(yCxC
yCxCxyCyxNCD
[Li et al 2003]
Comparing Compression Algorithms
• The better the compression the better the measure?
• Compared GZip with Prediction by Partial Matching (PPM)– GZip = Lempel-Ziv variant– PPM = adaptive statistical compressor
Results - % Error
2.4% 2.3%2.1% 2.0%
0.1% 0.2% 0.2% 0.2%
2.4%
1.9%2.2%
2.5%
1.4%1.1%
1.6% 1.7%
GZip PPM(2) PPM(4) PPM(8)
Dataset 1
Dataset 2Dataset 3
Dataset 4
Results
• Little difference in classification error– Compressor choice does not greatly
matter
• PPM is generally considered better at compression but on our datasets...– average of 59% compression for GZip– average 57% compression for PPM
• PPM computationally expensive– 180 times slower than GZip
GZip Speed Up
• GZip uses a 32 KByte sliding window
• Truncate each email to 16KB • Achieves speed ups of between 9.5%
to 25%
docX docY
32KB