special topics on text mining [ part i: text classification ]
DESCRIPTION
Special topics on text mining [ Part I: text classification ]. Hugo Jair Escalante , Aurelio Lopez , Manuel Montes and Luis Villaseñor. Classification algorithms and evaluation. Hugo Jair Escalante , Aurelio Lopez, Manuel Montes and Luis Villaseñor. Text classification. - PowerPoint PPT PresentationTRANSCRIPT
Special topics on text mining[Part I: text classification]
Hugo Jair Escalante, Aurelio Lopez, Manuel Montes and Luis Villaseñor
Classification algorithms and evaluation
Hugo Jair Escalante, Aurelio Lopez, Manuel Montes and Luis Villaseñor
Text classification
• Machine learning approach to TC: Recipe
1. Gather labeled documents2. Construction of a classifier
A. Document representationB. Preprocessing C. Dimensionality reductionD. Classification methods
3. Evaluation of a TC method
Machine learning approach to TC
• Develop automated methods able to classify documents with a certain degree of success
?Training documents
(Labeled)Learning machine
(an algorithm)
Trained machine
Unseen (test, query) document
Labeled document
Conventions
X={xij}
n
mxi
y ={yj}
a
wSlide taken from I. Guyon. Feature and Model Selection. Machine Learning Summer School, Ile de Re, France, 2008.
What is a learning algorithm?
• A function:
• Given:
: df C {1,..., }C K
1,...,{( , )}i i ND y x
;di iy C x
Classification algorithms
• Popular classification algorithms for TC are:– Naïve Bayes
• Probabilistic approach– K-Nearest Neighbors
• Example-based approach– Centroid-based classification
• Prototype-based approach– Support Vector Machines
• Kernel-based approach
Other popular classification algorithms
• Linear classifiers (including SVMs)
• Decision trees
• Boosting, bagging and ensembles in general
• Random forest
• Neural networks
Naïve Bayes• It is the simplest probabilistic classifier used to
classify documents– Based on the application of the Bayes theorem
• Builds a generative model that approximates how data is produced– Uses prior probability of each category given no
information about an item– Categorization produces a posterior probability
distribution over the possible categories given a description of an item.
Sec.13.2
A. M. Kibriya, E. Frank, B. Pfahringer, G. Holmes. Multinomial Naive Bayes for Text Categorization Revisited. Australian Conference on Artificial Intelligence 2004: 488-499
Naïve Bayes
• Bayes theorem:
• Why?– We know that:
– Then
– Then
( | ) ( )( | )( )
P B A P AP A BP B
( , ) ( | ) ( );P A B P A B P B ( , ) ( | ) ( )P A B P B A P A
( | ) ( ) ( | ) ( )P A B P B P B A P A
( | ) ( )( | )( )
P B A P AP A BP B
Naïve Bayes• For a document d and a class cj
( | ) ( )( | )
( )j j
j
P C P CP C
P
dd
d
Sec.13.2
1 | |
1 | |
( ,..., | ) ( )( ,..., )V j j
V
P t t C P CP t t
| |1 ( | ) ( )
( )
Vi i j jP t C P C
P
d
• Assuming terms are independent of each other given the class (naïve assumption)
| |1 ( | ) ( )V
i i j jP t C P C• Assuming each document is equally probable
t1
C
t2 t|V|. . . .
Bayes’ Rule for text classification
• For a document d and a class cj
Sec.13.2
| |
1
V
j j i ji
P C P C P t C
d
Bayes’ Rule for text classification
• For a document d and a class cj
• Estimation of probabilities
Sec.13.2
| |
1
V
j j i ji
P C P C P t C
d
| |
cj
j
NP C
D | |
1
1
| |
iji j V
kjk
NP t c
V N
Prior probability of class cj Probability of occurrence of word ti in class cj
Smoothing to avoid overfitting
Naïve Bayes classifier• Assignment of the class:
• Assignment using underflow prevention:– Multiplying lots of probabilities can result in floating-
point underflow– Since log(xy) = log(x) + log(y), it is better to perform all
computations by summing logs of probabilities rather than multiplying probabilities
| |
C C 1
arg max arg maxj j
V
j j i jC C i
class P C P C P t C
d
| |
C 1
argmax log log |j
V
j i jC i
class P C P t C
Comments on NB classifier• Very simple classifier which works very well on numerical and
textual data.
• Very easy to implement and computationally cheap when compared to other classification algorithms.
• One of its major limitations is that it performs very poorly when features are highly correlated.
• Concerning text classification, it fails to consider the frequency of word occurrences in the feature vector.
Naïve Bayes revisited
• For a document d and a class cj
• Estimation of probabilities
Sec.13.2
| |
1
V
j j i ji
P C P C P t C
d
| |
cj
j
NP C
D | |
1
1
| |
iji j V
kjk
NP t c
V N
Prior probability of class cj Probability of occurrence of word ti in class cj
What is the assumed probability distribution?
Bernoulli event model
• A document is a binary vector over the space of words:
• where B is a multivariate Bernoulli random variable of length |V| associated to document
| |
1
| (1 ) 1V
j i i j i i ji
P C B P t C B P t C
d
{0,1}iB A. McCallum, K. Nigam. A comparison of Event Models for Naïve Bayes Text Classification. Proceedings of the AAAI/ICML Workshop on Learning for Text Categorization, pp. 41—48, 1998
Bernoulli event model
• Estimation of probabilities:
• Problems with this formulation?– Word frequency occurrence is not taken into
account
| |
1
1
| |
iji j V
kjk
NP t C
V N
| |
cj
j
NP C
D
A. McCallum, K. Nigam. A comparison of Event Models for Naïve Bayes Text Classification. Proceedings of the AAAI/ICML Workshop on Learning for Text Categorization, pp. 41—48, 1998
Multinomial event model
• The multinomial model captures word frequency information in documents
• A document is an ordered sequence of word events drawn from the same vocabulary
• Each document is drawn from a multinomial distribution of words with as many independent trials as the length of the document
A. McCallum, K. Nigam. A comparison of Event Models for Naïve Bayes Text Classification. Proceedings of the AAAI/ICML Workshop on Learning for Text Categorization, pp. 41—48, 1998
Multinomial event model• What is a multinomial distribution?
1 21 1 2
1
( ,..., , ) ...,...,
kxx xk k
k
nf x x n p p p
x x
If a given trial can result in the k outcomes E1, …, Ek with probabilities p1, …, pk, then the probability distribution of the RVs X1, …, Xk, representing the number of occurrences for E1, …, Ek in n independent trials is:
# of ways in which the sequence E1, …, Ek can occur
Probability that event Ekoccurs
# times event Ek occur
R. E. Walpole, et al. Probability and Statistics for Engineers and Scientists. 8th Edition, Prentice Hall, 2007.
1 1
!,..., !,..., !k k
n nx x x x
Multinomial event model
• A document is a multinomial experiment with |d| independent trials
A. McCallum, K. Nigam. A comparison of Event Models for Naïve Bayes Text Classification. Proceedings of the AAAI/ICML Workshop on Learning for Text Categorization, pp. 41—48, 1998
| |
1
| (| |) | | !!
diN
Vi j
j di i
P t CP C P
N
d d d
:d
iN # occurrences of term ti in document d
Multinomial event model
• Estimation of probabilities:
• Then, what to do with real valued data?
I. Guyon. Naïve Bayes Algorithm in CLOP. CLOP documentation, 2005.
| |
1| || |
1 1
1
| |
cg
ch
Ddi
gi j DV
dk
k h
NP t C
V N
| |
cj
j
NP C
D
2,
| | 1/ 2( )
1
|
ji i
i j
tV
ji
P C e
dAssume a
probability density function (e.g., a Gaussian pdf)
KNN: K-nearest neighbors classifier
• Do not build explicit declarative representations of categories.– This kind of methods are called lazy learners
• “Training” for such classifiers consists of simply storing the representations of the training documents together with their category labels.
• To decide whether a document d belongs to the category c, kNN checks whether the k training documents most similar to d belong to c.– Key element: a definition of “similarity” between documents
KNN – the algorithm• Given a new document d:
1. Find the k most similar documents from the training set.
• Common similarity measures are the cosine similarity and the Dice coefficient.
2. Assign the class to d by considering the classes of its k nearest neighbors
• Majority voting scheme• Weighted-sum voting scheme
Common similarity measures
• Dice coefficient
• Cosine measure
wki indicates the weight of word k in document i
m
k kjmk ki
nk kjki
ji wwww
dds1
21
212
,
m
k kjm
k ki
n
k kjkiji
ww
wwdds
12
12
1,
2 | |,| | | |A Bs A B
A B
, cos( )|| || || ||
A Bs A BA B
KNN - comments• One of the best-performing text classifiers.
• It is robust in the sense of not requiring the categories to be linearly separated.
• The major drawback is the computational effort during classification.
• Other limitation is that its performance is primarily determined by the choice of k as well as the distance metric applied.
Centroid-based classification
• This method has two main phases:– Training phase: it considers the construction of one single
representative instance, called prototype, for each class.
– Test phase: each unlabeled document is compared against all prototypes and is assigned to the class having the greatest similarity score.
• Different from k-NN which represent each document in the training set individually.
How to compute the prototypes?H. Han, G. Karypis. Centroid-based Document Classification: Analysis and Experimental Results. Proc. of the 4th European Conference on Principles and Practice of Knowledge Discovery in Databases, pp. 424—431, 2000.
Centroid-based classification
T. Hastie, R. Tibshirani, J. Friedman. The Elements of Statistical Learning, Springer, 2009.
Calculating the centroids• Centroid as average
• Centroid as sum
• Centroid as normalized sum
• Centroid computation using the Rocchio formula
Comments on Centroid-Based Classification
• Computationally simple and fast model– Short training and testing time
• Good results in text classification
• Amenable to changes in the training set
• Can handle imbalanced document sets
• Disadvantages:– Inadequate for non-linear classification problems– Problem of inductive bias or model misfit
• Classifiers are tuned to the contingent characteristics of the training data rather than the constitutive characteristics of the categories
Linear models
• Idea: learning a linear function (in the parameters) that allow us to separate data f(x) = w x +b = Sj=1:n wj xj +b (linear discriminant)
f(x) = w F(x) +b = Sj wj fj(x) +b (the perceptron)
f(x) = Si=1:m ai k(xi,x) +b (Kernel-based methods)
Linear Discriminants and Support Vector Machines, I. Guyon and D. Stork, In Smola et al Eds. Advances in Large Margin Classiers. Pages 147--169, MIT Press, 2000.
Linear models• Classification of DNA micro-arrays
?
x1
x2
No Cancer
Cancer
( )f b x w x1 2,x x x
0b w x
0b w x
0b w x
?
Support vector machines (SVM)• A binary SVM classifier can be seen as a hyperplane
in the feature space separating the points that represent the positive from negative instances.– SVMs selects the hyperplane
that maximizes the marginaround it.
– Hyperplanes are fullydetermined by a small subsetof the training instances, calledthe support vectors.
Support vectors
Maximizemargin
Support vector machines (SVM)
• When data are linearly separable we have:
1min2
Tw w
( ( ) ) 1Ti iy bf w x
Subject to:
{1,..., }i m
1|| ||w 1
|| ||w
Non-linear SVMs• What about classes whose training instances
are not linearly separable?– The original input space can always be mapped to
some higher-dimensional feature space where the training set is separable.
• A kernel function is some function that corresponds to an inner product in some expanded feature space.
0 x
x2
SVM – discussion • The support vector machine (SVM) algorithm is very fast
and effective for text classification problems.
– Flexibility in choosing a similarity function• By means of a kernel function
– Sparseness of solution when dealing with large data sets• Only support vectors are used to specify the separating hyperplane
– Ability to handle large feature spaces• Complexity does not depend on the dimensionality of the feature
space
Decision trees
Select in each level the feature that reduces the entropy
All of the data
f1
f2
Choose f2
Choose f1
Random Forest, L. Breiman, Machine Learning (45):1, 5—32, 2001
54
Decision trees
Outlook Temperature Humidity Windy Play (positive) / Don't Play (negative)
sunny 85 85 false Don't Play
sunny 80 90 true Don't Play
overcast 83 78 false Play
rain 70 96 false Playrain 68 80 false Playrain 65 70 true Don't Play
overcast 64 65 true Play
sunny 72 95 false Don't Play
sunny 69 70 false Play
rain 75 80 false Play
sunny 75 70 true Play
overcast 72 90 true Play
overcast 81 75 false Play
rain 71 80 true Don't Play
55
Decision trees• Rule 1 suggests that if "outlook = sunny" and "humidity > 75" then "Don't Play".
Rule 2 suggests that if "outlook = overcast" then "Play".Rule 3 suggests that if "outlook = rain" and "windy = true" then "Don't Play".Rule 4 suggests that if "outlook = rain" and "windy = false" then "Play".Otherwise, "Play" is the default class.
Voted classification (ensembles)
k experts may be better than one if their individual judgments are appropriately combined
• Two main issues in ensemble construction: – Choice of the k classifiers– Choice of a combination function
• Two main approaches:– Bagging parallel approach– Boosting sequential approach
Voted classification (ensembles)
Robi Polikar. Ensemble Learning. Scholarpedia, 4(1):2776.
When do you think an ensemble can outperform any of the individual models?
Voted classification (ensembles)
• Idea: combining the outputs of different classification models:– Trained in different
subsets of data– Using different
algorithms – Using different
features
OriginalTraining data
....D1D2 Dt-1 Dt
D
Step 1:Create Multiple
Data Sets
C1 C2 Ct -1 Ct
Step 2:Build Multiple
Classifiers
C*Step 3:
CombineClassifiers
Bagging
• Individual classifiers are trained in parallel.
• To work properly, classifiers must differ significantly from each other:– Different document representations– Different subsets of features– Different learning methods
• Combining results by:– Majority vote– Weighted linear combination
Boosting
• Classifiers are trained sequentially using different subsets of the training set– Subsets are randomly selected– The probability of selecting an instance is not the same for
all; it depends on how often that instance was misclassified by the previous k-1 classifiers
• The idea is to produce new classifiers that are better able to correctly classify examples for which the performance of previous classifiers are poor– The decision is determined by a weighted linear
combination of the different predictions.
Evaluation of text classification
• What to evaluate?• How to carry out this evaluation?
– Which elements (information) are required?
• How to know which is the best classifer for a given task?– Which things are important to perform a fair
comparison?
Evaluation of text classification
• The available data is divided into three subsets:– Training (m1)
• used for the construction (learning) the classifier
– Validation (m2)• Optimization of parameters
of the TC method– Test (m3)
• Used for the evaluation of the classifier
Terms (N = |V|)
Docu
men
ts
(M)
m1
m2
m3
Evaluation – general ideas • Performance of classifiers is evaluated experimentally
• Requires a document set labeled with categories.– Divided into two parts: training and test sets– Usually, the test set is the smaller of the two
• A method to smooth out the variations in the corpus is the n-fold cross-validation. – The whole document collection is divided into n equal parts,
and then the training-and-testing process is run n times, each time using a different part of the collection as the test set. Then the results for n folds are averaged.
Aims at alleviating the lack of large data sets.
Used for estimating the generalization
performance of models.
(Do not mind overfitting!)
• Tradeoff between robustness and fit to data
68
PISIS research group, UANL, M
onterrey, Mexico,
27 de Noviem
bre de 2009
x1
x2
x1
x2
K-fold cross validation
69
Training data 5-fold CV
TrainTest Error fold 1
Error fold 2
Error fold 3
Error fold 4
Error fold 5
CV estimate
Performance metrics• Considering a binary problem
• Recall for a category is defined as the percentage of correctly classified documents among all documents belonging to that category, and precision is the percentage of correctly classified documents among all documents that were assigned to the category by the classifier.
What happen if there are more than two classes?
a bdc
Classifier YES
Classifier NO
Label YES Label NO
caa
(R) recall
dcbada
accuracy
baa
(P)precision RP
PRF
2
Micro and macro averages• Macroaveraging: Compute performance for each
category, then average.– Gives equal weights to all categories
• Microaveraging: Compute totals of a, b, c and d for all categories, and then compute performance measures.– Gives equal weights to all documents
Is it important the selection of the averaging strategy?What happen if we are very bad classifying the minority class?
Comparison of different classifiers
• Direct comparison– Compared by testing them on the same collection of
documents and with the same background conditions.– This is the more reliable method
• Indirect comparison– Two classifiers may be compared when they have been
tested on different collections and with possibly different background conditions if both were compared with a common baseline.
ROC Curve
100%
100%
For a given threshold on f(x), you get a point on the ROC curve.
Actual ROC
0
Positive class success rate
(hit rate, sensitivity)
1 - negative class success rate (false alarm rate, 1-specificity)
Random ROC
Ideal ROC curve
ROC Curve
Ideal ROC curve (AUC=1)
100%
100%
0 AUC 1
Actual ROC
Random ROC (AUC=0.5)
0
For a given threshold on f(x), you get a point on the ROC curve.
Positive class success rate
(hit rate, sensitivity)
1 - negative class success rate (false alarm rate, 1-specificity)
Want to learn more?
• C. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
• T. Hastie, R. Tibshirani, J. Friedman. The Elements of Statistical Learning, Springer, 2009.
• R. O. Duda, P. Hart, D. Stork. Pattern Classification. Wiley, 2001.
• I. Guyon, et al. Feature Extraction: Foundations and Applications, Springer 2006.
• T. Mitchell. Machine Learning. Mc Graw-Hill
Assignment # 3• Read a paper describing a classification approach or algorithm
for TC (it can be one from those available in the course page or another chosen by you)
• Prepare a presentation of at most 10 minutes, in which you describe the proposed/adopted approach *different of those seen in class*. The presentation must cover the following aspects:
A. Underlying and intuitive idea of the approachB. Formal descriptionC. Benefits and limitations (compared to the schemes seen in class)D. Your idea(s) to improve the presented approach
Suggested readings on text classification
• X. Ning, G. Karypis. The Set Classification Problem and Solution Methods. Proc. of International Conference on Data Mining Workshops, IEEE, 2008
• S. Baccianella, A. Esuli, F. Sebastiani. Using Micro-Documents for Feature Selection: The Case of Ordinal Text Classification. Proceedings of the 2nd Italian Information Retrieval Workshop, 2011
• J. Wang, J. D. Zucker. Solving the Multiple-Instance Problem: A Lazy Learning Approach. Proc. of ICML 200.
• A. Sun, E.P. Lim, Y. Liu. On Strategies for imbalanced text classification using SVM: a comparative study. Decision Support Systems, Vol. 48, pp. 191—201, 2009.