Prepared by: Mahmoud Rafeek Al-Farra
College of Science & TechnologyDep. Of Computer Science & ITBCs of Information Technology
Data MiningData Mining
2013www.cst.ps/staff/mfarra
Chapter 5: Evaluation
Course’s Out Lines
Introduction Data Preparation and Preprocessing Data Representation Classification Methods Evaluation Clustering Methods Mid Exam Association Rules Knowledge Representation Special Case study : Document clustering Discussion of Case studies by students
2
Out Lines
Definition of Evaluation Measure of interestingness Training versus Testing Cluster evaluation
3
Definition of Evaluation
After examining the data and applying automated
methods for data mining, we must carefully
consider the quality of the end-product of our
effort. This step is evaluation.
Evaluation evaluates the performance of the a
proposed solution to the data mining task.
4
Definition of Evaluation
A large number of patterns and rules exist in database. Many of them
has no interest to the user.
5
Data Cleaning
Data Integration
Databases
Data Warehouse
Task-relevant Data
Selection
Data Mining
Pattern Evaluation
Measure of interestingness6
Measure of interestingness has two approaches:
Objective: where the interestingness is measured in
term of its structure and underlying data used in the
discovery process.
Measure of interestingness7
Measure of interestingness has two approaches:
Subjective: Subjective measure do not depended only
in the structure of a rule and the data used , but also on
the user who examines the pattern. These measures
recognize that a pattern of interest to one user , may be
no interest to another user.
Training versus Testing8
“Just trust me!” does not work in evaluation. Error on the training data is not a good indicator of
performance on future data. Simple solution probably not be exactly the same
as the training that can be used if lots of (labeled) data is available.
Split data into training and test set.
Training versus Testing9
A strong and effective way to evaluate results is to hide some data and then do a fair comparison of training
results to unseen data. In this way it prevents poor results and gives the
developers time to extract the best performance from the application system.
Many kinds of splitting data into training and testing most common holdout and cross validation
Cluster evaluation10
One type of measure allows us to evaluate different sets of clusters without external knowledge and is called an internal quality measure; it is used when
we don't have external knowledge about the clustering data.
Overall similarity is an example for internal quality measure and will be discussed below.
Cluster evaluation11
The second type of measures lets us evaluate the quality of clustering by comparing the clusters
produced by the clustering techniques to known classes (external knowledge).
This type of measure is called an external quality measure and we will discuss two external qualities
which are entropy and F-measure.
Cluster evaluation12
There are many different quality measures and the performance and relative ranking of different
clustering algorithms can vary substantially depending on which measure is used.