data mining and statistics clustering techniques

Data Mining and Statistics, Clustering Techniques 4.1

COT5230 Data Mining

Week 4

Data Mining and StatisticsClustering Techniques

M O N A S HA U S T R A L I A ’ S I N T E R N A T I O N A L U N I V E R S I T Y


References

Elder, John F. IV; Pregibon, Daryl; A Statistical Perspective on KDD; pp.87-93. Proceedings of the First International Conference on Knowledge Discovery & Data Mining (Ed. Fayyad, U.M. & Uthurusamy, R.), AAAI Press, Menlo Park, California 1995.

Berry& Linoff (1997) Data Mining Techniques: For Marketing, Sales, and Customer Support, Wiley.

Berson A. & Smith S.J. (1997) Data Warehousing, Data Mining and OLAP, McGraw-Hill.


The Link between Pattern and Approach

Data mining aims to reveal knowledge about the data under consideration

This knowledge takes the form of patterns within the data which embody our understanding of the data

– Patterns are also referred to as structures, models and relationships

The approach chosen is inherently linked to the pattern revealed


A Taxonomy of Approaches to Data Mining - 1

It is not expected that all the approaches will work equally well with all data sets

Visualization of data sets can be combined with, or used prior to, modeling and assists in selecting an approach and indicating what patterns might be present


A Taxonomy of Approaches to Data Mining - 2

Verification-driven Discovery-driven

Predictive(Supervised)

Informative(Unsupervised)

Query and reportingStatistical analysis Regression

Classification

ClusteringAssociationDeviationdetection(outliers)


Verification-driven Data Mining Techniques - 1

Verification data mining techniques require the user to postulate some hypothesis

– Simple query and reporting, or statistical analysis techniques then confirm this hypothesis

Statistics has been neglected to a degree in data mining in comparison to less traditional techniques such as

– neural networks, genetic algorithms and rule-based approaches to classification

Many of these “less traditional” techniques also have a statistical interpretation


Verification-driven Data Mining Techniques - 2

The reasons for this are various

Statistical techniques are most useful for well-structured problems

Many data mining problems are not well structured:

– the statistical techniques breakdown or require large amounts of time and effort to be effective


Problems with Statistical Approaches - 1

Traditional statistical models often highlight linear relationships but not complex non-linear relationships

Exploring all possible higher dimensional relationships, often (usually) takes an unacceptably long time

– the non-linear statistical methods require knowledge about

» the type of non-linearity» the ways in which the variables interact

– This knowledge is often not known in complex multi-dimensional data mining problems


Problems with Statistical Approaches - 2

Statisticians have traditionally focussed on model estimation, rather than model selection

For these reasons less traditional, more exploratory, techniques are often chosen for modern data mining

The current high level of interest in data mining centres on many of the newer techniques, which may be termed discovery-driven

Lessons from statistics should not be forgotten. Estimation of uncertainty and checking of assumptions is as important as ever!


Discovery-driven Data Mining Techniques

Discovery-driven data mining techniques can also be broken down into two broad areas:

– those techniques which are considered predictive, sometimes termed supervised techniques

– those techniques which are termed informative, sometimes termed unsupervised techniques

Predictive techniques build patterns by making a prediction of some unknown attribute given the values of other known attributes


Informative techniques do not present a solution to a known problem

– they present interesting patterns for consideration by some expert in the domain

– the patterns may be termed “informative patterns”

The main predictive and informative patterns are:– Regression

– Classification

– Clustering

– Association


Regression

Regression is a predictive technique which discovers relationships between input and output patterns, where the values are continuous or real valued

Many traditional statistical regression models are linear

Neural networks, though biologically inspired, are in fact non-linear regression models

Non-linear relationships occur in many multi-dimensional data mining applications


An Example of a Regression Model - 1

Consider a mortgage provider that is concerned with retaining mortgages once taken out

They may also be interested in how profit on individual loans is related to customers paying off their loans at an accelerated rate

– For example, a customer may pay an additional amount each month and thus pay off their loan in 15 years instead of 25 years

A graph of the relationship between profit and the elapsed time between when a loan is actually paid off and when it was originally contracted to be paid off appears on the next slide



Profit

0

0 7Years Early Loan Paid Off

Non-linear

linear



Linear regression on the data does not match the real pattern of the data

The curved line represents what might be produced by a non-linear approach (perhaps a neural network)

This curved line fits the data much better. It could be used as the basis on which to predict profitability

– Decisions on exit fees and penalties for certain behaviors may be based on this kind of analysis.


Exploratory Data Analysis (EDA)

Classical statistics has a dogma that the data may not be viewed prior to modeling [Elde95]

– aim is to avoid choosing biased hypotheses

During the 1970s the term Exploratory Data Analysis (EDA) was used to express the notion that both the choice of model and hints as to appropriate approaches could be data-driven

Elder and Pregiban describes the dichotomy thus:

“On the one side the argument was that hypotheses and the like must not be biased by choosing them on the basis of what the data seemed to be indicating. On the other side was the belief that pictures and numerical summaries of data are necessary in order to understand how rich a model the data can support.”


EDA and the Domain Expert - 1

It is a very hard problem to include common sense based on some knowledge of the domain in automated modeling systems

– chance discoveries occur when exploring data that may not have occurred otherwise

– these can also change the approach to the subsequent modeling


EDA and the Domain Expert - 2

The obstacles to entirely automating the process are:

– It is hard to quantify a procedure to capture “the unexpected” in plots

– Even if this could be accomplished, one would need to describe how this maps into the next analysis step in the automated procedure

What is needed is a way to represent meta-knowledge about the problem at hand and the procedures commonly used


An Interactive Approach to DM

A domain expert is someone who has meta-knowledge about the problem

An interactive exploration and a querying and/or visualization system guided by a domain expert goes beyond current statistical methods

Current thinking on statistical theory recognizes such an approach as being potentially able to provide a more effective way of discovering knowledge about a data set


Automatic Cluster Detection

If the are many competing patterns, a data set can appear to contain just noise

Subdividing a data set into clusters where patterns can be more easily discerned can overcome this

When we have no idea how to define the clusters automatic cluster detection methods can be useful

Finding clusters is an unsupervised learning task


Automatic Cluster Detection - example

The Hehrtzsprung-Russell diagram which graphs a stars luminosity against temperature reveals three clusters

– It is interesting to note that each of the clusters has a different relationship between luminosity and temperature.

In most data mining situations the variables to consider and the clusters that may be formed are not so easily determined


The Hehrtzsprung-Russell diagram

Luminosity(Sun=1)

Temperature (Degrees Kelvin)

Red Giants

Main SequenceWhite Dwarves

1

40,0002,500


The K-Means Technique

K, the number of clusters that are to be formed, must be decided before beginning

– Step 1» Select K data points to act as the seeds (or initial centroids)

– Step 2» Each record is assigned to the centroid which is nearest, thus

forming a cluster

– Step 3» The centroids of the new clusters are then calculated. Go back

to Step 2

– This is continued until the clusters stop changing


Assign Each Record to the Nearest Centroid

X2

X1


Calculate the New Centroids

X2

X1


Determine the New Cluster Boundaries

X2

X1


Similarity, Association and Distance

The method just described assumes that each record can be described as a point in a metric-space

– This is not easily done for many data sets (e.g. categorical and some numeric variables)

The records in a cluster should have a natural association. A measure of similarity is required.

– Euclidean distance is often used, but it is not always suitable

– Euclidean distance treats changes in each dimension equally, but in databases changes in one field may be more important than changes in another


Types of Variables

Categories– e.g. Food Group: Grain, Dairy, Meat, etc.

Ranks– e.g. Food Quality: Premium, High Grade, Medium, Low

Intervals– e.g. The distance between temperatures

True Measures– The measures have a meaningful zero point so ratios

have meaning as well as distances


Measures of Similarity

Euclidean distance

Angle between two vectors (from origin to data point

The number of features in common

Mahalanobis distance


Weighting and Scaling

Weighting allows some variables to assume greater importance than others.

– The domain expert must decide if certain variables deserve a greater weighting

– Statistical weighting techniques also exist

Scaling attempts to apply a common range to variables so that differences are comparable between variables

– This can also be statistically based


Variations of the K-Means Technique

There are problems with simple K-means method– It does not deal well with overlapping clusters.

– The clusters can be pulled of centre by outliers.

– Records are either in or out of the cluster so there is no notion of likelihood of being in a particular cluster or not

A Gaussian Mixture Model varies the approach already outlined by attaching a weighting based on a probability distribution to records which are close to or distant from the centroids initially chosen. There is then less chance of outliers distorting the situation. Each record contributes to some degree to each of the centroids


Agglomeration Methods - 1

A true unsupervised technique would not pre-determine the number of clusters

A hierarchical technique would offer a hierarchy of clusters from large to small. This can be achieved in a number of ways

An agglomerative technique starts out by considering each record as a cluster and gradually building larger clusters by merging the records which are near each other


Agglomeration Methods - 2

An example of an agglomerative cluster tree:


Evaluating Clusters

We desire clusters to have members which are close to each other and we also want the clusters to be widely spaced

Variance measures are often used. Ideally, we want to minimize within-cluster variance and maximize between-cluster variance

But variance is not the only important factor, for example it will favor not merging clusters in an hierarchical technique


Strengths of Automatic Cluster Detection

Strengths– is an undirected knowledge discovery technique

– works well with many types of data

– is relatively simple to carry out

Weaknesses– Can be difficult to choose the distance measures and

weightings

– Can be sensitive to initial parameter choices

– The clusters found can be difficult to interpret

data mining and statistics clustering techniques

Documents