data mining-knowledge presentation—id3 algorithm prof. sin-min lee department of computer science
TRANSCRIPT
Data Mining-Knowledge Presentation—ID3 algorithm
Prof. Sin-Min Lee
Department of Computer Science
Data Mining Tasks
Predicting onto new data by using rules or patterns and behaviors – Classification– Estimation
Understanding the groupings, trends, and characteristics of your customer– Segmentation
Visualizing the Euclidean spatial relationships, trends, and patterns of your data– Description
Stages of Data Mining Process
1. Data gathering, e.g., data warehousing.2. Data cleansing: eliminate errors and/or bogus
data, e.g., patient fever = 125.3. Feature extraction: obtaining only the
interesting attributes of the data, e.g., “date acquired” is probably not useful for clustering celestial objects, as in Skycat.
4. Pattern extraction and discovery. This is the stage that is often thought of as “data mining” and is where we shall concentrate our effort.
5. Visualization of the data.6. Evaluation of results; not every discovered
fact is useful, or even true! Judgment is necessary before following your software's conclusions.
Clusters of Galaxies
Skycat clustered 2x109
sky objects into stars, galaxies, quasars, etc. Each object was a point in a space of 7 dimensions, with each dimension representing radiation in one band of the spectrum. The Sloan Sky Survey is a more ambitious attempt to catalog and cluster the entire visible universe
Cholera outbreak in London
Clustering: Examples
Decision trees are an alternative way of structuring rule information.
humidity windy
outlook
sunny rainovercast
normal true false
N NP P
P
A Classification rule based on the tree
if outlook = overcast
outlook =
sunny &
humidity = normal
outlook = rain
&
windy = false
then P
if outlook = overcast
then P
if outlook = sunny &
humidity = normal
then P
if outlook = rain &
windy = false
then P
Outlook
Sunny Overcast Rain
Humidity
High Normal
No Yes
Each internal node tests an attribute
Each branch corresponds to anattribute value node
Each leaf node assigns a classification
Top-Down Induction of Decision Trees ID3
1. A the “best” decision attribute for next node
2. Assign A as decision attribute for node3. For each value of A create new
descendant 4. Sort training examples to leaf node
according to the attribute value of the branch5. If all training examples are perfectly
classified (same value of target attribute) stop, else iterate over new leaf nodes.
Which Attribute is ”best”?
A1=?
True False
[21+, 5-] [8+, 30-]
[29+,35-] A2=?
True False
[18+, 33-] [11+, 2-]
[29+,35-]
Entropy
S is a sample of training examples
p+ is the proportion of positive examples
p- is the proportion of negative examples
Entropy measures the impurity of S
Entropy(S) = -p+ log2 p+ - p- log2 p-
Entropy(S)= expected number of bits needed to encode class (+ or -) of randomly drawn members of S (under the optimal, shortest length-code)
Why?
Information theory optimal length code assign
–log2 p bits to messages having probability p.
So the expected number of bits to encode
(+ or -) of random member of S:
-p+ log2 p+ - p- log2 p-
Entropy
• Gain(S,A): expected reduction in entropy due to sorting S on attribute A
A1=?
True False
[21+, 5-] [8+, 30-]
[29+,35-] A2=?
True False
[18+, 33-] [11+, 2-]
[29+,35-]
Gain(S,A)=Entropy(S) - vvalues(A) |Sv|/|S| Entropy(Sv)
Entropy([29+,35-]) = -29/64 log2 29/64 – 35/64 log2 35/64 = 0.99
Information Gain
Information Gain
A1=?
True False
[21+, 5-] [8+, 30-]
[29+,35-]
Entropy([21+,5-]) = 0.71Entropy([8+,30-]) = 0.74Gain(S,A1)=Entropy(S) -26/64*Entropy([21+,5-])
-38/64*Entropy([8+,30-])
=0.27
Entropy([18+,33-]) = 0.94Entropy([8+,30-]) = 0.62Gain(S,A2)=Entropy(S) -51/64*Entropy([18+,33-]) -13/64*Entropy([11+,2-])
=0.12
A2=?
True False
[18+, 33-] [11+, 2-]
[29+,35-]
Day Outlook Temp. Humidity Wind Play TennisD1 Sunny Hot High Weak NoD2 Sunny Hot High Strong NoD3 Overcast Hot High Weak YesD4 Rain Mild High Weak YesD5 Rain Cool Normal Weak YesD6 Rain Cool Normal Strong NoD7 Overcast Cool Normal Weak YesD8 Sunny Mild High Weak NoD9 Sunny Cold Normal Weak YesD10 Rain Mild Normal Strong YesD11 Sunny Mild Normal Strong YesD12 Overcast Mild High Strong YesD13 Overcast Hot Normal Weak YesD14 Rain Mild High Strong No
Training Examples
Selecting the Next Attribute
Humidity
High Normal
[3+, 4-] [6+, 1-]
S=[9+,5-]E=0.940
Gain(S,Humidity)=0.940-(7/14)*0.985 – (7/14)*0.592=0.151
E=0.985 E=0.592
Wind
Weak Strong
[6+, 2-] [3+, 3-]
S=[9+,5-]E=0.940
Gain(S,Wind)=0.940-(8/14)*0.811 – (6/14)*1.0=0.048
Outlook
Sunny Rain
[2+, 3-] [3+, 2-]
S=[9+,5-]E=0.940
Gain(S,Outlook)=0.940-(5/14)*0.971 -(4/14)*0.0 – (5/14)*0.0971=0.247
E=0.971
E=0.971
Overcast
[4+, 0]
E=0.0
Selecting the Next Attribute
Outlook
Sunny Overcast Rain
Yes
[D1,D2,…,D14] [9+,5-]
Ssunny=[D1,D2,D8,D9,D11] [2+,3-]
? ?
[D3,D7,D12,D13] [4+,0-]
[D4,D5,D6,D10,D14] [3+,2-]
Gain(Ssunny , Humidity)=0.970-(3/5)0.0 – 2/5(0.0) = 0.970Gain(Ssunny , Temp.)=0.970-(2/5)0.0 –2/5(1.0)-(1/5)0.0 = 0.570Gain(Ssunny , Wind)=0.970= -(2/5)1.0 – 3/5(0.918) = 0.019
ID3 Algorithm
Outlook
Sunny Overcast Rain
Humidity
High Normal
Wind
Strong Weak
No Yes
Yes
YesNo
[D3,D7,D12,D13]
[D8,D9,D11] [D6,D14][D1,D2] [D4,D5,D10]
The ID3 AlgorithmGiven
• a set of disjoint target classes {C1, C2, …, Ck},
• a set of training data, S, containing objects of more than one class.
Let T be any test on a single attribute of the data, with O1, O2, …, On
representing the possible outcomes of applying T to any object x (written as T(x)).
T produces a partition {S1, S2, …, Sn} of S such that
Si = { x | T(x) = Oi}
Proceed recursively to replace each Si with a decision tree.
Crucial factor: Selecting the tests.
S1 Sn
S
O1 OnO2
S2
…
…
In making this decision, Quinlan employs the notion of uncertainty
(entropy from information theory).
M = {m1, m2, …, mn} Set of messages
p(mi) Probability of the message mi being received
I(mi) = -log p(mi) Amount of information of message mi
U(M) = i p(mi) I(mi) Uncertainty of the set M
Quinlan’s assumptions:
•A correct decision tree for S will classify objects in the same proportion as their representation in S.
•Given a case to classify, a test can be regarded as the source of a message about that case.
Let Ni be the number of cases in S that belong to a class Ci:
p(cCi) = Ni / |S|
The uncertainty, U(S), measures the average amount of information
needed to determine the class of a random case, cS.
Uncertainty measure after S has been partitioned.
UT(S) = i (|Si| / |S|) U(Si)
Select the test T that gains the most information, i.e., where
GS(T) = U(S) – UT(S)
is maximal.
Evaluation of ID3The ID3 algorithm tends to favor tests with a large number of outcomes
over tests with a smaller number.
Its computational complexity depends on the cost of choosing the next test to branch on;
It was adapted to deal with noisy and incomplete data;
It is a feasible alternative to knowledge elicitation if sufficient data of the right kind are available;
However this method is not incremental.
Further modification were introduced in C4.5, e.g :
•pruning the decision tree in order to avoid overfitting
•Better test selection heuristic
Search Space and Search Trees
• Search space is logical space composed of – nodes are search states– links are all legal connections between search
states• e.g. in chess, no link between states where W castles
having previously moved K.
– always just an abstraction– think of search algorithms trying to navigate
this extremely complex space
Search Trees• Search trees do not summarise all possible searches– instead an abstraction of one
possible search• Root is null state• edges represent one choice
– e.g. to set value of A first
• child nodes represent extensions– children give all possible choices
• leaf nodes are solutions/failures• Example in SAT
– algorithm detects failure early– need not pick same variables
everywhere
B(a B )
Im p oss ib le
b(a b )
Im p oss ib le
a(a )
C h oose B
B(A B C )
Im p oss ib le
b(A b C )
S o lu tion
C(A C )
C h oose B
cA c
im p oss ib le
A(A )
C h oose C
A B C , A B c , A b C , A b c , aB C , ab C , ab cs ta te = ()C h oose A
Definition
• A tree shaped structure that represents a set of decisions. These decisions are used as a basis for predictions.
• They represent rules for classifying datasets. Useful knowledge can be extracted by this classification.
DT Structure
• Node Types– Decision nodes: specifies some test to
be carried out on a single attribute value.• Each outcome is assigned to a branch that
connects to a leaf node or another decision node.
– Leaf nodes: indicates the classification of an example.
An Example
Growing a Decision Tree
Iterative Dichotomiser 3 (ID3) Algorithm
• Invented by J. Ross Quinlan in 1975• Based on a greedy search algorithm.
ID3 Cont.
• The goal is to create the best possible tree that works on all available data.– An example is strictly either belonging to
one class or the other.– Need to select the attribute that best
classify the example (i.e. need to select the attribute with smallest entropy on all the example.)
– The lower the entropy the higher the Information Gain. We desire high IG.
Entropy
• A quantitative measurement of the homogeniety of a set of examples.
• It tells us how well an attribute separate the training examples according to their target classification.
Entropy cont.
• Given a set S with only positive or negative examples (2 class case):
Entropy(S) = -PPlog2PP – Pnlog2Pn
Where PP = proportion of positive examples
PN = proportion of negative examples
Entropy cont.
• Ex. Given 25 examples with 15 positive and 10 negative.Entropy(S) = -(15/25)log2(15/25)
-(10/25)log2(10/25) = .97
If Entropy(S) = 0 all members in S belong to strictly one class.
If Entropy(S) = 1 (max value) members are split equally between the two classes.
In General…
• In general, if an attribute takes more than two values
entropy(S) =
Where n is the number of values
n
iii pp
1
)log(
Information Gain
Where A is an attribute of S Value(A) is the set of possible value of A
v is a particular value in Value(A) Sv is a subset of S having of v’s on value(A).
Actual AlgorithmID3 (Examples, Target_Attribute, Attributes)Create a root node for the tree If all examples are positive, Return the single-node tree Root, with label = +. If all examples are negative, Return the single-node tree Root, with label = -. If number of predicting attributes is empty, then Return the single node tree Root, with label = most common value of the target attribute in the examples. Otherwise Begin A = The Attribute that best classifies examples. Decision Tree attribute for Root = A. For each possible value, vi, of A,
Add a new tree branch below Root, corresponding to the test A = vi. Let Examples(vi), be the subset of examples that have the value vi for A If Examples(vi) is empty
Then below this new branch add a leaf node with label = most common target value in the examples
Else below this new branch add the subtree ID3 (Examples(vi), Target_Attribute, Attributes – {A})
End Return Root
Independent Attributes Dependent
Outlook Temperature Humidity Windy Play
sunny 85 85 FALSE Don’t play
sunny 80 90 TRUE Don’t play
overcast 83 78 FALSE Play
rain 70 95 FALSE Play
rain 68 80 FALSE Play
rain 65 70 TRUE Don’t play
overcast 64 65 TRUE Play
sunny 72 95 FALSE Don’t play
sunny 69 70 FALSE Play
rain 75 80 FALSE Play
sunny 75 70 TRUE Play
overcast 72 90 TRUE Play
overcast 81 75 FALSE Play
rain 71 80 TRUE Don’t play
We choose Play as our dependent attribute.Don’t play = NegativePlay = Positive
Independent Attributes Dependent
Outlook Temperature Humidity Windy Play
sunny 85 85 FALSE Don’t play
sunny 80 90 TRUE Don’t play
overcast 83 78 FALSE Play
rain 70 95 FALSE Play
rain 68 80 FALSE Play
rain 65 70 TRUE Don’t play
overcast 64 65 TRUE Play
sunny 72 95 FALSE Don’t play
sunny 69 70 FALSE Play
rain 75 80 FALSE Play
sunny 75 70 TRUE Play
overcast 72 90 TRUE Play
overcast 81 75 FALSE Play
rain 71 80 TRUE Don’t play
Outlook playdon’t play total Entropy
sunny 2 3 5 0.34676807
ovecast 4 0 4 0
rain 3 2 5 0.34676807
total 0.69353614
Temp
>70 5 4 9 0.63712032
<=70 4 1 5 0.25783146
total 0.89495179
Humidity
>70 6 4 10 0.69353614
<=70 3 1 4 0.23179375
total 0.92532989
Windy
TRUE 3 3 6 0.42857143
FALSE 6 2 8 0.4635875
total 0.89215893
Independent Attributes Dependent
Outlook Temperature Humidity Windy Play
sunny 85 85 FALSE Don’t play
sunny 80 90 TRUE Don’t play
sunny 72 95 FALSE Don’t play
sunny 69 70 FALSE Play
sunny 75 70 TRUE Play
play don’t play total Entropy
Temp
>70 1 3 4 0.6490225
<=70 1 0 1 0
total 0.6490225
Humidity
>70 0 3 3 0
<=70 2 0 2 0
total 0
Windy
TRUE 1 1 2 0.4
FALSE 1 2 3 0.5509775
total 0.9509775
Independent Attributes Dependent
Outlook Temperature Humidity Windy Play
rain 70 95 FALSE Play
rain 68 80 FALSE Play
rain 65 70 TRUE Don’t play
rain 75 80 FALSE Play
rain 71 80 TRUE Don’t play
play don’t play total Entropy
Temp
>70 1 1 2 0.4
<=70 2 1 3 0.5509775
total 0.9509775
Humidity
>70 1 0 3 0
<=70 3 1 4 0.6490225
total 0.6490225
Windy
TRUE 0 2 2 0
FALSE 3 0 3 0
total 0
Independent Attributes Dependent
Outlook Temperature Humidity Windy Play
overcast 83 78 FALSE Play
overcast 64 65 TRUE Play
overcast 72 90 TRUE Play
overcast 81 75 FALSE Play
play don’t play total Entropy
Temp
>70 2 0 2 0
<=70 0 2 2 0
total 0
Humidity
>70 3 0 3 0
<=70 1 0 1 0
total 0
Windy
TRUE 2 0 2 0
FALSE 2 0 2 0
total 0
ID3 Summary
• Step 1: Take all unused attribute and count their entropy concerning test samples.
• Step 2: Choose the attribute with smallest entropy.
• Step 3: Make a node containing that attribute.
Growth stops when:
• Every attribute already exist along the path thru the tree.
• The training examples associated with a leaf all have the same target attribute.
References
• An Overview of Data Mining Techniques – http://www.thearling.com/text/
dmtechniques/dmtechniques.htm
• Decision tree– http://en.wikipedia.org/wiki/Decision_tree
• Decision Trees – http://dms.irb.hr/tutorial/tut_dtrees.php