fuzzy interpretation of discretized intervals dr. xindong wu andrea porter april 11, 2002

Fuzzy Interpretation of Discretized Intervals

Dr. Xindong Wu

Andrea Porter

April 11, 2002

Plan For Presentation

Introduction to Problem, HCVDiscretization Techniques/Fuzzy

BordersA Hybrid Solution for HCVExperiments and ResultsConclusion

Introduction

Real-world data contains both numerical and nominal data, must be able to deal with different types of data.

Existing systems discretize numerical domains into intervals and treat intervals as nominal values during induction.

Problems occur if test examples are not covered in training data (no-match, multiple match)

The solution is a hybrid approach using fuzzy intervals for no-match problem.

HCV

Attribute based rule induction algorithm, extension matrix approach Divide positive examples into intersecting groups Find a heuristic conjunctive rule in each group that

covers all PE and no NE

HCV can find a rule in the form of variable-valued logic

More compact than the decision trees/rules of ID3 and C4.5

Variable Valued Logic and Selectors

Represents decisions where variables can take a range

Selector:

[ X # R ]

X = attribute

# = relational operator ( = , <, >, . . . )

R = Reference, list of 1 or more values

e.g [ Windy = true] [Temp > 90]

, , ,...

HCV Software

C++ implementationCan work with noisy and real-valued

domains as well as nominal and noise-free databases

Provides a set of deduction facilities for the user to test the accuracy of the produced rules on test examples

Example DB ORDER X1 X2 X3 X4 CLASS

1 1 a a 1 F

2 1 a b 1 F

3 1 a c 1 F

4 1 a a 0 F

5 1 b c 1 T

6 0 b b 0 T

7 0 a c 1 T

8 1 b a 0 T

9 1 b a 1 T

10 1 c c 0 F

11 1 c b 1 F

12 0 c b 0 T

13 0 a a 0 T

14 0 c c 1 F

15 0 c a 0 T

16 1 a b 0 F

17 0 a a 1 T

18 0 b a 1 T

C4.5 Results vs. HCV

C4.5:The T classX2 = b

X1 = 0 & X3 = a X1 = 0 & X3 = b X1 = 0 & X2 = a

HCV:The T classX2 = b

X1 = 0 & X2 = a

X1 = 0 & X4 = 0

C4.5:The F classX1 = 1 & X2 = a

X1 = 1 & X2 = c X2 = c & X3 = c

Deduction of Induction Results

Induction generates knowledge from existing data Deduction applies induction results to interpret

new data. With real-world data, induction can not be

assumed to be perfect Three cases:

1) no-match (measure of fit)2) single-match3) multiple-match (estimate of probability)

Discretization

Occurs during rule induction Discretize numerical domains into intervals and

treat similar to nominal values. The challenge is to find the right borders for the

intervals Possible Methods:

1) Simplest Class-Separating Method

2) Information Gain Heuristic (implemented in HCV)

Simplest Class- Separating Method:

Interval Borders are places between each adjacent pair of examples which have different classes.

If attribute is very informative - method is efficient and useful.

If attribute is not informative - method produces too many intervals

Information Gain Heuristic

Use IGH to find more informative border.

x = (xi + xi+1)/2 for (i = 1, …, n-1) x is a possible cut point if xi and xi+1 are of different

classes. Use IGH to find best x Recursively split on left and right To stop recursive splitting:

1) stop if IGH is same on all possible cut points.2) stop if # of examples to split is less than a predefined number3) limit the number of intervals

Fuzzy Borders

Discretization of continuous domains does not always fit accurate interpretation.

Instead of using sharp borders, use a membership function, measures the degree of membership.

A value can be classified into a few different intervals at the same time (e.g. single to multiple match)

Fuzzy Borders (2)

Fuzzy matching - deduction with fuzzy borders of discretized intervals.

Take the interval with the greatest degree as the value’s discrete value.

3 functions to fuzzify borders:

1) linear2) polynomial3) arctan

Definitionss = spread parameter l = length of original

xleft, xright = left/right sharp borders

xleft xright

l

a = -kxleft + 1/2 b = kxright + 1/2

linleft(x) = kx + a

lin right(x) = -kx + b

lin(x) = MAX(0, MIN(1,linleft(x),linright(x)))

Linear Membership Function

k = 1/2sl

xleft xright

l

sl

Arctan Membership Function

*Polynomial Membership Function

polyside(x) = asidex3 + bsidex2 + csidex + dside

aside = 1/(4(ls)3)

bside = -3asidexside side {left,right}

cside = 3aside(xside2 - (ls)2)

dside = -a(xside3 -3xside(ls)2 + 2(ls)3)

polyleft(x), if xleft -ls x xleft + ls

poly(x) = polyright(x), if xright -ls x xright +ls

1, if xleft +ls x xright -ls

0, otherwise

Match Degree

Selector method - take the max membership degree of the value in all the intervals involved. If 2 adjacent intervals have the same class, values close to the border will have low membership.

Conjunction method - adds with fuzzy plus

ab=a + b - ab

No-Match Resolution

Largest Class

Assign all no match examples to the largest class, the default class.

Works well, if the number of classes in a training set is small and one class is clearly larger.

Deteriorates if there is a larger number of classes and the examples are evenly distributed

No-Match Resolution

Measure of FitCalculate the measure of fit for each class:

1) calculate MF for each selector (sel)MF(sel, e) = 1, if sel is satisfied by e

n/|x|, otherwise2) calculate MF for each conjunctive rule(conj)MF(conj, e) = MF(sel, e) * n(conj)/N

No-Match Resolution

Measure of Fit (2)3) calculate MF for each class c

MF(c, e) = MF(conj1, e) + MF(conj2, e) - MF(conj1,e)MF(conj2,e)

* For more than two rules, apply formula recursively.

* Find maximum MF - determines which class is closest to the example

Multiple-Match

Caused by over-generalization of the training examples at induction time

Example (X1 = a, X2 = 1)

All PE cover X1 = a

All NE cover X2 = 1 Multiple Match

Multiple-Match Resolution

First Hit Use first rule which classifies the example Produces reasonable results if the rules from

induction have been ordered according to a measure of reliability

Advantages - straightforward, efficient Disadvantages - have to sort rules at

induction time

Multiple-Match Resolution

Largest Rule

Similar to largest class method from no-match resolution

Choose conjunctive rule that covers the most examples in the training set.

Multiple-Match Resolution

Estimation of Probability Assign EP value to each class based on the

size of the satisfied conjunctive rules.

1) Find EP for each conjunctive rule (conj):EP(conj, e)= { n(conj)/N, if conj is satisfied

by e0, otherwise

n(conj) = number of examples covered by conjN = number of total examples

Multiple-Match Resolution

Estimation of Probability (2)

2) Find EP value for each class:

EP(c, e) = EP(conj1, e) + EP(conj2, e) - EP(conj1,e)EP(conj2,e).

* For more rules, apply formula recursively

* Choose class with highest EP value

Hybrid Interpretation

Used because fuzzy borders only add conflicts because they don’t reduce the number rules that are applicable

HCV - use sharp borders during induction and use fuzzy borders only during deduction

Algorithm:* Single match - use class indicated by rules* Multiple match - use estimation probability

(EP) with sharp borders* No match - use fuzzy borders with polynomial

membership function to find closest rule

The Data

Used 17 databases from the Machine Learning Database Repository, U. of California, Irvine.

Databases selected because:

1) All include numerical data

2) All lead to situations where no rules clearly apply.

Results – Predictive Accuracy

Results (cont.)

The results shown for C4.5 and NewID are the pruned ones These were usually better than the

unpruned ones in this experimentHCV did not fine tune different

parameters because this would be loss of generality and applicability of the conclusions

Accuracy Results

HCV(hybrid) - 9 databasesC4.5 (R 8) - 7 databasesC4.5 (R 5) - 6 databasesHVC (large) - 3 databasesHCV (fuzzy) - 2 databases

HCV Comparison

HCV (fuzzy) generally performs better than the simple largest class method

HCV’s performance improves significantly when the fuzzy borders (for no match) are combined with probability estimation (for multiple match) in HCV (hybrid)

Conclusions

Fuzzy borders are constructed and used at deduction time only when a no match case occurs.

This hybrid method performs more accurately than several other current deduction programs.

Fuzziness is strongly domain dependent, HCV allows the user to specify their own intervals and fuzzy functions.