Fuzzy Rules Generation using GeneticAlgorithms with Self-adaptive Selection

Marcos Evandro Cintra, Heloisa de Arruda CamargoFederal University of Sao Carlos (UFSCar)

Computer Science Departament-Sao Carlos, Brasil

E-mail: {marcos cintra, heloisa}@dc.ufscar.br

Abstract

The definition of the Rule Base is one of the most impor-tant and difficult tasks when designing Fuzzy Systems. Amethod for the generation of fuzzy rule bases using geneticalgorithm, including a phase of preselection of candidaterules, has been proposed by the authors. The selection ofcandidate rules uses criteria based on heuristics related tothe degree of coverage of the rules. This paper proposes theuse of a self-adaptive algorithm for the fitness calculationin the genetic algorithm, as an improvement of the referredmethod. The algorithm proposed emphasises the usefulnessof compact rule bases as a means of transparency enhance-ment. Some experiment results are presented with a briefdiscussion of the advantages of the proposal.

1 Introduction

Systems based on Fuzzy Logic, generally called FuzzySystems (FS) have been successfully used for the solutionof problems in many areas, including pattern classification,optimization, control of processes and the design of systems[11, 26, 27].

The FS of interest in this work are those known as RuleBased Fuzzy Systems (RBFS). A RBFS has two main com-ponents: a Knowledge Base (KB) and an Inference Machine(IM). The KB comprises the Fuzzy Rule Base (FRB), whichhas the fuzzy rule set that represents a given problem andthe Fuzzy Data Base (FDB), which contains the definitionsof the fuzzy sets related to the linguistic variables used inthe FRB. The IM is responsible for the application of a rea-soning process that uses inferences to derive the output, orconclusion, of the system, based on both, the rules and theknown facts.

Various approaches have been used for the automaticgeneration of the KB from numerical data representing sam-ples, or examples, of a problem. Clustering algorithms

[22], gradient-based methods [25], neural networks [20]and Genetic Algorithms (GA) [10] are among the mostwell-succeeded techniques.

Recently there has been a considerable research effort fo-cusing on the use of GA [12] in the design of FSs. This ini-tiative coined the term Genetic Fuzzy System (GFS), whichare, basically, FSs with a learning process based on a GA[10].

A very promising approach is the use of GAs to generatea FRB using previously defined and fixed fuzzy sets [13, 16,17]. This approach was adopted by Castro & Camargo whoproposed a method consisting of three stages: an attributeselection process, a GA to induce the rules and, in sequence,another GA to eliminate unnecessary rules [4].

However, depending on the number of variables and setsin the defined partition for a given problem, the total numberof possible rules can be extremely large, making it difficultto generate and codify the chromosomes and, consequently,the whole genetic learning process is overloaded.

As an alternative approach to deal with the dimensional-ity problem, Cintra & Camargo have recently proposed theapproach of genetic generation of FRBs from a set of can-didate rules preselected by a heuristic criteria based on theDegree of Coverage (DoC) as described in [5].

In this work, the method proposed in [5] is refinedthrough the enhancement of the fitness calculation with theuse of a self-adaptive selection process, aiming at obtainingFRB with small number of rules. The adaptive algorithmdescribed here uses reference values for the best number ofrules and best correct classification rates that are updatedin each GA iteration to penalize the individuals with largernumber of rules. The proposed strategy increases the se-lection probability of individuals with less rules and similarperformance.

This paper is organized as follows. In Section 2 themain characteristics of Fuzzy Classification Systems arepresented. In Section 3 Rule Based Genetic-Fuzzy Sys-tems are discussed. Section 4 presents the methodology and

2611-4244-1500-4/07/$25.00 ©2007 IEEE

relevant implementation details, followed by the results inSection 5. Finally, the conclusions and perspectives are dis-cussed in Section 6.

2 Fuzzy Classification Systems

Classification is an important task employed in many dif-ferent areas, such as pattern recognition, decision makingand data mining.

A classification task can be roughly described as: Givena set of objects E = {e1, e2, ..., eM} , also named pat-terns, assign a class Cj from a set of classes C ={C1, C2, ..., CJ} to an object ep, ep = {ap1, ap2, ..., apn}which is described by n attributes.

Fuzzy Classification Systems (FCS) are RBFS designedto perform a classification task, that requires the attributedomains to be granulated by means of fuzzy partitions. Thelinguistic variables in the antecedent part of the rules repre-sent attributes and the consequent part represents the class.

A typical classification fuzzy rule can be expressed by:

Rk : IFX1is A1l1AND...ANDXnis AnlnTHEN Class = Cj

(1)in which Rk is the rule identifier, X1, ..., Xn are the at-tributes of the pattern considered in the problem (repre-sented by linguistic variables), A1l1 , ..., Anln are the lin-guistic values used to represent the values of these attributesand Cj is the class, fuzzy or not, the pattern belongs to.

An inference mechanism applies the FRB to the patternto be classified, determining the class it belongs to. Most ofthe FCS use the Classic Fuzzy Reasoning Method (CFRM)[13], which classifies a pattern using the rule that has thehighest compatibility degree with the pattern, as describednext.

Let ep = {ap1, ap2, ..., apn} be a pattern andR1, R2, ..., RS , a set of S rules of a classification system,each with n antecedents. Let Aili(api), with i = 1, ..., n,be the membership degree of attribute api, of pattern ep,in the i-th fuzzy set of fuzzy rule Rk as defined in (1). TheCFRM applies the following steps to classify the pattern ep:

1. Calculates the compatibility degree between pattern ep

and each rule Rk, for k = 1, ..., S.

Compat(Rk, ep) =t(A1l1(ap1), A2l2(ap2), ..., Anln(apn))

in which t denotes a t-norm.

2. Finds the rule Rkmax with the highest compatibilitydegree with the pattern:

max{Compat(Rk, ep)}, k = 1, 2, ..., S

3. Assigns class Cj to pattern ep , so that Cj is the classpredicted by rule Rkmax found in the previous step.

3 Rule Based Genetic-Fuzzy Systems

This work focuses on a particular type of GFS, namelythe Rule Based Genetic-Fuzzy Systems (RBGFS), whichare RBFS equiped with GA learning capabilities.

The topic of the RBGFS, although quite recent, has al-ready originated an expressive number of works and, as aresearch area, continues to grow today. In spite of the dif-ficulty in classifying the recent approaches that come intolight every year, bringing innovative focuses, it is alreadypossible to identify some classical approaches, as well asnew tendencies.

The organization of the existing RBGFS in categories isbased on the definition of two basic aspects:

1. which class of problems is being dealt with - adapta-tion of previously defined elements of the KB or designof KB elements without previous definition;

2. which part of the KB is the target of optimization byGA.

Thus, the methods that combine the genetic and fuzzyapproaches for the generation of KBs can be divided intotwo main groups: methods that adjust the components ofthe KB (Genetic Adaptation) and methods that build KBcomponents (Genetic Construction).

Included in the Genetic Adaptation group are the meth-ods that initiate the process with an existing FRB or FDBand use GA to improve the performance of the systemby adjusting of adapting one or more parts of the KB.These methods can be divided in two subgroups accord-ing to the adopted focus: Genetic Adaptation of fuzzy sets[2, 3, 6, 14, 15] and Genetic Optimization of the number ofrules [6, 18].

The group here called Genetic Construction includes themethods that use GA to effectively build, or design, oneor more components of the KB. The previous definition ofthe KB components can be necessary if the goal of the ge-netic process is the generation of the other component ofthe KB. This group of methods is very active and gener-ated the largest number of proposals. It can be dividedinto three subgroups: Genetic Construction of the FRB[13, 16, 17, 29], Genetic Construction of the DB [8], andGenetic Construction of the KB [1, 7, 23, 21, 28].

Besides the classical approaches just described, new pro-posals continuously emerge in the literature. For instance,the proposal described in [19] uses GA to derive the FRBfrom a set of candidate rules, which were generated usingthe well-known evaluation measures of confidence and sup-port from data-mining.

The present work focuses the approach of FRB construc-tion using a predefined FDB. Thus, the genetic generationof the fuzzy rules is preceded by a process that selects can-didate rules from the set of all possible rules. The selection

262

of fuzzy rules is based on their DoC, as described in the nextsection.

4 Genetic Generation of Fuzzy Rules

In the automatic generation of FRB using GA, the searchspace is defined by sets of a certain number of rules andpotentially includes all the possible rules, that is, any ruleresulting from the combination of the possible values forthe variables can appear in the FRB.

As the number of variables increases, the set of possiblerules increases exponentially, interfering in the result of thelearning process, and even, in some situations, making itunfeasible. For instance, if the classification problem has 10input variables and the partition for the problem has 3 fuzzysets for each of the variables, the number of possible ruleswill be over 59,000. If the problem has 14 input variables,this number will represent over 4.7 million possible rules.

The method presented in [5] uses a criteria based onheuristic knowledge for the preselection of candidate rulesto be considered by the GA, so that the dimension of thesearch space can be reduced and the codification of the FRBin the chromosomes can be simplified.

The heuristic knowledge is associated to the DoC of therules. The main idea is that, although the DoC by itself isnot a good selection parameter to select rules to be part ofthe FRB, empirical studies show that it allows the discardof a large number of the possible rules without any qualityloss for the FRB generated by the whole learning process.The rules to be discarded are the ones with low or null DoCvalues. The calculation of the DoC is presented next.

Let E = {e1, e2, ..., eM} be a set of examples. The DoCof rule R with relation to E (DoCR) is defined as:

DoCR =M∑

i=1

{DoC(R,ei)}

so that DoC(R,ei)is the DoC of rule R with respect to exam-ple ei, obtained through the aggregation of the membershipdegrees of the attribute values of ei in the correspondingfuzzy sets appearing in the antecedent part of R.

In the method described here, once the fuzzy partitionsof the attribute domains are defined, the DoC value is cal-culated for all possible rules, which are then sorted in de-creasing order, by value of DoC.

The method proposed by Wang & Mendell (WM) [30,31] was used as a reference for the definition of several pa-rameters in the proposed method. The two criteria used topreselect the candidate rules were:

1. Consider the ordered rule set from top, up to the pointwhere all the rules presented in the FRB generated bythe WM method are included;

2. Consider the ordered rule set containing all the ruleswith non-null DoC.

The two criteria gave rise to two different versions ofthe proposed method concerning the preselection phase. Inboth cases, the set of candidate rules is used as a reducedsearch space for generation of the FRB using GA.

The codification scheme and the fitness calculation aredescribed next.

4.1 Chromosome codification

The preselection of candidate rules used here allows eachrule to be uniquely identified by its position in the sortedrules. The identification induces a very simple binary codi-fication of FRB in each chromosome and, consequently, theuse of very simple processes to create and handle the chro-mosomes.

The size of each chromosome was set as the total numberof preselected rules with a direct correspondence betweenthe rule position in the sorted rules list and the correspond-ing gene position in the chromosome. The rules that are inthe rule base are represented in the chromosome with digit1 in its corresponding gene, meaning that the rule is ac-tive. Rules that are inactive, or do not belong to the FRBrepresented by the chromosome, are coded by 0 in the cor-responding gene.

Figure 1 presents a chromosome with 10 positions, rep-resented in the binary system, with rules 1, 4, 5, 6 and 9 ac-tive, all the others inactive.

Figure 1. Binary Chromosome Representinga Complete Rule Base

For the initial population the chromosomes were createdwith a percentage of active rules based on the number ofrules generated by the WM method. The chromosomeswere randomly generated and conflicting rules were elimi-nated by discarding the chromosomes with conflicting rulesand generating new ones.

4.2 Fitness Calculation

Ever since the first proposals in the field of genetic gen-eration of FS started to appear, special attention has beendevoted to the problem of generating models that are bothtransparent and accurate. The number of rules in a rule baseis one of the parameters that favors transparency, as long asa FRB with a small number of rules can make the modeleasily understood by the user.

263

Several approaches in the field of RBGFS cope with theproblem of reducing the FRB size and yet maintaining itsaccuracy in different ways. Some of them first generate anaccurate FRB that may contain irrelevant rules and, in thesequel, apply an independent process, usually also basedon GA, with the specific goal of reducing the FRB size,by eliminating the irrelevant rules [3, 9]. Other approachessearch for FRB that are accurate and transparent in one step,defining the fitness calculation as a balance between para-meters that measure the RFB performance and parametersthat measure the FRB size.

In this work, the method proposed by the authors in [5]is reviewed and refined with the specific objective of im-proving the evaluation step performed by the GA in orderto reduce the number of rules in the final FRB during thesearch process. The fitness value is calculated using theCorrect Classification Rate (CCR) and the number of rulesin the FRB represented by each chromosome.

The evaluation process developed here and presented inpseudo code Algorithm 1 uses a self-adaptive algorithm toperform fitness evaluation. The algorithm updates the refer-ence values of ideal CCR and number of rules (NR), so thatthe NR is used in a penalization mechanism that decreasesthe initial fitness value when the number of rules in the FRBis bigger than the current reference number.

The initial values of the reference number of rules(Best NR) and the reference Correct Classification Rate(Best CCR) are set as the corresponding values found bythe WM method. The values are updated in each generationand then used in the fitness calculation.

The general evaluation process applied in each GA iter-ation can be described as:

1. Calculate the CCR for each chromosome and set it asthe initial fitness value;

2. Update the referential values Best NR andBest CCR using the NR and CCR found in theprevious step;

3. Calculate the final fitness value of each chromosome,applying the penalization mechanism.

The penalization rates are shown in Table 1. These rateswere defined empirically. Further studies on the definitionof these rates will be included in future works.

The algorithm presented in Algorithm 1 describes in de-tail the evaluation process.

Table 1. Penalization rates for the fitness ofthe chromosomes according to their numberof active rules

Number of Rules Fitness Value≤ NRWM CCR≤ NRWM ∗ 1.5 CCR/1.25≤ NRWM ∗ 2 CCR/1.5≤ NRWM ∗ 3 CCR/2> NRWM ∗ 3 CCR/3

Algorithm 1 Algoritm for the definition of the optimalnumber of rules of the generated RBs

C = chromosomeNR = number of rulesCCRWM = CCR of the RB generated by WMNRWM = NR of rules in the RB generated by WMBest NR = reference number of rulesBest CCR = reference CCR

1: Best NR = NRWM2: Best CCR = CCRWM3: for a = 0; a <number of chromosomes; a ++ do4: Fitness of Ca = CCR of Ca

5: end for6: for a = 0; a <number of chromosomes; a ++ do7: if NR of Ca ≤ Best NR then8: if CCR of Ca ≤ Best CC then9: Best NR = NR of Ca;

10: Best CCR = CCR of Ca;11: end if12: end if13: end for14: for a = 0; a <number of chromosomes; a ++ do15: if NR of Ca > Best NR then16: if NR of Ca ≤ Best NR ∗ 1.5 then17: Fitness of Ca = Fitness of Ca/1.2518: else19: if NR of Ca ≤ Best NR ∗ 2 then20: Fitness of Ca = Fitness of Ca/1.521: else22: if NR of C a ≤ Best NR ∗ 3 then23: Fitness of Ca = Fitness of Ca/224: else25: Fitness of Ca = Fitness of Ca/326: end if27: end if28: end if29: end if30: end for

In the next section, the results obtained in the experi-

264

ments are presented and discussed.

5 Experiments and Results Analysis

In this section the data sets and the obtained results arediscussed. All experiments with GAs were performed with250 iterations, elitism rate of 5%, crossing rate of 70% andmutation rate of 5%. The learning process used a 5-foldcross validation strategy.

The domains used are available at the UCI MachineLearning repository [24]. The choice of each data set wasbased on their attribute type (numerical-valued ones). Only4 attributes were used in the process. In those data sets withlarger number of attributes, the ones used were randomly se-lected. Table 2 summarizes the domain characteristics giv-ing the total number of instances as well as the number ofattributes (class included).

Table 2. Domain characteristicsDomain # Instances # Attributes (class included)Diabetes 724 5MPeG 392 5

Iris 150 5Machine 209 5

For each one of the sets, 3 distinct partitions were de-fined, with 3, 5 and 7 fuzzy sets for each input attribute,totalizing 12 different experiments.

Table 3 shows the total number of possible rules for eachpartition and the average number of rules (for all folds) inthe FRB produced by each of the experiments. The numeri-cal suffix added to each domain name represents the numberof fuzzy sets for each input variable.

Table 3. Number of rules.Domain Total GA I GA II WM

Diabetes 3 162 14.2 28.4 24Diabetes 5 1250 51 67.4 84.4Diabetes 7 4802 38.6 75.4 159.2

MPeG 3 243 13 16.4 20MPeG 5 3125 36.8 46 46.2MPeG 7 16807 34.4 51.2 76.6

Iris 3 243 7.6 13.8 15Iris 5 1875 15 41.2 44.8Iris 7 7203 54.4 61.4 67.2

Machine 3 243 6.2 12.8 13.8Machine 5 3125 21.6 25.4 29Machine 7 16807 26 27.8 33

Table 4 presents the CCRs for the FRB generated in eachexperiment.

Table 4. Correct classification ratesDomain GA I GA II WM

Diabetes 3 100 99.77 91.10Diabetes 5 99.97 99.39 89Diabetes 7 92.90 94.90 86.90Average 97.62 97.53 89

MPeG 3 87.51 86.32 79.30MPeG 5 78.69 74.12 76.80MPeG 7 64.80 51.81 62.30Average 77 70.15 72.8

Iris 3 99.71 98.70 100Iris 5 100 100 100Iris 7 98.31 97.30 94.70

Average 99.34 98.67 98.23

Machine 3 94.20 95.10 93.70Machine 5 94.30 92.00 95.60Machine 7 91.40 86.90 92.80Average 93.3 91.33 94.03

The results demonstrate that the two versions of the pro-posed method, GA I and GA II, generated a number o rulesconsiderably smaller than the WM method. The obtainedCCRs are also better in almost all experiments for GA Iwhen compared to either GA II or WM method.

The number of rules and CCRs shown in Table 3 and4 demonstrate that the preselection of fuzzy rules by theheuristic method is a promising approach to cope with theproblem of dimensionality.

The use of the self-adaptive algorithm for the definitionof the ideal number of active rules in each chromosome hasimproved the learning process as a whole guiding the gen-eration method to a better result with a faster convergencetime.

6 Conclusion

This work presents an improvement to the method pro-posed earlier by the authors to generate fuzzy rules usingGA and preselection of candidate rules. The process of pre-selection is divided in two stages. In the first one, all possi-ble rules for a given problem are generated. In the secondstage, these rules are classified and ordained according totheir DoC, and the most representative rules are preselectedto be used by a GA for the generation of FRBs. The im-provement presented here refers to the use of a self-adaptivealgorithm for fitness calculation which speeds up the search

265

process, favors the FRB size reduction, and enhances theKB performance.

The method demonstrated to be a promising approach tocope with the relevant problem of finding a balance betweenKB accuracy and transparency.

References

[1] P. P. Angelov. An evolutionary approach to fuzzy rule-basedmodel synthesis using indices for rules. Fuzzy Sets and Sys-tems, 137:325–338, 2003.

[2] P. S. C. Y. T. Bonissone, P.; Khedkar. Genetic algorithms forautomated tuning of fuzzy controllers, a transportation ap-plication. IEEE International Conference on Fuzzy Systems,FUZZ-IEEE’96:674–680, 1996.

[3] O. Casillas, J.; Cordon, M. Jesus, and F. Herrera. Genetictuning of fuzzy rule deep structures preserving interpretabil-ity and its interaction with fuzzy rule set reduction. IEEETransactions on Fuzzy Systems, 13:13–29, 2005.

[4] P. A. D. Castro and H. A. Camargo. Focusing on inter-pretability and accuracy of genetic fuzzy systems. IEEE In-ternational Conference on Fuzzy Systems, pages 696–701,2005.

[5] M. E. Cintra and H. A. Camargo. Fuzzy rules generationwith pre-selection of candidate rules. submitted for publica-tion - in Portuguese, 2007.

[6] O. Cordon and F. Herrera. A three-stage evolutionaryprocess for learning descriptive and approximate fuzzy logiccontroller knowledge bases from examples. InternationalJournal of Approximate Reasoning, 7:369–407, 1997.

[7] O. Cordon and F. Herrera. Hybridizing genetic algorithmswith sharing scheme and evolution strategies for designingapproximate fuzzy rule-based systems. Fuzzy Sets and Sys-tems, 118:235–255, 2001.

[8] O. Cordon, F. Herrera, and P. Villar. Generating the knowl-edge base of a fuzzy-based system by the genetic learningof the data base. IEEE Transactions on Fuzzy Systems,9(4):667–674, 2001.

[9] O. Cordon and F. Herrera. A tree-stage evolutionary processfor learning descriptive and aproximate fuzzy logic con-troller knowledge bases from examples. International Jour-nal of Approximate Reasoning, 7(4):369–407, 1997.

[10] O. Cordon, F. Herrera, F. Gomide, F. Hoffmann, and L. Mag-dalena. Special issue on genetic fuzzy systems. Fuzzy Setsand Systems, 141, 2004.

[11] D. Dumitrescu, B. Lazzerini, and L. Jair. Fuzzy Sets andTheir Applicatiion to Clustering and Training. CBC Press,2000.

[12] D. E. Goldberg. Genetic Algorithms in Search, Optimizationand Machine Learning. Addilson-Wesley, 1989.

[13] A. Gonzales and R. Peres. Slave: A genetic learning sys-tem based on an interative approach. IEEE Transactions onFuzzy Systems, 7:176–191, 1999.

[14] H. B. Gurocak. A genetic-algorithm-based method for tun-ing fuzzy logic controllers. Fuzzy Sets and Systems, 108-1:39–47, 1999.

[15] F. Herrera, M. Lozano, and J. L. Verdegay. Tuning fuzzylogic controllers by genetic algorithms. International Jour-nal of Approximate Reasoning, 12:299–315, 1995.

[16] F. Hoffmann. Combining boosting and evolutionary algo-rithms for learning of fuzzy classification rules. Fuzzy Setsand Systems, 141:47–58, 2004.

[17] H. Ishbuchi, T. Nakashima, and T. Murata. Performanceevaluation of fuzzy classifier systems for multidimensionalpattern classification problems. IEEE Transaction on FuzzySystems, Man and Cybernetics, 29(5):601–618, 1999.

[18] H. Ishibuchi, T. Murata, and I. B. Turksen. Single-objectiveand two-objective genetic algorithms for selecting linguis-tic rules for pattern classification problems. Fuzzy Sets andSystems, 89:134–150, 1997.

[19] H. Ishibuchi and T. Yamamoto. Fuzzy rule selection bymulti-objective genetic local search algorithms and ruleevaluation measures in data mining. Fuzzy Sets and Systems,141:59–88, 2004.

[20] S. R. Jang, C. T. Sun, and E. Mizutani. Neuro-Fuzzy andSoft Computing. Prentice Hall, 1997.

[21] M. Kim, C. Kim, and J. Lee. Evolving compact and inter-pretable takagi-sugeno fuzzy models with a new encodingscheme. IEEE Transactions on Systems, Man, and Cyber-netics - Part B: Cybernetics, 36:1006–1023, 2006.

[22] T. W. Liao, A. K. Celmins, and R. J. Hammell. A fuzzy c-means variant for the generation of fuzzy term sets. FuzzySets and Fuzzy Systems, 135:241–257, 1997.

[23] L. Magdalena and F. Monasterio. A fuzzy logic controllerwith learning through the evolution of its knowledge base.International Journal of Approximate Reasoning, 16:335–358, 1997.

[24] C. J. Merz and P. M. Murphy. UCI repository of machinelearning databases, [http://www.ics.uci.edu]. Irvine, CA,UCLA., 1998.

[25] H. Nomura, L. Hayashi, and N. Wakami. A learning methodof fuzzy inference rules by descent method. In IEEE In-ternational Conference on Fuzzy Systems, pages 203–210,1992. San Diego - USA.

[26] W. Pedrycz. Fuzzy Modelling: Paradigms and Practice.Kluwer Academic Press, 1996.

[27] W. Pedrycz. Computational Intelligence: An Introduction.CRC Press, 1998.

[28] V. M. Rivas, J. J. Merelo, I. Rojas, G. Romero, P. A. Castillo,and J. Carpio. Evolving two-dimensional fuzzy systems.Fuzzy Sets and Systems, 138:381–398, 2003.

[29] D. Spiegel and T. Sudkamp. Sparse data in the evolution-ary generation of fuzzy models. Fuzzy Sets and Systems,138:363–379, 2003.

[30] L. Wang. IEEE Trans. on Fuzzy Systems, 11:768–782, 2003.[31] L. Wang and J. Mendel. Generating fuzzy rules by learning

from examples. IEEE Trans. on SMC, 22:414–1427, 1992.

266