for user modeling adaptive bayessweet.ua.pt/gladys/papers/eunite2002.pdf · adaptive bayes for user...

31
Adaptive Bayes Adaptive Bayes for for User Modeling User Modeling . João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 1 EUNITE2002. Algarve. Portugal EUNITE2002. Algarve. Portugal Adaptive Adaptive Bayes Bayes for User Modeling for User Modeling João Gama * and Gladys Castillo ** * LIACC, FEP University of Porto e-mail: [email protected] ** Department of Mathematics University of Aveiro e-mail: [email protected]

Upload: others

Post on 26-Jun-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 1

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Adaptive Adaptive BayesBayesfor User Modelingfor User Modeling

João Gama*

andGladys Castillo**

* LIACC, FEP University of Porto

e-mail: [email protected]** Department of Mathematics

University of Aveiroe-mail: [email protected]

Page 2: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 2

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Goals & MotivationsGoals & Motivations

Goals:� to present Adaptive Bayes, an extension to

the Naïve Bayes for user modeling.� to show the advantages of the adaptive

models in comparison against their non-adaptive versions

Motivations:� User modeling is a very dynamic modeling

task, so it is important learning algorithms be able to quickly adapt the model in order to reflect the changes of the user behaviors.

One approach:to implement adaptive models for user modeling tasks

1. Introduction

���� Goals & MotivationsGoals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 3: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 3

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Machine LearningMachine Learning

What to learn?Some of the main tasks that can be learned are:� Classification:

� Supervised learning → concept learning� Unsupervised learning → clustering

� Prediction:� learning to predict future data trends

(actions)� Problem solving:

� learning to solve new problems

Machine Learning is concerned with automatic computing procedures that learn a task from a series of examples

1. Introduction

���� Goals & Motivations

���� ML for User ModelingML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 4: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 4

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Machine Learning for User ModelingMachine Learning for User Modeling

User Modeling involves inferring unobservable information about a user from observable information about him/her actions.

Two main approaches: �content-based: the behavior of a user is predicted from his/her

past behavior�collaborative: the behavior of a user is predicted from the

behavior of a community of users with commoncharacteristics and interests

Main applications fields of ML for UM:� Adaptive Hypermedia Systems

� Tutoring Systems: to build cognitive and predictive models for the students

� Information retrieval� Knowledge Discovery

User Modeling is one of the most challenging application field for machine learning

1. Introduction

���� Goals & Motivations

���� ML for User ModelingML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 5: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 5

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

A typical prediction task for UM (1)A typical prediction task for UM (1)

TrainingTrainingExamplesExamples

ML ML MethodMethod

PredictivePredictiveModelModel

1º. 1º. Learning Process:Learning Process:

� Training Examples: observations of the user behaviors

1. Introduction

���� Goals & Motivations

���� ML for User ModelingML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works� Model Induction: A ML method uses training

examples to build a predictive model of user future actions

Page 6: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 6

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

A typical prediction task for UM (2)A typical prediction task for UM (2)

2º.2º. Prediction taskPrediction task: given the user behaviours the predictive model infers the future user actions

3º. 3º. System AdaptationSystem Adaptation: the predicted user actions are used to adapt the behavior of the system

PredictivePredictiveModelModelII11 II22 IInn... PP11 PP22 PPnn...

Inputs: User behavioursInputs: User behaviours Outputs: Predicted user actionsOutputs: Predicted user actions

AdaptationAdaptationAlgorithmAlgorithm

AdaptiveAdaptiveSystemSystem

1. Introduction

���� Goals & Motivations

���� ML for User ModelingML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

For instance:

� modifying its dialogue strategy

� to recommend objects a user may be interested

� to perform actions on behalf of a user (ex: pre-sending WWW pages, forwarding email, etc.)

Page 7: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 7

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Limitations of Machine Learning for Limitations of Machine Learning for User ModelingUser Modeling

Some of main limitations of machine learning methods for real-world user modeling tasks are:

� a lot of data is required to learnthe user model cannot wait to acquire thousands of data before learning can begin, it must start learning with very few interactions

� labeled data for supervised classificationthe correct labels may not be readily apparent from simple observation of the user’s behavior

� a lot of computation time before learning can occurmost user modeling tasks must be done in real time, thus the computations must be rather quick.

� concept driftuser modeling is a very dynamic task, concept drift can occur, so, learning algorithm ought to be able to adjust the model parameters quickly as new data arrives

1. Introduction

���� Goals & Motivations

���� ML for User ModelingML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 8: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 8

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Supervised ClassificationSupervised Classification

�Supervised Classification = Concept Learning� given: a training set of labelled examples � induce: a classifier (concept description) that

accurately classifiers unlabeled examples

�There are two phases to constructing a classifier:� learning phase: the training set is used to build the

classifier (i.e., to determine how the parameters ought to be weighted and combined in order to separate the various classes of objects).

� application phase: the classifier is applied to a set of objects that do not have known classes in order to determine what their classes are belonging.

Any classification method uses a set of features (attribute-values) to characterize each object (example).In supervised classification each example is categorized (labelled ) in a class.

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Naïve BayesBayes ClassifierClassifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 9: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 9

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Supervised ClassificationSupervised Classification

�Methods for Supervised Classification:� Decision Tree Induction� Logistic Regression� Support Vector Machine� Neural Networks� K- nearest-neighbour classifiers� Bayesian classifiers →→→→ statistical classifiers

they predict class membership probabilities, i.e., the probability that a given example belongs to a particular class

� Bayesian Networks � Naïve Bayes Classifier

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Naïve BayesBayes ClassifierClassifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 10: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 10

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

The Naïve The Naïve BayesBayes ClassifierClassifier

NB Classifier usesNB Classifier uses:�Bayes theorem to determine the probability that an

example x belongs to class Ci

P(Ci | x) = P(Ci ) P(x | Ci ) / P(x) it provides a simple quantitative method to compute theposterior probability using the results of prior information

⇒ P(Ci | x) ∝ P(Ci ) P(x | Ci )P(x) can be ignored (it is the same for all the classes)

�Attributes independent assumption → so Naïve !!!

⇒ P(Ci | x) ∝ P(Ci ) ∏ P(xj | Ci )

Given a training set of labelled examples how to determine the probability that a given example belongs to a particular class?

C = argmaxi P(Ci | x)

Classification taskClassification task:1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Naïve BayesBayes ClassifierClassifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Then:Then:

The class C attached to example x is given by:

Page 11: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 11

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

The Naïve The Naïve BayesBayes ClassifierClassifierImplementation DetailsImplementation Details

� to compute the prior probability of each class Ci

All the required prior probabilities are computed from the training examples in one step

How are the prior probabilities computed?

examplestobelongingexamplesCP i

i #C #)( =

� to compute the conditional probability P(xj | Ci )Case 1- nominal attributes:

the following counting method is used:

i

jiij Ctobelongingexamples

xvalueattributethehavingCofexamplesCxP

# #

)|( =

Case 2 - continuous attributes: Two possibilities:1. A Gaussian distribution is assumed for this attribute2. The attribute is discretized into a number of intervals

and treaty as nominal.

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Naïve BayesBayes ClassifierClassifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 12: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 12

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Naïve Naïve BayesBayes & User Modeling& User Modeling

Naïve Bayes is one of the machine learning algorithm more used in UMVantages:

� it is simple – a very intuitive technique� quick learning- don’t require large amount of

data to learn� low computation to make decision� it is easy to apply- the decision is given as

probability distributionSome related works:

� M. Stern et al, Naïve Bayes classifier for user modeling, 1999application: for prediction tasks in the web-based

instructional system MANIC� K.Nigam et al. Text classification from labeled and unlabeled

documents using EM, 1999� M.Pazzani et al. Learning and revising user profiles: the

identification of interesting web sites, 1997

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Naïve BayesBayes & User Modeling& User Modeling

2. Iterative Bayes

���� Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 13: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 13

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Iterative Iterative BayesBayes

� Aim: to improve the class probability associatedwith Naïve Bayes predictions

� Main algorithm idea:� Init:

the set of contingency tables built by Naïve Bayes� Iterative Procedure:

iteratively cycle trough all the training examples�for each example an adaptation process is running:

the corresponding entries in the contingency tables areupdated in order to increase the confidence onthe correct class

� Stop:A stop criteria is defined

Note: Experimental evaluation of Iterative Bayes on 27 benchmark datasets show consistent gain in accuracyin the previous work: João Gama, Iterative Bayes, Intelligent Data Analysis 4,

475-488, IOS Pres, 2000

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Iterative BayesBayes

���� Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 14: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 14

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Adaptation Algorithm for Iterative Adaptation Algorithm for Iterative BayesBayes

Procedure AdaptModel (Model, Example, Observed, Predicted, Step)

// 1º. Compute the increment DeltaDelta = (1-P(Predicted|Example)) ← a heuristic, intuition !!! If Predicted <> Observed then Delta=-1 x Delta

// 2º. The increment is used to update the contingency tables:// Model(k+1) = Model (k) + Step X Delta

For each Attribute // for each contingency tableFor each Class

if (Observed = Predicted) then // to increment the entryModel(Attribute, Class, AttributeValue)+= Step x Delta

else // to decrease the entriesModel(Attribute, Class, AttributeValue)-= Step x Delta /

#Classesif Model(Attribute, Class, AttributeValue) < 1.0 then

Model(Attribute, Class, AttributeValue) = 1.0 Next Class

Next AttributeReturn: Model

End

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation AlgorithmAdaptation Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 15: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 15

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

The BalanceThe Balance--scale Problemscale Problem

� Each example has 4 continuous attributes: � the left weight (Left_W)� the left distance (Left_D)� the right weight (Right_W)� right distance (Right_D)

discretization is applied: each attribute is mapped to 5 intervals

This is an artificial problem from the UCI repository. The dataset was generated to model psychological experimental results

LeftLeft_W_W

RightRight_W_W

LeftLeft_D_D

RightRight_D_D

� Each example is classified into 3 classes:the balance-scale:� tip to the right (Right)� tip to the left (Left)� is balanced (Balanced)

� The rules to find the class are:� If LD x LW > RD x RW ⇒ tip to the left� If LD x LW < RD x RW ⇒ tip to the right� If LD x LW = RD x RW ⇒ it is balanced

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative exampleAn illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 16: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 16

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Naïve Naïve BayesBayes for the Balancefor the Balance--Scale ProblemScale Problem

Learning

Phase

A counter for each class is computed. For each attribute a contingency table is built to reflect the distribution on the training set of the attribute-values over the classes

565 examples

Contingency Tables

AttributeAttribute: : LeftLeft_W_W

2534496686RightRight

9108810BalancedBalanced

7271614214LeftLeft

I5I5I4I4I3I3I2I2I1I1ClassClass

AttributeAttribute: : LeftLeft_W_W

2534496686RightRight

9108810BalancedBalanced

7271614214LeftLeft

I5I5I4I4I3I3I2I2I1I1ClassClass

AttributeAttribute: : LeftLeft_D_D

2737495790RightRight

8109108BalancedBalanced

7770593816LeftLeft

I5I5I4I4I3I3I2I2I1I1ClassClass

AttributeAttribute: : LeftLeft_D_D

2737495790RightRight

8109108BalancedBalanced

7770593816LeftLeft

I5I5I4I4I3I3I2I2I1I1ClassClass

AttributeAttribute: : RightRight_W_W

7970583716RightRight

8910108BalancedBalanced

2833496387LeftLeft

I5I5I4I4I3I3I2I2I1I1ClassClass

AttributeAttribute: : RightRight_W_W

7970583716RightRight

8910108BalancedBalanced

2833496387LeftLeft

I5I5I4I4I3I3I2I2I1I1ClassClass

AttributeAttribute: : RightRight_D_D

8267573717RightRight

9108108BalancedBalanced

2535446591LeftLeft

I5I5I4I4I3I3I2I2I1I1ClassClass

AttributeAttribute: : RightRight_D_D

8267573717RightRight

9108108BalancedBalanced

2535446591LeftLeft

I5I5I4I4I3I3I2I2I1I1ClassClass

260

LeftLeft

Classes Classes CountersCounters

56526045

TotalTotalRightRightBalancedBalanced

260

LeftLeft

Classes Classes CountersCounters

56526045

TotalTotalRightRightBalancedBalanced

Balanced2643

...

2

1

LeftLeft_W_W

BalanceBalance--Scale DataSetScale DataSet

............

Left235

Right245

ClassClassRightRight_D_DRightRight_W_WLeftLeft_D_D

Balanced2643

...

2

1

LeftLeft_W_W

BalanceBalance--Scale DataSetScale DataSet

............

Left235

Right245

ClassClassRightRight_D_DRightRight_W_WLeftLeft_D_D

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative exampleAn illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Training data set

Page 17: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 17

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Naïve Naïve BayesBayes for the Balancefor the Balance--Scale ProblemScale Problem

Predicted = argmaxi P(Ci | x)

Classification Phase:To predict the class for the following example:

1

LeftLeft_W_W

Right245

ClassClassRightRight_D_DRightRight_W_WLeftLeft_D_D

1

LeftLeft_W_W

Right245

ClassClassRightRight_D_DRightRight_W_WLeftLeft_D_D

The output of the Naive Bayes for this example is:

0.277796

P(P(LeftLeft|x)|x)

0.5869780.135227

P(P(RightRight|x)|x)P(P(BalancedBalanced|x)|x)

Predicted = Right

where P(Ci |x) ∝ P(Ci ) x P(Left_W=1 | Ci) x P(Left_D=5 | Ci )

x P(Right_W=4 | Ci ) x P(Right_W=4 | Ci ),Ci ∈ {Left, Balanced, Right}

Correct classification

The class counters and contingency tables are used to compute the posterior probabilities for each class.

59% - Low confidence ?

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative exampleAn illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 18: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 18

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

--+

Iterative Iterative BayesBayes for the Balancefor the Balance--Scale ProblemScale Problem

1

LeftLeft_W_W

Right245

ClassClassRightRight_D_DRightRight_W_WLeftLeft_D_D

1

LeftLeft_W_W

Right245

ClassClassRightRight_D_DRightRight_W_WLeftLeft_D_D

Attribute: Left_WAttribute: Left_W

2534496686RightRight

9108810BalancedBalanced

7271614214LeftLeft

I5I5I4I4I3I3I2I2I1I1ClassClass

Attribute: Left_WAttribute: Left_W

2534496686.41RightRight

910889.80BalancedBalanced

7271614213.80LeftLeft

I5I5I4I4I3I3I2I2I1I1ClassClass

0.277796

P(P(LeftLeft|x)|x)

0.5869780.135227

P(P(RightRight|x)|x)P(P(BalancedBalanced|x)|x)

0.277796

P(P(LeftLeft|x)|x)

0.5869780.135227

P(P(RightRight|x)|x)P(P(BalancedBalanced|x)|x)

ActualModel

Output:class prediction

Input: an example

Observed=RightPredicted=Right

1º. Compute the increment

Delta = (1- (Predicted|Example)Delta= 0.413022 > 0 if correct

2º. Update the entries of the column Left_W= 1

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative exampleAn illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Start Adaptation

Page 19: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 19

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Iterative Iterative BayesBayes for the Balancefor the Balance--Scale ProblemScale Problem

for the actual model:

Observed=RightPredicted=Right

59% confidence levelIterative Bayescycles through training examples

0.277796

P(Left|x)P(Left|x)

0.5869780.135227

P(Right|x)P(Right|x)P(P(BalancBalanc.|x).|x)

AttributeAttribute: : LeftLeft_W_W

2534496686RightRight

9108810BalancedBalanced

7271614214LeftLeft

I5I5I4I4I3I3I2I2I1I1ClassClass

AttributeAttribute: : LeftLeft_W_W

2534496686RightRight

9108810BalancedBalanced

7271614214LeftLeft

I5I5I4I4I3I3I2I2I1I1ClassClass

AttributeAttribute: : LeftLeft_W_W

20.8937.0137.0162.63105.64RightRight

1.081.01.01.01.06BalancedBalanced

96.7092.2675.9842.517.06LeftLeft

I5I5I4I4I3I3I2I2I1I1ClassClass

AttributeAttribute: : LeftLeft_W_W

20.8937.0137.0162.63105.64RightRight

1.081.01.01.01.06BalancedBalanced

96.7092.2675.9842.517.06LeftLeft

I5I5I4I4I3I3I2I2I1I1ClassClass0.210816

P(Left|x)P(Left|x)

0.7890090.000175

P(Right|x)P(Right|x)P(P(BalancBalanc.|x).|x)

Observed=RightPredicted=Right

79% confidence level

after the iterative procedure:

The confidence level of the predict class

increases and on the other classes decreases

1

LeftLeft_W_W

Right245

ClassClassRightRight_D_DRightRight_W_WLeftLeft_D_D

1

LeftLeft_W_W

Right245

ClassClassRightRight_D_DRightRight_W_WLeftLeft_D_D

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative exampleAn illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 20: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 20

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Iterative Iterative BayesBayes: Discussion: Discussion

This parameter estimation problem can be formulated as the following non-linear optimization problem:

)|(Pr1(1),(minarg1

*∑

=

−==n

iixedictedP

nDWfW

where n - number of examples, Predicted – the predicted class for the example xi

based on the current set of contingency tables

Iterative Bayes implements a hill-climbing search for a parameter estimation task.

Given a dataset D and a initial set of contingency tables W0 built by the Naïve Bayes to determine the optimal set of contingency tables W* that minimizes the following evaluation function:

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative example

���� Some discussionSome discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 21: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 21

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Iterative Iterative BayesBayes: Discussion: Discussion

There are two main iterative approaches to solve a parameter estimation problem:

� gradient descent(ascent) - also known as “hill-climbing” � EM (Expectation/Maximization) – in the case of missing

values or hidden variables

Both algorithms perform a greedy local search on the surface defined by the model’s parameters and converge to a local optimum.

Some Related Works:� Nigam et al. – combine EM and Naïve Bayes classifier

for learning from labeled and unlabeled documents� Backpropagation algorithm for a multi-layer feed-

forward neural network is a “hill climbing” method� In parameter learning for Bayesian Networks

� Gradient approaches: Adaptive Probabilistic Networks S. Rusell et al., 1995; F. Jensen, 1999, E.Bauer et al. , 1997

� EM approaches: Lauritzen, 1991,1995; Spiegelhalter et al, 1993, Heckerman, 1995

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative example

���� Some discussionSome discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 22: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 22

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Adaptive Adaptive BayesBayes

The Iterative Bayes is a batch classifier (it learn from a pre-accumulated set of examples) that iteratively cycles over the training set to generate the model

What about an incremental scheme?

A fact:

The Naïve Bayes classifier is naturally incremental

but

in the case of concept drift, i,.e, when the concept description could change over time, the influence on pastinformation should be reduced to obtain a betterperformance

One idea: Based on the Iterative Bayes adaptation schemeto develop an incremental learning algorithm,that could also work on an on-line framework

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Adaptive BayesBayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 23: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 23

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Adaptive Adaptive BayesBayes

� We consider two adaptive versions:� Incremental Adaptive Bayes

it build a model from a training set updating the model once after seeing each example.A batch classifier too, but here is not iterative cycling over the training set

� On-Line Adaptive Bayesit works on an on-line framework, i.e., the model is continuously adapted when a new case arrives.

Adaptive Bayes is an incremental learning algorithm that implements the adaptation schema of the Iterative Bayes

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Adaptive BayesBayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 24: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 24

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Incremental Adaptive Incremental Adaptive BayesBayes

Procedure IncrementalBayes (TrainingSet, Model, Adaptive) // Initialization:

Model ← all counters are initialized to zero// Iteration:// cycle trough all the training examples

For each Example in TrainingSet

// 1º. Increment the countersIncrementCounters(Model, Example, Observed)

if Adaptive = True then // 2º. Predict the class

Predicted ← PredictClass (Model, Example)

// 3º. Adapt the modelAdaptModel(Model, Example, Observed, Predicted, Step)

Next ExampleReturn: Model

End

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Incremental BayesBayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 25: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 25

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

AdaptiveAdaptiveModelModel

LearningLearning//AdaptationAdaptation

3º. 3º. AdaptationAdaptation

The onThe on--line framework for Adaptive line framework for Adaptive BayesBayes

ObservedObserved

OnOn--Line Line SystemSystem

2º. 2º. FeedbackFeedback1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine BayesOnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

ActualActualModelModelxx11 xx22 xxnn...

PredictedPredicted

Input: new caseInput: new case

1º. 1º. ClassificationClassification

Page 26: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 26

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

OnOn--Line Adaptive Line Adaptive BayesBayes

Initialization: Model ← randomly to initialize all counters

Procedure OnLinelBayes (Example, Model, Adaptive)

// 1º. Make predictionPredicted ← PredictClass (Model, Example)

// 2º. FeedbackObserved ← ExampleClass

// 3º. Increment the countersIncrementCounters(Model, Example, Observed)

if Adaptive = True then // 4º. Adapt the model

AdaptModel(Model, Example, Observed, Predicted, Step)

Return: ModelEnd

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine BayesOnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future Works

Page 27: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 27

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Comparison between Incremental andComparison between Incremental andOnOn--line Naïve line Naïve BayesBayes against Adaptive against Adaptive Bayes Bayes

� on 23 benchmark problems from the UCI repository� evaluation statistic: the error rate

(percentage of misclassified examples) �� for incremental versionsfor incremental versions::

� Evaluation: a standard 10-fold cross validation.1. Training: the model is incrementally built from

each training data fold.2. Evaluation: the model is used to classify the test

set computing the error rate.� Result: the average of the 10 error rates

�� for onfor on--line versions:line versions:� Evaluation: all the available data is presented to the

algorithm in sequence 1. Each example is classified with the actual model2. The example is evaluated (misclassified?)3. The actual model is updated

The process is repeat 10 times using different permutations of the dataset

� Result: the average of the 10 error rates

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental EvaluationExperimental Evaluation

� Results

5. Conclusions and

Future Works

Page 28: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 28

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Summary of ResultsSummary of Results

Statistics show the advantages of using Adaptive Bayes

On-LineIncremental

00.179WilcoxonTest

182115Sign. Wins

194149Wins

0.8510.961ErrorRatio

1.1671.8331.391.61AverageRank

15.4718.5813.1913.98GeometricMean

18.5121.5616.6017.09ArithmeticMean

AdaptiveNaïve Bayes

AdaptiveNaïve BayesDataset

On-LineIncremental

00.179WilcoxonTest

182115Sign. Wins

194149Wins

0.8510.961ErrorRatio

1.1671.8331.391.61AverageRank

15.4718.5813.1913.98GeometricMean

18.5121.5616.6017.09ArithmeticMean

AdaptiveNaïve Bayes

AdaptiveNaïve BayesDataset

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� ResultsResults

5. Conclusions and

Future Works

Page 29: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 29

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

ConclusionsConclusions

� We have presented Adaptive Bayes, a new incremental algorithm based on the Naive Bayes and on the adaptation scheme used in Iterative Bayes

� In a set of benchmark datasets Adaptive Bayes shows clear advantages over the standard Naive Bayes both on predictive and on-line frameworks

� Learning algorithms for user modeling tasks should work on-line and be capable to quickly adapt the model to accommodate the changes of the user.The on-line framework of Adaptive Bayes seems to be a good and simple choice for the prediction tasks in user modeling

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. ConclusionsConclusions

Future Works

Page 30: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 30

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Future WorkFuture Work

� to implement and evaluate the on-line Adaptive Bayes in various prediction tasks included in a Web-based Adaptive Course in GeometryThis adaptive course is being developed by the project Geometrix sited in the Department of Mathematics of the Aveiro University

� These prediction tasks are:� In diagnostic testing: considering the student’s

knowledge level, to predict the future answers of the student to a set of questions to decide about the next question that will show to the student

� In problem solving: considering the student’s knowledge level and the previous student actions in the problem solving, to decide what is the best hint to make to the student to guarantee that a student can reach the solution in less steps

� to adapt the course topic presentation: considering the student’s knowledge level and his/her learning preferences to decide about what kind of teaching materials should be show to the student.

1. Introduction

���� Goals & Motivations

���� ML for User Modeling

���� Naïve Bayes Classifier

���� Naïve Bayes & User Modeling

2. Iterative Bayes

���� Adaptation Algorithm

���� An illustrative example

���� Some discussion

3. Adaptive Bayes

���� Incremental Bayes

���� OnLine Bayes

4. Experimental Evaluation

� Results

5. Conclusions and

Future WorkFuture Work

Page 31: for User Modeling Adaptive Bayessweet.ua.pt/gladys/Papers/Eunite2002.pdf · Adaptive Bayes for User Modeling. João Gama & Gladys Castillo. 2 EUNITE2002. Algarve. Portugal Goals &

Adaptive Bayes Adaptive Bayes for for User ModelingUser Modeling. João Gama & Gladys Castillo. . João Gama & Gladys Castillo. 31

� � � �

EUNITE2002. Algarve. PortugalEUNITE2002. Algarve. Portugal

Thank you for Thank you for

your attentionyour attention