euro xxii presentation improving the quality of customer satisfaction measurements of musa method...

8/13/2019 EURO XXII Presentation Improving the Quality of Customer Satisfaction Measurements of MUSA Method Using Clustering Data Mining Techniques

1/33


2/33

2

Structure:

MUSA Method

Research objectives

Data mining approach

The experiment

Application of results

Conclusions


3/33

3

MUSA MethodThe main objective of the method is the aggregation of individual judgementsinto a collective value functionassuming that clients global satisfaction

depends on a set of ncriteria representing service characteristic dimensions.

Customers Global Satisfaction

Satisfactionaccording to the

1-st criterion

The MUSA method assesses global and partial satisfaction functions Y* and X*I

respectively, given customers judgements Y and Xi.

1b

XbY

n

1i

i

n

1i

*

ii

*

where the value functions Y* and X*Iare normalised in the

interval [0,100], and biis the weight of the i-th criterion

Satisfactionaccording to the2-nd criterion

Satisfactionaccording to the

n-th criterion

MUSA Method (1)

Grigoroudis and Siskos (2002)


4/33

4

MUSA Method

MUSA Method (2)

CRITERIA GLOBALPREFERENCE

disaggregation

aggregation

AggregationModel

Aggregation

Model?

MUSA uses a preference disaggregation model. In the traditional aggregation

approach, the criteria aggregation model is known a priori, while the global

preference is unknown. On the contrary, the philosophy of disaggregation

involves the inference of preference models from given global preferences.


5/33

5

MUSA Method

MUSA Method (3)

Customer's global satisfaction

y1 y2 ym y

y*2

y*m

Y*

Yy*1

.

.

.

.

.

.

y*

... ...

Global Added Value Function

Satisfaction according to the 1st criterion

x1

1 x1

2 x1

k x11

x1*2

x1

*m

X1

*

X1

x1*i

x1

*1

.

.

.

.

.

.

... ...

Satisfaction Function for

the 1st Criterion

Satisfaction according to the 2nd criterion

xi1 x

i2 x

ik x

ii

xi*2

xi*m

Xi*

Xi

xi*i

xi*1

.

.

.

.

.

.

... ...


the 2nd Criterion

Satisfaction according to the n-th criterion

xn

1 xn

2 xn

k xnn

xn*2

xn

*m

Xn

*

Xn

xn*n

xn

*1

.

.

.

.

.

.

... ...


the n-th Criterion

...


6/33


7/33

7

Research ObjectivesLets say that we have two equal divided, farraginous

groups of customers in our sample. The first isconsisted of demanding customers and the second onehas non-demanding customers. MUSA will produce aresult describing neutral customers.

Research objectives (2)

Demanding Non-demanding Neutral


8/33

8

Research ObjectivesIn the case of different importance of the criteria (criteria

wights) given by farraginous groups of customers willlead us to similar problems.


1st Group of Customers

15.0% 15.0%

35.0% 35.0%

0%

10%

20%

30%

40%

50%

Criterion 1 Criterion 2 Criterion 3 Criterion 4

2nd Group of Customers

35.0% 35.0%

15.0% 15.0%

0%

10%

20%

30%

40%

50%


MUSA results

25.0% 25.0% 25.0% 25.0%

0%

10%

20%

30%

40%

50%



9/33

9

Research ObjectivesMUSA gives as internal measures evaluating the quality of

its results. The reliability evaluation of the results ismainly related to the following quantitative measures:

the fitting level to the customer satisfaction data(Average Fitting IndexAFIand Overall PredictionLevel-OPL)

the stability of the near-optimality analysis results(Average Stability IndexASI).



10/33

10

Research ObjectivesThe Overall Prediction Level (OPL) is based onthe sum of the main diagonal cells of theprediction table, and it represents thepercentage of correctly classified customers:

: the number of customers that havedeclared to belong to global satisfaction levelm1, while the model classifies them to levelm2

: the percentage of customers ofactual global satisfaction level m1, that themodel classifies to level m

2

: the percentage of customers ofestimated global satisfaction level m1, thathave declared to belong to level m2


1~y 2~y y~

1y

2y

y

Nij R

ij

Cij

N11

R11

C11

N12

R12

C12

...

jy~

N1j

R1j

C1j

...N

1 R

1

C1

N21

R21

C21

N22

R22

C22

...N

2j R

2j

C2j

...N

2 R

2

C2

.

.

.

Na1

Ra1

Ca1

Na2

Ra2

Ca2

...

Naj

Raj

Caj

...

Na

Ra

Ca

... ...N

i R

i

Ci

Ni1 R

i1

Ci1

Ni2 R

i2

Ci2

iy

.

.

.

.

.

.

.

.

.

Predicted global satisfaction level

Actualglobals

atisfactionlevel

1 2m m

N

1 2m mR

1 2m mC

1 1 1 2

1 1 21 1 1

m m m m

m m m

OPL N N


11/33

11

Data Mining ApproachSurveys data will be processed following a clustering

(unsupervised learning or segmentation) approach.

Data mining approach (1)

Data from

questionnaires

......

Preprocessed

Data

MUSAfor eachcluster

Clusters(2,, n)

DataSelection:

Valid Answers

DataPreprocessing:

Transformations

DataMining

Labeling: Basedon demographic

data


12/33


13/33

13

Transformations


For different Demand, DCr(i) nominal:

If Cr(i)(T+thr) Then DCr(i)=INT(Cr(i)-(T+thr))

If(T-thr)Cr(i)(T+thr) Then DCr(i)=0

where T is the declared total satisfaction of the customer, Cr(i)is thesatisfaction regarding his/her satisfaction on i criterion and thris athreshold.

For different Criteria Weights, W1Cr(i), W2Cr(i), numeric:

W1Cr(i)=ABS(Cr(i)-T)and

W2Cr(i)= W1Cr(i)*(crc(a)-(ABS([(a-1)/2-T]/(a-1)/2))

where T is the declared total satisfaction of the customer, Cr(i)is thesatisfaction regarding satisfaction on i criterion, crc(a)is a correction

parameter and a is the number of global satisfaction levels


14/33

14

EM algorithm


The EMalgorithm can be seen as a generalizedversion of K-means clustering

A hard membership is adopted in the K-meansalgorithm, (i.e., a data pattern is assigned to onecluster only).

This is not the case with the EMalgorithm, wherea soft membership is adopted, (i.e., themembership of each data pattern can bedistributed over multiple clusters)


15/33

15

EM algorithm


Similarly to K-means, first select the cluster parameters (A, A,P(A)) or guess the classes of the instances, then iterate

Each cluster A is defined by a mean (A)and a standarddeviation (A)

Samples are taken from each cluster A with a specifiedprobability of sampling P(A)

Adjustment needed: we know cluster probabilities, not actualclusters for each instance. So, we use these probabilities asweights

For cluster A:

Stop when the difference between two successive iterationbecomes negligible (i.e. there is no improvement of clusteringquality).

We measure that by:


16/33

16

EM algorithm



17/33

17

The experiment

The experiment (1)

For the development of the transformation procedureand for the evaluation of our research results wedesigned and we implemented an experiment.

Steps:

1. Generation of synthetic dataDataSet Generator

2. Evaluate clusters generationWEKA DM tool

3. Evaluate MUSA results on new segments

4. Select the most appropriate transformations


18/33

18

Generation of synthetic data

The experiment (2)

A dataset generator, developed by our team for MUSA software evaluation,was used for the production of different data sets. The generator is ableto produce data (answers to surveys) that have specific characteristics.


19/33

19

Generation of synthetic data

The experiment (3)

1stData Set Produce two segments regarding different customers demand:Criterion 1 Criterion 2 Criterion 3 Criterion 4

Weights 25% 25% 25% 25%Sets (500) Non Demanding Non Demanding Non Demanding Non Demanding (500) Demanding Demanding Demanding DemandingSatisfaction LevelsA (Global) 5a(i) (per criterion) 5 5 5 5

2nd

Data Set

Produce two segments regarding different criteria weights:Criterion 1 Criterion 2 Criterion 3 Criterion 4Demand Neutral Neutral Neutral NeutralSets (500) 15% 15% 35% 35% (500) 35% 35% 15% 15%Satisfaction LevelsA (Global) 5a(i) (per criterion) 5 5 5 5


20/33

20

Evaluate clusters generation

The experiment (4)

WEKA, a Java Data Mining Tool developed in University of Waikato, wasused for classes to clusters evaluation.


21/33

21

Evaluate clusters generation

The experiment (5)

Evaluation of 1stdata set using DCr(i) transformation:

Assigned to Cluster

Initial Classes 0 1

I 164 336

II 446 54

Cluster 0


22/33

22

Evaluate MUSA results

The experiment (6)

Evaluation of 1stdata set using DCr(i) transformation:

MUSA's Internal Quality Measures

Samples - Data Sets Customers AFI ASI OPL

Generator Data Set I 500 97.17% 96.59% 95.00%

Generator Data Set II 500 96.29% 95.68% 95.40%

Generator Data Set I + II 1000 91.29% 88.77% 56.50%

Cluster 1 --> I 390 95.21% 96.13% 86.15%

Cluster 0 --> II 610 94.75% 94.63% 92.95%

Criteria Weights Demanding Indices

Samples - Data Sets Cr 1 Cr 2 Cr 3 Cr 4 Global Cr 1 Cr 2 Cr 3 Cr 4Generator Data Set I 25.04% 25.92% 24.61% 24.44% -55.25% -60.43% -57.28% -16.13% -61.86%Generator Data Set II 26.28% 25.43% 24.16% 24.13% 41.82% 59.962 65.41% 30.55% 12.99%Generator Data Set I + II 25.34% 27.69% 23.61% 23.36% -27.96% -19.57% -27.40% -5.16% -54.35%Cluster 1 --> I 25.24%

26.23%

24.89%

23.64%

-52.34%

-57.61%

-57.92%

-20.09%

-65.90%

Cluster 0 --> II 25.54% 25.01% 24.86% 24.59% 40.26% 54.48% 62.63% 35.28% 19.14%


23/33

23

Evaluate MUSA results

The experiment (7)

Evaluation of 2nddata set using W1Cr(i), W2Cr(i) transformations:



Generator Data Set I 500 96.45% 97.64% 89.60%

Generator Data Set II 500 96.51% 96.63% 91.00%

Generator Data Set I + II 1000 92.52% 91.97% 60.30%

Cluster 0 --> I 498 94.35% 94.85% 80.92%

Cluster 1 --> II 502 95.32% 94.65% 93.43%

Criteria Weights Demanding Indices

Samples - Data Sets Cr 1 Cr 2 Cr 3 Cr 4 Global Cr 1 Cr 2 Cr 3 Cr 4Generator Data Set I 16.45% 16.34% 33.67% 33.53% 28.92% 26.97% 51.05% 33.58% 36.29%Generator Data Set II 34.31% 34.23% 16.50% 14.96% 29.80% -36.15% 32.27% -49.20% 46.51%Generator Data Set I + II 25.31% 25.08% 24.35% 25.27% 27.06% -7.87% 37.47% 23.23% 31.45%Cluster 0 --> I 19.78%

17.65%

35.35%

27.22%

14.56%

24.30%

-20.66%

22.10%

3.70%

Cluster 1 --> II 33.91% 34.14% 16.11% 15.83% -6.48% -17.50% -20.52% 15.42% 25.44%


24/33

24

Application of results

Application of results (1)

The clustering procedure was applied on two realworld surveys in order to be further evaluated.The measure of success would be the

improvement of MUSAs internal quality measuresthrough the proper segmentation of the initialsample.

Survey 1: Policemen Satisfaction in Greece(sample: 1508, criteria: 8)

Survey 2: Tourists Satisfaction in Skopelos Island

(sample: 599, criteria: 5)

li i f l (2)


25/33

25

Survey 1


Evaluation of segments production using DCr(i) transformation:



Initial Sample 1508 93.02% 78.02% 56.63%

Cluster 0 537 92.76% 82.65% 77.09%

Cluster 1 971 95.16% 74.46% 72.81%

Cluster 0 526 94.98% 66.42% 82.32%

Cluster 1 835 96.07% 83.14% 88.38%

Cluster 2 147 85.03% 81.96% 52.38%

Cluster 0 720 96.87% 80.19% 90.83%

Cluster 1 105 82.10% 79.15% 40.95%

Cluster 2 494 96.03% 85.06% 90.28%

Cluster 3 189 93.51% 67.67% 82.01%

A li ti f lt (3)


26/33

26

Survey 1


Evaluation of segments production using W1Cr(i), W2Cr(i) transformations:



Initial Sample 1508 93.02% 78.02% 56.63%

Cluster 0 579 90.39% 79.65% 34.72%

Cluster 1 929 95.60% 75.63% 77.40%

Cluster 0 533 94.59% 84.22% 54.60%

Cluster 1 440 90.29% 80.52% 47.50%

Cluster 2 535 96.30% 76.01% 90.84%

Cluster 0 374 93.99% 82.40% 42.51%

Cluster 1 188 85.98% 83.47% 27.13%

Cluster 2 547 94.96% 74.41% 73.13%

Cluster 3 399 97.04% 78.24% 91.73%


27/33

A li ti f lt (5)


28/33

28

Survey 2



` MUSA's Internal Quality Measures


Initial Sample 599 93.18% 62.64% 53.59%

Cluster 0 214 89.48% 60.51% 57.48%

Cluster 1 385 96.25% 90.69% 77.66%

Cluster 0 186 89.04% 59.65% 56.45%

Cluster 1 342 96.69% 74.61% 84.21%

Cluster 2 71 92.31% 93.86% 39.44%


29/33



30/33

30

Survey 2



Demanding Indices

Clusters Global Cr 1 Cr 2 Cr 3 Cr 4 Cr 5

Initial Sample -60.37% -49.56% -55.99% -38.34% -12.14% -75.29%

Cluster 0 -64.50% -54.00% -54.00% -54.00% -54.00% -54.00%

Cluster 1 -55.10% -40.09% -56.97% -29.22% -10.32% -76.54%

Criteria Weights

Clusters Cr 1 Cr 2 Cr 3 Cr 4 Cr 5

Initial Sample 17.84% 20.45% 14.60% 11.24% 36.87%

Cluster 0 20.00% 20.00% 20.00% 20.00% 20.00%

Cluster 1 15.36% 21.38% 13.00% 10.26% 40.01%

Labelling: Tourists staying in hotels turn to belong into Cluster 1 while on the contrary the

ones chose to stay in rooms to let seem to belong in cluster 0.

Conclusions (1)


31/33

31

Conclusions

Conclusions (1)

Data Mining Clustering procedure led to morehomogeneous segments of customers both insynthetic datasets and in real world surveys results.

DCr(i) transformation seems to work better thanW1Cr(i), W2Cr(i) transformations.

The labelling of the produced clusterssegments is

not always obviousMaybe more attention shouldbe paid during the designing of the survey to includemore demographical information.

Conclusions (2)


32/33

32

Future research

Conclusions (2)

Some improvements regarding the data mining procedure mayinclude:

Further experiments using the dataset generator evaluating theresults should be undertaken. Real world surveys should beused as well.

Other MUSA internal quality measures, recently proposed,should be also considered.

The transformations regarding the different criteria weightsshould be improved, if it is possible.

Other or new similarity metrics should be studied.

The labelling procedure should be thoroughly examined.


33/33

33

Thank you

euro xxii presentation improving the quality of customer satisfaction measurements of musa method...

Documents