Hierarchical Recommender System for Improving NPS.

www.kdd.uncc.edu

CCI, UNC-Charlotte

Research sponsored by:presented by

Zbigniew W. Ras

CONSULTING COMPANY in Charlotte

Client 1 Client 2 Client 3 Client 4

Build Recommender Systemfor each client (34 clients) helping toincrease its revenue

Build Personalized Recommender Systemfor each shop helping toincrease its revenue

Services(heavy equipment repair)

Parts CUSTOMERS

shops shops shops

What we have to do

Net Promoter Score (NPS) – today’s standard for measuring customer loyalty

Promoter - {9,10}

Passive – {8,7}

Detractor – {1,2,3…,6}

NPS of a company is correlated with its revenue growth

NPS rating for all clients

NPS rating for all shops

What is our goal?

Build recommender system giving possibly the simplest and the best advice to every client/shop on the changes they need to make in order to improve their NPS ratings.

Initial Dataset – provided by Consulting CompanyAbout 100,000 records representing answers to a questionnaire collected from over 35,000 randomly chosen customers in 2010 -2013.Customers use services provided by 34 clients

A questionnaire consists of:

Information about the customerCustomer’s name, location, phone number…

Information about the serviceClient’s name, invoice amount, service type…

Information on customers’ feeling about the servicewas the job completed correctlyare you satisfied with the joblikelihood to refer to friends…

Benchmarks

Personal

BenchmarkAllOverallSatisfaction 35

BenchmarkAllLikelihoodtobeRepeatCustomer 34

BenchmarkAllDealerCommunication 33

BenchmarkServiceRepairCompletedCorrectly 32

BenchmarkReferralBehavior 31

BenchmarkServiceFinalInvoiceMatchedExpectations 31

BenchmarkEaseofContact 30

BenchmarkAllDoesCustomerhaveFutureNeeds 28

BenchmarkServiceTechPromisedinExpectedTimeframe 26

BenchmarkServiceRepairCompletedWhenPromised 26

BenchmarkServiceTimelinessofInvoice 25

BenchmarkServiceAppointmentAvailability 24

BenchmarkServiceTechEquippedtodoJob 23

BenchmarkAllContactStatusofFutureNeeds 22

BenchmarkServiceTechArrivedWhenPromised 21

BenchmarkAllHasIssueBeenResolved 19

BenchmarkAllContactStatusofIssue 17

BenchmarkServiceTechnicianCommunication 6

BenchmarkServiceContactPreference 3

BenchmarkCBCallansweredpromptly 1

BenchmarkServiceReceivedQuoteforRepair 1

BenchmarkCBAutoattendantansweredbycorrectdepartment 1

BenchmarkServiceCallAnsweredQuickly 1

BenchmarkAllMarketingPermission 1

Randomly chosen customers are asked to complete Questionnaire – It has questions concerning personal data + 30 benchmarks

To compute NPS we calculate average score of all benchmarks for all customers. Knowing thenumber of promoters and detractors we know NPS.

Data

Data Preprocessing (including reduction of benchmarks)

Knowledge Extraction (Classical Tools - WEKA + Our Software for Extracting Action Rules & Their Triggers)

Recommender Systems

Classical Approach

ClassificationSelection of Best Classification Algorithm

11

J48

RandomForest

KNN

NaiveBayes

BayesNet

RBF

PART

0 10 20 30 40 50 60 70 80

0.4 0.405 0.41 0.415 0.42 0.425 0.43 0.435

AccuracyTime Taken(secs)

  PART RBF BayesNet NaiveBayes KNN RandomForest J48

Accuracy 2 3 4 1 5 7 6

Time Taken 2 4 6 7 3 1 5

Total score 4 7 10 8 8 8 11

Initial Clients Datasets Recommender Systems local approach global approach

Using Information about 34 Clients to Enlarge these Datasets

2 global approach

More Powerful Recommender Systems

1

2

Two Options

Semantic Distance between Clients

attr1

attr2

attr3 attr4

attr3

attr2

attr2

attr1

attr3 attr4

attr3

attr1

Vector describing customer 1

Vector describing customer 2

Vector describe ing customer 3

Dataset for Client-1Vector describing customer 1

Vector describing customer 2

Vector describe ing customer 3

Dataset for Client-2J48 classifier extracted from dataset D1 of Client-1

J48 classifier extracted from dataset D2 of Client-2

More similar these two classification trees, more close semantically the clients are

FIRST APPROACH for Dataset Enlargement

Semantic distance based dendrogramfor Service

n5

n6

n7n3

n2n1

Recommender System Engine based on Semantic Similarity of Clients

NPS(n1)Table1Classifier1F-score1

NPS(n2)Table2Classifier2F-score2

Start with n1.If NPS(n1) has to be improved, “move to n3”If NPS(n2) > NPS(n1), thenTable3 := Table1 Table 2 is assigned to n3.Classifier from Table3 is extracted and itsF-score is computed. If F-score3 > F-score1, then “move to n5”, otherwise STOP

\

Table 5, Classifier5

Table3Classifier3F-score3

For each client (node) we need to identify the best ancestor to be used for action rules discovery

• Extract meta-actions from relevant comments that are associated with values of attributes used in action rules;

• Find smallest sets of meta-actions triggering these action rules;• Identify all customers supporting these action rules (needed to compute the

increase in revenue);• Search for smallest groups of meta-actions that trigger maximal increase in

revenue (we present them in descending order for same-size groups of meta-actions).

• effect(m) = num(distinct records in dataset involved with m) × conf(r*)• Effect of a set of meta-action indicates the number of customers who are

expected to become promoters when these meta-actions are executed.• Organize those groups of meta-actions hierarchically.

Extracting Meta-Actions /sentiment analysis/: We use Stanford Typed Dependencies Manual & Stanford Parser to generate grammatical relationshttp://nlp.stanford.edu:8080/parser/

r1 = (A, a1 a2) (B, b1->b2) (C, c1 c2) (detractor promoter)

Sup(r1) – number of customers supporting action rule r1 in a shop which NPS has to be improved

Number of detractors for a given shop having the properties (A,a1), (B,b1), (C,c1) - we target them

Before we start searching for triggers (meta-actions) supporting r1, this number is enlarged by adding customerssemantically similar and geographically close to our shop who also support rule r1.

Op1, Op2, Op3, Op4, Op5 – comments/sentiment about the service provided by this enlarged set of customers.Comments about the service from promoters who satisfy the description (A,a2), (B,b2), (C,c2) are also listed.

Op1 - positive opinion related to AOp2 - negative opinion related to C and BOp3 – positive opinion related to BOp4 – negative opinion related to AOp5 – negative opinion related to C

Customers sentiment associated with rule r1 –{Op1, Op2, Op3, Op4, Op5}

Trigger for A -> extracted from Op1 and not(Op4)Trigger for B -> extracted from Op3 and not(Op2) Trigger for C -> extracted from not(Op2) and not(Op5)

We search for minimum set of triggers covering A, B, C

invoice+++++,outstanding bill=141invoice+++++,they billed properly=65invoice+++++,fine invoice=61invoice-----,refused pay bills=29invoice+++++,happy with bill=12

price-----,not fair pricing=108price+++++,good price=108price+++++,fair pricing=87price-----,aggressive pricing=66price-----,unreasonable charge fee=37price-----,not satisfied with price=35price+++++,better pricing=27price-----,expensive amount charged=12price+++++,the charged fairly=12

staff+++++,best manager=112staff-----,bad experience because diagnosis=108staff+++++,good technician=96staff+++++,nice guy=87staff+++++,excellent mechanic=85staff+++++,great guy=83staff+++++,excellent technician=79staff+++++,good guy=74staff+++++,wonderful dealer=71staff+++++,good team=66staff+++++,honest guy=60staff+++++,pleased with manager=40staff-----,not available technician=34staff-----,wrong diagnosis=16staff+++++,best mechanic=12staff+++++,knowledgeable mechanic=6

Examples of Comments with Sentiment Orientation

Threshold for adding triggers = 6

Seeds

How to construct optimal sets of meta-actions (triggers activating changes in customer NPS)

{TR5} {TR1,TR4}We add single triggerone by one followingtheir order with respectto support

Sup=160

{TR5,TR1} {TR5, TR2} TR5, TR3}

Sup=180 Sup=165 Sup=163

{TR5,TR1,TR2} {TR5,TR1,TR3} {TR5,TR1,TR4}Sup=184 Sup=189 Sup=185

1 2 3 4

TR5 {TR1,TR4} {TR5,TR1,TR3}

{TR5,TR1}

Triggers activating changes in customer NPS

24

• Client-2 is selected to be our target.

Client{2,4,6,1} by HAMIS

NPS 0.77 (73 detractors) 0.852

Action rules extracted 28,126

Meta-actions extracted 15

Number of meta-nodes 447(*But we only focus on the top 10 in each

size category of meta-nodes.)Maximum effect in meta-nodes 45.83 (out of 73 detractors)

Search for the best meta-actions

idx Meta-actions effect

0 • ensure service done correctly 2.00

1 • ensure service done correctly• improve price competitiveness

7.62

2 • ensure service done correctly• properly set invoice

expectations(slightly high)

5.0

3 • ensure service done correctly• sufficient staff

2.0

7 • ensure service done correctly• competitive price• improve price competitiveness

11.13

8 • ensure service done correctly• improve price competitiveness• sufficient staff

10.66

9 • ensure service done correctly• improve price competitiveness• properly set invoice

expectations(slightly high)

9.99

10 • ensure service done correctly• improve price competitiveness• keep proactive communication

8.62

Search for the best sets of meta-actionsfor Client-2

idx Meta-actions effect

22 • ensure service done correctly• competitive price• improve price competitiveness• sufficient staff

15.07

23 • ensure service done correctly• improve price competitiveness• properly set invoice

expectations(slightly high)• sufficient staff

15.00

24 • ensure service done correctly• competitive price• improve price competitiveness• reasonable invoice

13.13

25 • ensure service done correctly• improve price competitiveness• reasonable invoice• sufficient staff

11.66

46 • ensure service done correctly• competitive price• improve price competitiveness• properly set invoice

expectations(slightly high)• sufficient staff

19.41

26

• Why HAMIS helps?1. 2) Extra meta-actions extracted from datasets extended by HAMIS help improve effect further.

Set of meta-actions Expected effect

competitive priceensure service done correctlyimprove price competitiveness

sufficient staff 9.87

decrease dealer response time* 11.13competitive priceensure service done correctlyimprove price competitivenesskeep proactive communicationnice technician

reasonable invoice 7.59

knowledgeable technician* 13.3

Search for the best meta-actions

QUESTIONS?