using ai to make sense of customer feedback
TRANSCRIPT
Using AI to Make Sense of
Customer Feedback
Alyona Medelyan
@zelandiya
Correct Understanding of Customer Feedback
Can Save Millions
2015: Tens of Thousands of New Zealanders
were Surveyed About the new Flag
Government Reported
the Results of Manual Feedback Analysis
Actual Responses
Two costly & unnecessary referendum followed. Outcome: NZ kept the current flag
Millions could have been saved!
People wanted to ”keep the current flag”
1. Types of customer feedback
2. Why analyzing customer feedback is important
3. Why is it hard
4. Approaches
5. Applying AI to customer feedback analysis
6. Demo
Different Types
of Customer Feedback
Types of Customer Feedback
one-on-one interviews / focus groups
call centre logs / complaints
social media
open-ended survey questions / reviews
quantitate survey questions
UX tests / analytics
unstructured
structured
Collection Analysis Insight
one-on-one interviews / focus groups hard hard good
call centre logs / complaints easy hard limited
social media easy hard limited
open-ended survey questions / reviews easy medium good
quantitate survey questions easy easy limited
UX tests / analytics medium easy limited
unstructured
structured
Comparing Types of Customer Feedback
Why Understanding
Customer Feedback
is More Important than Ever
Customer Experience
is the New Marketing
It’s Measured Using
Net Promoter Score Surveys
Image credit
The number of “Net Promoter Score”
searches on Google since 2004
1. Growing Number of
Satisfaction Surveys and Reviews
v
¯\_(ツ)_/¯
2. The Need to Explain
the Why’s Behind the Scores
Net Promoter Score by month over time
3. Scores can be Cheated
Unstructured Feedback, not so Much
Why Analyzing
Customer Feedback is Hard
Common Misconception:
Sarcasm Makes Analysis Hard
One of Many Sarcastic Tui Beer Adverts
Sarcasm is Hard: Even People Struggle
I’ll keep it in
mind
They’ll do itI’ve
forgotten
already
Sarcasm is Rarer Than You Think
Dataset Sarcasm Example
NPS Survey 1%I’m so disappointed! What a great
customer service you have!
Social Media
comments5% Very helpful answer. Troll.
The Actual Challenges
With Customer Feedback
Challenge 1: Messy Data
How many ways there are to say
‘wet paper’?
Challenge 2: Synonyms and Paraphrases
Hundreds of
possible variations
of the same theme
wet
dripping
soaking
soaked
damp
drenched
paper
papers
newspaper
news paper
newspapers
news papers
+
Paraphrasing the Same Theme
Challenge 3: Negation
Positive or Negative?
My coffee was great positive
My coffee was awful negative
My coffee was not great negative
My coffee was not that great neutral?
I did not think my coffee was great negative
I did not expect my coffee to be this great positive
I was disappointed with the quality of the coffee negative
I was not disappointed with the quality of the coffee positive
Approaches to
Customer Feedback Analysis
Manual Coding
1.
Figure out the Code Frame, Apply, Repeat
What is the meaning of life?
1 2 3 4 5
What is the meaning of life?
42
Friends and family
Making a difference in the world
Happiness
Finding happiness
To achieve, to conquer
Family
…
What is the meaning of life?
42
Friends and family
Making a difference in the world
Happiness
Finding happiness
To achieve, to conquer
Family
…
1
2
3
4
4
5
2
Sentiment in a Manual Code Frame
Customer Service
Positive Negative
Timely Nice Helpful Didn’t fix issue Rude
Word Clouds
2.
“Every time I see a word cloud presented as insight,
I die a little inside.”
– J. Harris, journalist
Word Clouds Lack
Interpretation, Context, Meaning
“Overall the language
focuses on sweeping
statements focusing on
the state of the nation.”
Kalev Leetaru (Forbes)
You wouldn’t create a Word Cloud from your Numbers,
why is it ok from Text?
Rule-based Approaches
3.
It’s Hard to Find a Rule That Works Well
I was impressed by how friendly the person
on the other end of the line wasStaff friendliness ✔
The lady who helped me was friendly Staff friendliness ✔
Friendliness of staff Staff friendliness ✔
Your website is very user friendly Staff friendliness ✘
The young man on the phone was very pleasant Other ✘
friendly OR friendliness –> Staff friendliness
Text Categorization
4.
old
customer
responses
categories
new
customer
responses
Machine
Learning
Algorithm
Predictive
Model categories
Need for Sufficient Training Data,
and Clear Categories
Customer Feedback Analysis
Needs to be ‘Unsupervised’
Thanks to an unsupervised approach, Facebook found
Candi Crash Saga causes low App Store reviews
Topic Modeling
5.
21
3
A Topic can be Hard to Interpret
2
???ok
Source: Ben Fields
Sentiment
1. Rule-based (dictionary)
2. Text categorization (positive / negative)
Two Sentiment Detection Approaches
Advances in AI > Customer Feedback
Messy Data
Paraphrases
Negation
AI > Challenges
Word2vec*
Deep Learning
*See also: Conceptnet.io
Knowledge Representation
Word2Vec
Image source: ericbern.com
Best Intro: Word2Vec Udacity Youtube
Knowledge Representation
Deep Learning
Precision Recall F-Measure Errors
People 84 73 75 <1
Dictionaries 61 57 54 8
Linear Regression 65 56 47 3
Deep Learning 62 57 49 2
Sentiment Analysis is not about maximizing F-Measure,
it’s about reducing true Errors: positive confused with negative
Theme Extraction
6.
From Words to Complex Themes
Applying Customer Feedback Analysis
Google: Sentiment by Theme
Thematic Demo
