social data is now predictive – now...

© 2014 Converseon Inc. Proprietary and Confidential

Social Data is Now Predictive – Now What?

Sentiment Symposium 2015


Today’s speakers

2

ROB KEY Founder & CEO of Converseon @robkey

JOEL RUBINSON Former Chief Research Office at The Advertising Research Foundation Senior Strategist and Advisor with Converseon @joelrubinson

Over a decade of social intelligence and consulting leadership. Starts where listening platforms stop. “Top score consulting/research (5 out of 5), data processing and sentiment analysis (Forrester Wave Q1 2014 Enterprise Social Listening) Forrester Research Top Innovator (ConveyAPI technology) in Social Data Mining “for its ability to provide near human level precision at the speed and scale that only software can provide.” Dataweek


Consumers become social…with brands

Marketers begin social marketing and social business programs

Marketing research becomes “intrigued”

Social media moves from curiosity to quantified impact

3

The journey marketers and researchers have been on to become social

How can we engage in the conversation?

What insights can we gain from social media conversation

Are the data trustworthy?

Does social media have quantitative value

What can we learn about user interests?

How can we drive sales?

Curiosity

Core


The top priority for marketing research…

…Into your framework for brand success …into your brand research data strategies ….to reinvent brand tracking …into your research modalities

….get serious about integrating digital, especially social data – and capturing voice of customer in new ways

© 2014 Converseon Inc. Proprietary and Confidential 5

The challenge: truth is that for too long, much social data has been “coin flip.” Greater insights must begin with better data


A case in point..


The criticism of social data is widespread among insight professionals

7

Current State

!  Imprecise – average sentiment accuracy is 60%

! Analyst bias

Data quality remains an issue. When asked about their satisfaction with general data quality, 74% of respondents reported positive results. But when we dug deeper and asked about the specifics of the data, many changed their tunes. In fact, the five responses showing the most dissatisfaction all centered on data: the ability to weed out spam; the accuracy of the tool’s automated sentiment analysis; influencer identification tools; multilingual and international capabilities; and the tool’s integration capabilities. These data challenges make a direct call to CI teams to get involved and bring their past data management experience to the table.

– Forrester Research

“ ”

! No sample frame

!  “One size fits all”

!  “Same old metrics”

! High irrelevancy

! Don’t know who is speaking

!  “Garbage in, garbage out”


…Many Market Researchers Are Relying on these Monitoring Tools for Insights

Source: Forrester Research

Yet…


Recently, Researchers at CMU and McGill Recently recognized the issue

9

“Far from being unfixable, however, miscalculations in social-media analyses can already be fixed using methods developed to fix similar problems in studies in epidemiology, statistics and machine learning.” - ComputerWorld

But they also recognized the solution


ConveyAPI: Our Approach to Solving the Data Issues

Knowledge-based

Resources

Machine Learning

System Test Data

Training Data

Semi Supervision: Keeps “humans in the loop” for continuous training

Customizable: Trains to domain and brands (“small” may be good for selling smartphones, bad for hotel rooms)

Accurate: Close approximation of human performance at scale (humans that agree with each other) – generally 90-95%

Scalable: Now allows the accuracy of human coding at large scale and speed

Vertical and Brand/Domain Specific

Custom Classifiers: Enables unlimited number of custom classifiers (intent, purchase phase, etc.)

Vertical and brand specific

High Relevancy and Recall: Isolates key data sets rapidly and at most detailed level.

Data approach as represented by Converseon’s ConveyAPI technology


Three Levels of Intelligence

Sentiment (target level)

Emotion Intensity

Relevancy Confidence

Industry specific Brand Specific

Function specific

Customer journey

Intent

Standard Classifiers Domain/Brand Specific Custom Classifiers

Purchase stage, etc.

Ability to create nearly unlimited additional Classifiers to isolate and analyze and Address specific insight requirements.

Highly precise “out of box” Standard classifiers Record, sentence and entity

Ability to rapidly choose and create classifiers based on specific industries, brands, needs. Auto, pharma, CPG, etc.

Recent test of performance versus 10,000 human coded records found A variance of less than 5%


We can then infuse the data into role specific models

Custom enrichment and tuning to the brand / product

Custom classifiers

Standard classifiers


Such as customer experience mapping

11% 14% 13% 19%

54% 35%

14% 22%

41%

14%

22% 42% 42%

27%

0%

25%

50%

75%

100%

Problem Recognition Information Search Competitive Evaluation Purchase Decision Post- Purchase

Fear

Distraction

Apprehension

Pensiveness

Acceptance

Trust

Serenity

Surprise

Interest

Sadness

Annoyance

Disgust

Anger

Anticipation

Joy

New classifiers unlock deeper, more actionable insights such as those

Based on Plutchik’s Wheel of Emotion


All acne mentions n=469,000; relevant acne mentions n=262,000, acne sufferer mentions n=3,200. Source: Converseon analysis of public online records.

Relevancy helps separate signal from noise

Category Exploration: % Relevant Acne Conversation

Irrelevant: “Gonna order a acne tee, plaid pants, & loafers for my birthday.”

All Acne Mentions 100%

Relevant Acne Mentions ~57%

Acne Sufferer Mentions >1%

Not

-Rel

evan

t

Spam: “Whitening Cream, Acne Series, Acne Treatment Aman tanpa efek, Alami, glowing , bebas jerawat recommended..”

“Has anyone used Aveeno baby…to help with baby acne? Does anyone that uses this in general think it's better than j&j or noticed a difference?”

“I need to find some new face scrubs and masks. Cause obviously the ones I been using just don't work anymore”

“are you sure that's accuttane? It worked so well for me. Cleared my face from having SEVERE cystic acne”

Relevant mentions and voices can act as a panel for companies looking to identify the questions and issues that consumers have.


•  Relevance feedback allows you to be in control •  The best boolean query achieved only 15% relevancy •  Machine learning trained custom classifier raised

relevancy to 85% in less than an hour

To get there, you have to go beyond “booleans”


Custom Classifiers Unlock New Insights

Fact / Opinion Product Application Consumer Intent

Brand Personality Unmet Needs Influencers

Patient Journey

Sentiment

Adverse Events


Which has helped lead us to the beginning of a new era

17

Characteristics

!  Limited Complexity !  Free tools !  Experimental !  People working in

their spare time

!  “Dictionary Based” !  Easy to understand ! Difficult to expand to different

languages ! Directional ! Declarative ! Heuristic ! Qualitative ! Challenging for slang,

sarcasm and limited expressions

!  Siloed

! Higher precision !  Embraces language

evolution ! Domain specific !  Trainable !  Adaptable ! Customizable ! Quantitative/Predictive !  Integrated into measurement

frameworks ! Modeled

Proactive/Enterprise •  Business Intelligence •  Marketing Mix Modeling •  Brand Tracking •  Product Development,

etc.

!  Experimental Business Value

1 2

Precision Recall Relevancy

Word spotting “Rules Based” NLP + Machine Learning

Reactive/Limited •  crisis comm •  PR •  Campaign tracking

2015

Predictive & Quantitative

Reactive & Qualitative

Business Value


When Social Data has proven to be quantitative and predictive

18

Professor Wendy Moe and David Schweidel, conducted analysis of social conversation versus offline brand tracking using Converseon data. .

New WOMM Media Mixed Modeling Study (utilizing Converseon data)


Applying to Brand Tracking

0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Prop

ortio

n of

Pos

itive

C

omm

ents

Observation Month

Blog

Forum

Microblog

Aggregate

0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Dis

trib

utio

n of

Com

men

ts

Observation Month

Blog

Forum

Microblog

Other

Venue Correlation

Blogs .197 Forums -‐.231 Microblogs -‐394 Average .008

Simple average sentiment Had no correlation with offline brand tracking survey


What influenced expressed sen9ment?

General Brand Impression (GBI)

Venue Venue-specific dynamics

Message topic (relevancy)


Product and A=ribute Effects

How much variance exists across focal topics related to the brand?


GBI and Offline Brand Tracking Surveys

Potential for GBI as a lead indicator Correlation with survey (t) ! GBI = .376 ! Avg sentiment =.008 ! Blogs = .197 ! Forums = -.231 ! Microblogs = .394 Correlation with survey (t+1) ! GBI = .881 ! Avg sentiment = .169 ! Blogs = .529 ! Forums = .213 ! Microblogs = .722

8.75

8.8

8.85

8.9

8.95

9

9.05

9.1

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

1 2 3 4 5 6 7 8 9 10

Aver

age

Surv

ey R

espo

nse

GB

I

Month of Overlap Period (t)

GBI in month t-1 Survey in month t

Similar findings published by TNS


Groundbreaking Marketing Mixed Modeling Study: WOMMA

,

WOMMA (Word of Mouth Marketing Association) Study


Key findings

Using adaption Structural Equation Modeling (by Analytics Partners) and data from six brands (and social online data provided by Converseon/offline by Keller Fay) •  WOM Impression equal to between 5 – 200x paid impression •  Contributes 13% of sales (or $6 trillion dollars) – greater impact the greater the consideration •  Amplifies the impact of paid media by 15% •  Has the most immediate impact on performance (over other media)

So now that we KNOW that social media data are DATA, now what?

Image source: WOMMA


So let’s stop treating social media as a hobby

25

3. All the way to bright •  Measure and analyze social media to demonstrate its importance to the markeFng organizaFon

•  ContribuFon to sales •  PredicFve value •  Listening for the unexpected •  Segment conversaFons by

customer groups

Use social media as a way of transforming brand tracking •  Lighten the survey load by

tracking brand beliefs via social

•  Be agile…as the marketplace changes, no need to fear trend disrupFon from adding aPributes

Turn social media into trustworthy informaFon •  Establish rigorous standards

for determining conversaFon relevance and senFment

•  Use the same data for modeling, KPIs, and brand tracking


Parting Thoughts

•  The challenges of social listening, research and intelligence are being tackled head on.

•  Predictive models require high threshold of relevancy, recall (entity) and precision

•  New technologies and models are creating breakthrough results

•  Leading brands are indeed beginning to truly mainstream this data across the organization, globally

•  It’s important to seek a “single truth”

•  Not all companies are the same, and requiring custom/bespoke NLP approaches. •  Measure the passion you have created around the brand and how they discuss it in the context of the

need it solves in people’s lives

•  Assuming you find the same positioning picture coming from social media as from attribute ratings, cut your tracker expense by eliminating continuous tracking of attributes.

•  Integrate this data both broadly and specifically


Thank You For questions or more details, please contact [email protected]

social data is now predictive – now...

Documents