world throughnlp coloredglasses-final
DESCRIPTION
Argus Insights CEO's presentation from the 2011 Sentiment Analysis Symposium in San Francisco. John walks through why it is important to validate sentiment measurements of social media and gives an example from a leading vender where 1 + 1 do not eTRANSCRIPT
Looking at the world through NLP-colored glasses
Dr. John Feland, CEO, Argus Insights
Sentiment Analysis Symposium Research Workshop 8 Nov 2011
Tweet it! @argusinsights
Page 2© 2011 Argus Insights, Inc. Confidential: Do Not Distribute
If you can not measure it, you can not improve it.
-Lord Kelvin
Always Check your math
-my First Grade Teacher
Trust, but Verify
- Russian Proverb (Доверяй, но проверяй)
Page 5© 2011 Argus Insights, Inc. Confidential: Do Not Distribute
Most Corporate Clients Treat Sentiment as a Black Box
MagicSentimentAnalysis
Raw musings of consumers
Numerical Market Sentiment
• Corporate Decision Makers blinded by precision• Using these numbers to drive performance
• What is the impact of getting it wrong?
Page 6© 2011 Argus Insights, Inc. Confidential: Do Not Distribute
How we checked the math
• Over 1900 user reviews taken from the Tablet Market• Sources include most of the major online retailers• Leveraged best-in-class sentiment provider
– Scored entire verbatim using tuned sentiment model– Usage is outside of vendor’s recommended use
• Surprised at the results
Unnamed sentiment provider
Page 7© 2011 Argus Insights, Inc. Confidential: Do Not Distribute
Initial compared sentiment scores directly
• All scales normalized (linearly) to -1 to +1• SV Sentiment Score within Review Range 7% of the time• SV Sentiment within correct Positive, Neutral or
Negative territory 51% of the time• SV results are biased towards the negative for Tablets
-1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 10
0.1
0.2
0.3
0.4
0.5
0.6Actual Distribution SV Distribution
Consumer Sentiment
Perc
enta
ge D
istr
ibuti
on
Page 8© 2011 Argus Insights, Inc. Confidential: Do Not Distribute
Scaled so that SV distro matches User distro
-1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 10
0.1
0.2
0.3
0.4
0.5
0.6
Actual Distribution SV Distribution
Consumer Sentiment
P{er
cent
age
of D
istr
ibuti
on
• Trimmed and scaled SV results to match actual distribution
• Signficant performance improvements– Correct Bin: 33% -> almost 5X improvement– Correct Sign: 74% -> almost 50% improvement
Page 9© 2011 Argus Insights, Inc. Confidential: Do Not Distribute
Comparing actual to scored by product
• Calculated sentiment could lead to the wrong competitive understanding
• Selecting the wrong target can be disastrous for the enterprise
• Responsibility to our clients demand transparency
Reviews Raw SV Score Corrected SV Score1 ASUS Eee Slate RIM PlayBook RIM PlayBook2 Iconia Andr Tablet iPad 2 iPad 23 Motorola Xoom Iconia Andr Tablet Iconia Andr Tablet4 iPad 2 Samsung Galaxy Tab 7 Motorola Xoom5 RIM PlayBook Motorola Xoom Samsung Galaxy Tab 76 Samsung Galaxy Tab 7 ASUS Eee Slate ASUS Eee Slate7 Iconia Win Tablet Iconia Win Tablet Iconia Win Tablet8 HP TouchPad HP TouchPad HP TouchPad
Reviews Raw SV Score1 ASUS Eee Slate RIM PlayBook2 Iconia Andr Tablet iPad 23 Motorola Xoom Iconia Andr Tablet4 iPad 2 Samsung Galaxy Tab 75 RIM PlayBook Motorola Xoom6 Samsung Galaxy Tab 7 ASUS Eee Slate7 Iconia Win Tablet Iconia Win Tablet8 HP TouchPad HP TouchPad
Reviews1 ASUS Eee Slate2 Iconia Andr Tablet3 Motorola Xoom4 iPad 25 RIM PlayBook6 Samsung Galaxy Tab 77 Iconia Win Tablet8 HP TouchPad
Page 10© 2011 Argus Insights, Inc. Confidential: Do Not Distribute
New sentiment models - where the math checks out
• Argus Insights uses customer reported sentiment• Proven predictor of market performance• Full transparency to clients, enabling them to trust and verify
Page 11© 2011 Argus Insights, Inc. Confidential: Do Not Distribute
With great power comes great responsibility
-Voltaire
Questions?