![Page 1: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/1.jpg)
Real-Time Social Analytics Ian Hersey
CTO and EVP, Products
![Page 2: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/2.jpg)
The Business Challenge
The BIG DATA wave
Driven by conversations on the Internet, Social Media, Mobile Apps
• 300 Million: Tweets per day • 250 Billion: Emails per day • 800 Million: Facebook users • 126 Million: Blogs • 1.97 Billion: Internet users
worldwide • 5 Trillion: SMS messages
annually • Millions of CRM Records • 100s of Millions of Survey
Verbatims
![Page 3: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/3.jpg)
Social Media: “Human Sensors”
• News, firsthand, secondhand, thirdhand…
• Natural disasters
• Military movements
• Networks, velocity, acceleration
• Opinions
• Products
• Services
• Popular culture (TV, film, music)
• Politics
• Conversations, Comments, Recommendations
• Can sometimes predict (or explain) outcomes
![Page 4: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/4.jpg)
![Page 5: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/5.jpg)
Predictive Power
• Don’t take it to Vegas…
• 90+% success rate if data volumes are sufficient
• Successful business uses involve not just prediction, but engagement
• Product feedback
• Direct customer service
• Analysis of marketing campaign effectiveness (TV, film, music)
• Political outreach/mobilization
• Science Art is still in its infancy
• Equally or more important are the “whys” behind the predictions/outcomes
![Page 6: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/6.jpg)
GOP Florida – Newt Gingrich
Selected Newt Gingrich topics were discussed more at length throughout the day. For example,
being sued by the band Foreign for using the song “Eye Of The Tiger” since 2009 captured the
imagination of Twittersphere …
Another topic that had mileage throughout the day particularly around mid-afternoon. A Ron Paul supporter
was somewhat roughed up at a Gingrich rally. Later in the day Ron Paul’s team
demanded an apology…
There was a sustained campaign to drum up support for Newt Gingrich
![Page 7: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/7.jpg)
GOP Florida – Mitt Romney
There was a sustained campaign to drum up support for Newt Gingrich
One of the key criticisms were jabs aimed at Romney’s wealth in the sense that money and privilege can win you
leadership…
The most consistent theme throughout the day was Romney being a Populist.
![Page 8: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/8.jpg)
Some Major Technical Challenges
• Data scale and rates
• NLP – no “one size fits all” technology
• Multi-channel content acquisition, coverage and quality
• Domain and customer specificity in the metadata
• Combining structured and full-text queries
• Operation by non-linguists
![Page 9: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/9.jpg)
Data Scale and Rates
• Experience with Hadoop, HBase and Solr
• Biggest issues
• “Enterprise friendliness”
• Cannot support low-latency processing
• No current commercial offerings with both SQL and full-text front ends
• “Real-time” analysis scenario
• Match a tweet according to an initial filter
• Do further analysis to determine whether it is “actionable” vs “just a mention”
• Figure out who to route it to with what kind of priority
• All within a handful of seconds from the time it was tweeted
• 2500 times or more per second
• Required development of real-time ingestion and orchestration framework
![Page 10: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/10.jpg)
Real-Time Processing Flow
Firehose
Pipeline
Command Center
Custom Apps
Analyze
Respond
Harvest & Harmonize 150+ Million Sources
Sensemaking & Annotation
![Page 11: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/11.jpg)
Real-Time Content Aggregation
Direct API Access Scrapers, Crawlers, RSS Collectors Aggregators and Syndicators Structured and Unstructured
![Page 12: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/12.jpg)
Social Analytics Application Stack
![Page 13: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/13.jpg)
Natural Language Processing “Reads” Every Communication
I bought an iPad2 for my mom last week. She loves the weight, but
doesn’t like the color. She wishes it came in blue. She says if it came in
blue, then she’d buy one for all her friends.
Entities (brands, people, locations, times, products…)
Events and relationships (purchasing event, my mom…)
Sentiment
Suggestions
Intent (to purchase, to leave)
I:have:mom
I:buy[past]:Product.apple_product.iPad2
![Page 14: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/14.jpg)
Limitations of NLP
• Irony, sarcasm
• “slanguage”
• Who’s talking/tweeting?
• Agendas
• Impact (“opinions are like…”)
• Cross-/multi-language
• Single posts vs. “body of work”
![Page 15: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/15.jpg)
Annotated Data Streams Feed Downstream Applications
![Page 16: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/16.jpg)
Real-Time Processing Pipeline
Advanced Topic Creator
Geotagging
Language ID
Reach
Klout
Entity, Event, Sentiment Tagging
Topic Matcher
Message Tracking
Worker Libraries
All standard content (Twitter, Google+, Facebook, Forums, Blogs, Online News)
SDK Kit and API documentation
![Page 17: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/17.jpg)
Advanced Topic Creator https://smcc.attensity.com/ui
![Page 18: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/18.jpg)
Command Center Concepts and Overview
The Command Center is a highly branded shared experience providing a lens to real-time social media conversations
Command Center screens use a responsive design for the following resolutions
1920x1080 (Most televisions)
1024x728 (Compatible with desktop computers and tablets)
A Command Center implementation is made up of multiple Dashboards
Implementations are hosted by Attensity
Dashboards contain multiple Widgets
Widgets are configurable with lots of options
Endless combinations
![Page 19: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/19.jpg)
![Page 20: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/20.jpg)
![Page 21: Real-World Challenges of Real-Time Social Analytics](https://reader034.vdocument.in/reader034/viewer/2022042713/548261ccb4af9f82328b45ed/html5/thumbnails/21.jpg)
Thank you