![Page 1: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/1.jpg)
Why Twitter Is All The Rage:A Data Miner's PerspectiveMatthew A. Russell - @ptwobrussell - http://MiningTheSocialWeb.com
PyTN - 23 February 2014
1
![Page 2: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/2.jpg)
Overview
Intro
Twitter as a Platform for Data Science
Applications of Firehose Analysis (#Syria circa last)
Understanding the Amazon Prime Air Reaction (IPython Notebook Walk Through)
Q&A
2
![Page 3: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/3.jpg)
Intro
3
![Page 4: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/4.jpg)
Hello, My Name Is ... Matthew
4
Background in Computer Science
Data mining & machine learning
CTO @ Digital Reasoning Systems
Data mining; machine learning
Author @ O'Reilly Media
5 published books on technology
Principal @ Zaffra
Selective boutique consulting
![Page 5: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/5.jpg)
Transforming Curiosity Into Insight
5
An open source software (OSS) project
http://bit.ly/MiningTheSocialWeb2E
A book
http://bit.ly/135dHfs
Accessible to (virtually) everyone
Virtual machine with turn-key coding templates for data science experiments
Think of the book as "premium" support for the OSS project
![Page 6: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/6.jpg)
Mining the Social Web ToCChapter 1 - Mining Twitter
Chapter 2 - Mining Facebook
Chapter 3 - Mining LinkedIn
Chapter 4 - Mining Google+
Chapter 5 - Mining Web Pages
Chapter 6 - Mining Mailboxes
Chapter 7 - Mining GitHub
Chapter 8 - Mining the Semantically Marked-Up Web
Chapter 9 - Twitter Cookbook
6
![Page 7: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/7.jpg)
Anatomy of Each ChapterBrief Intro
Objectives
API Primer
Analysis Technique(s)
Data Visualization
Recap
Suggested Exercises
Recommended Resources
7
![Page 8: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/8.jpg)
Opportunities for Data Alchemy
A model for the world: signal and sinks
Growth in data exhaust is accelerating
Digital fingerprints of the "real world" are accumulating
Lots of opportunities for motivated Python hackers
"Software is eating the world"
8
![Page 9: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/9.jpg)
Social Media Is All the Rage
World population: 7B people
Facebook: 1B+ users
Twitter: 650M users
Google+ 500M users
LinkedIn: 260M users
250M+ blogs (conservatively?)
9
![Page 10: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/10.jpg)
But what does it all mean, Basil?
It's a platform for data science and the frontier for predictive analytics
Understanding world events
Swaying political elections
Modeling human behavior
Analyzing sentiment
Making intelligent recommendations
10
![Page 11: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/11.jpg)
Twitter & Data Science
11
![Page 12: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/12.jpg)
Data Science
12
Data => Actionable information
Highly interdisciplinary
Nascent
Necessary
http://wikipedia.org/wiki/Data_science
![Page 13: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/13.jpg)
Another View of Data Science
13
![Page 14: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/14.jpg)
14
![Page 15: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/15.jpg)
Twitter Is All the Rage
It satisfies fundamental human desires
We want to be heard
We want to satisfy our curiosity
We want it easy
We want it now
Accessible, rich, and (mostly) "open" data
RESTful APIs and JSON responses
Great proving ground for predictive analytics about the real world
15
![Page 16: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/16.jpg)
Twitter's Network Dynamics
~650M curious users
A collective consciousness
Real-time communication
Short, sweet, ... and fast
Asymmetric Following Model
An interest graph
16
![Page 17: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/17.jpg)
Twitter Primitives
17
Accounts Types: "Anything"
"Following" Relationships
Favorites
Retweets
Replies
(Almost) No Privacy Controls
![Page 18: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/18.jpg)
Twitter and Facebook Compared
18
Accounts Types: "Anything"
"Following" Relationships
Favorites
Retweets
Replies
(Almost) No Privacy Controls
Accounts Types: People & Pages
Mutual Connections
"Likes"
"Shares"
"Comments"
Extensive Privacy Controls
![Page 19: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/19.jpg)
What's in a Tweet?
19
140 Characters ...
... Plus ~5KB of metadata!
Authorship
Time & location
Tweet "entities"
Replying, retweeting, favoriting, etc.
![Page 20: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/20.jpg)
What are Tweet Entities?
Essentially, the "easy to get at" data in the 140 characters
@usermentions
#hashtags
URLs
multiple variations
(financial) symbols
stock tickers
media
20
![Page 21: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/21.jpg)
API RequestsRESTful requests
Everything is a "resource"
You GET, PUT, POST, and DELETE resources
Standard HTTP "verbs"
Example: GET https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=SocialWebMining
Streaming API filters
JSON responses
Cursors (not quite pagination)
21
![Page 22: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/22.jpg)
Data Mining: Low Hanging Fruit
"Know thy data..."
Start with simple stats:
Count
Compare
Filter
Rank
Then, apply more complex analyses
22
![Page 23: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/23.jpg)
A Starting Point: Histograms
A chart that is handy for frequency analysis
They look like bar charts...except they're not bar charts
Each value on the x-axis is a range (or "bin") of values
Not categorical data
Each value on the y-axis is the combined frequency of values in each range
23
![Page 24: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/24.jpg)
24
Example: Histogram of Retweets
![Page 25: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/25.jpg)
25
Roberto Mercedes
Jorge
Ana
Nina
Social Network Mechanics
![Page 26: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/26.jpg)
Interest Graph Mechanics
26
Roberto Mercedes
Jorge
Ana
Nina
U2
Juan Luis
Guerra
Juan Luís
Guerra
![Page 27: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/27.jpg)
A (Social) Interest Graph
27
Roberto Mercedes
Jorge
Ana
Nina
U2
Juan Luis
Guerra
Juan Luís
Guerra
![Page 28: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/28.jpg)
A (Political) Interest Graph
28
Roberto Mercedes
Jorge
Ana
Nina
Johnny Araya
Rodolfo Hernández
![Page 29: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/29.jpg)
Measuring Influence Is Tricker Than It Looks
29
Spam bot accounts that effectively are zombies and can’t be harnessed for any utility at all
Inactive or abandoned accounts that can’t influence or be influenced since they are not in use
Accounts that follow so many other accounts that the likelihood of getting noticed (and thus influencing) is practically zero
The network effects of retweets by accounts that are active and can be influenced to spread a message
See also http://wp.me/p3QiJd-2a
![Page 30: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/30.jpg)
Justin Bieber vs Tea Party
30
![Page 31: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/31.jpg)
Realtime Analysis: #Syria
31
Monitor Twitter's firehose for realtime data using filters such as #Syria
Keep in mind the sheer volume of data can be considerable
Fuller analysis at http://wp.me/p3QiJd-1I
![Page 32: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/32.jpg)
#Syria: Who?
32
See http://wp.me/p3QiJd-1I
![Page 33: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/33.jpg)
#Syria: Who?
33
See http://wp.me/p3QiJd-1I
![Page 34: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/34.jpg)
#Syria: Who?
34
See http://wp.me/p3QiJd-1I
![Page 35: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/35.jpg)
#Syria: What?
35
See http://wp.me/p3QiJd-1I
![Page 36: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/36.jpg)
#Syria: What?
36
See http://wp.me/p3QiJd-1I
![Page 37: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/37.jpg)
#Syria: Where?
37
See http://wp.me/p3QiJd-1I
![Page 38: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/38.jpg)
#Syria: When?
38
See http://wp.me/p3QiJd-1I
![Page 39: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/39.jpg)
#Syria: Why?
39
That's for you (as the data scientist) to decide
Quantitative automation can amplify human intelligence
Qualitative analysis is still requires human intelligence
![Page 40: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/40.jpg)
Twitter Firehose Analysis with pandas
40
![Page 41: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/41.jpg)
MTSW Virtual Machine Experience
Goal: Make it easy to transform curiosity into insight
Vagrant-based virtual machine
Virtualbox or AWS
IPython Notebook User Experience
Point-and-click GUI
100+ turn-key examples and templates
Social web mining for the masses
41
![Page 42: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/42.jpg)
Social Media Analysis Framework
A memorable four step process to guide data science experiments:
Aspire
Acquire
Analyze
Summarize
42
![Page 43: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/43.jpg)
Goals
To understand how to capture data from Twitter's firehose
A understand basic pandas usage for tweets
To work through a data science experiment with a systematic 4-step process
To better understand the emotional reaction to the Amazon Prime Air announcement
To introduce some tools for data science
43
![Page 44: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/44.jpg)
Useful Links
Website
http://MiningTheSocialWeb.com
Twitter Data Mining Round Up
http://wp.me/p3QiJd-5H
All Source Code in IPython Notebook format (GitHub)
http://bit.ly/MiningTheSocialWeb2E
44
![Page 45: Why Twitter Is All The Rage: A Data Miner's Perspective (PyTN 2014)](https://reader030.vdocument.in/reader030/viewer/2022020122/554e819cb4c90545698b5343/html5/thumbnails/45.jpg)
Q&A
45