twitter power-user howto
TRANSCRIPT
✦Visualize content and connections
✦Ad-hoc queries
✦Exploration
✦Focus on specific profiles and posts
Goals
2
Why ?✦ Handles de-duplication and
merging
✦ Speaks JSON and REST
✦ Scales vertically and horizontally
✦ Because
6
Why ?✦ Handles de-duplication and
merging
✦ Relationships central to data model
✦ Built and optimized for graph traversal queries
✦ Because
7
Why ?
✦Micro-service friendly
✦Works the same way across OS, distribution, cloud provider
✦Build, ship, run anywhere with 3-5 shell commands (build, [tag,] push, [pull,] run)
9
Why ?✦Assemble pipelines of modular data providers and database writers
✦Normalize your social databases to community standards
✦Why waste time hunting for and wrangling new SDKs?
10
✦ Using linux? • apt-get / yum install docker.io
✦ Using a mac? ✦ Install Boot2Docker ✦ http://boot2docker.io/
• boot2docker ssh• ifconfig• note the bridged IP (eth1)
27
Docker
✦docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e PLUGINS=mobz/elasticsearch-head,elasticsearch/marvel/latest quay.io/steveblackmon/elasticsearch:1.4.4
✦docker run -d --name kibana -p 5601:5601 --link elasticsearch:elasticsearch quay.io/steveblackmon/kibana:4.0.0
✦docker run -d --name neo4j -p 7474:7474 quay.io/steveblackmon/neo4j:2.2.0.M4
28
Installation
✦ docker run -d --name linkurious -p 3000:3000 —link neo4j:neo4j quay.io/steveblackmon/linkurious:latest
** LINKURIOUS IS NOT OPEN-SOURCE **** PURCHASE A LICENSE AT http://linkurio.us **
** 20% OFF IF YOU MENTION THIS TALK **** 30-DAY MONEY-BACK GUARANTEE **
** ACCESS TO THIS DOCKER REPO IF YOU BUY**
29
Installation (Cont.)
✦developer.twitter.com
✦mytwitterid.com
✦mkdir streams
✦cd streams
Configuration (Part 0)
30
Configuration (Part 1)
twitter.userstream.json1. Add twitter oauth credentials2. Add own userid to twitter.follow3. Set elasticsearch.hosts4. Set graph.hostname
twitter.history.json5.Add twitter oauth credentials6.Add own userid to twitter.info7.Set elasticsearch.hosts
✦docker run -d --name twitter.userstream -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.userstream.json -cp /twitter-0.2-SNAPSHOT.jar twitter.UserstreamPipeline
✦docker run -d --name twitter.history -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.history.json -cp /twitter-0.2-SNAPSHOT.jar twitter.HistoryPipeline
Execution (Part 1)
32
twitter.friends.json1. Add twitter oauth credentials2. Add own userid to twitter.info3. Set graph.hostname
twitter.followers.json1. Add twitter oauth credentials2. Add own userid to twitter.info3. Set graph.hostname
Configuration (Part 2)
33
✦docker run -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.friends.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline
✦docker run -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.followers.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline
Execution (Part 2)
34
Configuration (Part 3)Export IDs of your connections with Cypher
•MATCH (v:page {handle:'handle'})-[:follow]-(x:page) RETURN DISTINCT SUBSTRING(x.id, 11);
twitter.connections.history.json1.Add twitter oauth credentials2.Add connection ids to twitter.info3.Set elasticsearch.hosts
twitter.connections.friends.json4.Add twitter oauth credentials 5.Add connection ids to twitter.info6.Set graph.hostname
35
✦docker run -v /home/ubuntu/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.connections.history.json -cp /twitter-0.2-SNAPSHOT.jar twitter.HistoryPipeline
✦docker run -v /home/ubuntu/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.connections.friends.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline
Execution (Part 3)
36