data hacking and the city: ucl casa conference 2013 workshop
TRANSCRIPT
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 1/42
Data Hacking and th
CASA Workshop
Steven GrayThursday 26th September 2013 - 9:30 - 12:30
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 2/42
OrderoftheMorning
9:30-10:00-Showcaseofwhatwedo
10:00-11:00-Coding--Collec8ngtheData11:00-12:00-MoreCoding-VisualisingData
12:00-12:30-Ques8onandWrapup
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 3/42
Letsstartcoding!
Butfirstafewthingsweneedtosortout
SeparateintoGroups
--SocialData(BestforBeginnerstocoding)
--TransportData
--EnvironmentalData
Rules
--Don’tbeafraidtoaskques8onsifyoudon’tunderstan
--HavefunandEnjoyyourself
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 4/42
QuickShowcaseofmywork
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 5/42
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 6/42
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 7/42
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 8/42
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 9/42
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 10/42
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 11/42
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 12/42
Letsstartcoding!
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 13/42
Mac--InstallXcodeCommandlinetoolsforyourMacOSVersion
--h$ps://developer.apple.com/downloads/ --InstallHomebrew(hp://brew.sh/)
--Runonterminal:
ruby-e"$(curl-fsSLhps://raw.github.com/mxcl/homebrew
--Followhp://madebyhoundstooth.com/blog/install-node-with-homebrew-on-os-x/
--brewinstallnode
NPMwillbeinstalledatthesame8measnode!
InstallNodeJS+NPM
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 14/42
Linux(Ubuntu)
--Followhps://github.com/joyent/node/wiki/Installing-Node.js-via-p
Windows
--Downloadandfollowinstruc8ons
InstallNodeJS+NPM
hp://nodejs.org/download/
OpenNode.jsfromStartMenu-Programs-Node.js-Node.jsCommandP
Orsearchfor“Node.jsCommandPrompt”onyoursystem(Windows,
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 15/42
InstallModulesviaNPM(AllSystems)
UseNPMtoinstallthefollowingmodulesusedbythecollector.
Checkthatyouhavenpminstalledbyrunning“npm-v”(itshouldreport1.3
Runthefollowingtoinstall
npminstalloauthcolors
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 16/42
CollecngSocialMediaData(Twier)-ARecipe
Whatyouwillneed:
1xTwierAccount
1xDeveloperKeyfromhps://dev.twier.com/
Sprinklewithsomepa8enceand8meandin8meyou’llbe
datafromTwier.
Instruc8onscomingupsoonaerasmallexplanta8on.
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 17/42
Understandingwhat’shappening
http://dev.twitter.com/doc
User Pro!les Tweet Database
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 18/42
H T T P
G E T
R e q
u e s t
R e s u l t s
{"results":[
{"text":"@twitterap
"to_user_id":396524
"to_user":"TwitterA
"from_user":"jkoum "
" metadata":{
"result_type":"pop
"recent_retweets":
},
"id":1478555574,
"from_user_id":1833
"iso_language_code"
....}
}
Returns tweets that:• match speci!ed query•
located in Lat/Lon box• Historical Tweets• User Pro!le Information• Return Public Timeline
Service is rate limited
JSON or A
MakingRequeststoTwier
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 19/42
Authencaon
2MethodsofAuthen8ca8on
--BasicHTTPAuthen8ca8on
--oAuth(MoreComplex
MostmoderndaywebservicesuseoAuth(includingTwierandGoogleSe
Keepsinforma8onsafefromthewrongpeoplebutalsoallowsprovidersto
iftheyareusingtoomanyresourcesormaking.
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 20/42
SohowdoesoAuthwork
Ser
Applica8on
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 21/42
SohowdoesoAuthwork
Ser
Applica8on
RequestaSessionTokenPassaIDandaSecret
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 22/42
SohowdoesoAuthwork
Ser
Applica8on
RequestaSessionTokenPassaIDandaSecret
AsksLoginwithIDServergivesaPinnumberforsession
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 23/42
SohowdoesoAuthwork
Ser
Applica8on
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 24/42
SohowdoesoAuthwork
Ser
Applica8on
Loginissuccessful:)
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 25/42
SohowdoesoAuthwork
Ser
Applica8on
Loginissuccessful:)
SendsAppTokensTheseneedtobesaved!
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 26/42
SohowdoesoAuthwork
Ser
Applica8on
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 27/42
SohowdoesoAuthwork
Ser
Applica8on
RequestusingtokensServerchecksifit’svalid
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 28/42
SohowdoesoAuthwork
Ser
Applica8on
RequestusingtokensServerchecksifit’svalid
DataReturned:)
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 29/42
Gotohps://dev.twier.com/
LoginusingyourcurrentTwieraccount
Clickyourprofilepicture(toprightcorner)andclickMyApplicaons
ClickCreateNewApplicaon
Fillinyourdetailsfortheapp
You’llbepresentedwithyourConsumerKeyandConsumerSecret
Savethesekeysforthenextstep
CollecngSocialMediaData(Twier)CreangyourTwierApplicaonKeys
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 30/42
LetsCollectsomedata
DownloadtheCollector
hps://gist.github.com/sjg/6712144
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 31/42
LetsCollectsomedata
Aerdownloadingthescripts8ckinyourkeysandsecretinthescriptbetwe
quota8onmarksandsavethefile
ConsumerKey
ConsumerSecre
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 32/42
LetsCollectsomedata
Copythescriptintoafolderthatyoucanrunthescriptfromandnavigatet
folderinyoursystemterminal.
Youcanedittheloca8onofthecollectorbychangingthesevaluesinthefi
Torunthecollectortype:
nodetwier_collect.js
“SearchTag”,“La8tude”,
-NB:“”willreturnalltw
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 33/42
LetsCollectsomedata
Ifallgoeswellandeverythingisworkingyoushouldseethefollowingscree
Gotothelinkinabrowser,logintotwierandtypethepinnumberin.Pre
andtweetswillstartcollec8nginafilecalled“tweets.csv”inthesamefold
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 34/42
Visualisingtheoutput
fi i li i h h ll d
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 35/42
Gotohps://mapsengine.google.com
ClickonNewMap
Nameyourmapbyclickingon“UntledMap”Click“Import”anduploadyourCSVfile
ClickBaseMaptochangethemapstyle
CreangyourfirstVisualisaonwiththeDataCollected
C fi Vi li i h h D C ll d
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 36/42
CreangyourfirstVisualisaonwiththeDataCollected
Oncethefilehasuploaded,choosethecorrectLa8tudeandLongitudecolu
thenclickCon8nue.ClicktheIDfield(leover)asthemarker8tleandpre
Theupdatedoutputfilewill,hopefully,autodetectthelat/lngfields.
C fi t Vi li ith th D t C ll t d
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 37/42
CreangyourfirstVisualisaonwiththeDataCollected
Styleyourmarkersbypressingthepaintbucket(highlightedinred)andsel
circleshapeandcolour.Experimentwithstylingyourmap.
C fi t Vi li ith th D t C ll t d
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 38/42
CreangyourfirstVisualisaonwiththeDataCollected
Ifallwentwellyourmapwilllooksomethinglikethis.Whynotshareitwit
world!
C fi t Vi li ith th D t C ll t d
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 39/42
CreangyourfirstVisualisaonwiththeDataCollected
Andyoucanclickthemarkersandseethetweets
Different Sources you can use to Visualise your data
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 40/42
DifferentSourcesyoucanusetoVisualiseyourdata
GoogleFusionTables
-hp://www.google.com/drive/apps.html#fusiontables
GoogleMapEngine
-hps://mapsengine.google.com
GoogleEarth
-hp://www.google.com/earth/index.html
MapTube
-hp://www.maptube.org
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 41/42
hp://download.bigdatatoolkit.org
h k
7/27/2019 Data Hacking and The City: UCL CASA Conference 2013 Workshop
http://slidepdf.com/reader/full/data-hacking-and-the-city-ucl-casa-conference-2013-workshop 42/42
Thanks
Steven [email protected]
@frogo+StevenGray