yql - christian heilmann open hack london presentation
TRANSCRIPT
![Page 1: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/1.jpg)
Remixing web data for your hacks the easy wayYQL – the can opener of the web
![Page 2: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/2.jpg)
Hacking together systems in 24 hours is a lot of fun.
![Page 3: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/3.jpg)
But you want to spend that time thinking about the interface.
![Page 4: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/4.jpg)
And not how you get to the right data in the right format.
![Page 5: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/5.jpg)
The web is full of juicy and long lasting data.
http://www.flickr.com/photos/clspeace/162336973/
![Page 6: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/6.jpg)
And there is a lot of it around.
http://www.flickr.com/photos/tudor/2981410947/
![Page 7: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/7.jpg)
However, our attempts to get to it can be clumsy.
http://www.flickr.com/photos/lumachrome/2140368742/
![Page 8: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/8.jpg)
What we need is an easy way to get to that data.
http://www.flickr.com/photos/careytilden/115435226/
![Page 9: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/9.jpg)
We had a way to do that for quite a while now.
![Page 10: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/10.jpg)
http://pipes.yahoo.com/
![Page 11: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/11.jpg)
Pipes, however is high end technology…
http://www.flickr.com/photos/axio/2346342672/
![Page 12: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/12.jpg)
We’re developers, not interface users.
http://www.flickr.com/photos/codepo8/2278641937/
![Page 13: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/13.jpg)
So for a long time people asked Yahoo for a command line version of pipes.
Can we have one?
![Page 14: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/14.jpg)
Yes, we can!
![Page 15: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/15.jpg)
The Yahoo Query Language, or short YQL is a unified interface language to the web.
http://developer.yahoo.com/yql/
![Page 16: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/16.jpg)
Using YQL, accessing the web and its services becomes as easy as SQL:
select {what} from {service} where {condition}
![Page 17: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/17.jpg)
Say you want kittens in your hack (who doesn’t?)
select * from flickr.photos.search where text="kitten"
![Page 18: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/18.jpg)
Say you want kittens in your hack (who doesn’t?)
![Page 19: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/19.jpg)
Say you only want 5 kittens
select * from flickr.photos.search where text="kitten" limit 5
![Page 20: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/20.jpg)
Say you only want 5 kittens
![Page 21: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/21.jpg)
Nice, but where can you get this?
![Page 22: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/22.jpg)
YQL is a REST API in itself, and it has two endpoints.
![Page 23: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/23.jpg)
The public endpoint does not need any authentication.
http://query.yahooapis.com/v1/public/yql?q={query}&format={format}
![Page 24: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/24.jpg)
The private endpoint needs oauth authentication.
http://query.yahooapis.com/v1/yql?q={query}&format={format}
http://developer.yahoo.com/yql/guide/authorization-access.html
![Page 25: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/25.jpg)
Output formats are XML or JSON. JSON also allows for a callback parameter to use the output directly as JSON-P.
![Page 26: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/26.jpg)
http://query.yahooapis.com/v1/public/yql?q=select%20*%20from flickr.photos.search%20where%20text=%22kitten%22limit%205&format=json&callback=meow
![Page 27: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/27.jpg)
![Page 28: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/28.jpg)
You can mix and match several web services using the in() command.
![Page 29: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/29.jpg)
Guess what this does:
select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text='london,uk') and license=4)
![Page 30: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/30.jpg)
Find photos in London, UK with a Creative Commons “By” license and give me all the information you have about them.
![Page 31: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/31.jpg)
select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text='london,uk') and license=4)
![Page 32: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/32.jpg)
Using a command like this and some out-of-the-box UI elements like Yahoo Maps and the YUI carousel, you can build something *very* quickly.
![Page 33: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/33.jpg)
http://isithackday.com/hacks/cantine/index.php?loc=covent+garden
![Page 34: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/34.jpg)
Instead of selecting all the information you can also limit the results:
select name,url from upcoming.venue where metro_id in (select id from upcoming.metro where search_text="stokey")
![Page 35: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/35.jpg)
select name,url from upcoming.venue where metro_id in (select id from upcoming.metro where search_text="stokey")
![Page 36: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/36.jpg)
The diagnostics part of the returned data shows you what happened and how long it took
![Page 37: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/37.jpg)
Here’s what Yahoo offers you in this format:
flickr.photos.exifflickr.photos.infoflickr.photos.interestingnessflickr.photos.recentflickr.photos.searchflickr.photos.sizesflickr.placesflickr.places.infogeo.placesgeo.places.ancestorsgeo.places.belongtosgeo.places.childrengeo.places.neighborsgeo.places.parentgeo.places.siblingsgeo.placetypesgnip.activity
local.searchmusic.artist.idmusic.artist.popularmusic.artist.searchmusic.artist.similarmusic.release.artistmusic.release.idmusic.release.popularmusic.release.searchmusic.track.idmusic.track.popularmusic.track.searchmusic.video.categorymusic.video.idmusic.video.popularmusic.video.searchmusic.video.similar
mybloglog.community.findmybloglog.membermybloglog.member.contactsmybloglog.member.newwithcontactsmybloglog.member.newwithmemybloglog.members.findmybloglog.stats.findupcoming.categoryupcoming.countryupcoming.eventsupcoming.events.bestinplaceupcoming.groupsupcoming.metroupcoming.stateupcoming.userupcoming.venue
![Page 38: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/38.jpg)
Here’s what Yahoo offers you in this format:
search.imagessearch.newssearch.siteexplorer.inlinkssearch.siteexplorer.pagessearch.spellingsearch.suggestsearch.termextractsearch.websocial.connectionssocial.contactssocial.presencesocial.profilesocial.updatesweather.forecastyahoo.identityyap.setsmallview
search.imagessearch.newssearch.siteexplorer.inlinkssearch.siteexplorer.pagessearch.spellingsearch.suggestsearch.termextractsearch.websocial.connectionssocial.contactssocial.presencesocial.profilesocial.updatesweather.forecastyahoo.identityyap.setsmallview
search.imagessearch.newssearch.siteexplorer.inlinkssearch.siteexplorer.pagessearch.spellingsearch.suggestsearch.termextractsearch.websocial.connectionssocial.contactssocial.presencesocial.profilesocial.updatesweather.forecastyahoo.identityyap.setsmallview
![Page 39: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/39.jpg)
You want even more?
http://www.flickr.com/photos/verylastexcitingmoment/3123597774/
![Page 40: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/40.jpg)
Alright, how about this?
atomcsvfeedhtml
jsonmicroformatsrssxml
![Page 41: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/41.jpg)
Telegraph’s headlines anyone?select * from html where url='http://www.telegraph.co.uk/' and xpath='//h3/a'
![Page 42: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/42.jpg)
Telegraph’s headlines anyone?
![Page 43: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/43.jpg)
You can dynamically create YQL queries to collate several sources…
![Page 44: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/44.jpg)
Then use cURL to pull them off the web with one single http request!
![Page 45: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/45.jpg)
YQL caches and compresses the results for you.
Pretty easy, isn’t it?
![Page 46: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/46.jpg)
Here’s another fun part:Anyone can be part of the YQL interface.
![Page 47: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/47.jpg)
All we need the data provider to do is to create a schema that explains their data structure.
http://developer.yahoo.com/yql/guide/yql-opentables-chapter.html
![Page 48: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/48.jpg)
And people do…amazonbitlydeliciousdopplretsyfriendfeedgithubGreaderguardianimdbiplocationlastfmnestorianetflix
nmmnytopensocialsearchshoppingsocialtwitterupdate.groovyweatherwesabewhitepagesyahooyelpzillow
http://github.com/spullara/yql-tables/tree/master
![Page 49: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/49.jpg)
For example the national maritime museum:
select * from nmm.archive.search where searchterm=‘horatio nelson'
![Page 50: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/50.jpg)
select * from nmm.archive.search where searchterm=‘horatio nelson'
![Page 51: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/51.jpg)
That’s pretty cool – allowing anyone to be part of this interface.
![Page 52: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/52.jpg)
The only shame is that you can’t do all the things in YQL that you can do in Pipes – for example string manipulation.
![Page 53: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/53.jpg)
We wondered how to make this possible.
One thing we didn’t want to sacrifice is the simplicity of the language itself.
![Page 54: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/54.jpg)
So instead of inventing an own language, we decided to piggy-back on one you already know.
![Page 55: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/55.jpg)
YQL execute allows you to embed JavaScript in the open table schema that runs on the YQL server and converts the data for you.
http://developer.yahoo.com/yql/guide/yql-execute-chapter.html
![Page 56: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/56.jpg)
For example you can augment an existing service to do something different.
The following example adds a rank to search results.
http://www.yqlblog.net/samples/searchrank.xml
![Page 57: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/57.jpg)
![Page 58: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/58.jpg)
![Page 59: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/59.jpg)
Stored as XML this can be used in a YQL query:
use 'http://yqlblog.net/samples/searchrank.xml' as searchrank; select * from searchrank where query='pizza' and dispurl like '%pizzahut%'
![Page 60: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/60.jpg)
![Page 61: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/61.jpg)
Anyways, the *easiest* way to start with YQL is to use the console.
http://developer.yahoo.com/yql/console/
![Page 62: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/62.jpg)
![Page 63: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/63.jpg)
![Page 64: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/64.jpg)
![Page 65: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/65.jpg)
![Page 66: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/66.jpg)
![Page 67: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/67.jpg)
![Page 68: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/68.jpg)
![Page 69: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/69.jpg)
![Page 70: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/70.jpg)
![Page 71: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/71.jpg)
Of course, you can also spend half the hack day reading API docs
![Page 72: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/72.jpg)
Check out some code examples on.
http://isithackday.com/hacks/ohd-london
![Page 73: YQL - Christian Heilmann Open Hack London presentation](https://reader031.vdocument.in/reader031/viewer/2022032514/55d72b52bb61ebad738b45bd/html5/thumbnails/73.jpg)
THANKS!
Chris Heilmannhttp://twitter.com/codepo8