Download - Web as a data resource
![Page 1: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/1.jpg)
`
Get me my Data !Use the web as a data source
![Page 2: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/2.jpg)
2
Hack U @ Georgia Tech - March 8 – 12 2011
![Page 3: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/3.jpg)
Subramanyan MuraliyahooMail Engineer Hacker, Photographer, Traveler @rmsguhan
![Page 4: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/4.jpg)
Data and hacks
4
![Page 5: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/5.jpg)
Putting something together in 24 hours can be a real
rush !5
![Page 6: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/6.jpg)
Need innovative ways to make the data into
information that is useful
7
![Page 7: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/7.jpg)
Any resource on the internet is a source of
data …
8
![Page 8: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/8.jpg)
… that means that there is a lot of data
9
![Page 9: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/9.jpg)
Loads and loads of data and data
services
10
![Page 10: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/10.jpg)
Data that is complex and in many formats
11
![Page 11: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/11.jpg)
Navigating through all the complexity can be hard !#$%&
12
![Page 12: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/12.jpg)
How to solve the data problem ?
13
![Page 13: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/13.jpg)
http://pipes.yahoo.com saves the day!
14
![Page 14: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/14.jpg)
15
Graphical interface to play with data !
![Page 15: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/15.jpg)
WYSIWYG = pipes
16
![Page 16: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/16.jpg)
Pipes is a visual tool, where is my command
line ??
18
![Page 17: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/17.jpg)
Put your hands up for YQL !
19
![Page 18: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/18.jpg)
Common data access language across the
webdeveloper.yahoo.com/y
ql
20
![Page 19: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/19.jpg)
SQL like syntaxStandard and easy to use
21
![Page 20: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/20.jpg)
select {what} from {service} where {condition} in ({sub-query} )
22
![Page 21: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/21.jpg)
Several services can be chained using in ()
23
![Page 22: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/22.jpg)
YQL Console
24
![Page 23: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/23.jpg)
Lots of Integrations and predefined tables
anyone can be part of the interface
25
![Page 24: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/24.jpg)
Fetching data from a resource has never
been simpler
!
26
![Page 25: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/25.jpg)
http://query.yahooapis.com/v1/ public/yql?q={query} &format={format}
27
![Page 26: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/26.jpg)
Irrespective of the source, output can be
standardized …
28
![Page 27: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/27.jpg)
29
… thus giving you a solid & consistent data foundation
![Page 28: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/28.jpg)
Input can be XML, HTML, CSV, micro-
formats, Atom feeds, JSON, RSS
30
![Page 29: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/29.jpg)
Output can be JSON or XML
31
![Page 30: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/30.jpg)
Mix and match YQL and Pipes ?
32
![Page 31: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/31.jpg)
Pipes provides you with processing
capabilities
33
![Page 32: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/32.jpg)
YQL is another feed source
34
![Page 33: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/33.jpg)
Interlinking YQL and Pipes =
fewer lines of code
35
![Page 34: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/34.jpg)
Not limited to Yahoo! sources
![Page 35: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/35.jpg)
select * from google.translate where q in (select text from twitter.search where q="georgia tech") and target='fr';
![Page 36: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/36.jpg)
Fire off multiple queries at once
![Page 37: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/37.jpg)
select * from query.multi where
queries=‘{query1};{query2}’;
![Page 38: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/38.jpg)
Create your own tables with Open
tables
www.datatables.org/
![Page 39: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/39.jpg)
http://www.yqlblog.net/
![Page 40: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/40.jpg)
Go explore !
developer.yahoo.com/yql/console/
![Page 41: Web as a data resource](https://reader034.vdocument.in/reader034/viewer/2022051612/54bd63be4a79595e238b4624/html5/thumbnails/41.jpg)
Happy Hacking !!
Photos @ http://flickr.com/photos/rmsguhan