web as a data resource

`

Get me my Data !Use the web as a data source

2

Hack U @ Georgia Tech - March 8 – 12 2011

Subramanyan MuraliyahooMail Engineer Hacker, Photographer, Traveler @rmsguhan

Data and hacks

4

Putting something together in 24 hours can be a real

rush !5

Need innovative ways to make the data into

information that is useful

7

Any resource on the internet is a source of

data …

8

… that means that there is a lot of data

9

Loads and loads of data and data

services

10

Data that is complex and in many formats

11

Navigating through all the complexity can be hard !#$%&

12

How to solve the data problem ?

13

http://pipes.yahoo.com saves the day!

14

15

Graphical interface to play with data !

WYSIWYG = pipes

16

Pipes is a visual tool, where is my command

line ??

18

Put your hands up for YQL !

19

Common data access language across the

webdeveloper.yahoo.com/y

ql

20

SQL like syntaxStandard and easy to use

21

select {what} from {service} where {condition} in ({sub-query} )

22

Several services can be chained using in ()

23

YQL Console

24

Lots of Integrations and predefined tables

anyone can be part of the interface

25

Fetching data from a resource has never

been simpler

!

26

http://query.yahooapis.com/v1/ public/yql?q={query} &format={format}

27

Irrespective of the source, output can be

standardized …

28

29

… thus giving you a solid & consistent data foundation

Input can be XML, HTML, CSV, micro-

formats, Atom feeds, JSON, RSS

30

Output can be JSON or XML

31

Mix and match YQL and Pipes ?

32

Pipes provides you with processing

capabilities

33

YQL is another feed source

34

Interlinking YQL and Pipes =

fewer lines of code

35

Not limited to Yahoo! sources

select * from google.translate where q in (select text from twitter.search where q="georgia tech") and target='fr';

Fire off multiple queries at once

select * from query.multi where

queries=‘{query1};{query2}’;

Create your own tables with Open

tables

www.datatables.org/

http://www.yqlblog.net/

Go explore !

developer.yahoo.com/yql/console/

Happy Hacking !!

Photos @ http://flickr.com/photos/rmsguhan