ql.io at nodepdx
TRANSCRIPT
![Page 1: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/1.jpg)
Feb 24, 2012
![Page 2: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/2.jpg)
2
Subbu Allamaraju@sallamar
http://www.subbu.org
Hello, I'm
![Page 3: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/3.jpg)
3
![Page 4: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/4.jpg)
4
https://github.com/ql-io/ql.ioOpen source (Apache 2)
http://ql.io
![Page 5: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/5.jpg)
5
XX,000,000,000req/day Pr
oduc
ers
Cons
umer
sAPIs
![Page 6: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/6.jpg)
6
XX,000,000,000
Prod
ucer
s
Cons
umer
sM
any APIs
![Page 7: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/7.jpg)
7
XX,000,000,000
Prod
ucer
s
Cons
umer
sM
any D
ialects
![Page 8: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/8.jpg)
8
Prod
ucer
s
Q1: Make them RESTful and consistent?
Many
Dialects
![Page 9: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/9.jpg)
9
Cons
umer
s
Prod
ucer
s
> 1 req per user action
![Page 10: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/10.jpg)
10
Q2: Optimize producers for every consumer?Co
nsum
ers
![Page 11: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/11.jpg)
Real code (randomized)
![Page 12: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/12.jpg)
12
![Page 13: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/13.jpg)
13
![Page 14: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/14.jpg)
14
![Page 15: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/15.jpg)
15
![Page 16: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/16.jpg)
16
![Page 17: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/17.jpg)
17
![Page 18: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/18.jpg)
18
![Page 19: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/19.jpg)
19
t1
t2
t3
Sprint 1: Blocking IO code with sequential requests
(t1+t2+t3)
Client Server-1 Server-2 Server-3
![Page 20: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/20.jpg)
20
max(t1,t2,t3)
Sprint 4: Parallelize independent requests
Client Server-1 Server-2 Server-3
![Page 21: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/21.jpg)
21
max(max(t1+t4),t2,t3)
Sprint "n": Fork/join dance
Client Server-1 Server-2 Server-3 Server-4
![Page 22: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/22.jpg)
22
Parallellizing SequencingJoiningNormalizing
I/OWriting such code once or twice is fun – writing tens of times is not.
![Page 23: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/23.jpg)
23
Bad for far-away clients
Native, mobileSingle page apps Server-1 Server-2 Server-3 Server-4
![Page 24: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/24.jpg)
24
![Page 25: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/25.jpg)
25
Easy and fast HTTP consumption
![Page 26: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/26.jpg)
26
HFRwni G NIxGNs TSMeb7 A9On vtwZhQoJGnFQFqgkV9 3WFgC 93TbEBy6 Q ocpBxgpH3 Pu4ju fiZsKb W RkIs5b z UAsS QK3nyJ68IhTSB0aTufR98ymV evsX7 tUH 8i4fwR S Hut69mnCHAOufyx w CZLOtN 9 PvTU sPd2lMVDV42tRAfIoPM56H1hE tGz5s kmekNeyrai5SklC 5 TstTKDhFb OLy 5KQ5oz A MiZzQJSCbEvaLr068KLleE X q8cwPm 5 nZpH 3jpeWcIpkTTIjGsZovq7 fR4Hn dz3Lhl o MfdTDqpFVdhaiPOsj2fO9 w fWD3mv p ORHX Bq4xIMvLGjMrgnC6JpBw1S5 HDwoI CwhI09 z 742rMEqx626ZH0qwtN g boU4fU W QYKf F24BKGrFfg0sfhkc8U4aZfL4bn kUNmG vm6odt 6 YaC6b0Ff4gGOx4Jh0 6 aXtsEg G LUlJL3k2O WeRAMe d 9GlF1XJM8 9oicQwaHnMp7n U Pjnojj5kdhD0sZzh Pz3HHpnBy L OlVQMpHAILCH RF3vwaFHarZR Q i2Ofa38U9ylvvecE
The same in ql.io (randomized)
![Page 27: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/27.jpg)
27
Lines of code for 5 API calls Data size (k)
before
after
![Page 28: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/28.jpg)
28
before
after
Lines of code for API calls Data size (k)
![Page 29: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/29.jpg)
29
What is ql.io
![Page 30: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/30.jpg)
30
A DSL for HTTPAn HTTP gateway
Built on node.js
![Page 31: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/31.jpg)
31
create table for each resource select to read insert into to add or create update to update delete to delete
HTTP CRUD to SQLish CRUD
# HTTP requests with one line of code
select long_url from bitly.shorten where shortUrl = 'http://bit.ly/uZIvmY'
select Url from bing.soap.search where q = "ql.io"
![Page 32: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/32.jpg)
32
HTTP Resources as Tablescreate table bitly.shorten on insert get from 'http://api.bitly.com/v3/shorten?login={^login}&apiKey={^apikey}&longUrl={^longUrl}&format={format}' using defaults apikey = "…", login = "…", format = "json" resultset 'data.url' on select get from http://api.bitly.com/v3/expand?login={^login}&apiKey={^apikey}&shortUrl={^shortUrl}&format={format} using defaults apikey = "…", login = "…", format = "json" resultset 'data.expand'
![Page 33: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/33.jpg)
33
Any HTTP Method// POST on a resource bound to SELECTcreate table bing.soap.search on select post to 'http://api.bing.net/soap.asmx' using defaults appid = '…' using bodyTemplate 'bing.soap.xml.mu' type 'application/xml' resultset 'soapenv:Envelope.soapenv:Body.SearchResponse.parameters.Web.Results.WebResult';
![Page 34: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/34.jpg)
34
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"> <soapenv:Header/> <soapenv:Body> <SearchRequest xmlns="http://schemas.microsoft.com/LiveSearch/2008/03/Search"> <parameters> {{#params}} <Query>{{q}}</Query> <AppId>{{appid}}</AppId> {{/params}} <Sources> <SourceType>Web</SourceType> </Sources> </parameters> </SearchRequest> </soapenv:Body></soapenv:Envelope>
Body Templates
![Page 35: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/35.jpg)
35
No Async Mind-Bending-- Sequentialminis = select * from finditems where keywords = 'mini cooper' limit 10;return select PictureURL from details where itemId = "{minis.itemId}";
-- Or parallelkeyword = "ql.io";web = select * from bing.search where q = "{keyword}";tweets = select id as id, from_user_name as user_name, text as text from twitter.search where q = "ql.io"; return { "keyword": "{keyword}", "web": "{web}", "tweets": "{tweets}"}
![Page 36: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/36.jpg)
36
Implicit Fork-Joinprodid = select ProductID[0].Value from eBay.FindProducts where QueryKeywords = 'macbook pro';details = select * from eBay.ProductDetails where ProductID in ('{prodid}') and ProductType = 'Reference';reviews = select * from eBay.ProductReviews where ProductID in ('{prodid}') and ProductType = 'Reference'; return select d.ProductID[0].Value as id, d.Title as title, d.ReviewCount as reviewCount, r.ReviewDetails.AverageRating as rating from details as d, reviews as r where d.ProductID[0].Value = r.ProductID.Value
![Page 37: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/37.jpg)
37
How to Use
![Page 38: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/38.jpg)
38
Client apps S-1 S-2 S-3 S-4
ql.io as an HTTP
Gateway
Optional streamingthrough WebSockets
As a Gateway
![Page 39: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/39.jpg)
39
Routingprodid = select ProductID[0].Value from eBay.FindProducts where QueryKeywords = 'macbook pro';details = select * from eBay.ProductDetails where ProductID in ('{prodid}') and ProductType = 'Reference';reviews = select * from eBay.ProductReviews where ProductID in ('{prodid}') and ProductType = 'Reference'; return select d.ProductID[0].Value as id, d.Title as title, d.ReviewCount as reviewCount, r.ReviewDetails.AverageRating as rating from details as d, reviews as r where d.ProductID[0].Value = r.ProductID.Value via route '/myapi' using method get;
![Page 40: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/40.jpg)
40
-- Show daily deals - use siteId=0 for US -- and siteId=3 for UK.
dailyDealsResponse = select * from dailydeals where siteId="{siteId}";dailyDeals = "{dailyDealsResponse.$..Item}";itemDetails = select ItemID as itemId, Title as title, GalleryURL as pic, Seller.UserID as sellerUserId, Seller.FeedbackScore as feedback, HitCount as pageViews from details where itemId in (dailyDeals.ItemID);
return itemDetails via route '/deals/{siteId}' using method get;
![Page 41: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/41.jpg)
41
var Engine = require('ql.io-engine'), fs = require('fs');var engine = new Engine({tables : __dirname + '/tables'});var script = fs.readFileSync(__dirname + '/myapi.ql', 'UTF-8');engine.execute(script, function(emitter) { emitter.on('prodid', function(data) { console.log('found ' + data.length + ' product IDs'); }); emitter.on('details', function(data) { console.log('found ' + data.length + ' details'); }); emitter.on('reviews', function(data) { console.log('found ' + data.length + ' reviews'); }); emitter.on('end', function(err, result) { console.log(result.body); });});
Node.js Module/> npm install ql.io-engine
![Page 42: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/42.jpg)
42
mkdir myappcd myappcurl -L "http://tinyurl.com/7cgglby"| bash
bin/start.sh
![Page 43: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/43.jpg)
43
We're HiringDM @sallamar
![Page 44: ql.io at NodePDX](https://reader035.vdocument.in/reader035/viewer/2022070315/554f4e8bb4c905524c8b4c95/html5/thumbnails/44.jpg)
http://ql.io