#bdam: cask market - big data's app store
TRANSCRIPT
![Page 1: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/1.jpg)
cask.cocask.co1
Big Data’s App Store
Albert ShauBDAM 2017-02-08
![Page 2: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/2.jpg)
cask.cocask.co
The Big Data PromiseSolving new Problems with Powerful
Infrastructure
2
![Page 3: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/3.jpg)
cask.cocask.co
The Big Data PromiseSolving new Problems with Powerful
Infrastructure
3
Anomaly DetectionNatural Language Processing
ClassificationRecommendation Systems
Realtime Event Monitoring
![Page 4: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/4.jpg)
cask.cocask.co
From Infrastructure to Applications
4
1.Easy to write powerful applications
2. Easy to distribute, discover, and install applications
![Page 5: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/5.jpg)
cask.cocask.co
Cask Market
5
Distribute, discover and install applications, plugins, and data
![Page 6: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/6.jpg)
cask.cocask.co
Architecture
6
![Page 7: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/7.jpg)
cask.cocask.co
Demo
7
![Page 8: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/8.jpg)
cask.cocask.co
Terminology
8
Package - A collection of jars, applications, datasets, streams, and configuration that can be installed on a CDAP instance
Package Spec - Metadata about a package and a list of installation steps
Catalog - List of packages in the market
![Page 9: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/9.jpg)
cask.cocask.co
API
9
GET market.cask.co/v1/packages.json
[ { "name": "datapack-access-log", "version": "1.0.0", "description": "Sample access logs in Combined Log Format (CLF).", "label": "Access Log Sample", "author": "Cask", "org": "Cask Data, Inc.", "created": 1473901763, "beta": false, "categories": [ "datapack" ] }, ...]
![Page 10: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/10.jpg)
cask.cocask.co
GET market.cask.co/v1/packages/usecase-sms-spam-pipeline/1.0.0/spec.json
{ "actions": [ ... { "type": "load_datapack", "label": "Labeled Messages Sample", "arguments": [ { "name": "name", "value": "labeledSMSTexts" }, { "name": "files", "value": [ "labeled_texts.tsv" ] } ] }, ...}
API
10
![Page 11: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/11.jpg)
cask.cocask.co
{ "actions": [ ... { "type": "create_pipeline", "label": "Spam Classifier Trainer", "arguments": [ { "name": "artifact", "value": { "scope": "system", "name": "cdap-data-pipeline", "version": "4.0.0" } }, { "name": "config", "value": "trainer.json" }, ... ] }}
API
11
![Page 12: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/12.jpg)
cask.cocask.co
Execution
12
CDAP
Big Data Infrastructure
PUT /v3/namespaces/default/streams/smsTexts
![Page 13: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/13.jpg)
cask.cocask.co
Execution
13
CDAP
Big Data Infrastructure
POST /v3/namespaces/default/streams/smsTexts/batch
![Page 14: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/14.jpg)
cask.cocask.co
Execution
14
CDAP
Big Data Infrastructure
PUT /v3/namespaces/default/apps/SpamTrainer
![Page 15: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/15.jpg)
cask.cocask.co
Hosting
15
Just serve static content through HTTP/v1/packages.json…/v1/packages/hydrator-plugin-solrsearch/1.5.0/spec.json/v1/packages/hydrator-plugin-solrsearch/1.5.0/solrsearch-plugins-1.5.0.json/v1/packages/hydrator-plugin-solrsearch/1.5.0/solrsearch-plugins-1.5.0.jar.../v1/packages/usecase-sms-spam-pipeline/1.0.0/spec.json/v1/packages/usecase-sms-spam-pipeline/1.0.0/trainer.json/v1/packages/usecase-sms-spam-pipeline/1.0.0/classifier.json/v1/packages/usecase-sms-spam-pipeline/1.0.0/texts.txt/v1/packages/usecase-sms-spam-pipeline/1.0.0/labeled_texts.txt
![Page 16: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/16.jpg)
cask.cocask.co
Looking Forward
16
Improve discoverability
Formalize external contribution process
Package dependencies
More Applications!
![Page 17: #BDAM: Cask Market - Big Data's App Store](https://reader036.vdocument.in/reader036/viewer/2022062412/58ac3b671a28ab145e8b628d/html5/thumbnails/17.jpg)
cask.cocask.co
Summary
17
Cask Market is a way to distribute, discover, and install Big Data applications
Cask hosts a public market, but enterprises can easily host their own
Move from infrastructure to applications