cfsummit: data science on cloud foundry
TRANSCRIPT
![Page 1: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/1.jpg)
![Page 2: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/2.jpg)
Data Science on Cloud Foundry Ian Huston @ianhuston
Alexander Kagoshima @akagoshima
![Page 3: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/3.jpg)
Who are we?
• Data Scientists at Pivotal Labs • Using Cloud Foundry since 2013 • Working with enterprises to get value out
of their data
![Page 4: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/4.jpg)
Image by Drew Conway: http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram
![Page 5: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/5.jpg)
Data Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician.
- Josh Wills
![Page 6: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/6.jpg)
Typical Projects
Risk Analysis
Predictive Maintenance
Understanding Your Customer
![Page 7: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/7.jpg)
![Page 8: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/8.jpg)
Data Services Easy control of incoming data
![Page 9: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/9.jpg)
Data Services Bind and scale system services
– Databases, NoSQL, message queues etc. $ cf create-‐service rediscloud PLAN_NAME INSTANCE_NAME $ cf bind-‐service APP_NAME INSTANCE_NAME
Add User Provided Services – Standalone Hadoop or Apache Spark cluster,
Big Data System $ cf cups SERVICE_INSTANCE -‐p "host, port, username, password"
Data Service
App App App
App App App
![Page 10: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/10.jpg)
Deploy a Model Prediction API Control distributed computation
![Page 11: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/11.jpg)
h"ps://github.com/ihuston/python-‐conda-‐buildpack Install PyData packages with binary builds using conda
![Page 12: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/12.jpg)
h"ps://github.com/alexkago/cf-‐buildpack-‐r R interpreter and package setup, ready for RShiny
![Page 13: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/13.jpg)
Siloed Data
Siloed Systems
Distributed Big Data Platform
HOW TO DEPLOY
MODELS? Data Extract
?
(Model development
happens here!)
(Business needs model predictions
here!)
![Page 14: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/14.jpg)
App
App
App
App
App
Big Data Platform Big Data Storage
![Page 15: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/15.jpg)
R E S T A P I
Send data as JSON
Data Ingest
Model
Create Model
Redis
Kicking off periodic
retraining
Save training data
Save model object Send JSON data
without label
Receive prediction from trained model
instance
Deployed at: http://dsoncf.cfapps.io
Code: https://github.com/pivotalsoftware/ds-cfpylearning
PREDICTION API ARCHITECTURE
$ cf create-‐service rediscloud PLAN_NAME INSTANCE_NAME
![Page 16: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/16.jpg)
MODEL INTERFACE
![Page 17: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/17.jpg)
Data Driven Applications
![Page 18: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/18.jpg)
SIMPLE HTML + JS MODEL PREDICTIONS
http://ds-demo-transport.cfapps.io
![Page 19: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/19.jpg)
RSHINY APP INTERACTIVE EXPLORATION
https://ak-insurance-demo.cfapps.io:4443/
![Page 20: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/20.jpg)
Show off your data science related Cloud Foundry apps:
Twitter: @dsoncf http://dsoncf.com
![Page 21: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/21.jpg)
@ianhuston @akagoshima
![Page 22: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/22.jpg)
R E S T A P I
Send data as JSON
Data Ingest
Model
Create Model
Redis
Kicking off periodic
retraining
Save training data
Save model object
Send JSON data without label
Receive prediction from trained model
instance
Deployed at: http://dsoncf.cfapps.io
Code: https://github.com/pivotalsoftware/ds-cfpylearning
Visualization
PREDICTION API ARCHITECTURE
![Page 23: CFSummit: Data Science on Cloud Foundry](https://reader033.vdocument.in/reader033/viewer/2022052913/55ba8ec3bb61eb7d5a8b45e2/html5/thumbnails/23.jpg)
Data Services Bind and scale system services – Databases, NoSQL, message queues etc. $ cf create-‐service rediscloud PLAN_NAME INSTANCE_NAME $ cf bind-‐service APP_NAME INSTANCE_NAME
Add User Provided Services – Standalone Hadoop or Apache Spark cluster, Big Data System $ cf cups SERVICE_INSTANCE -‐p "host, port, username, password"