10 measures and kpis for ml success
TRANSCRIPT
10 measures and KPIs for ML successBuilding the business case for MLOps and management
algorithmia.com | @algorithmia
We’ll get started in just a moment. While we wait for everyone to join, enjoy getting to know our speaker better as we ask him questions about his youth 😊
Send over your questions! 🎉
Diego Oppenheimer, CEO, Algorithmia
Diego Oppenheimer is co-founder and CEO of Algorithmia. Previously, he designed, managed, and shipped some of Microsoft’s most used data analysis products including Excel, Power Pivot, SQL Server, and Power BI. He holds a Bachelor’s degree in Information Systems and a Master’s degree in Business Intelligence and Data Analytics from Carnegie Mellon University.@doppenhe | linkedin.com/in/diego
Speaker
Survey questions while we wait for people to join…
1. How many ML models do you have in production?
a) None yetb) 1-5c) 5-25d) 25+e) Do not know
2. What’s the typical time it takes you to deploy a new ML model?
a) 0-7 daysb) 8-30 daysc) 31-90 daysd) 91-365 dayse) 1+ yearsf) Do not know
3. What is your infrastructure cost of operations for all your models in production?
a) Under $50kb) $50k-$250kc) $250k-$500kd) $500k or moree) Do not know
2020 report findings
From the 2020 state of enterprise ML report:
● 55% of companies developing machine learning have never deployed a model.
● 22% of companies spend between 31 and 90 days deploying a model.
● 25% of data scientist time is spent on deployment.
algorithmia.com/state-of-ml
The machine learning lifecycle
Data, network, model securityInfosec complianceRegulatory compliance
Customerapplications
ValidationVerificationPreventive controls
Measures and KPIs
Leverage – Measure impact
● Shorten time to market
● Reduce IT ops overhead
● Rapid iteration and model optimization
● Model reusability
● Infrastructure scalability and cost
Risk – KPIs to measure value
● Time to deployment
● Models in production
● New models deployed each month
● API calls
● Chargebacks and showbacks
1. Shorten time to market
● Automate deployment paths
● Impact of model in production
● Monitoring and reporting 2020 State of Enterprise Machine Learning Survey
31+ days Average time to deploy for 59% of organizations.
1. Shorten time to marketChallenge: A customer wanted to use ML for fraud detection and compliance, but was taking up to 18 months to deploy a single model.
Criminal patterns and behavior would change during that time, making any models obsolete at deployment.
Solution:● Algorithmia streamlined the developer
workflow for machine learning.
● Enabled modular pipeline for model reuse, allowing quick editing and deployment into client applications.
● Deployed 100+ models with 10 different open-source frameworks and more than 75 libraries in 5 months.
Top 4 consulting firm
Platform approach to ML ops provides KPIs:
● Time to deploy
● Business result
● Performance
2. Reduce IT Ops overhead
Model registration
● Self-service
● Documentation
● Model operations
2x growthNumber of AI projects doubling in the next year by 2021Gartner predicts the future of AI technologies, Feb 2020
Gartner predicts the future of AI technologies, Feb 2020
2x Number of new AI projects by 2021.
2. Reduce IT Ops overhead
Operational KPIs:
● Models in production
● Call and errors
● Hardware utilization
Challenge: In experimenting with computer vision and ML for rapid prototyping, Toyota’s insurance group, MS&AD, found that deploying each model as a “one-off” was time-consuming and resource-intensive.
Solution:● Toyota increased models in production
by 75% with a consolidated ML architecture running on Algorithmia.
● The Algorithmia platform provided a central space to serve the company’s ML innovation needs and made collaboration across teams easier.
3. Rapid iteration and model operation
● Rapid iteration
● Freedom to choose
● Pipelining80% will face production delaysFor enterprises moving ML to production, due to lack of collaboration and IT process immaturity.
“Accelerate your ML & AI Journey” Gartner, 2019
v.1.0.0
v.2.0.0
CONSUMING APPS:
API
“Accelerate your ML & AI Journey” Gartner, 2019
80%
Of enterprises will face production delays due to lack of collaboration and IT process immaturity.
Production model KPIs:
● New models deployed each month
● Models in production
● Model versioning
3. Rapid iteration and model operation
Challenge: HappyMoney needed to integrate user data and apply ML models in real time to provide advice on how to pay down personal debt efficiently.
Solution:● Algorithmia integrated real-time
and batch data from financial services firms to run the models and generate savings recommendations.
● Algorithmia chained versions of HappyMoney’s cash-flow model projections using different frameworks from real-time bank transaction data and risk analysis from different data science tools.
4. Model reusability
● Reusability
● Future-proofing
● Model maintenance
2020 State of Enterprise Machine Learning Survey
Top 3 use casesReduce company costsProcess automation for internal organizationImprove customer experience
4. Model reusability
Model activity KPIs:
● API calls
● Avg call duration
● Models in production
Challenge: Wanted to deliver customer engagements without long consulting engagements, software purchases, sourcing external data, or repeatedly building out applications.
Solution:● They added Algorithmia into the
workflow as a repository for reusable models and a model deployment and serving engine.
● Their customers can now integrate data connectors and pipeline models into one end-to-end analytics process using proprietary industry algorithms.
Logistics Consulting company
Traditional capacity planning
● Wasteful and expensive
● Hard capacity limits
Local maximum planning
● Still inefficient
● Increased management overhead
Serverless (Algorithmia)
● Massive efficiency boost
● Substantially lower cost
● Infinite scalability
5. Infrastructure cost and scalability
5x increaseBy 2023, cloud-based AI will increase 5x, making AI one of the top cloud services.Gartner predicts the future of AI technologies, Feb 2020
Gartner predicts the future of AI technologies, Feb 2020
5xThe amount cloud-based AI will increase by 2023. Making AI one of the top cloud services.
5. Infrastructure cost and scalability
Infrastructure consumption KPIs:
● Chargebacks and showbacks reports
● Auditability: Who called what model,
at what time, with which data
Measures and KPIs
Leverage – Measure impact
● Shorten time to market
● Reduce IT ops overhead
● Rapid iteration and model optimization
● Model reusability
● Infrastructure scalability and cost
Risk – KPIs to measure value
● Time to deployment
● Models in production
● New models deployed each month
● API calls
● Chargebacks and showbacks
Algorithmia Enterprise
Model deploymentConnect, load, catalog, version, and validate models for production
Model operationsManage costs, control infrastructure usage, monitor operations, deliver models and services at high velocity
Governance and securityAdvanced security enabling rule-based control over users, data, infrastructure and models
Building versus buying:how to achieve ML operations
Join us on 9 June for our next webinar that will help you evaluate a build vs buy decision on investing in your ML operations and management capacity.
Webinar
Register today: info.algorithmia.com/build-vs-buy-webinar
Kenny DanielCTO Algorithmia
Sam CharringtonTWIML AI Podcast Host
Q&A