machine learning success: the key to easier model management

© 2017 MapR Technologies 1

Machine Learning Success:

The Key to Easier Model Management


Contact Information

Ellen Friedman, PhD

Principal Technologist, MapR Technologies

Committer Apache Drill & Apache Mahout projects

O’Reilly author

Email [email protected] [email protected]

Twitter @Ellen_Friedman

mailto:[email protected]

mailto:[email protected]


Machine Learning Everywhere

Image courtesy Mtell used with permission.Images © Ellen Friedman.


Traditional View


Traditional View: This isn’t the whole story


90% of the effort in successful

machine learning isn’t the

algorithm or the model…

It’s the logistics


Why?

• Just getting the training data is hard

– Which data? How to make it accessible? Multiple sources!

– New kinds of observations force restarts

– Requires a ton of domain knowledge

• The myth of the unitary model

– You can’t train just one

– You will have dozens of models, likely hundreds or more

– Handoff to new versions is tricky


What Machine Learning Tool is Best?

• Most successful groups keep several “favorite” machine

learning tools at hand

– No single tool is best in every situation

• The most important tool is a platform that supports logistics well

– Don’t have to do everything at the application level

– Lots of what matters can be handled at the platform level

• A good design can make a big difference


Rendezvous Architecture

Input Scores

RendezvousModel 1

Model 2

Model 3

request

response

Results


Rendezvous to the Rescue: Better ML Logistics

• Stream-1st architecture is a powerful approach with surprisingly

widespread advantages

– Innovative technologies emerging to for streaming data

• Microservices approach provides flexibility

– Streaming supports microservices (if done right)

• Containers remove surprises

– Predictable environment for running models


Rendezvous: Mainly for Decisioning Type Systems

• Decisioning style machine learning

– Looking for a “right answer”

– Simpler than interactive machine learning (such as in self-driving car)

• Examples include:

– Fraud detection

– Predictive analytics / market prediction

– Churn prediction (as in telecommunications)

– Yield optimization

– Deep learning in form of speech or image recognition, in some cases


Why Stream?

Munich surfing wave Image © 2017 Ellen Friedman


Streaming data has value beyond

real-time insights


Heart of Stream-1st Architecture: Message Transport

Real-time analytics

EMRPatient Facilities

management

Insurance audit

A

B

Medical tests

C

Medical test results

The right messaging tool

supports multiple classes of use

cases (A, B, C in figure)

Image © 2016 Ted Dunning & Ellen Friedman from Chap 1 O’Reilly

book Streaming Architecture used with permission


Stream Transport that Decouples Producers & Consumers

P

P

P

C

C

C

Transport Processing

Kafka /

MapR Streams


MapR Streams in the MapR Converged Data Platform

Enterprise StorageMapR-FS MapR-DB MapR Streams

Database Event Streaming

Global Namespace High Availability Data Protection Self-healing Unified Security Real-time Multi-tenancy

• Helps build a global data fabric

• Multiple types of storage engineered into one technology

• Under the same security & administration


With MapR, Geo-Distributed Data Appears Local

streamData

sourceConsumer


With MapR, Geo-Distributed Data Appears Local

stream

streamData

source

Consumer


With MapR, Geo-distributed Data Appears Local

stream

streamData

source

ConsumerGlobal Data Center

Regional Data Center


Stream transport supports microservices


Stream-1st Architecture: Basis for MicroServices

Stream instead of database as the shared “truth”

POS 1..n

Fraud detector

Last card use

Updater

Card analytics

Other

card activity

Image © 2016 Ted Dunning & Ellen Friedman from Chap 6 of O’Reilly book Streaming Architecture used with permission


Features of Good Streaming

• It is Persistent– Messages stick around for other consumers

– Consumers don’t affect producers

– Consumer doesn’t have to be online when message arrives

• It is Performant– You don’t have to worry if a stream can keep up

• It is Pervasive– It is there whenever you need it, no need to deploy anything

– How much work is it to create a new file? Why harder for a stream?


Raw data is gold!


Raw Data & Training Data Are Key to Success

Model 1

Model 2

Model 3

request

Raw

Add external

dataInput

Database

The world

Raw data may contain features you’ll want in future


Quality & Reproducibility of Input Data is Important!

• Recording raw-ish data is really a big deal

– Data as seen by a model is worth gold

– Data reconstructed later often has time-machine leaks

– Databases were made for updates, streams are safer

• Raw data is useful for non-ML cases as well (think flexibility)

• Decoy model records training data as seen by models under

development & evaluation


Decoy Model in the Rendezvous Architecture

InputScores

Decoy

Model 2

Model 3

Archive

• Looks like a server, but it just archives inputs

• Safe in a good streaming environment, less safe without good isolation


Scores

ArchiveDecoy

m1

m2

m3

Features / profiles

Input Raw


ResultsRendezvousScores

ArchiveDecoy

m1

m2

m3

Features / profiles

Input Raw


MetricsMetrics

ResultsRendezvousScores

ArchiveDecoy

m1

m2

m3

Features / profiles

Input Raw


Models in production live in the

real world:

Conditions may (will) change


How to Do Better – Deployment in Production

• Keep models running “in the wings” – Don’t wait until conditions change to start building the next model

– Keep new models ready

• Hot hand-off– With rendezvous: just stop ignoring the model of interest

• Deploy a canary server– Keep an old model active as a reference

– If it was 90% correct, difference with any better model should be small

– Score distribution should be roughly constant


Advantages of Rendezvous Architecture

Real

model∆

Result

Canary

Decoy

Archive

Input


DataOps: Brings Flexibility & Focus

• You don’t have to be a data scientist to contribute to machine learning

• Software engineer/ developer plays a role: but you need good data skills


Example: Tensor Chicken

Label

training

data

Run the

model

Deploy

model

Gather

training

data

Labeled

image files

Train

model

Update

model

Deep learning project by

software engineer Ian Downard

(see blog + @tensorchicken)


Rendezvous Architecture

Input Scores

RendezvousModel 1

Model 2

Model 3

request

response

Results


How to Do Better

• Data + the right question + domain knowledge matter!

• Prioritize – put serious effort into infrastructure

– DataOps requires more than just data science

• Persist – use streams to keep data around

• Measure – everything, and record it

• Meta-analyze – understand and see what is happening

• Containerize – make deployment repeatable, easy

• Oh… don’t forget to do some machine learning, too


Sign Up for ML Logistics Workshop Series

Three deep-dive machine learning workshops

by Ted Dunning, Chief Applications Architect at MapR:

1. A New Architecture for Machine Learning Logistics: How to use streaming, containers & a microservices design

2. Machine Learning Evaluation: How to do model-to-model comparisons

3. Machine Learning in the Enterprise: How to do model management in production

http://bit.ly/mapr-machine-learning-logistics-series

http://bit.ly/mapr-machine-learning-logistics-series


Additional Resources

O’Reilly report by Ted Dunning & Ellen Friedman © March 2017

Read free courtesy of MapR:

https://mapr.com/geo-distribution-big-data-and-analytics/

O’Reilly book by Ted Dunning & Ellen Friedman

© March 2016


https://mapr.com/streaming-architecture-using-

apache-kafka-mapr-streams/



O’Reilly book by Ted Dunning & Ellen Friedman

© June 2014


https://mapr.com/practical-machine-learning-

new-look-anomaly-detection/

O’Reilly book by Ellen Friedman & Ted Dunning

© February 2014


https://mapr.com/practical-machine-learning/



by Ellen Friedman 8 Aug 2017 on MapR blog:

https://mapr.com/blog/tensorflow-mxnet-caffe-h2o-which-ml-best/

by Ted Dunning 13 Sept 2017 in

InfoWorld:

https://www.infoworld.com/article/3223

688/machine-learning/machine-

learning-skills-for-software-

engineers.html


New book:

O’Reilly book by Ellen Friedman & Ted Dunning © Sept 2017

Pre-register for a free pdf copy of book when it becomes

available 25th September, courtesy of MapR:

http://info.mapr.com/2017_Content_Machine-Learning-

Logistics_eBook_Prereg_RegistrationPage.html


Please support women in tech – help build

girls’ dreams of what they can accomplish

© Ellen Friedman 2015#womenintech #datawomen


Thank you !


Q&A

@mapr

Maprtechnologies

[email protected]

ENGAGE WITH US

@ Ellen_Friedman

machine learning success: the key to easier model management

Data & Analytics