how to train your product owner
TRANSCRIPT
How to Train Your Product Owner
David Murgatroyd ( @dmurga)MassTLC ML Dev DayJanuary 24, 2018
(please don’t sue me for copyright infringement, DreamWorks!)
Agile & Machine LearningDavid Murgatroyd (@dmurga)(ML Chapter Lead in Quest)
Your Product Owner
Agile & Machine LearningDavid Murgatroyd (@dmurga)(ML Chapter Lead in Quest)
Your Product Owner
Your Product
@dmurga
@dmurga
What do Product Owners Do?Lead team in: ‣ establishing vision/hypotheses for
what product should become‣ prioritizing work to get there
@dmurga
What do Product Owners Do?Lead team in: ‣ establishing vision/hypotheses for
what product should become‣ prioritizing work to get there
Your Product
@dmurga
Your Product
Machine Learning
@dmurga
Outside In
Machine Learning
Your Product
@dmurga
Outside In … Now to Then
Machine LearningMachine Learning
Your ProductYour Future Product
@dmurga
Outside In … Now to Then
Machine Learning
Problem Metrics Data Models Prioritizing OrganizingDesign
Machine Learning
Your ProductYour Future Product
Train Your PO…Outside In
@dmurga
Machine Learning
Problem Metrics Data Models
Outside In
Design
Your Product
@dmurga
Machine Learning
Problem Metrics Data Models
Picking a Problem
Design
Your Product
1. Tweak It2. Think It
@dmurga
Problem Metrics Data Models
Picking a Problem: Tweak It
Design
‣ What’s the business goal of your product?
‣ What fuzzy decision does it make that impacts that goal?
Your Product
@dmurga
‣ Perception: a person can do it in less than a second.
‣ Prediction: done over and over‣ Personalization: similar need but
met in different ways
Problem Metrics Data Models
Picking a Problem: Think It
Design
Your Product
?
@dmurga
@dmurga
@dmurga
1. The role of ‘why’ of ML output?2. Coping with errors?3. Designing for varied output?4. New user behaviors that ML
enables?
Problem Metrics Data Models
Designing with ML
Design
Your Product
@dmurga
Design
‣ Trading off why vs. right‣ Granularity of ‘why’‣ Creepy vs. personal‣ First step: designer gives concrete
example on concrete data
Problem Metrics Data Models
Design: the role of ‘why’
Your Product
@dmurga
‣ What quality is needed to user maintain trust or delight them?
‣ Rank multiple outputs?‣ Provide fallback behavior?‣ Explicit feedback mechanisms?
Problem Metrics Data Models
Design: coping with errors
Design
Your Product
@dmurga
‣ Get realistic output samples from a variety of users (e.g., synthetic, persona)
‣ Watch for subtle assumptions
Problem Metrics Data Models
Design: coping with variety
Design
Your Product
@dmurga
‣ Users might anthropomorphize ML Products
‣ Users might express more of themselves or test the limits of the systems
‣ Avoid “user bubbles” by encouraging discovery and crafting metrics
Problem Metrics Data Models
Design: new user behavior
Design
Your Product
@dmurga
‣ Alignment to business value‣ Effort to measure‣ Useful for team’s decisions‣ Useful for model’s training
Problem Metrics Data Models
Metrics: Properties
Design
Your Product
Machine Learning
@dmurga
Problem Metrics Data Models
Metrics: Matrix of Metrics
Design
Where / What
Heuristic Modeled
Online Online Heuristic OnlineModeled
Offline Offline Heuristic Offline Modeled
Your Product
Machine Learning
@dmurga
Problem Metrics Data Models
Metrics: Online vs Offline
Design
Online More aligned with business value
Offline Generally less effort to measure
Your Product
Machine Learning
@dmurga
Problem Metrics Data Models
Metrics: Heuristic vs Modeled
Design
Heuristic Modeled
More useful for team’s decisions interpretable
More useful for training the model directly
Your Product
Machine Learning
@dmurga
Problem Metrics Data Models
Metrics: Matrix of Metrics
Design
Where / What
Heuristic Modeled
Online Online Heuristic OnlineModeled
Offline Offline Heuristic Offline Modeled
Your Product
Machine Learning
@dmurga
‣ Just look at some data before/after‣ Alignment to business value: OK‣ Effort to measure: OK (one-off)‣ Useful for team’s decisions: OK‣ Useful for model’s training: Bad
Problem Metrics Data Models
Metrics: … vs Subjective
Design
Your Product
Machine Learning
@dmurga
1. What data do you need?2. Where can you get it?3. What biases does it carry?
Problem Metrics Data Models
Data
Design
Your Product
@dmurga
‣ Raw input data‣ Metadata / reference‣ Has metadata that is a source for
measurement
Problem Metrics Data Models
What data do you need?
Design
Your Product
@dmurga
‣ Data is the new Wireframe‣ Product Owner provides example
inputs / outputs‣ Use these to also vet metrics
Problem Metrics Data Models
Specifying with Data
Design
Your Product
Machine Learning
@dmurga
‣ Logging‣ Proxies and other 3rd Parties‣ Annotation‣ Watch out for drift in the data set‣ Take advantage of PO’s domain
knowledge!
Problem Metrics Data Models
Where can you get it?
Design
Your Product
@dmurga
‣ Is it skewed?‣ Is it tainted?‣ Is it likely to stereotype?
Problem Metrics Data Models
What biases does it carry?
Design
Your Product
@dmurga
1. Teach tasks rather than techniques2. Teach trade-offs rather than tools3. Start simple
Problem Metrics Data Models
Models
Design
Your Product
Machine Learning
@dmurga
‣ Classification‣ Clustering‣ Regression‣ (and Embeddings if you must :- )
Problem Metrics Data Models
Models: Tasks
Design
Your Product
Machine Learning
@dmurga
‣ Simplicity‣ Interpretability‣ Confidence‣ Accuracy‣ Adaptability‣ Speed‣ Space‣ Scale
Problem Metrics Data Models
Models: Trade-offs
Design
Your Product
Machine Learning
@dmurga
‣ Rules before baseline models‣ Baselines before adapted models‣ Adapted models before end-to-end
models
Problem Metrics Data Models
Models: Err toward Simplicity
Design
Your Product
Machine Learning
Train Your PO…Now to Then
@dmurga
Outside In … Now to Then
Machine Learning
Problem Metrics Data Models Prioritizing OrganizingDesign
Machine Learning
Your ProductYour Future Product
@dmurga
Now to Then
Prioritizing Organizing
Machine Learning
Your Future Product
@dmurga
Prioritizing
Prioritizing Organizing
1. Exploration2. Experimentation3. Error Analysis4. ML Product Lifecycle 5. Longer term Goals (OKRs)
Your Backlog
@dmurga
Prioritizing: Exploration
Prioritizing Organizing
Your Backlog
@dmurga
Prioritizing: Experiments
Prioritizing Organizing
Your Backlog
@dmurga
Prioritizing: Error Analysis
Prioritizing Organizing Bug ErrorYour Backlog
@dmurga
Prioritizing: Error Analysis
Prioritizing Organizing
Your Backlog
@dmurga
Prioritizing: Prod Lifecycle
Prioritizing Organizing
Your Backlog
Stage Characteristics
Exploration by hand examples/rules
Pre-MVP (0.1% / early Beta)
measurable & inspectable
MVP (1%, Beta) accurate, not slow, & documented
v1 (100% / GA) simple & fast
Post-v1 handle new domains
@dmurga
Prioritizing: Goals (OKRs)
Prioritizing Organizing
Your Backlog
Stage OKRs
Exploration amount of analysis
Pre-MVP (0.1% / early Beta)
having metrics, amount of experiments
MVP (1%, Beta) moving core metrics, amount of experiments
v1 (100% / GA) moving all metrics
Post-v1 moving all metrics on new data
@dmurga
Organizing
Prioritizing Organizing
1. Team Structure2. Roles
Your Backlog
@dmurga
Organizing: Team Structure
Organizing
‣ Aligned to Product/Org Maturity‣ Exploration: Centralized Team
Your Backlog
Prioritizing
@dmurga
Organizing: Team Structure
Organizing
‣ Aligned to Product/Org Maturity‣ Exploration: Centralized Team‣ Pre-MVP: Cross-functional
Your Backlog
Prioritizing
@dmurga
Organizing: Team Structure
Organizing
‣ Aligned to Product/Org Maturity‣ Exploration: Centralized Team‣ Pre-MVP: Cross-functional‣ MVP to v1: Cross-functional with
separate work stream
Your Backlog
Prioritizing
@dmurga
Organizing: Team Structure
Prioritizing Organizing
‣ Aligned to Product/Org Maturity‣ Exploration: Centralized Team‣ Pre-MVP: Cross-functional‣ MVP to v1: Cross-functional with
separate work stream‣ Post v1: Dedicated Sibling Team
Your Backlog
@dmurga
‣ Applied ML Eng
‣ ML Tool Eng
‣ Core Researcher
Organizing: Roles
Prioritizing Organizing
@dmurga
Outside In … Now to Then
Machine Learning
Problem Metrics Data Models Prioritizing OrganizingDesign
Machine Learning
Your ProductYour Future Product
Thanks! Questions?
David Murgatroyd (@dmurga)
Suggestions:
What about different kinds of testing?
What are common features ML-based products have?
More on identifying metrics?
Machine Learning vs. Data Science?
Hiring in Boston, NYC, London, and Stockholm!