designing to learn: creating effective mvp experiments - lean ux nyc 2014
DESCRIPTION
Designing MVP workshop for Lean UX NYC 2014.TRANSCRIPT
Designing to LearnCreating Effective MVP Experiments
Melissa Perri @lissijean
Consultant Lean Product Manager UX Designer Teacher Twitter addict (@lissijean) Blogger (melissaperri.com)
Hi! I’m Melissa.
@lissijean
• What is an MVP (and why should we care)?
• Setting Up MVP Experiments
• Types of MVPs and How to Choose
• Iterating on MVPs
• Agile + MVP: How does it work together?
Agenda
@lissijean
Why MVP?
@lissijean
“Match your taste with the exceptional finds of industry insiders.”
@lissijean
@lissijean
My way is faster.
This isn’t what I needed.
I don’t know how to use it.
The information I want to see isn’t on here.
@lissijean
@lissijean
@lissijean
@lissijean
Validate Before Building
@lissijean
Build
Measure
Learn
@lissijean
Iterate your way to Product/Market Fit
your users
@lissijean
MVPMinimum Viable Product
@lissijean
MVP Smallest amount of work you can do
to test your idea.
@lissijean
@lissijean
@lissijean
@lissijean
@lissijean
LEARN
@lissijean
MVP = Minimum Feature Set
Agile Methodology MVPs
@lissijean
Focus on problem not features.
@lissijean
MYTH
@lissijean
Crappy products
Crappy products everywhere
@lissijean
Good MVPs offer a great experience to the customer.
@lissijean
@lissijean
MVP =
Experiment
@lissijean
Experiment Process Set Up !
Design !
Run !
Feedback !
Iterate
@lissijean
@lissijean
Setting up MVP Experiments
PART I
@lissijean
1. Define Customer & Problem 2. Investigate assumptions 3. Design a test 4. Measure customer behavior 5. Evaluate success 6. Pivot or persevere
Process
@lissijean
@lissijean
DefineDefine
@lissijean
Customers want twitter-like messages in their feed.
@lissijean
Customers want twitter-like messages in their feed.
Not a problem.
@lissijean
Real ProblemCustomers don’t think the
curators are authentic.
@lissijean
Investigate
@lissijean
Assumptions
Shoppers care that curators are authentic.
Shoppers will engage more with real messages on site.
Shoppers think tweets are authentic.
@lissijean
Riskiest Assumption
Shoppers care that curators are authentic.
Shoppers will engage more with curator messages on site.
Shoppers think tweets are authentic.
@lissijean
Design a Test
@lissijean
Setting up a test1. What do we want to learn? 2. How do we measure that? 3. What’s the least we can build to learn?
@lissijean
Will shoppers engage more with curator messages?
What do we want to learn?
@lissijean
Engagement Clicks, purchases, added to cart
How do we measure that?
@lissijean
What is the least we can build to learn?
Wizard of OzLooks real to the customer,
completely manual on the backend.
@lissijean
@lissijean
@lissijean
Measure
@lissijean
How do we measure engagement?
@lissijean
Clicks on product.
Purchases. Add to cart.
@lissijean
What do we consider successful?
@lissijean
Minimum Success CriteriaWhat is the minimum number we need to
hit to consider this validated?
@lissijean
Time Salary# Peoplex x
Cost to build completely
@lissijean
Minimum Success Criteria
Improvement needed to justify cost to build.
@lissijean
25% increase in engagement in A/B test.
@lissijean
@lissijean
Evaluate
@lissijean
@lissijean
0% change over other products.
@lissijean
INVALIDATED
@lissijean
But why?
@lissijean
Why didn’t users engage?
What do we want to learn?
@lissijean
User Testing Sessions watch them use it.
How do we find that out?
@lissijean
Emails.
@lissijean
Pivot
@lissijean
strategic change in direction based on learning.
Pivot
@lissijean
Riskiest Assumption
Shoppers will engage more with curator messages.
@lissijean
Will shoppers engage more with curator messages
through email?
What do we want to learn?
@lissijean
Engagement Opens, Click throughs, purchases.
How do we measure that?
@lissijean
What is the least we can build to learn?
@lissijean
How do we determine success?
30% increase in engagement over other emails.
@lissijean
Success!2x Open Rate 3x Purchases
@lissijean
Now, we automate. Now, we invest.
@lissijean
workshop one:
Setting up MVP Experiments
@lissijean
@lissijean
We’ve recently received a flurry of complaints from Macy’s shoppers.
@lissijean
“Every time I come into Macy’s, I spend HOURS I don’t have and can never seem to find what I’m looking for. I know exactly what type of item I want, but options are all over the place. It’s daunting.”
@lissijean
“I needed a blue tie to go with my suit for a friend’s wedding, so I went to find it but there were too many options. Where do I even start? I don’t even know how to choose.”
@lissijean
Who is the customer?
@lissijean
“OpenSky shoppers who visit the site at least once a month, but rarely buy.”
@lissijean
What is their problem?
@lissijean
“Do not think the curators are authentic.”
@lissijean
10 minutes
In your teams, write down your customer and their problem for Macy’s.
@lissijean
Shoppers who are purchasing something specific out of necessity, not browsing.
Customer Hypothesis
@lissijean
Cannot find what they need efficiently.
Problem Hypothesis
@lissijean
What are we assuming about the customer and the problem?
@lissijean
Riskiest AssumptionWhat are we the most uncertain about?
@lissijean
10 minutes
In your teams, write down your assumptions and pick the riskiest one.
@lissijean
Assumptions
Shoppers are overwhelmed by the amount of options.
Shoppers do not want to browse.
Shoppers want help finding items.
@lissijean
Assumptions
Shoppers are overwhelmed by the amount of options.
Shoppers do not want to browse.
Shoppers want help finding items.
@lissijean
Now, we design a test for that riskiest assumption.
@lissijean
Types of MVPs
PART II
@lissijean
WHAT SHOULD YOU ALWAYS DO FIRST?
@lissijean
InterviewsGet out of the building and talk to people.
@lissijean
• First thing you should do. • Talk to at least 10 people. • Verify the problem and the customer hypothesis. • Verify as many assumptions as you can.
When to do Interviews
@lissijean
ConciergeDelivering a service manually.
Delivering a service manually.Concierge
@lissijean
@lissijean
@lissijean
The Good The Bad• Very close to the
customer, easy to learn. • Can adjust easily based on
feedback. • Very easy and cheap to
get started.
• Not scalable. • Time consuming to run.
Concierge
@lissijean
• If you are testing a service. • Trying to determine what customers will respond to. • Testing and refining with a small number of people.
When to use Concierge.
@lissijean
Wizard of OzLooks real, but manual on the back end.
@lissijean
@lissijean
Subscription
@lissijean
@lissijean
@lissijean
The Good The Bad
• Looks exactly real to customers, so very accurate response to if you would build it.
• Need to code to get started. • Can’t tell you why customers
don’t use it.
Wizard of Oz
@lissijean
• Running a test with a large audience. • Trying to sell a feature or service before it is built,
but pretend it is already there. • If you are concerned about sacrificing brand.
When to use Wizard of Oz.
@lissijean
Landing PageA pitch of your product, to gauge idea reception.
@lissijean
@lissijean
@lissijean
The Good The Bad
• Just a pitch, don't need to refine the whole idea
• Can launch in a matter of minutes
• MOST PEOPLE FORGET THEY NEED TO DIRECT PEOPLE HERE THROUGH MARKETING.
Landing Page
@lissijean
• Looking for initial customers to test. • Trying to gauge how much interest there is
in the idea. • Great way to pitch to a large audience.
When to use Landing Page.
@lissijean
VideosDemo what your product can do through hacking
it together for a video.
@lissijean
@lissijean
The Good The Bad
• Can demonstrate complicated things.
• Show a prototype before producing them all.
• More complicated to put together.
• Needs to be a certain amount done before you can make the video.
Videos
@lissijean
• If your product does something very easily. • When your product sounds complicated. • When people need to see it to believe it.
When to use Videos.
@lissijean
What if I’m not solving a dire need?
@lissijean
workshop two:Design your test.
@lissijean
You have already interviewed your customer,
they have that problem.
@lissijean
Now how are you going to solve that problem?
@lissijean
Choose an MVP strategy and plan out a test.
30 minutes
• What you are trying to learn? • Explain every little detail of the test. • What are you measuring? • Minimum success criteria.
@lissijean
Resources
Access to 2 Macy’s stores Access to Macy’s online page !
3 Store Interns $30,000 salary each 10 Sales Associates In Store $40,000 salary each 5 Developers $100,000 salary each 1 Designer $100,000 salary Yourself $100,000 salary
@lissijean
Every time you start with a minimum feature set,
a kitten dies.
@lissijean
Responding to Constraints &
Feedback
PART III
@lissijean
The goal of an MVP is to learn.
@lissijean
Customer Feedback
@lissijean
@lissijean
@lissijean
Constraints
@lissijean
Restricts medium. Restricts resources.
@lissijean
workshop three:
Alter your test based on constraints.
@lissijean
You learned that people only have this problem online.
How do you make an online MVP?
@lissijean
You learned that people only have this problem in store.
How do you make an in person MVP?
@lissijean
How would you change your MVP experiment based on these new
constraints?
5 minutes
@lissijean
MVPs in Agile Teams
PART IV
@lissijean
Gather Requirements
Spec features
Design Interfaces
Development
QA Process
Release
Waterfall
@lissijean
Gather Requirements
Spec features
Design Interfaces
Development
QA Process
Release
Product Manager UX Designer Developer
@lissijean
Agile
@lissijean
Individuals and Interactions over
Processes and Tools
@lissijean
Agile SprintsShort, goal oriented sprints formed around a problem running Build, Measure, and Learn.
@lissijean
Build
Measure
Learn
@lissijean
Product Manager
UX Designer Developer
Small, cross-functional teams
@lissijean
@lissijean
monday tuesday wednesday thursday friday
Reflect & Define Specify Build & Refine Build & Refine Customer Feedback
Sprint Structure
@lissijean
Problem Exploration
MVP Experiments
Feature Development
@lissijean
MVPs are only a test.
@lissijean
“My solution is too obvious, if I just build it people will use it.”
@lissijean
Customers will only use your features if it fills a need.
Customers will only use your features if it fills a need.
@lissijean
“Lean is just for startups, my company is too grown up.”
@lissijean
Lean applies to all companies.
@lissijean
Melissa Perri !
Want an MVP Guide? [email protected] !
produxlabs.com melissaperri.com !
!
@lissijean