designing & optimizing the dna of a killer app (dan olsen, startonomics sf 2008)
DESCRIPTION
Presentation by Dan Olsen, YourVersion, on how to plan, design, & optimize product development by focusing on measurable events & user actions, business value creation.TRANSCRIPT
Copyright © 2008 Olsen Solutions LLC
Designing & OptimizingDesigning & Optimizingthe DNA of a Killer Appthe DNA of a Killer AppDan Olsen, CEO & Founder, YourVersionDan Olsen, CEO & Founder, YourVersionStartonomics Startonomics · San Francisco · Oct 2, 2008 · San Francisco · Oct 2, 2008
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Building the DNA of a Killer App with Building the DNA of a Killer App with Metrics and OptimizationMetrics and Optimization
How to integrate metrics & optimizationHow to integrate metrics & optimization Using metrics to optimize value creationUsing metrics to optimize value creation
Business ValueBusiness Value Customer ValueCustomer Value UI DesignUI Design
Illustrate with real world examplesIllustrate with real world examples
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Adding Metrics and Optimization to your Product Process
PlanPlan
DesignDesign
DevelopDevelop
BusinessObjectives
ProductObjectives
Prioritized Feature List
Scoping
Requirements & Design
Code Test Launch
Site Level
Feature Level
OptimizeOptimize Metrics & User Feedback
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Optimization through Iteration:Optimization through Iteration:Continuous ImprovementContinuous Improvement
Measurethe metric
Analyzethe metric
Identify top opportunities
to improve
Design & develop the enhancement
Launch theenhancement
Learning
Gaining knowledge:
• Market
• Customer
• Domain
• Usability
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
OptimizingOptimizingBusiness Value Creation Business Value Creation
with Metricswith Metrics
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Approaching Business as an Approaching Business as an Optimization ExerciseOptimization Exercise
Given reality as it exists today,Given reality as it exists today,optimize our business resultsoptimize our business resultssubject to our resource constraints.subject to our resource constraints.
Profit = Profit = RevenueRevenue - Cost - Cost
Unique VisitorsUnique Visitors x x Ad Revenue per VisitorAd Revenue per Visitor
Impressions/VisitorImpressions/Visitor x Effective CPM / 1000 x Effective CPM / 1000
Visits/Visitor x Pageviews/Visit x Impressions/PVVisits/Visitor x Pageviews/Visit x Impressions/PV
New VisitorsNew Visitors + Returning Visitors + Returning Visitors
Invited VisitorsInvited Visitors + Uninvited Visitors + Uninvited Visitors
# of Users Sending Invites x Invites Sent/User x Invite Click-through Rate# of Users Sending Invites x Invites Sent/User x Invite Click-through Rate
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Define the Equation of your BusinessDefine the Equation of your Business“Peeling the Onion”“Peeling the Onion”
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
How to Track Your MetricsHow to Track Your Metrics Track each metric as daily time seriesTrack each metric as daily time series
Create ratios from primary metrics: X / YCreate ratios from primary metrics: X / Y Example: How good is your registration page?Example: How good is your registration page? Okay:Okay: # of registered users per day# of registered users per day Better:Better: registration conversion rate =registration conversion rate =
# registered users / # uniques to reg # registered users / # uniques to reg pagepage
DateDateUnique Unique VisitorsVisitors
Page Page viewsviews
Ad Ad RevenueRevenue
New User New User Sign-upsSign-ups ……
4/24/084/24/08 10,10010,100 29,60029,600 2525 490490
4/27/084/27/08 10,50010,500 27,10027,100 2424 480480
……
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Sample Signup Page Yield DataSample Signup Page Yield Data
Daily Signup Page Yield vs. TimeNew Registered Users divided by Unique Visitors to Signup Page
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10
Da
ily S
ign
up
Pa
ge
Yie
ld
Changedmessaging
Added questionsto signup page
Started requiringregistration
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Using Metrics to Optimize the Using Metrics to Optimize the Equation of your BusinessEquation of your Business
What are the metrics for your business?What are the metrics for your business? Where is current value for each metric? Where is current value for each metric? How many resources to “move” each metric?How many resources to “move” each metric?
Developer-hours, time, moneyDeveloper-hours, time, money Which metrics have highest ROI opportunities?Which metrics have highest ROI opportunities?
Retu
rnRe
turn
InvestmentInvestment
Retu
rnRe
turn
InvestmentInvestmentRe
turn
Retu
rnInvestmentInvestment
Metric AMetric AGood ROIGood ROI
Metric BMetric BBad ROIBad ROI
Metric CMetric CGreat ROIGreat ROI
• Which metric has highest ROI opportunity?
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Case Study:Case Study:Optimizing Viral Loop MetricsOptimizing Viral Loop Metrics
ActiveUsers
Prospective Users
Invite Click
Succeed
Inviteclick-through rate
Conversion rate
Don’t Click
Fail
Invites per sender
% of users sending invites
• Multiplied together, these metrics determine your viral ratio
Users
% of users who are active
= 15%= 2.3
= 85%
Registration Process
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Maximum Upside Potential of a MetricMaximum Upside Potential of a Metric
0
100%
0
100%
0
?
Registration conversion rate
% of users sending invitations
Avg # of invites sent per sender
2.3
85%
15%
Max possible improvement
0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Doubling Number of Invitations Sent per Sender by Doubling Number of Invitations Sent per Sender by Adding Address Book ImporterAdding Address Book Importer
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
OptimizingOptimizingCustomer Value Creation Customer Value Creation
with Metricswith Metrics
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
The Fuzzy Math of Customer ValueThe Fuzzy Math of Customer Value You create customer value with a product thatYou create customer value with a product that
Satisfies customers’ needsSatisfies customers’ needs Is easy to useIs easy to use Has a good priceHas a good price Is better than other alternativesIs better than other alternatives
Customer value = customer satisfactionCustomer value = customer satisfaction Not fuzzy: # of users and frequency of useNot fuzzy: # of users and frequency of use Understanding user needs and satisfactionUnderstanding user needs and satisfaction
Quantitative: site analytics, usage metrics, surveysQuantitative: site analytics, usage metrics, surveys Qualitative: support emails, usability sessions (best)Qualitative: support emails, usability sessions (best)
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
What is Your Value Proposition?What is Your Value Proposition? Which user benefits are most important?Which user benefits are most important? How well does your product deliver?How well does your product deliver?
Benefit 1
ImportancImportancee to User to User
Current Current UserUser
SatisfactionSatisfaction
Benefit 2
User BenefitUser BenefitUpsideUpside
PotentialPotential
Benefit 4
Benefit 5
Benefit 6
High
High
Med
Low
Low
Low
High – +
Doesn’t matter
Benefit 3 Lowopportunity
opportunity
?
?
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Analyzing Product Ideas by ROIAnalyzing Product Ideas by ROI
Investment (developer-weeks)
Ret
urn
(V
alu
e C
reat
ed)
Idea C
Idea B
Idea D
Idea A
Idea F
1
1
2 3 4
2
3
4?
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Prioritizing Product Ideas by ROIPrioritizing Product Ideas by ROI
Investment (developer-weeks)
Ret
urn
(V
alu
e C
reat
ed)
Idea B
Idea C
Idea A
1
1
2 3 4
2345678
5
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
The UI Design IcebergThe UI Design Iceberg
VisualDesign
InteractionDesign
InformationArchitecture
ConceptualDesign
Recommended reading: Jesse James Garrett’s“Elements of User Experience” chart, free at www.jjg.net
What most people seeand react to
What good PMs and Designers think about
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
UI questions are never yes/no! (not binary)UI questions are never yes/no! (not binary) Should ask: “What percentage of users …?”Should ask: “What percentage of users …?” UI changes impact your metricsUI changes impact your metrics
Impact can be positive, negative, small, largeImpact can be positive, negative, small, large Seek high-ROI UI changesSeek high-ROI UI changes
Approaching UI Design AnalyticallyApproaching UI Design Analytically Typical UI design question:Typical UI design question:
““When using web pages, do users scroll down?”When using web pages, do users scroll down?”- Yes- Yes- No- No
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Put Key Conversion Actions Above The FoldPut Key Conversion Actions Above The Fold
Landing Page ALanding Page A Landing Page BLanding Page B
Key conversion action is above the fold
Key conversion action is below the fold
The Fold
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
The Fold Isn’t Binary EitherThe Fold Isn’t Binary Either
600 px
768 px
1024 px
Data courtesy of ClickTaleData courtesy of ClickTaleFree trial at www.clicktale.comFree trial at www.clicktale.com
% o
f Use
rs The chrome steals about 170 pixels
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Case Study: Account Signup Process Case Study: Account Signup Process RedesignRedesign
100%
62.3%58.8%
50.9%
34.4% 32.7%
0%
20%
40%
60%
80%
100%
% o
f Use
rs%
of U
sers
Sign in /Sign in /RegistrationRegistration
Account TypeAccount Type Cash vs.Cash vs.MarginMargin
5 Partner5 PartnerPagesPages
3 Partner3 PartnerPagesPages
Biggest drop
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
OpenAccount
Sign in
Account Selection
Register
56%
44%
Forget Password
Registration Process
45% drop off(20% of total)
36% overall drop off for
this step
70%(32% of Total)
17% drop off (10% of total)
20% drop off(6% of total)
30%(14% of Total)
80%(26% of Total)
55%(24% of Total)
64%of Total
Analysis of Sign In/Registration FlowAnalysis of Sign In/Registration Flow
Change Password
83%(46% of Total)
Abandonment Rate (7 Day Moving Average)
0%
10%
20%
30%
40%
50%
60%
70%
80%
10/7
/02
10/1
4/02
10/2
1/02
10/2
8/02
11/4
/02
11/1
1/02
11/1
8/02
11/2
5/02
12/2
/02
12/9
/02
12/1
6/02
12/2
3/02
12/3
0/02
1/6/
03
1/13
/03
1/20
/03
Ab
and
on
men
t R
ate
(7 D
ay M
ovi
ng
Ave
rag
e)
Steps 1-2
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Redesigned User Flow Improved Redesigned User Flow Improved Registration Conversion RateRegistration Conversion Rate
37% improvement in conversion rate
ReleasedNew Design
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Summary: Building the DNA of a Killer Summary: Building the DNA of a Killer AppApp
Define what success meansDefine what success means Customer value propositionCustomer value proposition Equation of your businessEquation of your business
Instrument your site and track key metricsInstrument your site and track key metrics Identify opportunities and prioritize by ROIIdentify opportunities and prioritize by ROI Launch, learn, and iterateLaunch, learn, and iterate
Copyright © 2008 Olsen Solutions LLC
[email protected]@yourversion.com
Questions?Questions?
www.yourversion.comwww.yourversion.com
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
AppendixAppendix
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Taking Optimization to The Next LevelTaking Optimization to The Next Level
Multivariate testing, aka A/B testingMultivariate testing, aka A/B testing New challenges emergeNew challenges emerge
Are we at local vs. global optimum?Are we at local vs. global optimum? How do we identify top new ideas to test? How do we identify top new ideas to test?
(creativity and innovation)(creativity and innovation) Are there assumptions we should Are there assumptions we should
challenge?challenge? Are we missing the forest for the trees?Are we missing the forest for the trees?
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Is the site up when I want to use it?
Is the site fast enough?
Does the functionality work?
How compelling and easy to use is the functionality?
Olsen’s Hierarchy of Web User Needs(adapted from Maslow)
Customer’s Perspective What does it mean to us?
Uptime
Page Load Time
Absence of Bugs
Feature Set
Usability & Design
Decreasing
Dissatisfaction
Increasing S
atisfaction
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Kano Model: User Needs & SatisfactionUser SatisfactionUser Satisfaction
User DissatisfactionUser Dissatisfaction
Performance Performance (more is better)(more is better)
Delighter (wow)Delighter (wow)
NeedNeednot metnot met
NeedNeedfully metfully met
Must HaveMust Have
Needs & features Needs & features migrate over timemigrate over time
Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC
Importance vs. Satisfaction Ask Users to Rate for Each Feature
98
8784
8679 847055 80
7280
75
4150
55
60
65
70
75
80
85
90
95
100
40 50 60 70 80 90 100
Satisfaction
Imp
ort
an
ce
Recommended reading: “What Customers Want” by Anthony Ulwick
BadBad
GreatGreat