world-class web metrics by dan olsen
Post on 17-Oct-2014
10.335 views
DESCRIPTION
Best practices in web metrics and product managementTRANSCRIPT
Copyright © 2009 YourVersion
Dan OlsenDan OlsenCEO, YourVersionCEO, YourVersionJune 17, 2009June 17, 2009
World‐Class Web Metrics:How to Optimize Your Product & Business
Copyright © 2009 YourVersion
World‐Class Metrics:Amazon.com
$37 B Market Cap$19 B Revenue ‘08
•Automated testing of page elements & suggested products
•High degree of user personalization
•Real‐time recommendations
Copyright © 2009 YourVersion
World‐Class Metrics:Google
$137 B Market Cap$22 B Revenue ‘08
•Auto‐testing ofUI elements & search results
•Automatically learns from user mistakes
•Optimize ads to maximize revenue
Copyright © 2009 YourVersion
OverviewOverview
What is being World‐Class in Metrics?
Optimizing Business Value
Optimizing the Customer Value ofYour Product
Copyright © 2009 YourVersion
My Web Metrics EpiphanyQuicken (1998 – 2001)
New version every year: shrink‐wrapped CD‐ROMPre‐launch
Before development: Validated product ideas with usersDuring development: Usability testing & user feedback
Post‐launchNo “real‐time” metrics dataSales data: 1‐2 month lag, not actionableUser surveys: actionable, but not until next product cycle
Quicken Brokerage (2001 – 2003)Online brokerage launched in September 2002The day after launch: LOTS of metrics data → sought moreMuch faster iteration: metrics → hypothesis → improvement
Copyright © 2009 YourVersion
Metrics Maturity ModelUnaware Aspiring World‐Class
MetricsTracking None Sporadic Comprehensive
breadth & depth
Analysis None Ad hoc Recurring
Testing & Optimization None Some A/B
tests
Automated multivariate
testing
Organization Lack of ownership
Partial ownership
Clear, dedicated owners
Culture Intuition‐based
Data‐driven
Continuous improvement
Copyright © 2009 YourVersion
Optimizing Business ValueOptimizing Business Value
Copyright © 2009 YourVersion
Approaching Business as an Approaching Business as an Optimization ExerciseOptimization Exercise
Given reality as it exists today,Given reality as it exists today,
optimize our business resultsoptimize our business results
subject to our resource constraints.subject to our resource constraints.
Copyright © 2009 YourVersion
( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conve( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conversion Ratersion Rate
Paying UsersPaying Users x x Revenue per Paying UserRevenue per Paying User
New Paying UsersNew Paying Users + + Repeat Paying UsersRepeat Paying Users
Previous Paying Users x ( 1 Previous Paying Users x ( 1 –– Cancellation Rate )Cancellation Rate )Trial UsersTrial Users x x ConvConv RateRate
Profit = Profit = RevenueRevenue ‐‐ CostCost
Define the Equation of your BusinessDefine the Equation of your Business““Peeling the OnionPeeling the Onion””
Copyright © 2009 YourVersion
How to Track Your MetricsTrack each metric as daily time series
Create ratios from primary metrics: X / YExample: How good is your registration page?Okay: # of registered users per dayBetter: registration conversion rate =
# registered users / # uniques to reg page
DateUnique Visitors
Page views
Ad Revenue
New User Sign‐ups …
4/24/08 10,100 29,600 25 490
4/25/08 10,500 27,100 24 480
…
Copyright © 2009 YourVersion
Sample Signup Page Yield DataSample Signup Page Yield Data
Daily Signup Page Yield vs. TimeNew Registered Users divided by Unique Visitors to Signup Page
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10
Dai
ly S
ignu
p Pa
ge Y
ield
Changedmessaging
Added questionsto signup page
Started requiringregistration
Copyright © 2009 YourVersion
Identifying the Identifying the ““Critical FewCritical Few”” MetricsMetrics
What are the metrics for your business?Where is current value for each metric? How many resources to “move” each metric?
Developer‐hours, time, moneyWhich metrics have highest ROI opportunities?
Return
Return
InvestmentInvestment
Return
Return
InvestmentInvestmentRe
turn
Return
InvestmentInvestment
Metric AMetric AGood ROIGood ROI
Metric BMetric BBad ROIBad ROI
Metric CMetric CGreat ROIGreat ROI
Copyright © 2009 YourVersion
Turning Metrics into Meaning & Action
• Clear owner for each key metric
•Focus on trend & changes
•Discuss business & customer impact
•Propose ideas to improve
From Avinash Kaushik’s Occam’s Razor at www.kaushik.net
Copyright © 2009 YourVersion
Optimizing the Customer Optimizing the Customer Value of Your ProductValue of Your Product
Copyright © 2009 YourVersion
Using Analytics to Clarify TheFuzzy Math of Customer Value
You create customer value with a product thatSatisfies customers’ needs
Is easy to use
Has a good price
Is better than other alternatives
Offline: hard to measure
Online: can measure everythingMeasuring value: # of users, repeat use, revenue
Can achieve clarity with metrics and analytics
Copyright © 2009 YourVersion
Once You’re Measuring Everything, Where Do You Focus?
How do you decide which user benefits and features you should focus on?
Need framework to make decisions“Problem space” vs. “Solution space”
Importance vs. SatisfactionImportance of user need (problem space)
Satisfaction with how well your product meets the user’s need (solution space)
Copyright © 2009 YourVersion
Importance vs. SatisfactionAsk Users to Rate for Each Feature
98
8784
8679 847055 80
7280
75
4150
55
60
65
70
75
80
85
90
95
100
40 50 60 70 80 90 100
Satisfaction
Impo
rtan
ce
Recommended reading: “What Customers Want” by Anthony Ulwick
BadBad
GreatGreat
Copyright © 2009 YourVersion
Kano Model: User Needs & Satisfaction
User SatisfactionUser Satisfaction
User DissatisfactionUser Dissatisfaction
Performance (more is better)
Delighter (wow)
NeedNeednot metnot met
NeedNeedfully metfully met
Must Have
Needs & features migrate over time
Copyright © 2009 YourVersion
Is the site up when I want to use it?
Is the site fast enough?
Does the functionality work?
Does the functionality meet my needs?
Olsen’s Hierarchy of Web User Needs(adapted from Maslow)
Customer’s Perspective What does it mean to us?
Uptime
Page Load Time
Absence of Bugs
Feature Set
Usability & Design
Decreasing
Dissatisfaction
Increasing S
atisfaction
How easy to use is it?
Copyright © 2009 YourVersion
Using Web Analytics Tools to Understand Your Users
Tracking visitors and traffic
Seeing where users are clicking
Measuring key conversions
Monitoring user feedback
Copyright © 2009 YourVersion
Basic Tracking of Traffic:Google Analytics
•Unique visitors
•New vs. returning
•Pageviews
•Time on site
•Top referrers
•Top geos
Copyright © 2009 YourVersion
Seeing Where Users are Clicking: CrazyEgg Heatmap
•Shows Click Density
•Color indicates % of clicks
•See which links perform best
•See impact of UI changes
Copyright © 2009 YourVersion
Measuring Key Conversions:Conversion Funnel
•Tie user actions to business goals
•Instrument key steps in user flow
•See where users are dropping off
•Quantify improvement from changes
Copyright © 2009 YourVersion
Monitoring User Feedback Quantitatively with Kampyle
Simple popup
UnobtrusiveSolicitation Average Grade over time
Average Grade for each page on your site
Copyright © 2009 YourVersion
The UI Design IcebergThe UI Design Iceberg
VisualDesign
InteractionDesign
InformationArchitecture
ConceptualDesign
Recommended reading: Jesse James Garrett’s“Elements of User Experience” chart, free at www.jjg.net
What most people seeand react to
What good PMs and Designers think about
Copyright © 2009 YourVersion
UI questions are never yes/no! (not binary)Should ask: “What percentage of users …?”UI changes impact your metrics
Impact can be positive, negative, small, largeSeek high‐ROI UI changes
Typical UI design question:“When using web pages, do users scroll down?”
‐ Yes‐ No
Approaching UI Design Analytically
Copyright © 2009 YourVersionCopyright Copyright ©© 2009 Olsen Solutions LLC2009 Olsen Solutions LLC
Put Key Conversion Actions Above The FoldPut Key Conversion Actions Above The Fold
Landing Page ALanding Page A Landing Page BLanding Page B
Key conversion action is above the fold
Key conversion action is below the fold
The Fold
Copyright © 2009 YourVersion
Marrying UI Design with MetricsThink of “Ease of use”as distinct from the functionalityUser Interface elements matter
PositionLayoutSizeColorFontText copyInteraction designNavigation
Copyright © 2009 YourVersion
Levels of Optimization Maturity
A/B testing vs. Multivariate testingA/B: test 2 alternatives (ideally same time)
Multivariate: test multiple variables at once
Degree of automation & frequencyManual → Automated → Continuous real‐time
Level of personalizationUniversal: same for all users
Personalized: varying based on user, user attributes, or user segment
Copyright © 2009 YourVersion
Automated Multivariate Testing:Google Website Optimizer
Copyright © 2009 YourVersion
Organization and CultureImpact Product Success
Roles and Skills requiredProduct Management (PM)UI DesignDevelopmentQuality Assurance (QA)Operations
Cultural traits requiredCustomer‐centricData‐driven decision‐making with clear ownershipRapid, iterative developmentContinuous improvement mindset
Copyright © 2009 YourVersion
The Product Manager’s Job:A Successful Product
Be the expert on the market and the customer
Translate business objectives and customer needs into product requirements
Be the clearinghouse for all product ideas
Work with team to design & build great product
Define and track key metrics
Use metrics to drive continuous improvement
Identify, plan & prioritize product ideas to maximize ROI on engineering resources
Copyright © 2009 YourVersion
Prioritizing Product Ideas by ROIPrioritizing Product Ideas by ROI
Investment (developer‐weeks)
Return (V
alue
Created
)
Idea C
Idea B
Idea D
Idea A
Idea F
1
1
2
3
4
2 3 4
?
Copyright © 2009 YourVersion
OpenAccount
Sign in
Account Selection
Register
56%
44%
Forget Password
Registration Process
45% drop off(20% of total)
36% overall drop off for this step
70%(32% of Total)
17% drop off (10% of total)
20% drop off(6% of total)
30%(14% of Total)
80%(26% of Total)
55%(24% of Total)
64%of Total
Case Study 1: Quicken BrokerageCase Study 1: Quicken BrokerageOptimizing Sign In/Registration FlowOptimizing Sign In/Registration Flow
Change Password
83%(46% of Total)
Copyright © 2009 YourVersion
Redesigned User Flow Improved Redesigned User Flow Improved Registration Conversion Rate 37%Registration Conversion Rate 37%
37% improvement in conversion rate
ReleasedNew Design
Copyright © 2009 YourVersion
• Which metric offers the highest ROI opportunity?
Case Study 2: FriendsterCase Study 2: FriendsterOptimizing Viral GrowthOptimizing Viral Growth
ActiveUsers
Prospective Users
Invite Click
Succeed
Inviteclick-through rate
Conversion rate
Don’t Click
Fail
Invites per sender
% of users sending invites
• Multiplied together, these metrics determine your viral ratio
Users
% of users who are active
= 15% = 2.3
= 85%
Registration Process
= 20%
= 50%
Copyright © 2009 YourVersion
Case Study 2: FriendsterCase Study 2: FriendsterDoubled Number of Invitations Sent per SenderDoubled Number of Invitations Sent per Sender
Copyright © 2009 YourVersion
ContinuousContinuousImprovementImprovement
Copyright © 2009 YourVersion
Adding Metrics and Optimization to your Product Process
PlanPlan
DesignDesign
DevelopDevelop
BusinessObjectives
ProductObjectives
Prioritized Feature List
Scoping
Requirements & Design
Code Test Launch
Site Level
Feature Level
OptimizeOptimize Metrics & User Feedback
Copyright © 2009 YourVersion
Optimization through Iteration:Optimization through Iteration:Continuous ImprovementContinuous Improvement
Measurethe metric
Analyzethe metric
Identify top opportunitiesto improve
Design & develop the enhancement
Launch theenhancement
Learning
Gaining knowledge:
• Market
• Customer
• Domain
• Usability
Copyright © 2009 YourVersion
Summary:How to be World‐Class in Web MetricsDefine the equation of your business
Identify and track your key metrics
Use analytics to gain customer insights
Establish clear roles and data‐driven culture
Identify opportunities and prioritize by ROI
Launch, learn, and iterate