data-driven product development - agileaus20 · 2019-05-20 · product manager & their team =...
TRANSCRIPT
![Page 1: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/1.jpg)
1
Data-Driven
Product Development
![Page 2: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/2.jpg)
2
➤ Introduction
Approach
Implementation
Putting it in action
Going mobile
Learnings
![Page 3: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/3.jpg)
3
![Page 4: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/4.jpg)
4
![Page 5: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/5.jpg)
5
Desktop website
![Page 6: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/6.jpg)
6
Desktop Website
![Page 7: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/7.jpg)
7
iPhone
![Page 8: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/8.jpg)
8
Android
![Page 9: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/9.jpg)
9
Mobile website
![Page 10: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/10.jpg)
10
Domain Now
● Major improvements, with no major releases
● While growing audience and revenue
So how did we do it?
![Page 11: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/11.jpg)
11
Introduction
➤ Approach
Implementation
Putting it in action
Going mobile
Learnings
![Page 12: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/12.jpg)
12
We believe.
● Product manager & their team = success
● Identify success with metrics & measurement.
● If we can’t measure it, we can’t improve it.
● Must deliver fast & learn fast.
![Page 13: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/13.jpg)
13
Initial approach
● Set goals & use actionable metrics.
● Instrument to measure, track & learn.
● Identify the features to improve first.
● Improve these features quickly & iteratively.
![Page 14: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/14.jpg)
14
Feature analysis.
High Low
IMPORTANCE
Which features did
we start with ?
![Page 15: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/15.jpg)
15
High
High Low
FREQUENCY OF USE
IMPORTANCE
Low
Feature Analysis
![Page 16: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/16.jpg)
16
High
Low
Low
High value features.
Invest
Low value
features.
Sunset
Property
Homepage
FREQUENCY OF USE
IMPORTANCE
High
Navigation
Search
Feature Analysis
![Page 17: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/17.jpg)
17
Types of ideas.
⅓ positive effect
⅓ neutral effect
⅓ negative effect
Book: The Lean Enterprise
![Page 18: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/18.jpg)
18
Our Sprints.
Each sprint idea’s (stories) move through 4 steps.
We ship instrumented experiments !
IDEA
Something you believe
people want and have a
vision for but no proof.
METRICS
Define what success looks
like for your experiment.
What is the goal?
HYPOTHESIS
You Believe Something will results in an
outcome for your target customer and
you want to test it.
MVP
Create a minimum
viable product, an
experiment to test your
hypothesis with
customers
![Page 19: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/19.jpg)
19
We love & trust data.
● Validate ideas at scale in production.
● Many real users vs a few users.
● Use data gleaned from actual usage.
● New insights unearthed for future planning.
So how did we get started ?
![Page 20: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/20.jpg)
20
Introduction
Approach
➤ Implementation
Putting it in action
Going mobile
Learnings
![Page 21: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/21.jpg)
21
Getting started
Web Analytics Database Logs Server Logs
![Page 22: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/22.jpg)
22
● Large dev effort to re-build
● GA data showed very low
usage (0.3% of searches)
● ‘Schools’ was most popular
Killing Radar Search
![Page 23: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/23.jpg)
23
Optimization Platforms
![Page 24: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/24.jpg)
24
![Page 25: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/25.jpg)
25
Swapping Links for Buttons
Hypothesis: Changing “Create home alert” to a text link will achieve higher
conversion.
![Page 26: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/26.jpg)
26
Calling out the login button
Hypothesis: Adding a call out to the “Log In” link will generate more log ins.
![Page 27: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/27.jpg)
27
Optimization Platforms
✔ Quick to implement
✔ Quick to get results
✔ Pre-built reports
✔ No dev skills required
✘ Easy to make mistakes
✘ Performance hit
✘ Presentation glitches
✘ No dev skills required
Time to get serious....
![Page 28: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/28.jpg)
28
Feature Flagging
![Page 29: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/29.jpg)
29
Implementation
New logic
Original logic
Check the flag
![Page 30: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/30.jpg)
30
Domain Switchboard
![Page 31: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/31.jpg)
31
Rules
● On/Off
● Percentage
● Logged In Status
● User Groups
● .. and more to come
See: http://tech.domain.com.au/2015/05/feature-flagging-framework/
![Page 32: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/32.jpg)
32
Introduction
Approach
Implementation
➤ Putting it in action
Going mobile
Learnings
![Page 33: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/33.jpg)
33
Visible: Homepage opt-in/out
SHOWN FIRST
TO ONLY 10%
OF USERS.
USERS CAN
CHOOSE TO
TRY A NEW
EXPERIENCE
MEASURED
OPT-INS
![Page 34: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/34.jpg)
34
MEASURED HOW
MANY USERS
WENT BACK
“OPT OUT”
MONITORED
FEEDBACK
TRENDS...
![Page 35: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/35.jpg)
35
MAP IS BACK
&
USAGE IS
MEASURED
![Page 36: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/36.jpg)
36
Invisible: 5 more results
● Insight: Users don’t go beyond pg 1
● Idea: 15 results p.page v 20 results
● Metrics: leads & views - CR
● Exposed to small % of audience
SEARCH RESULTS PAGE NUMBER - VIEWS
![Page 37: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/37.jpg)
37
Invisible: 5 more results
● Search Page Views - Dropped 5.5%
● Property Page Views - Increased 4.6%
● Email Enquiries - Increased 3.4%
Small change + Big Impact + Valuable Insights
![Page 38: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/38.jpg)
38
Introduction
Approach
Implementation
Putting it in action
➤ Going mobile
Learnings
![Page 39: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/39.jpg)
39
Mobile A/B Testing
![Page 40: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/40.jpg)
40
Mobile - Button to nowhere test
![Page 41: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/41.jpg)
41
Android A/B Test
![Page 42: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/42.jpg)
42
Android A/B Test
![Page 43: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/43.jpg)
43
Mobile Tech
![Page 44: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/44.jpg)
44
Introduction
Approach
Implementation
Putting it in action
Going mobile
➤ Learnings
![Page 45: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/45.jpg)
45
Getting Started
![Page 46: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/46.jpg)
46
Changing culture
![Page 47: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/47.jpg)
47
Changing culture
![Page 48: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/48.jpg)
48
Conclusion / Highlights
Data-driven product development:
● Agile + Tools + Data.
● Move quickly and learn things.
● Cultural change. A new way of working.
![Page 49: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/49.jpg)
49
Thanks
![Page 50: Data-Driven Product Development - AgileAus20 · 2019-05-20 · Product manager & their team = success ... Killing Radar Search . 23 Optimization Platforms . 24 . 25 Swapping Links](https://reader033.vdocument.in/reader033/viewer/2022043006/5f90a346c4fab73cf0742b69/html5/thumbnails/50.jpg)
50
Lyndon Maher
@lyndon_maher
Paul McManus
@paul_macca
http://tech.domain.com.au
Thanks